Continuous Application Off-Heap Memory Growth until OOM

Problem:

Application off-heap memory usage continuously grows in sudden/unexplained spurts, typically every 1-3 days after startup.  The continuous growth continues for approximately 1 hour and can be approximately ~2GB in size.  These growth-spurts typically re-occur every 1-3 days and if not addressed, will cause the JVM to be killed by the OOM (Out-Of-Memory) killer. 
 
This is what the growth (1 instance) typically looks like:

Cause:

The cause is due to Linux THP (Transparent Huge Page) configuration set to "always" rather than the recommended "madvise" setting. This potential memory waste with THP=always is (unfortunately) a known and studied phenomena.
 
The cause is due to Linux's lazy memory funding mechanisms, where mapped virtual memory regions are only "funded" with actual physical memory, backing them when that physical memory is first needed (typically upon first modification).  Basically, when using THP, funded physical memory pages are 2MB in size, versus 4KB without THP. This is typically not an issue when THP is applied to memory regions that are contiguous and are fully in use.
 
However, when memory regions are contiguously mapped but sparsely used, the difference in committed physical memory can become dramatic.  For example, thread stacks or malloc arenas that have lots of mapped but never used pages, can grow from a few 4KB pages of physical memory use to 2MB of physical memory use when THP is applied.  When using the "madvise" setting, only memory mappings that actually request THP (by passing MADV_HUGEPAGE to madvise(2)) will utilize the THP 2MB pages. 
 
But when using THP="always," any contiguous aligned 2MB range in any virtual memory mapping will attempt to apply THP by default. When those ranges are funded (on the first touch), they then attempt to be funded with the full 2MB physical page if one is available and will use 4KB pages if not, or if the initial mappings were not 2MB-aligned. The kernel will also, as a background task, defragment and compact memory mappings, looking for 2MB-aligned ranges of virtual memory that are not mapped to 2MB physical pages and shuffling physical memory around to "update" to a contiguous 2MB mapping. 
 
This background compaction is beneficial for virtual memory mapping efficiency but can be wasteful of physical memory when applied to sparsely populated regions. Specifically, compaction can "upgrade" a mapping from using a few 4KB physical  pages for its actual used memory portions, to using an entire 2MB of physical memory.
 
This issue is not specific to Azul's JVMs or even to Java in general, and has been noted publicly over the past decade. Here are some examples:
 

• NodeJS being impacted by huge memory waste when THP=always, with the suggested solution being "disable transparent huge pages, or switch them to madvise mode": https://github.com/nodejs/node/issues/11077
• Transparent Huge Pages set to [always] is sub-optimal for many applications: https://www.digitalocean.com/blog/transparent-huge-pages-and-alternative-memory-allocators

 
And here is some analysis and a reproducer from 2015:

• https://bugzilla.kernel.org/show_bug.cgi?id=93111

Solution:

It is recommended to set OS Kernel Setting THP=madvise (for all hosts which are currently configured to THP=always).  This will have no impact on memory regions within application that are set to use THP via madvise. Memory regions that did not request THP will continue to function normally (with the same 4KB mappings they often start with, but will no longer get "upgraded" transparently to using 2MB physical pages, eliminating the root cause of the observed "mysterious background memory use growth".
 
The technical details on how to configure THP=madvise, is to run the following commands:

$ echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
$ echo advise | sudo tee /sys/kernel/mm/transparent_hugepage/shmem_enabled
$ echo defer | sudo tee /sys/kernel/mm/transparent_hugepage/defrag
$ echo 1 | sudo tee /sys/kernel/mm/transparent_hugepage/khugepaged/defrag

 
However, the above commands temporary set THP-madvise.  To permanently enable the recommended THP settings for Zing, run the following:

$ sudo apt install sysfsutils
$ sudo tee /etc/sysfs.d/zingthp.conf <<EOF
kernel/mm/transparent_hugepage/enabled=madvise
kernel/mm/transparent_hugepage/shmem_enabled=advise
kernel/mm/transparent_hugepage/defrag=defer
kernel/mm/transparent_hugepage/khugepaged/defrag=1
EOF
$ sudo systemctl restart sysfsutils

 
For more information on the above recommended settings, please refer our THP Documentation page.