Memory management: unable to fund java heap account

This article refers to using Zing with the ZST component. Since ZVM 19.07 the ZST component is optional and depending on your use case Zing without ZST might be the preferred choice now as by its different memory management it is not affected by the problem described here. For large heapsizes (-Xmx setting) of hundreds of GBytes the ZST component adds performance advantages for most applications.

If you get this error when trying to launch an application it means that Zing was unable to reserve enough memory to launch the application.

When you launch an application on Zing it reserves some of your system's RAM for its own use. At a minimum Zing will reserve a chunk of memory equal to your heap size (Xmx). If no other Zing apps are already running it will reserve an additional chunk of memory, usually about 7% of system RAM, for the Zing Contingency Memory and Pause Prevention pools.

If Zing can't reserve all of the memory that it needs it will throw this error: Memory management: unable to fund java heap account and your application won't start.

You can check system memory usage with the 'free' and 'zing-ps -s' command. The zing-ps command also shows the absolute maximum upper limit available for -Xmx values. Those are usually defined as 75% or 85% of the system RAM in /etc/zing/pmem.conf depending on the Zing version.

This failure to start can also happen if there are not enough free 2MB pages and 4MB pages for Zing available due to memory fragmentation. To check that, please run "grep Normal /proc/buddyinfo" and look at the last two columns to see if there are sufficient 2 and 4 MBytes pages.

Example of one machine with zero 2M and 4M pages available:

$ grep Normal /proc/buddyinfo
Node 0, zone   Normal  168 69 184 186 182  109  71 386  492  0  0

If no large other processes are occupying memory, then the cause of memory fragmentation might be a high usage activity of the Linux page cache (file system cache). In that situation you can tell the operating system to write back the cache to disk to free memory. Please take care when using the following on a production system if an application is already running in parallel on it as writing all file system cache content back to disk can take seconds to a few minutes on machines with hundreds of GBytes of RAM and temporarily increases I/O load significantly!

To flush the page cache (file system cache):

sync
echo 1 | sudo tee /proc/sys/vm/drop_caches

Since Zing ZST 5.21.7 this situation is resolved much better with Zing automatically. Please consider upgrading the Zing ZST component if you experience this problem often. Zing ZST is available for download on https://www.azul.com/software-downloads-and-documents/. In addition since ZST 5.21.7, an enhanced memory defragmentation can be enabled in Zing: https://docs.azul.com/zing/index.htm#ZingAppA_EnhancedLinuxMemoryDefragmentation.htm

If the recommendations above don't work for you, reboot the system or contact our support: https://www.azul.com/support/

 

Add Comment

Comments

6 comments
  • > top 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
      919 www-data  20   0 43.916t 586576   7652 S 172.5  0.9  14955:18 /opt/zing/zing-jdk1.8.0-16.01.3.0-3-x86_64/bin/java -server -DAPPID=me.ele.arch.das.capricorn -DHOSTNAME=xg-d+

     

    >  sudo /opt/zing/zing-jdk1.8.0-16.01.3.0-3-x86_64/bin/jps
    Zing Warning:  ulimit value of core file size (-c) is 0 and should be 'unlimited'.
    Error occurred during initialization of VM
    Memory management: unable to fund java heap account

     

    > sudo /opt/zing/zing-jdk1.8.0-16.01.3.0-3-x86_64/bin/jinfo 919
    Zing Warning:  ulimit value of core file size (-c) is 0 and should be 'unlimited'.
    Error occurred during initialization of VM
    Memory management: unable to fund java heap account

     

    Could you explain why the helper tool failed and how to fix?

    0
  • Tasks: 415 total,   1 running, 414 sleeping,   0 stopped,   0 zombie
    %Cpu(s):  5.1 us,  1.5 sy,  0.0 ni, 93.0 id,  0.0 wa,  0.0 hi,  0.4 si,  0.0 st
    KiB Mem : 65681124 total,  4261548 free, 44162172 used, 17257404 buff/cache
    KiB Swap: 16383996 total, 16380720 free,     3276 used. 17828708 avail Mem

    0
  •  free -h
                  total        used        free      shared  buff/cache   available
    Mem:            62G         42G        4.4G        3.1G         16G         16G
    Swap:           15G        2.9M         15G
    [dev-super@xg-dal-c6-21 ~]$ cat /proc/meminfo
    MemTotal:       65681124 kB
    MemFree:         4597716 kB
    MemAvailable:   17820088 kB
    Buffers:               8 kB
    Cached:         16183868 kB
    SwapCached:         1768 kB
    Active:         12025144 kB
    Inactive:        8213616 kB
    Active(anon):    4876868 kB
    Inactive(anon):  2401228 kB
    Active(file):    7148276 kB
    Inactive(file):  5812388 kB
    Unevictable:           0 kB
    Mlocked:               0 kB
    SwapTotal:      16383996 kB
    SwapFree:       16380996 kB
    Dirty:             24392 kB
    Writeback:             0 kB
    AnonPages:       4055220 kB
    Mapped:            40124 kB
    Shmem:           3222436 kB
    Slab:             736776 kB
    SReclaimable:     599592 kB
    SUnreclaim:       137184 kB
    KernelStack:       15120 kB
    PageTables:        18388 kB
    NFS_Unstable:          0 kB
    Bounce:                0 kB
    WritebackTmp:          0 kB
    CommitLimit:    49224556 kB
    Committed_AS:    7579448 kB
    VmallocTotal:   34359738367 kB
    VmallocUsed:     4583016 kB
    VmallocChunk:   34321070580 kB
    HardwareCorrupted:     0 kB
    AnonHugePages:   3069952 kB
    HugePages_Total:       0
    HugePages_Free:        0
    HugePages_Rsvd:        0
    HugePages_Surp:        0
    Hugepagesize:       2048 kB
    DirectMap4k:      196864 kB
    DirectMap2M:    21696512 kB
    DirectMap1G:    47185920 kB
    [dev-super@xg-dal-c6-21 ~]$ zing-ps -s


     System Zing Memory reserved at configuration (reserve-at-config)

         Memory Usage Summary      Physical        Linux         Zing   Reservable  Contingency    PausePrev
                         Used:        59668        21184        38484        33792            0            0
                         Free:         4474         4474            0          844         1924         1924
                        Total:        64142        25658        38484        34636         1924         1924

    Found 1 process
    USER           PID       TIME    LRSS    ZRSS     Xmx     CTG      PP    JSYS CMD
    www-data       919 10-10:35:10     573    6377   33792       0       0      18 java -server -classpath *: me.ele.jarch.athena.netty.AthenaServer

    0
  • Hi Xu,

    I have opened a support ticket for this issue. Just for the  sake of the community, I am just answering the query here

    The reason why you are unable to launch is you seem to have only 844M left in the Reservable Pool.

     

    [dev-super@xg-dal-c6-21 ~]$ zing-ps -s


     System Zing Memory reserved at configuration (reserve-at-config)

    MemSummary  Physical  Linux  Zing  Reservable  Contingency    PausePrev
            Used:        59668    21184 38484    33792            0                   0
             Free:         4474       4474    0          844         1924               1924
             Total:        64142    25658 38484   34636       1924               1924

    In order to launch any process in Zing, there should be minimum one gig (-Xmx1024M) of Reservable Memory available.

    0
  • Thanks Meivin. We just verified that the helper tools works after reduce the reserved memory of the zing vm process. but the max reserved memory is recommanded by zing-configure-pmem. It would be strange if the helper tool did not work with the recommaned values.

    0
  • Xu,

    If you use the 'system-config-zing-memory' wizard to configure zing memory, it by default sets 75% of the RAM for the zing partition. Out of that 75% it sets aside 10% by default for the Pause Prevention(5%) and Contingency memory(5%) pools. So, basically you get 65% of the Zing memory as "Reservable Memory" which can be used to set -Xmx. 

    If you would like to run additional helper tools like jps, jstat etc, it is recommended to leave aside 1-2G of Reservable Memory for its usage while setting Xmx. 

    0

Please sign in to leave a comment.

Was this article helpful?
0 out of 0 found this helpful