This article refers to using Azul Platform Prime with the ZST component. Since Azul Zulu Prime JVM 19.07 the ZST component is optional and depending on your use case Azul Zulu Prime without ZST might be the preferred choice now as by its different memory management it is not affected by the problem described here. For large heapsizes (-Xmx setting) of hundreds of GBytes the ZST component adds performance advantages for most applications.
If you get this error when trying to launch an application it means that Azul Zulu Prime JVM was unable to reserve enough memory to launch the application.
When you launch an application on Azul Zulu Prime JVM it reserves some of your system's RAM for its own use. At a minimum Azul Zulu Prime JVM will reserve a chunk of memory equal to your heap size (Xmx). If no other Azul Zulu Prime JVM apps are already running it will reserve an additional chunk of memory, usually about 7% of system RAM, for the Azul Zulu Prime JVM Contingency Memory and Pause Prevention pools.
If Azul Zulu Prime JVM can't reserve all of the memory that it needs it will throw this error: Memory management: unable to fund java heap account and your application won't start.
You can check system memory usage with the 'free' and 'zing-ps -s' command. The zing-ps command also shows the absolute maximum upper limit available for -Xmx values. Those are usually defined as 75% or 85% of the system RAM in /etc/zing/pmem.conf depending on the Azul Zulu Prime JVM version.
This failure to start can also happen if there are not enough free 2MB pages and 4MB pages for Azul Zulu Prime JVM available due to memory fragmentation. To check that, please run "grep Normal /proc/buddyinfo" and look at the last two columns to see if there are sufficient 2 and 4 MBytes pages.
Example of one machine with zero 2M and 4M pages available:
$ grep Normal /proc/buddyinfo
Node 0, zone Normal 168 69 184 186 182 109 71 386 492 0 0
If no large other processes are occupying memory, then the cause of memory fragmentation might be a high usage activity of the Linux page cache (file system cache). In that situation you can tell the operating system to write back the cache to disk to free memory. Please take care when using the following on a production system if an application is already running in parallel on it as writing all file system cache content back to disk can take seconds to a few minutes on machines with hundreds of GBytes of RAM and temporarily increases I/O load significantly!
To flush the page cache (file system cache):
sync
echo 1 | sudo tee /proc/sys/vm/drop_caches
Since ZST 5.21.7 this situation is resolved much better with Azul Zulu Prime JVM automatically. Please consider upgrading the ZST component if you experience this problem often. ZST is available for download at http://www.azul.com/software-downloads-and-documents/.
In addition since ZST 5.21.7, an enhanced memory defragmentation can be enabled in Azul Zulu Prime JVM: https://docs.azul.com/zing/index.htm#ZingAppA_EnhancedLinuxMemoryDefragmentation.htm
If the recommendations above don't work for you, reboot the system or contact our support: https://www.azul.com/support/
Add Comment
Comments
> top
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
919 www-data 20 0 43.916t 586576 7652 S 172.5 0.9 14955:18 /opt/zing/zing-jdk1.8.0-16.01.3.0-3-x86_64/bin/java -server -DAPPID=me.ele.arch.das.capricorn -DHOSTNAME=xg-d+
> sudo /opt/zing/zing-jdk1.8.0-16.01.3.0-3-x86_64/bin/jps
Zing Warning: ulimit value of core file size (-c) is 0 and should be 'unlimited'.
Error occurred during initialization of VM
Memory management: unable to fund java heap account
> sudo /opt/zing/zing-jdk1.8.0-16.01.3.0-3-x86_64/bin/jinfo 919
Zing Warning: ulimit value of core file size (-c) is 0 and should be 'unlimited'.
Error occurred during initialization of VM
Memory management: unable to fund java heap account
Could you explain why the helper tool failed and how to fix?
Tasks: 415 total, 1 running, 414 sleeping, 0 stopped, 0 zombie
%Cpu(s): 5.1 us, 1.5 sy, 0.0 ni, 93.0 id, 0.0 wa, 0.0 hi, 0.4 si, 0.0 st
KiB Mem : 65681124 total, 4261548 free, 44162172 used, 17257404 buff/cache
KiB Swap: 16383996 total, 16380720 free, 3276 used. 17828708 avail Mem
free -h
total used free shared buff/cache available
Mem: 62G 42G 4.4G 3.1G 16G 16G
Swap: 15G 2.9M 15G
[dev-super@xg-dal-c6-21 ~]$ cat /proc/meminfo
MemTotal: 65681124 kB
MemFree: 4597716 kB
MemAvailable: 17820088 kB
Buffers: 8 kB
Cached: 16183868 kB
SwapCached: 1768 kB
Active: 12025144 kB
Inactive: 8213616 kB
Active(anon): 4876868 kB
Inactive(anon): 2401228 kB
Active(file): 7148276 kB
Inactive(file): 5812388 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 16383996 kB
SwapFree: 16380996 kB
Dirty: 24392 kB
Writeback: 0 kB
AnonPages: 4055220 kB
Mapped: 40124 kB
Shmem: 3222436 kB
Slab: 736776 kB
SReclaimable: 599592 kB
SUnreclaim: 137184 kB
KernelStack: 15120 kB
PageTables: 18388 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 49224556 kB
Committed_AS: 7579448 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 4583016 kB
VmallocChunk: 34321070580 kB
HardwareCorrupted: 0 kB
AnonHugePages: 3069952 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 196864 kB
DirectMap2M: 21696512 kB
DirectMap1G: 47185920 kB
[dev-super@xg-dal-c6-21 ~]$ zing-ps -s
System Zing Memory reserved at configuration (reserve-at-config)
Memory Usage Summary Physical Linux Zing Reservable Contingency PausePrev
Used: 59668 21184 38484 33792 0 0
Free: 4474 4474 0 844 1924 1924
Total: 64142 25658 38484 34636 1924 1924
Found 1 process
USER PID TIME LRSS ZRSS Xmx CTG PP JSYS CMD
www-data 919 10-10:35:10 573 6377 33792 0 0 18 java -server -classpath *: me.ele.jarch.athena.netty.AthenaServer
Hi Xu,
I have opened a support ticket for this issue. Just for the sake of the community, I am just answering the query here
The reason why you are unable to launch is you seem to have only 844M left in the Reservable Pool.
[dev-super@xg-dal-c6-21 ~]$ zing-ps -s
System Zing Memory reserved at configuration (reserve-at-config)
MemSummary Physical Linux Zing Reservable Contingency PausePrev
Used: 59668 21184 38484 33792 0 0
Free: 4474 4474 0 844 1924 1924
Total: 64142 25658 38484 34636 1924 1924
In order to launch any process in Zing, there should be minimum one gig (-Xmx1024M) of Reservable Memory available.
Thanks Meivin. We just verified that the helper tools works after reduce the reserved memory of the zing vm process. but the max reserved memory is recommanded by zing-configure-pmem. It would be strange if the helper tool did not work with the recommaned values.
Xu,
If you use the 'system-config-zing-memory' wizard to configure zing memory, it by default sets 75% of the RAM for the zing partition. Out of that 75% it sets aside 10% by default for the Pause Prevention(5%) and Contingency memory(5%) pools. So, basically you get 65% of the Zing memory as "Reservable Memory" which can be used to set -Xmx.
If you would like to run additional helper tools like jps, jstat etc, it is recommended to leave aside 1-2G of Reservable Memory for its usage while setting Xmx.
Please sign in to leave a comment.