VMware Balloon Driver

Azul Platform Prime runs on most Virtual Machines. 

The following detail needs to be taken care of only for production systems if you use Azul Platform Prime with the ZST component.  Since Azul Zulu Prime JVM 19.07 the ZST component is not required any more and is only an optional tuning. Depending on your use case Azul Platform Prime without ZST might be the preferred choice now, and if you do use Azul Platform Prime without ZST, you do not need the instructions here.

VMware, VirtualBox, Xen and other hypervisor systems try to utilize the hardware resources as efficient as possible. Sometimes, overcommitting CPU and memory resources is activated, which is obviously dangerous for productions systems and usually limited to test and developer systems. But even without overcommitting activated, the VM Balloon Driver is running on most standard Linux systems by default. This driver tries to mark rarely used VM memory for reassigning it either for overcommitting or also for other actions like creating snapshots or relocation of VMs to other hosts or offering more memory to the host system if the host is used for other tasks in parallel, like using a VM on a small system like a notebook.

Unfortunately, the Balloon Driver is unaware of  memory pages locked by other processes. This is a known bug in VMware tools and also Open-VM-Tools: VMware KB 1003586

As the Azul Zulu Prime JVM is using locked memory pages provided by the Azul Zulu Prime System Tools (ZST) and the Balloon Driver is ignoring them, wrong assumptions are made over the actual used memory. If the hypervisor now needs more memory, for example when other applications are used on a laptop in parallel to VMware, the result will be most likely a killed JVM process inside the VM.

The reason for the killed process is the memory pressure the balloon driver creates by over-expanding its virtual balloon as it considers the memory used by Azul Zulu Prime JVM as free to be reclaimable by itself. The virtual balloon won't overwrite the amount of Azul Zulu Prime memory defined by the -Xmx parameter, but it would fill the remaining free Linux system memory completely. So other than in normal use cases of the balloon driver where it leaves some Linux memory remaining as buffer, none is left here. As a consequence of no free memory available any more, the Linux Out Of Memory Process Killer activates itself, selecting the largest process, which is most likely the Azul Zulu Prime JVM process and thereby terminating the java process.

This is visible by the following messages listed by "dmesg" or logged in related files like /var/log/syslog:

java invoked oom-killer
Out of memory: Kill process 6465 (java) score 38 or sacrifice child
Killed process 6465 (java) total-vm: ...



Disable the Balloon Driver,
either by changes in the hypervisor or by changing the VM configuration file. 

For VMware add the following line to the VMX file and restart the VM:
sched.mem.maxmemctl = "0"

Alternatively, you can disable the balloon driver within the VM:

Find the module name: /sbin/lsmod | grep balloon
Remove the module: sudo modprobe -r vmw_balloon

Balloon Driver on VirtualBox: https://www.virtualbox.org/manual/ch04.html#guestadd-balloon

Balloon Driver on Xen: https://lists.gt.net/xen/users/314755


Add Comment



Please sign in to leave a comment.

Was this article helpful?
0 out of 0 found this helpful