VMware Balloon Driver

Zing runs on most Virtual Machines. For details see Does Zing run on a Virtual Machine? Only the following detail needs to be taken care of for production systems if you use Zing with the ZST component.  Since ZVM 19.07 the ZST component is not required any more and only an optional tuning and depending on your use case Zing without ZST might be the preferred choice now as by its different memory management it is not affected by the problem described here.

VMware, VirtualBox, Xen and other hypervisor systems try to utilize the hardware resources as efficient as possible. Sometimes, overcommitting CPU and memory resources is activated, which is obviously dangerous for productions systems and usually limited to test and developer systems. But even without overcommitting activated, the VM Balloon Driver is running on most standard Linux systems by default. This driver tries to mark rarely used VM memory for reassigning it either for overcommitting or also for other actions like creating snapshots or relocation of VMs to other hosts or offering more memory to the host system if the host is used for other tasks in parallel, like using a VM on a small system like a notebook.

Unfortunately, the Balloon Driver is unaware of  memory pages locked by other processes. This is a known bug in VMware tools and also Open-VM-Tools: VMware KB 1003586

As the Zing Java Virtual Machine (ZVM) is using locked memory pages provided by the Zing System Tools (ZST) and the Balloon Driver is ignoring them, wrong assumptions are made over the actual used memory. If the hypervisor now needs more memory, for example when other applications are used on a laptop in parallel to VMware, the result will be most likely a killed JVM process inside the VM.

The reason for the killed process is the memory pressure the balloon driver creates by over-expanding its virtual balloon as it considers the memory used by Zing as free to be reclaimable by itself. The virtual balloon won't overwrite the amount of Zing memory defined by the -Xmx parameter, but it would fill the remaining free Linux system memory completely. So other than in normal use cases of the balloon driver where it leaves some Linux memory remaining as buffer, none is left here. As a consequence of no free memory available any more, the Linux Out Of Memory Process Killer activates itself, selecting the largest process, which is most likely the Zing process and thereby terminating the java process.

This is visible by the following messages listed by "dmesg" or logged in related files like /var/log/syslog:

java invoked oom-killer
Out of memory: Kill process 6465 (java) score 38 or sacrifice child
Killed process 6465 (java) total-vm: ...

 

Solution

Disable the Balloon Driver,
either by changes in the hypervisor or by changing the VM configuration file. 

For VMware add the following line to the VMX file and restart the VM:
sched.mem.maxmemctl = "0"

Alternatively, you can disable the balloon driver within the VM:

Find the module name: /sbin/lsmod | grep balloon
Remove the module: sudo modprobe -r vmw_balloon

Balloon Driver on VirtualBox: https://www.virtualbox.org/manual/ch04.html#guestadd-balloon

Balloon Driver on Xen: https://lists.gt.net/xen/users/314755

 

Add Comment

Comments

0 comments

Please sign in to leave a comment.

Was this article helpful?
0 out of 0 found this helpful