More than half of the memory in use right after rebooting-Collection of common programming errors

I am trying to do some tests with Azul’s ZingVM, which needs to pre-allocate large amount of ram.

The problem is, on my server (running Cent OS 5.8) which has 16GB ram, more than 8GB will be used right after rebooting… like this (I reboot the server, ssh login to it, and run free -m at once):

             total       used       free     shared    buffers     cached
Mem:         16048       8281       7767          0          1         19
-/+ buffers/cache:       8260       7788
Swap:        18047          0      18047

Obviously the buffers and cached don’t contribute to the used 8GB ram. This is an application server which runs Apache Tomcat everyday (for tests), and I am sure that the Tomcat won’t be started automatically.

I tried to find out which process is using such a large amount of memory with top, but only to find that the top-RES-using process only uses 4460k…

However I have another server (with Cent OS 5.8) running mysql. The memory usage of this database sever is normal. Now I am testing ZVM on the database server, and with Zing allocating 65% ram on startup, after a fresh rebooting, the free -m shows:

             total       used       free     shared    buffers     cached
Mem:         16048      10942       5105          0          3         55
-/+ buffers/cache:      10883       5165
Swap:        18047          0      18047

So… If we ignore the 65% of 16GB — about 10GB — used by Zing, the used memory after rebooting should be just several MBs.

Now I am just testing if I can get Tomcat working under ZingVM. Then I need to run real tests on another two Cent OS servers, and unfortunately, those two servers have the same problem. That is to say, we have 4 servers, and 3 servers have this problem.

What may be the reason of this problem? Can I resolve it without re-installing the OS on those machines?

Update:

here is what zing-system-tool reports when I try to set it to use 75%(it’s default value) of the memory. Exactly the same pattern of output when I specify 85%.

Fatal error:  Not enough free memory (memoryForAzulBytes (12620660736) > MemFree (8142327808)).
zing-memory: ERROR: Configure pmem failed
Fatal error:  os::safeSystem:  command '/sbin/service zing-memory restart' failed.

And here is a successful allocation on the database server (I specified 68% this time, and the old value is 65%; ZST will release the memory reserved by itself):

Info:  azulPmemPages: 5456.
INFO: az_pmem_reserve_pages (num2mPages 5456) succeeded
INFO: az_pmem_fund_transfer (to 7, from 0, bytes 11442061312) succeeded
INFO: az_pmem_fund_transfer (to 1, from 7, bytes 570425344) succeeded
INFO: az_pmem_fund_transfer (to 3, from 7, bytes 570425344) succeeded
INFO: az_pmem_fund_transfer (to 0, from 7, bytes 10301210624) succeeded
Info:  You can now start Java processes up to -Xmx9824m or -Xmx9g.
Info:  Azul pmem initialized successfully.
zing-memory: INFO: start successful
zing-memory: INFO: restart successful

Update 2:

I saw this in /proc/meminfo on the problematic server:

HugePages_Total:  4096
HugePages_Free:   4096
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

while on the database server, it’s all 0. Could this be the reason of the used memory?