64GB Addressable Memory Limit on ESX 3.x Host

Prior to ESX 3.5 Update 3, the ability to address more than 64GB of memory on ESX Server 3.5 is suppressed by default. In a standard installation, a 36bit MTRR mask is forced, even though the machine may support 40bit mask values. This means that the ESX Server may see any memory above 64GB as memory that is in use. For example, if an ESX server has 256GB of RAM, the Memory Usage counter displays 192GB in use and only 64GB free. If you attempt to create a virtual machine using memory exceeding the available 64GB of memory, you see an Insufficient Memory error. This condition is documented in the following location: http://www.vmware.com/support/vi3/doc/vi3_esx35u3_rel_notes.html

For more details & Resolution refer Jason Boche Article below..

http://www.boche.net/blog/index.php/2010/05/24/esx-3-x-host-64gb-addressable-memory-limit/