VMware KBTV - vCenter Heartbeat Installation & Validation

Are you interested to check How to install & validate -> vCenter Server Heartbeat.
Check out below VMware KBTV video.

VMware vCenter Server Heartbeat delivers high availability for VMware vCenter Server,

Protecting virtual infrastructure from application, configuration, operating system, network and hardware-related problems. Protect VMware vCenter Server and its database against all types of planned and unplanned downtime with seamless, rapid failover and failback on both physical and virtual platforms. Proactively monitor VMware vCenter Server and ensure seamless failover and failback of for in local and remote locations.

In a Nutshell, vCenter Heartbeat is “Cluster suite” to provide HA for vCenter Server.

ESX host Maintenance mode from ESX CLI

1) login to ESX host to execute below.

To enter Maintenance Mode, at the ESX console:

vimsh -n -e /hostsvc/maintenance_mode_enter

To exit Maintenance Mode :

vimsh -n -e /hostsvc/maintenance_mode_exit

To display whether the ESX Server is currently in maintenance mode or not type:

vimsh -n -e"hostsvc/hostsummary" | grep inMaintenanceMode

Using system libcrypto, version 90810F

inMaintenanceMode = false, 

(False - means "not in maintenance mode" / True - Means "in maintenance mode") 

Troubleshooting Performance Related Problems in vSphere 4.1 Environments

Source : communities.vmware.com

The hugely popular Performance Troubleshooting for VMware vSphere 4 guide is now updated for vSphere 4.1 . This document provides step-by-step approach for troubleshooting most common performance problems in vSphere-based virtual environments. The steps discussed in the document use performance data and charts readily available in the vSphere Client and esxtop to aid the troubleshooting flows. Each performance troubleshooting flow has two parts:

1. How to identify the problem using specific performance counters.
2. Possible causes of the problem and solutions to solve it.

New sections that were added to the document include troubleshooting performance problems in resource pools on standalone hosts and DRS clusters, additional troubleshooting steps for environments experiencing memory pressure (hosts with compressed and swapped memory), high CPU ready time in hosts that are not CPU saturated, environments sharing resources such as storage and network, and environments using snapshots.

This document by no means covers the entire breadth of performance-related problems. We request the readers of this document, including VMware performance community members and vSphere administrators, to help us enhance this document by letting us know about all the performance problems they encounter in their vSphere-based virtual environments, including those that require elaborate troubleshooting steps.  We hope that the community will actively contribute by engaging in live discussions, providing feedback, and asking questions. All this input will serve as the topics for future updates.

vsphere41-performance-troubleshooting.pdf (755.0 K) 

Troubleshooting Virtual Machine snapshot problems

A nice guide on Troubleshooting VM snapshot problems --> http://is.gd/ckwDc

This troubleshooting guide explains basic concepts about Virtual Machine snapshots and different troubleshooting paths depending on the problem. This guide was designed for ESX3.5 and extra considerations have to be taken if working with ESX3.5i or ESX4(i). The formulas and most of the procedures described in this document were created by Ruben as part of a continuous troubleshooting improvement process.

Ruben is also creator of SnapVMX utility.

While troubleshooting Virtual Machine (VM) snapshot problems sometimes it is important to retrieve a lot of information in order to take the most appropriate decision in accordance with the situation. That collection and arrangement of information may take a long time especially if the VM has many snapshots.


SnapVMX was created to speed up the troubleshooting process bringing you instantaneously all the information that you need to evaluate the situation and take the correct decision, reducing the downtime to the bare minimum needed to solve the problem,
 
Source : Eric Blog

Also Try below VMware KB TV - Consolidating snapshots (VMware KB 1007849) - video below

Using MOB- (Managed Object Browser) --> Ops Panel for ESX

I was recently reading Eric Sloof article,
Where you can go and edit (index.html - which opens as ESX server home page) customize by using MOB (Via Java Scripting).. here is the cool video by Eric.

The Operations Panel is a script tool, which runs on the client browser and extends the default ESX server web page with a short list of all available virtual machines. It gives the user the ability to perform simple power operations (start, stop, suspend, resume). Easily accessible user interface for some of the most common operations on an ESX host, available directly from the ESX home page.


Ops Panel for ESX from Eric Sloof NTPRO.NL on Vimeo.

64GB Addressable Memory Limit on ESX 3.x Host

Prior to ESX 3.5 Update 3, the ability to address more than 64GB of memory on ESX Server 3.5 is suppressed by default. In a standard installation, a 36bit MTRR mask is forced, even though the machine may support 40bit mask values. This means that the ESX Server may see any memory above 64GB as memory that is in use. For example, if an ESX server has 256GB of RAM, the Memory Usage counter displays 192GB in use and only 64GB free. If you attempt to create a virtual machine using memory exceeding the available 64GB of memory, you see an Insufficient Memory error. This condition is documented in the following location: http://www.vmware.com/support/vi3/doc/vi3_esx35u3_rel_notes.html

For more details & Resolution refer Jason Boche Article below..

http://www.boche.net/blog/index.php/2010/05/24/esx-3-x-host-64gb-addressable-memory-limit/

How to power off an unresponsive VMware ESX virtual machine

Sometimes Guest OS will go stale / hang due to various reasons..
follow the below VMware KBTV Article to handle the situation..

How to connect to an ESX host using a SSH Client

How to connect to an ESX host using a SSH Client

This article details steps for connecting to an ESX host using an SSH Client. This video uses the PuTTY SSH Client, but a number of different clients work similarly. The steps in the video are the same for different versions of ESX, but for the video, we used ESX 4.
For more information and context, continue reading Connecting to an ESX host using a SSH client

How to connect to an ESX host using a SSH Client

VMware has released the vSphere 4.0 Hardening Guide

VMware would like to announce the availability of the final release of the vSphere 4.0 Security Hardening Guide. This version incorporates the extensive feedback from the VMware community on the previous draft release, which was published in January. We would like to thank all the people who took the time to go through the draft release and provide their comments.

vSphere Hardening Guide provides guidance on how to securely deploy vSphere 4.0 in a production environment. The focus is on initial configuration of the virtualization infrastructure layer, which covers the following:

•The virtualization hosts (both ESX and ESXi)
•Configuration of the virtual machine container (NOT hardening of the guest OS or any applications running within)
•Configuration of the virtual networking infrastructure, including the management and storage networks as well as the virtual switch (but NOT security of the virtual machine’s network)
•vCenter Server, its database, and client components
•VMware Update Manager (included because the regular update and patching of the ESX/ESXi hosts and the virtual machine containers is essential to maintaining the security of the environment)

http://communities.vmware.com/docs/DOC-12306

VMware KB video showing patching of ESX Host. vSphere 4 Update Manager in action.

If you never used VMware vSphere 4 update manager before, this video shows you how to get started and how to successfully apply a necessary security patches to your ESX host.
The GUI interface seems to be difficult to understand, but hopefully with this video you’ll be able to find your way around. A terms like schedule patches definitions download, baselines, remediation,  ESX hosts, scan of ESX host,  maintenance mode will be shown. It’s an easy step-by-step video you can follow if you’re new to VMware and managing Virtual Infrastructure with VMware vSphere 4.
Popout

This video details installing patches or updates to your VMware ESX host using VMware vCenter Update Manager. This video was created using ESX 4.0 and Update Manager 4.0. The same basic steps apply to other versions of ESX. This video also shows you how to determine how often your host checks for patches or updates, where to schedule the task, and how to run the task manually.

Source: VMware KB

How to maintain a the same MAC address all the time within your vSphere 4 installation.

In VMware vSphere 4 environement, when you move your VM to another host or the VM has different path on the same host, the MAC adress of the VM is changed.
If you want to guarantee that the same MAC address is assigned to a given virtual machine every time, even if the virtual machine is moved, or if you want to guarantee a unique MAC address for each virtual machine within a networked environment you have 2 choices. You can do it on the VM configuration level or you can do it on the guest level.
On the VM level you can do it via GUI or by editing your VMX file (just make sure that you unregister the VM from vCenter first, otherwise vCenter will overwrite the value) you go to Menu VM > Edit Settings > Network Adapter and you set the radio button on Manual. Then you can assign a static MAC address there between the authorised range by VMware. The MAC address range is:
00:50:56:00:00:00-00:50:56:3F:FF:FF

If in some rare cases you need to keep the same MAC as for exaple on your Physical server (for some software license files tightened to a MAC address for exemple), and the MAC adress is outside of the range, you have the possibility to do it inside of your Windows (linux) VM.
On the Windows VM you do it in the properties of your NIC:
Start the VM and in the Windows system go to Control panel > network connections > Properties

Click the Configure button and go to Advanced Tab where on selecting Locally administered value you are able to enter your own value.

And for linux VM you can try this:
ifconfig eth0 down
ifconfig eth0 hw ether xx:xx:xx:xx:xx:xx
ifconfig eth0 up
There is reference webpage I found about MAC addresses ranges on VMware’s Website here. The page is is for VMware ESX Server 2.1, but it’s still valuable… There might be other resources on that, feel free to post a comment…

Source : www.vladan.fr Feeds

VMware Update manager does not scan hosts??? Disable Internet Explorer Enhanced Security Configuration on Win2k8

VMware update manager hated me and wouldn’t scan any of my hosts, one of the many fixes I was trying was making sure that my VMware Update Manager server could reach the patch stores to download updates and do patch checks and so decided to make sure of this I wanted to remove the painfully irritating Internet Explorer Enhanced Security Configuration. Now before I get loads of comments saying this is dangerous etc etc and that it’s a security risk,I know it is but for testing purposes I wanted to do it so I did. Now the point of this blog is that to disable it isn’t as easy as it used to be in 2003 as you used to just:

• Go to Control Panel
• Add/Remove Programs, 
• Add/Remove Windows Components on the left hand side

• Untick it from the drop down list

But for Windows Server 2008 I couldn’t find it anywhere in the features drop down lists which has replaced the Windows Components in Server 2008. Fortunately while on another Server 2008 server I noticed the place to disable it. In the Server Manager root page under the Security Information section is Configure IE ESC

 All you need to do is click on this link and disable it for either Administrators and/or for Users. I only needed for Administrators so this is what I selected for the testing.

Source : EMC Feeds

How to Decode MCE from PSOD (ESX dump) to identify the H/w Errors

Next time you see a PSOD (Pink Screen on Death) for ESX Server..
Look for any MCE (Machine Check Error) in the PSOD screen to isolate H/w issues.

MCE errors are always indication of some H/w failure..

Decoding the MCE:
For AMD CPUs you can use MCAT utility to decode the MCE for you..
www.amd.com/us-en/assets/content_type/utilities/mcatsetup.exe

For Intel CPUs you need to convert the MCE from
Hexadecimal to Binary and decode the error manually..

Watch the Video for how to decode the MCE for AMD Processors



Source : VMware KB - 1005184

Performance Monitoring of LUN, HBA, VM & for Which Threshold values, you should show concern

The esxtop utility can be used to measure how much I/O is moving across various devices. The esxtop utility is interactive. As such, pressing certain keys changes the view.

Configuring monitoring using esxtop

To monitor storage performance per HBA:

1. Start esxtop by typing esxtop at the command line.
   
2. Press d to switch to disk view (HBA mode).
   
3. Press f to modify the fields that are displayed.
   
4. Press b, c, d, e, h, and j to toggle the fields and press Enter.
   
5. Press s, then 2 to alter the update time to every 2 seconds and press Enter.
   
6. See Analyzing esxtop columns for a description of relevant columns.

 Click below --> "Read More" for Full Article (Tolerable Threshold Values in Real time)..

VMware ESXi 4 Log Files

Source : /www.simonlong.co.uk/blog/

Like most VI Admins, I've been using VMware ESXi quite a lot more lately and I'm slowly coming across things that are different to how they are in ESX. Log files being one of these differences.
With the absence of the Service Console, ESXi presents a slightly different architecture. If you haven't yet read The Architecture of VMware ESXi, I would recommend having a good read through.
Here is the common log file structure in ESX (Source)

 Click below --> "Read More" for Full Article (Log File Geometry & Snippets)..

Online Training - How to Configure VMware dvPortGroup + Cisco Switch + Private VLAN IDs + Trunking

Source - www.NTpro.nl

• This presentation will lead you trough configuring VLAN IDs at both the vSphere environment and on the physical Cisco switch.

• You’ll get a good understanding of trunking on a Cisco switch and dvPortGroup Settings.

• At the end you’re able to configure a Cisco switch using Cisco’s Network Assistant and understand the different dvPortGroup Settings.

Below Video is about for 40 minutes.  (right click on video for Full Screen)



Online Training - Configure Private VLAN IDs from Eric Sloof NTPRO.NL on Vimeo.

Re-Connect NFS Datastore without Restarting ESX Host (in case NFS Physical connection is lost & restored)

Try below VMware KB  or another work around below..
(Ensure vmkping x.x.x.x) - where x.x.x.x is your NAS IP - is pinging

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1005057


Another workaround - below

NFS is a great storage solution within a VMware vSphere environment. But when the connection to the NFS server is lost, you’re in big trouble. In most circumstances some virtual machines are still connected to the NFS Datastore and since it’s in use, you cannot delete and recreate it. I’ve tried this by using the vSphere Client and trough the command line but didn’t succeed. In the end there’s no other alternative than rebooting the ESX host, or is there… If you want to find out how to remount an NFS Datastore you may watch below video. By Eric Sloof

Actual Trick is - NFS data store IP was x.x.x.x, he created a host name (using /etc/hosts file or DNS or alias host name on DNS) to fool the vCenter, then added host name when adding NFS source. vCenter thinks it's getting NFS from different Source..

May be it's a little dragging video, wait till end of video to see the insight

How to troubleshoot Pink Screen on Death (PSOD) in ESX

Source : VMware KB, VMworld PSOD Guide, TechTarget

VMware might not want to get into Blues like Microsoft much famed BSOD (Blue Screen on Death). So they named it as PSOD (Pink Screen on Death).. LOL..

As you know PSOD crash normally could have no shortcut to find the source of problem, VMware tried it's best to make to straight, possible PSOD causes as mentioned below

1) Hardware Faults
2) Host Faults
3) VMM Faults
4) Guest Operating System Faults
5) Application Faults

  How to Capture the PSOD crash dump (By default PSOD dump is saved on /root)

1) type "vm-support" (on # Prompt) without any options. The utility will run and create a single Tar file that will be named  "esx-xxxxxx..tgz".

2) Alternatively, you can generate the same file by using the VMware Infrastructure Client (VI Client). Select  Administration, then Export Diagnostic Data, and select your host (VirtualCenter data optional) and a directory on your local PC to store the file that will be created. 


You may follow these steps to resolve the issue

 Click below --> "Read More" for Full Article

How CDP (Cisco Discovery Protocol) will benefit you to identify *** Which NIC connected to what External Device on ESX ***

Source : VMware KB and www.vmetc.com 
 

This is a handy little trick, starting with version 3.5, VMware added support for Cisco Discovery Protocol (CDP) on the ESX Server vSwitches. CDP support is enabled on a vSwitch with this command:

Verify the current CDP setting for vSwitch2
ESX# esxcfg-vswitch -b vSwitch2 

down 

The above output is down - which indicates CDP is currently not enabled vSwitch2  

To enable CDP on vSwitch2

ESX# esxcfg-vswitch -B both vSwitch2

To Verify the CDP Status (Expect -> Down, Listen, Advertise, Both)

ESX# esxcfg-vswitch -b vSwitch2

Both

Check CDP information on Physical Switch

Once CDP support is enabled on the vSwitch & physical switch - execute below command on Phy.Switch. Which will show the link between each physical switch port and the matching ESX Server NIC. The output will look something like this:

# show cdp neighbor 

 Capability Codes: R-Router, T-Trans Bridge, B-Source Route Bridge

                  S-Switch, H-Host, I-IGMP, r-Repeater, P-Phone

DeviceID  Local Intrfce  Holdtme  Capability  Platform      Port ID

s3        Gig   0/26        147      T S    WS-C3524-XFas 0/24

esx04     Gig   0/22        168       S     VMware        ESXvmnic0

esx04     Gig   0/21        168       S     VMware        ESXvmnic1

As you can see in the output above, the CDP output clearly links the physical switch port and the ESX Server NIC. This makes it incredibly easy to identify the NICs in the server. This is particularly helpful in blade situations, since you can’t exactly unplug the NIC and see which one goes down with “esxcfg-nics -l” (a common approach to identifying the NICs in the server). Of course, this requires CDP switches in the blade chassis. Since the internal port mappings on the blade chassis determine which NICs connect to which ports, this command adds the mapping within ESX Server and lets us quickly and definitively identify the NICs in the server as seen by ESX Server.

Check CDP information in VI Client

Now, using the VI Client you can go to the Configuration tab of a host and select Networking. To the right of each vSwitch is what I’ll describe as a dialogue icon.

Clicking this Call out icon (Green circle) brings up the switch port info you are seeking.

VMDK Related - Simple Stuff - but good to read & keep in mind

I guess, you may use Mr.Mattr workaround (below) in other cases, like data recovery / repair file system times as well. - Source : www.breathalize.co.uk - by mattr

I was upgrading the memory in a Windows 2000 VM to 8GB, and had to alter the boot.ini to use more than 4GB. Unfortunately I was doing this remotely and during the save process my VPN died. When I rebooted the server I was faced with the dreaded ‘NTBTLDR.EXE is missing’. 

Now I knew that the problem must lie with the boot.ini, but how to get at it? Because I was remote I did not have a Windows 2000 disk to get a recovery console up and I needed to get the server up and running again. I then realized that I did not need to boot from a CD to get to the VM’s hard drive.

I shut the VM off again, and switched to other VM on same host. In Edit Settings, I simply added the vmdk of the troubled VM to the other VM and rescanned for the disk in Disk Management. Lo and behold the disk appeared and I could access the boot.ini file – I had managed to add a space at the beginning of the file during the save process which was causing the problem.

I rectified the file, removed the disk from the spare server and then the vmdk from the client. I crossed my fingers and restarted the VM … and gratefully saw the windows splash screen appear. If this had been a physical box I would not have had a chance of fixing this remotely, and even on site there would have been much more time expended on finding a chassis with spare slots, array controller configurations etc. so it goes to show even the most basic functions of VMware are worth the effort of putting it in in the first place