Tuesday, February 18, 2014

ESXi 5.0 HP Smart Array P420i Controller issue (no local Storage and no Array Controller)

Today we had an issue with an ESXi 5.0 Host. Is a HP DL360p Gen8, so we needed to reinstall the ESXi 5.0.

These ESXI host have 4TB of local Storage, so we need to reinstall the ESXi without destroy the Local VMFS Storage.

Downloaded the latest ESXi 5.0.0.update03-1311175.x86_64 and reinstall the ESXi in the Flash Card(we have our ESXi's installed in internal 4Gb Flash Cards).

When reinstalling the ESXi recognize a ESXi 5.0 installation and we choose to install(not upgrade), then after the install we had a surprise, no Local Storage on ESXi.
Troubleshooting the problem, we notice that HP Smart Array P420i Controller hpsa driver was from original ESXi 5.0 ISO and doesn't work, so we need to use the HP bundle driver/VIB.

We have some options here:

First option: Use a The HP customized VMware ESXi that will install all HP drivers(VIP) and bundles. In the ISO is already included the hpsa that fix the issue(you should get this from your vendor, or HP account support).

Second option: Install a offline bundle from HP. This offline bundle will update, install and remove old VIBs/drives.

How to install HP offline bundle.

First you need to download the latest from HP Software Delivery Repository: http://downloads.linux.hp.com/SDR/index.html

For ESXi 5.0
For ESXi 5.5

In our case was the hp-esxi5.0uX-bundle-1.5-39.zip

Before any changes or install, we need to put the host in Maintenance Mode.

After download the VIB we need to copy to the ESXi Datastore, again we will use WinSCP for this. You can copy directly to /var/log/vmware(if doesn't exist, create).
Or copy to any folder and then copy to the /var/log/vmware

cp offline-bundle.zip /var/log/vmware

After we copy the file we can install the VIB.

Run this command to install drivers using the offline bundle (this requires an absolute path):

For example(in our case):

esxcli software vib install –d /var/log/vmware/hp-esxi5.0uX-bundle-1.5-39.zip

After is finish, you should see a list of the VIBs that were installed and removed(also the ones that were skipped)

Next reboot the ESXi host.

After the server is rebooted you can now list the VIB that are installed.

Run the command:

esxcli software vib list | grep -i (in this case HP)

You should get something like this:

char-hpcru 5.0.3.09-1OEM.500.0.0.434156 Hewlett-Packard PartnerSupported 2013-04-16
char-hpilo 500.9.0.0.9-1OEM.500.0.0.434156 Hewlett-Packard PartnerSupported 2013-04-15
hp-ams 500.9.3.5-02.434156 Hewlett-Packard PartnerSupported 2013-04-26
hp-build 5.20.43-434156 Hewlett-Packard PartnerSupported 2013-04-15
hp-smx-provider 500.03.02.10.4-434156 Hewlett-Packard VMwareAccepted 2013-04-26
hpacucli 9.40-12.0 Hewlett-Packard PartnerSupported 2013-04-16
hpbootcfg 01-01.02 Hewlett-Packard PartnerSupported 2013-04-15
hponcfg 04-00.10 Hewlett-Packard PartnerSupported 2013-04-15
scsi-hpsa 5.0.0-28OEM.500.0.0.472560 Hewlett-Packard VMwareCertified 2013-04-15
scsi-hpvsa 5.0.0-22OEM.500.0.0.406165 Hewlett-Packard PartnerSupported 2013-04-15
vmware-esx-hp_vaaip_p2000 2.1.0-2 Hewlett-Packard VMwareAccepted 2013-04-15
ata-pata-hpt3x2n 0.3.4-3vmw.500.1.11.623860 VMware VMwareCertified 2013-04-15
hpnmi 2.0.11-434156 hp PartnerSupported 2013-04-15

Note: In bold is the hpsa driver that we need to change to fix the HP Smart Array P420i Controller issue.

Connect to your vCenter(or host) with vSphere and check you Datastores on the host and/or Hardware Status(only with vCenter). You should see in Storage sensor your disks and also HP Smart Array P420i Controller in the bottom.

Remove the Maintenance Mode.

Third option: Install only the hspa driver(was what we initially decided to test if it fix the issue):

Same procedure as the Second Option, but in this case is only the hpsa VIB that we will install.

You can download from HERE

After you downloaded just follow the same tasks in the above option.

After you install you should see a list of the one(s) that were removed, and the new one(s) that was installed.

Note: To install/remove VIBs you can also check http://kb.vmware.com/kb/2005205

Wednesday, February 12, 2014

Troubleshooting ESXi 5.5 snapshots issues

Today we had some issues with some VMs regarding snapshots and some VM operations.

We had upgraded some several ESXi 5.0 hosts to a ESXi 5.5 and put them in a new vCenter 5.5b. Consequently the VMs HW were upgraded to v10.

I don't know if this snapshots issues was regarding these migrations, but we have seen them after this migrations. In around 200 VMs(50% of them have snapshots) we have find 10/15 with this problems.

1st problem:

When they try to revert to any snapshots we get "The operation is not supported on the object".
This one was easy to fixed, just remove the VM from the inventory and add again to vCenter. That fix the problem.

We may use the vim-cmd vmsvc/reload vmid option here(second option on this article).

2th problem:

When they try to use any of the Snapshots options('Take Snapshot', 'Snapshot Manager' and 'Revert to current Snapshot') Virtual machine operations were grayed out. There is virtual machine tasks running in the background(maybe any previous task hanging).

So first we need to check which tasks are running on the VM.

Just connect to your host(ssh) and list all VMs and their vmid.

Run: vim-cmd vmsvc/getallvms

You will get all VMs and vmid associated. After you get your VM vmid run:

vim-cmd vmsvc/get.tasklist vmid

The output is similar to:

(ManagedObjectReference) [
'vim.Task:haTask-112-vim.VirtualMachine.createSnapshot-3887']


If there is no task running on that virtual machine, you see output:

(ManagedObjectReference) []

But even you don't see any tasks running, it has a process running(or hanging). So we should reload the vmx file(that will create a new Inventory ID (Vmid) and this should be enough to fix this issue.

Run:

vim-cmd vmsvc/reload vmid

Now the Virtual Machine should be ok and you should have all options available again.

More information here about this commands:

http://kb.vmware.com/kb/1013003
http://kb.vmware.com/kb/2048748

UPDATE: Tested the vim-cmd vmsvc/reload on the first issue, and it works.