after upgrade Vcenter Operation manager (VCOPS) from 5.6 to 5.7, VCOPS license changed to Foundation

Task: remediate openSSL issue on VCOPS by upgrading the VCOPS from 5.6 to 5.8

issue: after upgraded the VCOPS license changed to Foundation

Solution: unregistered vCenter from VCOPS and re-registered. now the license changed to advanced license.

VM VDDK Error: Insufficient permissions in the host operating system (permission to perform this operation was denied. you do not hold privilege)

Issue: my backup team informed me that VADP level backup failed for one of my VM and error message showed as

Oct 13 19:02:00 s575mjikz005 logger: * VMname:*FULL* The following internal error occurred: An error was returned from nsrvddk.exe: Error opening disk, [datastore_] VM_test/VM_test-000001.vmdk: VDDK Error: Insufficient permissions in the host operating system

Findings: Up on checking the Vcenter, the VM looks fine and I can able to create  manual snapshot. but I cannot perform other VM features such as vMotion, clone, delete VM. when I tried to move VM to another folder given error message

 

 

the issue is here My domain account as well VADP backup account recently added to administrator group in vcenter windows machine ( windows local permission)  and in the vcenter level, power user permission has given to windows machine administrator group which propagated power user permission to my account as well VADP account. ( even though my account and VADP account has administrator permission in vcenter level)

Solution: logged in to vcenter windows machine and removed my account as well VADP account from administrator group and now as usual,  my account and VADP account got administrator permission and backup team able to perform VADP backup as well.

 

Dont ever patch/upgrade Cisco Vem module in ESXi hosts. the host will not be communicated to 1000V switch

Task: I was assigned to perform patching for  all the ESXi hosts on Japanese environment where the vcenter servers were in Japanese language.

Issue: I have created base line and patched one of the ESXI server. after patching, the ESXi server was out of synch from Cisco 1000V switch and all system up-link and vm up-link accessible VLAN was shown as 1. I logged to 1000V switch and checked the Module status by Show Module command. but unfortunately, this particular patched server connection was not available. So I logged in to the server and compared the version of CISCO VIB software with other server. up on checking I noticed that this issue server  CISCO 1000V VIB updated to latest and it was not connected to 1000V switch.

Solution: removed host from 1000V switch and uninstalled the latest version of VIB from host and installed the older version of VIB. after that the server has been connected back to 1000V switch and all the VLAN access available as normal.

Root cause: somehow because of language issue, I added the Cisco 1000V VEM patches with the base line policy.

Upgrade EMC powerpath from 5.7 to 5.9 SP1 on ESXi's 5.0

Task: I have been asked to upgrade the power path to the latest version.

Up-gradation Procedure:

1) downloaded the powerpath tool from EMC web site  ( required EMC Login creantial). https://download.emc.com/downloads/DL54746_PowerPath/VE-5.9.1.2-for-VMWARE-vSphere-Install-SW.zip

2) then you download you will get VE-5.9.1.2-for-VMWARE-vSphere-Install-SW.zip file.

3) unzip this file and copy EMCPower.VMWARE.5.9.SP1.P02.b054.zip file to your shared data store.

4) you cannot install with this VE-5.9.1.2-for-VMWARE-vSphere-Install-SW.zip file. if you do that it will give you error that index.xml file error. So use EMCPower.VMWARE.5.9.SP1.P02.b054.zip as up-gradation file

5) put the host in tomaintenance mode and migrate all the VM to another Hosts. and log in thru SSH and exeute below command

# esxcli software vib update -d /vmfs/volume/datastore/EMCPower.VMWARE.5.9.SP1.P02.b054.zip

datastore should be where you exported the EMC file.

6) reboot the ESXi host. 

before that bear in mind about the compatibility

Security scanner detected openSSL vulnerability on Vmware Vcenter, ESXi and other products

Issue: Recently received security vulnerability list from Security team that included openSSL vulnerability affected Vcenter and ESXi as below

Solution: VMware has released relative patches on their Security Advisories page: http://www.vmware.com/security/advisories/VMSA-2014-0006.html

kindly download appropriate patches and update it.

EOL/Obsolete Software: Microsoft XML Core Services 4.0 Service Pack 2 Detected on Windows Based Vcenter server ( Security Scanner Detected Vulnerability)

Vulnerability: EOL/Obsolete Software: Microsoft XML Core Services 4.0 Service Pack 2 Detected

This is relaed to Microsoft security Vulnerabilities:

THREAT:
The host is running Microsoft XML Core Services 4.0 with Service Pack 2 installed. Microsoft ended support for Microsoft XML 4.0 SP2 on April 13,
2009 and provides no further support for that specific service pack level.
IMPACT:
The system is at high risk of being exposed

Solution: Since Microsoft  XML core services 6.0 released, uninstall this Microsoft XML 4.0 SP2 and install Microsoft  XML core services 6.0.

how to delete Microsoft XML 4.0 SP2 and SP3?

created batch file with below command and execute it. 

REM Script to quietly uninstall MSXML 4.0 and updates.

REM Changelog

REM v0.1 - initial version

REM v0.2 - updated titles and spacing

REM Uninstalling MSXML 4.0 SP2 and updates

REM Uninstalling MSXML 4.0 SP2 Parser and SDK (Base Installer - msxml.msi)

MsiExec.exe /uninstall {716E0306-8318-4364-8B8F-0CC4E9376BAC} /quiet

REM Uninstalling KB925672 (MS06-061 - msxml4-KB925672-enu.exe)

MsiExec.exe /uninstall {A9CF9052-F4A0-475D-A00F-A8388C62DD63} /quiet

REM Uninstalling KB927978 (MS06-071 - msxml4-KB927978-enu.exe)

MsiExec.exe /uninstall {37477865-A3F1-4772-AD43-AAFC6BCFF99F} /quiet

REM Uninstalling KB936181 (MS07-042 - msxml4-KB936181-enu.exe)

MsiExec.exe /uninstall {C04E32E0-0416-434D-AFB9-6969D703A9EF} /quiet

REM Uninstalling KB954430 (MS08-069 - msxml4-KB954430-enu.exe)

MsiExec.exe /uninstall {86493ADD-824D-4B8E-BD72-8C5DCDC52A71} /quiet

REM Uninstalling KB973688 (Non Security Update - msxml4-KB973688-enu.exe)

MsiExec.exe /uninstall {F662A8E6-F4DC-41A2-901E-8C11F044BDEC} /quiet

REM Uninstalling MSXML 4.0 SP3 and updates

REM Uninstalling MSXML 4.0 SP3 Parser (Base Installer - msxml.msi)

MsiExec.exe /uninstall {196467F1-C11F-4F76-858B-5812ADC83B94} /quiet

REM Uninstalling KB973685 (Non Security Update - msxml4-KB973685-enu.exe)

MsiExec.exe /uninstall {859DFA95-E4A6-48CD-B88E-A3E483E89B44} /quiet

REM Uninstalling KB2721691 (MS12-043 - msxml4-KB2721691-enu.exe)

MsiExec.exe /uninstall {355B5AC0-CEEE-42C5-AD4D-7F3CFD806C36} /quiet

REM Uninstalling KB2758694 (MS13-002 - msxml4-KB2758694-enu.exe)

MsiExec.exe /uninstall {1D95BA90-F4F8-47EC-A882-441C99D30C1E} /quiet

After Networker VADP completed backup, snapshots not removed and alert triggered VM required snapshot consolidation

Issue: after Networker VADP snapshot level backup completed, snapshot are not deleted in the data store and triggered VM required snapshot consolidationissue


Esxi Version: 5.0 U2


Temporary solution:  1) perform management agent restart on ESXi host and re-do the consolidation by selecting VM-right click- snapshot - consolidate option.

How to restart managemnet agent:    http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1003490

permanent Solution: this issue related to VADP API version and EMC advised to upgrade the Networker 8.0  to Networker 8.0 SP1.

While detaching the unused LUN's ESXi 5.0 Host disconnected and isolated

Task: 
Recently I was assigned a task to remove 10 old unused LUN from ESXi 5.0 Hosts.

Issue: 

 when I tried to detach first 3 LUN,  it was successful but when I tried 4th LUN, task bar shows LUN detaching in progress for 30 Min's and Host Disconnected from vcenter.

After that I  SSH to the host and tried to execute esxcli commands. but unfortunately the whole command module was not available I cannot execute any cli esxcli commands. I re-started the management network and tried to rejoin back to vcenter but all failed.

 Temporary solution: 

I informed all the server owner and got downtime to shutdown all the VM's and rebooted the ESXi Hosts and re-connected to vcenter



up on checking the logs it's shown as below

2014-09-28T00:34:35.293Z cpu21:2263260)PowerPath: EmcpEsxLogEvent:1252: Info:emcp:MpxEsxVolProbe: Wait for volume to be unquiesced 2014-09-28T00:34:35.294Z cpu11:5668140)PowerPath: EmcpEsxLogEvent:1252: Info:emcp:MpxEsxVolProbe: Wait for volume to be unquiesced 2014-09-28T00:34:55.304Z cpu18:6691338)PowerPath: EmcpEsxLogEvent:1252: Info:emcp:MpxEsxVolProbe: Wait for volume to be unquiesced 2014-09-28T00:34:55.306Z cpu18:6691338)PowerPath: EmcpEsxLogEvent:1252: Info:emcp:MpxEsxVolProbe: Wait for volume to be unquiesced 2014-09-28T00:35:05.311Z cpu4:4316)PowerPath: EmcpEsxLogEvent:1252: Info:emcp:MpxEsxVolProbe: Wait for volume to be unquiesced 2014-09-28T00:35:05.312Z cpu4:4316)PowerPath: EmcpEsxLogEvent:1252: Info:emcp:MpxEsxVolProbe: Wait for volume to be unquiesced 2014-09-28T00:35:15.319Z cpu13:6692663)PowerPath: EmcpEsxLogEvent:1252: Info:emcp:MpxEsxVolProbe: Wait for volume to be unquiesced 2014-09-28T00:35:15.321Z cpu25:5114901)PowerPath: EmcpEsxLogEvent:1252: Info:emcp:MpxEsxVolProbe: Wait for volume to be unquiesced 2014-09-28T00:35:25.326Z cpu25:5114901)PowerPath: EmcpEsxLogEvent:1252: Info:emcp:MpxEsxVolProbe: Wait for volume to be unquiesced 2014-09-28T00:35:35.334Z cpu7:6692369)PowerPath: EmcpEsxLogEvent:1252: Info:emcp:MpxEsxVolProbe: Wait for volume to be unquiesced 2014-09-28T00:36:05.354Z cpu38:6692663)PowerPath: EmcpEsxLogEvent:1252: Info:emcp:MpxEsxVolProbe: Wait for volume to be unquiesced 2014-09-28T00:36:45.378Z cpu34:6692369)PowerPath: EmcpEsxLogEvent:1252: Info:emcp:MpxEsxVolProbe: Wait for volume to be unquiesced 2014-09-28T00:36:55.384Z cpu22:5668140)PowerPath: EmcpEsxLogEvent:1252: Info:emcp:MpxEsxVolProbe: Wait for volume to be unquiesced 2014-09-28T00:36:55.385Z cpu13:2263260)PowerPath: EmcpEsxLogEvent:1252: Info:emcp:MpxEsxVolProbe: Wait for volume to be unquiesced 2014-09-28T00:37:05.969Z cpu15:6691610)PowerPath: EmcpEsxLogEvent:1252: Info:emcp:MpxEsxVolProbe: Wait for volume to be unquiesced 2014-09-28T00:37:15.972Z cpu23:6691610)PowerPath: EmcpEsxLogEvent:1252: Info:emcp:MpxEsxVolProbe: Wait for volume to be unquiesced

Solution: Up on checking VMware KB, it shows that  this issue is not related to VMware and it's related to Powerpath.

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2016567

I upgraded the Power path to latest version and now I can able to detach the LUN with out any issue.