Complete installation of VMware tools caused Windows 2008 r2 server unexpected rebooted

Issue: I have received alert that one of the Windows 2008 R2 server unexpectedly rebooted

Findings:  upon checking the system logs found out that the system got unexpectedly rebooted and created Memory dump file. I copied the dump file and done analysis using win debugging tool.
                              after analyzing,  I found that vsepflt.sys caused this issue.

So when I search about this on internet, I found it isThe vShield Endpoint Thin Agent driver (vsepflt.sys).

 

 http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2082588

I confirmed it was installed together with vmwaretools.

cause: my server is already running with McAfee. So multiple antiviruses running on Windows system would  cause stack overflow issue( due to too many filters).


Solution: I uninstalled the vmware tools and reinstalled with typical setup. and confirmed that vshield endpoint thin agent driver is excluded.

undeleted Multiple snapshots of VMs would Kill your ESXi ( Heap COW cannot expand)

Issue: I have received alert that few VMs are rebooted and few VMs are hung

Findings: Up on checking the Vcenter, found that all these VMs are running on same Host  but Esxi Seems to be running fine and also some of the other VMs running fine on the same ESXi.

but when I checked all these VMs  events and  tasks  logs deeply,  noticed that when VADP triggered snapshots creation, all these VMs got same erroras below and failed to create snapshots and  follow by rebooted

Error: error message from ESXi Reason: 0 ( cannot allocate memory)

 

up on checking ESXi kernal warning Logs, its shown Heap COW already it's maximum size and cannot expand.

so I checked the current free size of the heap COW % e and it is 4%. (which means it's been utilized 96 %).

How to check current heap COW Size utilization:

solution:  in order to resolve this COW Heap memory issue, I restarted management services of ESXi and consolidated all the VM's existing snapshots manually. after that the Heap COW free % back to 95% ( which means now it's been utilizing 5% of Heap COW memory)



and also I have increased the default size of Heap COW in ESXi to maximum. http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004424 

So what is the relation between Heap COW and multiple snapshots

VMware solution is as below:

If you use snapshots on virtual machines running on an ESX host, each snapshot delta disk is a COW (Copy On Write) disk. For each one in use by running virtual machines, their data structures take up ESX kernel memory. This allocation is known as the COW heap. This memory is used to store cached metadata, pointing to where in a VMDK or in a chain of VMDK files disk data to be accessed resides.


http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1003156&sliceId=2&docTypeID=DT_KB_1_1&dialogID=161478702&stateId=1%200%20161488616

moreover In order to prevent this issue in feature. I have configured motoring for  consolidation failed issue and scripted to poll every minutes Heap COW memory utilization to data store( which  can be even polled to syslog server)

after upgrade Vcenter Operation manager (VCOPS) from 5.6 to 5.7, VCOPS license changed to Foundation

Task: remediate openSSL issue on VCOPS by upgrading the VCOPS from 5.6 to 5.8

issue: after upgraded the VCOPS license changed to Foundation

Solution: unregistered vCenter from VCOPS and re-registered. now the license changed to advanced license.

VM VDDK Error: Insufficient permissions in the host operating system (permission to perform this operation was denied. you do not hold privilege)

Issue: my backup team informed me that VADP level backup failed for one of my VM and error message showed as

Oct 13 19:02:00 s575mjikz005 logger: * VMname:*FULL* The following internal error occurred: An error was returned from nsrvddk.exe: Error opening disk, [datastore_] VM_test/VM_test-000001.vmdk: VDDK Error: Insufficient permissions in the host operating system

Findings: Up on checking the Vcenter, the VM looks fine and I can able to create  manual snapshot. but I cannot perform other VM features such as vMotion, clone, delete VM. when I tried to move VM to another folder given error message

 

 

the issue is here My domain account as well VADP backup account recently added to administrator group in vcenter windows machine ( windows local permission)  and in the vcenter level, power user permission has given to windows machine administrator group which propagated power user permission to my account as well VADP account. ( even though my account and VADP account has administrator permission in vcenter level)

Solution: logged in to vcenter windows machine and removed my account as well VADP account from administrator group and now as usual,  my account and VADP account got administrator permission and backup team able to perform VADP backup as well.

 

Dont ever patch/upgrade Cisco Vem module in ESXi hosts. the host will not be communicated to 1000V switch

Task: I was assigned to perform patching for  all the ESXi hosts on Japanese environment where the vcenter servers were in Japanese language.

Issue: I have created base line and patched one of the ESXI server. after patching, the ESXi server was out of synch from Cisco 1000V switch and all system up-link and vm up-link accessible VLAN was shown as 1. I logged to 1000V switch and checked the Module status by Show Module command. but unfortunately, this particular patched server connection was not available. So I logged in to the server and compared the version of CISCO VIB software with other server. up on checking I noticed that this issue server  CISCO 1000V VIB updated to latest and it was not connected to 1000V switch.

Solution: removed host from 1000V switch and uninstalled the latest version of VIB from host and installed the older version of VIB. after that the server has been connected back to 1000V switch and all the VLAN access available as normal.

Root cause: somehow because of language issue, I added the Cisco 1000V VEM patches with the base line policy.

Upgrade EMC powerpath from 5.7 to 5.9 SP1 on ESXi's 5.0

Task: I have been asked to upgrade the power path to the latest version.

Up-gradation Procedure:

1) downloaded the powerpath tool from EMC web site  ( required EMC Login creantial). https://download.emc.com/downloads/DL54746_PowerPath/VE-5.9.1.2-for-VMWARE-vSphere-Install-SW.zip

2) then you download you will get VE-5.9.1.2-for-VMWARE-vSphere-Install-SW.zip file.

3) unzip this file and copy EMCPower.VMWARE.5.9.SP1.P02.b054.zip file to your shared data store.

4) you cannot install with this VE-5.9.1.2-for-VMWARE-vSphere-Install-SW.zip file. if you do that it will give you error that index.xml file error. So use EMCPower.VMWARE.5.9.SP1.P02.b054.zip as up-gradation file

5) put the host in tomaintenance mode and migrate all the VM to another Hosts. and log in thru SSH and exeute below command

# esxcli software vib update -d /vmfs/volume/datastore/EMCPower.VMWARE.5.9.SP1.P02.b054.zip

datastore should be where you exported the EMC file.

6) reboot the ESXi host. 

before that bear in mind about the compatibility

Security scanner detected openSSL vulnerability on Vmware Vcenter, ESXi and other products

Issue: Recently received security vulnerability list from Security team that included openSSL vulnerability affected Vcenter and ESXi as below

Solution: VMware has released relative patches on their Security Advisories page: http://www.vmware.com/security/advisories/VMSA-2014-0006.html

kindly download appropriate patches and update it.