Quick and Dirty PowerCLI to Repair a Disk Space Problem

I ran into a case of a datastore that ran out disk space (why that happened is a whole different story – don’t go there… really…) but because of a whole strange chain of event this caused a number of VM’s to become corrupted.

So how did they become corrupt? There were a number of VM’s that were in the middle of committing a snapshot and there was no space on the volume. The VM’s became corrupt. I could not power them on, I couldn’t do anything with them. The only thing I could do was to restore them from backup - and that is what I did. (There were a number of other VM’s on this datastore as well – but they froze and were waiting for a prompt)

I was asked – how many VM’s were affected by this and which VM’s – and what I did want to share with you is - how I found the list of machines to restore.

First step was to get all the events that happened between 22.00-24.00

$a = Get-VIEvent -Start (Get-Date).Addhours(-14) -Finish (Get-Date).Addhours(-12) -MaxSamples 10000 

[12:00:07 PM] ~> (Get-Date).Addhours(-14) 
Wednesday, March 21, 2012 22:00:13 PM 

[12:00:13 PM] ~> (Get-Date).Addhours(-12) 
Thursday, March 22, 2012 00:00:19 AM 

I then looked for the events that happened on a machine that I knew had been affected

$a | ? {$_.ObjectName -like "MACHINE1*"} 

EventTypeId          : **com.vmware.vc.VmDiskFailedToConsolidateEvent**  
Severity             :                                                         
Message              :  
Arguments            :  
ObjectId             : vm-1346  
ObjectType           : VirtualMachine  
ObjectName           : MACHINE1  
Fault                :  
Key                  : 29721794  
ChainId              : 29716968  
CreatedTime          : 21/03/2012 22:56:59 PM  
UserName             : NDR-IL\vi3admin  
Datacenter           : VMware.Vim.DatacenterEventArgument  
ComputeResource      : VMware.Vim.ComputeResourceEventArgument  
Host                 : VMware.Vim.HostEventArgument  
Vm                   : VMware.Vim.VmEventArgument  
Ds                   :  
Net                  :  
Dvs                  :  
FullFormattedMessage : event.com.vmware.vc.VmDiskFailedToConsolidateEvent.fullFormat  
ChangeTag            :  
DynamicType          :  
DynamicProperty      :

One of the events was the one above. Now that we have found an EventTypeId that looks like the problematic one – I looked for all the machines that had this error. (VM Names of course have been altered to protect the innocent…)

$a | ? {$_.EventTypeId -eq "com.vmware.vc.VmDiskFailedToConsolidateEvent"} | select CreatedTime, ObjectName, FullFormattedMessage | ft 

CreatedTime                    ObjectName      FullFormattedMessage  
-----------                           ----------              --------------------  
21/03/2012 23:21:08 PM  MACHINE1        event.com.vmware.vc.VmDiskFailedToCo...  
21/03/2012 23:06:10 PM  MACHINE2        event.com.vmware.vc.VmDiskFailedToCo...  
21/03/2012 23:04:34 PM  MACHINE3        event.com.vmware.vc.VmDiskFailedToCo...  
21/03/2012 23:03:38 PM  MACHINE4        event.com.vmware.vc.VmDiskFailedToCo...  
21/03/2012 22:57:17 PM  MACHINE5        event.com.vmware.vc.VmDiskFailedToCo...  
21/03/2012 22:57:17 PM  MACHINE6        event.com.vmware.vc.VmDiskFailedToCo...  
21/03/2012 22:57:04 PM  MACHINE7        event.com.vmware.vc.VmDiskFailedToCo...  

Get the list of VM’s and on which datastore they were located so that I start a restore….

$a | ? {$_.EventTypeId -eq "com.vmware.vc.VmDiskFailedToConsolidateEvent"} | select ObjectName | sort ObjectName | % { 
$x = get-vm $_.ObjectName 
  Write-Host "$($x.Name) - $($x.ExtensionData.Config.Files.VmPathName)" 


Quick and dirty (and I am sure that it is not really my best coding) – but Oh So Useful!!!!

You see why you have to learn PowerCLI ??