Quick and Dirty PowerCLI to Repair a Disk Space Problem
I ran into a case of a datastore that ran out disk space (why that happened is a whole different story – don’t go there… really…) but because of a whole strange chain of event this caused a number of VM’s to become corrupted.
So how did they become corrupt? There were a number of VM’s that were in the middle of committing a snapshot and there was no space on the volume. The VM’s became corrupt. I could not power them on, I couldn’t do anything with them. The only thing I could do was to restore them from backup - and that is what I did. (There were a number of other VM’s on this datastore as well – but they froze and were waiting for a prompt)
I was asked – how many VM’s were affected by this and which VM’s – and what I did want to share with you is - how I found the list of machines to restore.
First step was to get all the events that happened between 22.00-24.00
$a = Get-VIEvent -Start (Get-Date).Addhours(-14) -Finish (Get-Date).Addhours(-12) -MaxSamples 10000
[12:00:07 PM] ~> (Get-Date).Addhours(-14)
Wednesday, March 21, 2012 22:00:13 PM
[12:00:13 PM] ~> (Get-Date).Addhours(-12)
Thursday, March 22, 2012 00:00:19 AM I then looked for the events that happened on a machine that I knew had been affected
$a | ? {$_.ObjectName -like "MACHINE1*"}
EventTypeId : **com.vmware.vc.VmDiskFailedToConsolidateEvent**
Severity :
Message :
Arguments :
ObjectId : vm-1346
ObjectType : VirtualMachine
ObjectName : MACHINE1
Fault :
Key : 29721794
ChainId : 29716968
CreatedTime : 21/03/2012 22:56:59 PM
UserName : NDR-IL\vi3admin
Datacenter : VMware.Vim.DatacenterEventArgument
ComputeResource : VMware.Vim.ComputeResourceEventArgument
Host : VMware.Vim.HostEventArgument
Vm : VMware.Vim.VmEventArgument
Ds :
Net :
Dvs :
FullFormattedMessage : event.com.vmware.vc.VmDiskFailedToConsolidateEvent.fullFormat
(com.vmware.vc.VmDiskFailedToConsolidateEvent)
ChangeTag :
DynamicType :
DynamicProperty :One of the events was the one above. Now that we have found an EventTypeId that looks like the problematic one – I looked for all the machines that had this error. (VM Names of course have been altered to protect the innocent…)
$a | ? {$_.EventTypeId -eq "com.vmware.vc.VmDiskFailedToConsolidateEvent"} | select CreatedTime, ObjectName, FullFormattedMessage | ft
CreatedTime ObjectName FullFormattedMessage
----------- ---------- --------------------
21/03/2012 23:21:08 PM MACHINE1 event.com.vmware.vc.VmDiskFailedToCo...
21/03/2012 23:06:10 PM MACHINE2 event.com.vmware.vc.VmDiskFailedToCo...
21/03/2012 23:04:34 PM MACHINE3 event.com.vmware.vc.VmDiskFailedToCo...
21/03/2012 23:03:38 PM MACHINE4 event.com.vmware.vc.VmDiskFailedToCo...
21/03/2012 22:57:17 PM MACHINE5 event.com.vmware.vc.VmDiskFailedToCo...
21/03/2012 22:57:17 PM MACHINE6 event.com.vmware.vc.VmDiskFailedToCo...
21/03/2012 22:57:04 PM MACHINE7 event.com.vmware.vc.VmDiskFailedToCo... Get the list of VM’s and on which datastore they were located so that I start a restore….
$a | ? {$_.EventTypeId -eq "com.vmware.vc.VmDiskFailedToConsolidateEvent"} | select ObjectName | sort ObjectName | % {
$x = get-vm $_.ObjectName
Write-Host "$($x.Name) - $($x.ExtensionData.Config.Files.VmPathName)"
}
MACHINE1 - [NFS_2] MACHINE1/MACHINE1.vmx
MACHINE2 - [NFS_3] MACHINE2/MACHINE2.vmx
MACHINE3 - [NFS_2] MACHINE3/MACHINE3.vmx
MACHINE4 - [NFS_3] MACHINE4/MACHINE4.vmx
MACHINE5 - [NFS_2] MACHINE5/MACHINE5.vmx
MACHINE6 - [NFS_2] MACHINE6/MACHINE6.vmx
MACHINE7 - [NFS_3] MACHINE7/MACHINE7.vmxQuick and dirty (and I am sure that it is not really my best coding) – but Oh So Useful!!!!
You see why you have to learn PowerCLI ??