I was performing some maintenance on a few VMFS LUNs and came across a few files for a VM that I knew were no longer in use. There were a couple of old snapshot files and a VMDK that had been renamed when doing a restore.
After confirming the files were no longer needed or in use, I attempted to remove them using the rm command. ESXi reported back the following error:
rm: cannot remove ‘.vmdk’: Resource temporarily unavailable
After some quick research, I realized it was likely a file lock that was causing the error. VMFS allows access from multiple ESXi servers at the same time by implementing per-file locking. That likely meant that an ESXi host other than the current owner of the VM had a lock on the file. The cluster was small enough that I was able to simply log in to each host and attempt the delete. After 3 attempts, I found the host with the lock to the files and was able to successfully delete them from the VMFS store.
I had tried removing the files via the datastore browser in vSphere hoping that it would be smart enough to know which host had the lock on the file, and issue the delete the command on that host – but no such luck. There was a way to detect which host had a lock on the files in ESX, but I have not found a similar mechanism in ESXi. Until then, trial and error will suffice.
Thanks for the post. Solved my issue by helping me think outside the box!
Hi, you can login to one of the ESXi hosts and run the following command to find which hosts owns the file:
vmkfstools -D /path/to/vmdkfile.vmdk
Lock [type 10c00001 offset 238067712 v 111, hb offset 3248128
gen 49, mode 0, owner 00000000-00000000-0000-000000000000 mtime 2444188
num 0 gblnum 0 gblgen 0 gblbrk 0]
Addr <4, 559, 28>, gen 41, links 1, type reg, flags 0, uid 0, gid 0, mode 600
len 73926705152, nb 31678 tbz 0, cow 0, newSinceEpoch 31678, zla 3, bs 1048576
Check the highlighted information in the above o/p. That is the MAC address or the server. You can use
arp -ato decode the MAC address back to IP Address.