VMFS Resource Temporarily Unavailable

I was performing some maintenance on a few VMFS LUNs and came across a few files for a VM that I knew were no longer in use. There were a couple of old snapshot files and a VMDK that had been renamed when doing a restore.

After confirming the files were no longer needed or in use, I attempted to remove them using the rm command. ESXi reported back the following error:

rm: cannot remove ‘.vmdk’: Resource temporarily unavailable

After some quick research, I realized it was likely a file lock that was causing the error. VMFS allows access from multiple ESXi servers at the same time by implementing per-file locking. That likely meant that an ESXi host other than the current owner of the VM had a lock on the file. The cluster was small enough that I was able to simply log in to each host and attempt the delete. After 3 attempts, I found the host with the lock to the files and was able to successfully delete them from the VMFS store.

I had tried removing the files via the datastore browser in vSphere hoping that it would be smart enough to know which host had the lock on the file, and issue the delete the command on that host – but no such luck. There was a way to detect which host had a lock on the files in ESX, but I have not found a similar mechanism in ESXi. Until then, trial and error will suffice.