VMFS Resource Temporarily Unavailable

I was performing some maintenance on a few VMFS LUNs and came across a few files for a VM that I knew were no longer in use. There were a couple of old snapshot files and a VMDK that had been renamed when doing a restore.

After confirming the files were no longer needed or in use, I attempted to remove them using the rm command. ESXi reported back the following error:

rm: cannot remove ‘.vmdk’: Resource temporarily unavailable

After some quick research, I realized it was likely a file lock that was causing the error. VMFS allows access from multiple ESXi servers at the same time by implementing per-file locking. That likely meant that an ESXi host other than the current owner of the VM had a lock on the file. The cluster was small enough that I was able to simply log in to each host and attempt the delete. After 3 attempts, I found the host with the lock to the files and was able to successfully delete them from the VMFS store.

I had tried removing the files via the datastore browser in vSphere hoping that it would be smart enough to know which host had the lock on the file, and issue the delete the command on that host – but no such luck. There was a way to detect which host had a lock on the files in ESX, but I have not found a similar mechanism in ESXi. Until then, trial and error will suffice.

Multithreaded Powershell Port Scanner

We recently had to perform a hardware upgrade of a perimeter firewall. Doing so is a major undertaking, and while we have very good documentation, it’s always important to do some real-world testing.

To facilitate this, we needed to perform some port scanning from outside our network to ensure that A) All of our firewall rule documentation matched what was actually configured, and B) Ensure a smooth transition to the new hardware. Most port scanners I found were capable of scanning a port range for a given IP set. But I wasn’t able to find much of anything that could take specific IP/port data and return the results. I had previously written a simple ASP.NET application to do this, but it wasn’t designed for testing large datasets.

ASP.NET Port Scanner

So, I decided Powershell was the best bet. There were several available examples, but nothing that truly did what we needed. I was able to pull several resources together and came up with the attached Powershell script. Credit for the port detection goes to Boe Prox, to Gaurhoth for the IP range powershell functions, and to Oisin Grehan for the multithreading code.

The result is a script that takes a CSV input and outputs the results to CSV. You can specify IP addresses (eg. 192.168.100.1), CIDR subnets (192.168.1.0/24, 10.254.254.16/28, and/or IP ranges (10.0.1.1-10). The services.xml file in the bin folder contains a powershell object with port settings for various well-know ports and can be modified to meet your needs. Port cans be specified using their well-known name (eg. SMTP, RDP, HTTP) or in a protocol/portNum format (eg. tcp/80, udp/53, tcp/4900-4910).

Scanning is fairly quick:

PS D:\temp\portscanner> .\PortScanner.ps1 Importing Data from .\externalrules.csv
Imported 3033 targets
Flattening targets into endpoints
There are 3996 to scan
Begin Scanning at 07/08/2011 15:58:31
Waiting for scanning threads to finish...
We scanned 3996 endpoints in 399.1811698
Exporting data to .\results.csv

Happy networking!

portscanner