True Cloud Storage – Storage Spaces Direct

Posted on May 8, 2015 by jeff

Let me start off by saying this is one of the most exciting new features in the next version of Windows Server. Prior to joining Microsoft, as a customer, I had been begging for this feature for years. Well, the product team has delivered – and in a big way. All of the testing I have performed on this has shown that this is a solid technology, with minimal performance overhead. You can expect to see near line-rate network speeds and full utilization of disk performance. It is as resilient as you would expect, and as flexible as you will need. Recently announced at Ignite 2015, Storage Spaces Direct (S2D) is available in the Server 2016.

The previous version of storage spaces relied on a shared SAS JBOD being connected to all storage nodes. While this helps reduce the cost of storage by replacing expensive SANs with less expensive SAS JBOD’s, it still is a higher cost than leveraging DAS for a few reasons: Dual-port SAS drives are required in order to provide redundant connectivity to the SAS backplane, the JBOD itself has additional cost, and the datacenter costs associated with the rack space, power, and cabling for the JBOD shelf. Storage Spaces Direct overcomes these hurdles by leveraging local disks in server chassis, thereby creating cloud storage out of the lowest cost hardware. You can finally leverage SATA SSD’s with Storage Spaces and still maintain redundancy and failover capabilities. Data is synchronously replicated to disks on other nodes of the cluster leveraging SMB3, meaning you can take advantage of SMB Multichannel and RDMA capable nics for maximum performance.

Here’s a quick look at a Storage Spaces Direct deployment in my lab:

Storage Spaces Direct

Clustering creates virtual enclosures using the local disks in each storage node. Data is the automatically replicated and tiered according to the policy of the virtual disk. There are several recommendations for this release:

A minimum of 4 nodes is recommended to ensure availability of data in servicing and failure scenarios
10Gbps network with RDMA is the recommended backplane between nodes
Three-way mirrored virtual disks are recommended
The server HBA must be capable of pass-through or JBOD mode

All of the other features you know and love about Storage Spaces work in conjunction with this feature. So how do we configure this? Powershell of course. Standard cluster configuration applies, and it’s assumed you have a working cluster, network configured, and disks ready to go. The process of configuring S2D has been streamlined to a single command in Server 2016: Enable-ClusterS2D. This command will scan all available storage on cluster nodes, add the physical disks to a new storage pool, and create two storage tiers based on the disk characteristics. Alternatively, you can manually create the storage pool and tiers by skipping the eligibility checks when enabling S2D.

We can confirm S2D has been enabled by looking at the cluster properties:

Storage Spaces Direct Cluster Properties

And that enclosures have been created, and all disks are recognized by the cluster:

Disks and Enclosures

The storage pool and tiers should also now be visible, I chose to create mine manually (see my post on SSD’s and Storage Spaces Performance).

Storage Spaces Direct Storage Pool

If the media type of your disks shows up as “Unspecified” which can happen with a guest cluster on a 3rd party hypervisor like ESXi, you can set them manually to HDD/SSD now that there are in a storage pool like so:

Get-StoragePool pool1 | Get-PhysicalDisk | Set-PhysicalDisk -MediaType SSD

The enclosures, disks and storage pool will now appear in Failover Cluster Manager:

Failover Cluster Manager Enclosures and Disks

Lastly, we’ll create a virtual disk (you can do this through the GUI as well). You’ll want to tweak the number of columns for your environment (hint: for number of columns, it’s best not to exceed the number of local disks per virtual enclosure):

New-Volume -StoragePoolFriendlyName “pool1” -FriendlyName “vm-storage1” -PhysicalDiskRedundancy 2 -FileSystem CSVFS_NTFS -ResiliencySettingName Mirror -Size 250GB

Viola! We have a mirrored virtual disk that’s been created using storage spaces direct:

Virtual Disk

Rebooting a node will cause the pool and virtual disk to enter a degraded state. When the node comes back online, it will automatically begin a background Regeneration storage job. The fairness algorithm we set earlier will ensure sufficient disk IO access to workloads while regenerating the data as quickly as possible – this will saturate the network bandwidth, which, in this case, is a good thing. You can view the status of the running job by using the Get-StorageJob cmdlet. Disk failures are handled the same there were previously, retire and then remove the disk.

Regarding resiliency, I ran several IO tests while performing various simulated failures:

Node Reboot – 33% performance drop for 20s then full performance, 10% performance drop during regeneration

Node Crash (hard power cycle) – Full IO pause for 20s then full performance, 10% performance drop during regeneration

Disk Surprise Removal – Full IO pause for 4s then full performance, 10% performance drop during regeneration

Note that when using Hyper-V over SMB, the IO pause does not impact running VM’s. They will see increased latency, but the VM’s will continue to run.

Control Scale-Out File Server Cluster Access Point Networks

Posted on October 27, 2014 by jeff

Scale-out File Server (SOFS) is a feature that allows you to create file shares that are continuously available and load balanced for application storage. Examples of usage are Hyper-V over SMB and SQL over SMB. SOFS works by creating a Distributed Network Name (DNN) that is registered in DNS to all IP’s assigned to the server for which the cluster network is marked as “Cluster and Client” (and if the UseClientAccessNetworksForSharedVolumes is set to 1, then additionally networks marked as “Cluster Only”). DNS natively serves up responses in a round robin fashion and SMB Multichannel ensures link load balancing.

If you’ve followed best practices, your cluster likely has a Management network where you’ve defined the Cluster Name Object (CNO) and you likely want to ensure that no SOFS traffic uses those adapters. If you change the setting of that Management network to Cluster Only or None, registration of the CNO in DNS will fail. So how do you exclude the Management network from use on the SOFS? You could use firewall rules, SMB Multichannel Constraints, or the preferred method – the ExcludeNetworks parameter on the DNN cluster object. Powershell to the rescue.

First, let’s take a look at how the SOFS is configured by default:

Scale-out file server cluster access point

In my configuration, 192.168.7.0/24 is my management network and 10.0.3.0/24 is my storage network where I’d like to keep all SOFS traffic. The CNO is configured on the 192.168.7.0/24 network and you can see that both networks are currently configured for Cluster and Client use:

Cluster networks

Next, let’s take a look at the advanced parameters of the object in powershell. “SOFS” is the name of my Scale-out File Server in failover clustering. We’ll use the Get-ClusterResource and Get-ClusterParameter cmdlets to expose these advanced parameters:

Powershell cluster access point

You’ll notice a string parameter named ExcludeNetworks which is currently empty. We’ll set that to the Id of our management network (use a semicolon to separate multiple Id’s if necessary). First, use the Get-ClusterNetwork cmdlet to get the Id of the “Management” network, and then the Set-ClusterParameter cmdlet to update it:

Powershell cluster set-parameter

You’ll need to stop and start the resource in cluster manager for it to pick up the change. It should show only the Storage network after doing so:

Scoped cluster access point

Only IP’s in the Storage network should now be registered in DNS:

PS C:\Users\Administrator.CONTOSO> nslookup sofs
Server: UnKnown
Address: 192.168.7.21

Name: sofs.contoso.int
Addresses: 10.0.3.202
10.0.3.201
10.0.3.203

Managed Service Accounts in Server 2012 R2

Posted on October 7, 2014 by jeff

Managed Service Accounts were first introduced in Server 2008 R2. They are a clever way to ensure lifecycle management of user principals of windows services in a domain environment. Passwords for these accounts are maintained in Active Directory and updated automatically. Additionally, they simplify SPN management for the services leveraging these accounts. In Server 2012 and above, these can also be configured as Group Managed Service Accounts which are useful for server farms. A common scenario for using a managed service account may be to run a the SQL Server service in SQL 2012.

There are a few steps involved in creating these managed service accounts on Server 2012 R2. First, there is a dependency on the Key Distribution Service starting with Server 2012 (in order to support group managed service accounts, though it’s now required for all managed service accounts). You must configure a KDS Root Key. In a production environment, you must wait 10 hours for replication to complete after creating the key, but in lab scenarios with single domain controllers, you can force it to take effect immediately:

Add-KdsRootKey -EffectiveTime ((get-date).addhours(-10))

Once the key has been created, you can create a managed service account from a domain controller. You will need to import the AD Powershell module. We’ll create a MSA named SQL01MSSQL in the contoso.int domain for use on a server named SQL01

Import-Module ActiveDirectory

New-ADServiceAccount -Name SQL01MSSQL -Enable $true -DNSHostName SQL01MSSQL.contoso.int

Next, you’ll need to specify which computers have access to the managed service account.

Set-ADServiceAccount -Identity SQL01MSSQL -PrincipalsAllowedToRetrieveManagedPassword SQL01$

Lastly, the account needs to be installed on the computer accessing the MSA. You’ll need to do this as a domain admin and the AD Powershell module installed and loaded there as well:

Enable-WindowsOptionalFeature -FeatureName ActiveDirectory-Powershell -Online -All

Import-Module ActiveDirectory

Install-ADServiceAccount SQL01MSSQL

You can now use the MSA in the format of DOMAINNAME\ACCOUNTNAME$ with a blank password when configuring a service.

SSD’s on Storage Spaces are killing your VM’s performance

Posted on June 3, 2014 by jeff

We’re wrapping up a project that involved Windows Storage Spaces on Server 2012 R2. I was very excited to get my hands on new SSDs and test out Tiered Storage Spaces with Hyper-V. As it turns out, the newest technology in SSD drives combined with the default configuration of Storage Spaces is killing performance of VM’s.

First, it’s important to understand sector sizes on physical disks, as this is the crux of the issue. The sector size is the amount of data the physical disk controller inside your hard disk actually writes to the storage medium. Since the invention of the hard disk, sector sizes have been 512 bytes for hard drives. Many other aspects of storage are based on this premise. Up until recently, this did not pose an issue. However, with larger and larger disks, this caused capacity problems. In fact, the 512-byte sector is the reason for the 2.2TB limit with MBR partitions.

Disk manufacturers realized that 512-byte sector drives would not be sustainable at larger capacities, and started introducing 4k sector, aka Advanced Format, disks beginning in 2007. In order to ensure compatibility, they utilized something called 512-byte emulation, aka 512e, where the disk controller would accept reads/writes of 512 bytes, but use a physical sector size of 4k. To do this, internal cache temporarily stores the 4k of data from physical medium and the disk controller manipulates the 512 bytes of data appropriately before writing back to disk or sending the 512 bytes of data to the system. Manufacturers took this additional processing into account when spec’ing performance of drives. There are also 4k native drives which use a physical sector size of 4k and do not support this 512-byte translation in the disk controller – instead they expect the system to send 4k blocks to disk.

The key thing to understand is that since SSD’s were first released, they’ve always had a physical sector size of 4k – even if they advertise 512-bytes. They are by definition either 512e or 4k native drives. Additionally, Windows accommodates 4k native drives by performing these same Read-Modify-Write, aka RMW, functions at the OS level that are normally performed inside the disk controller on 512e disks. This means that if the OS sees you’re using a disk with a 4k sector size, but the system receives a 512b, it will read the full 4k of data from disk into memory, replace the 512 bytes of data in memory, then flush the 4k of data from memory down to disk.

Enter Storage Spaces and Hyper-V. Storage Spaces understands that physical disks may have 512-byte or 4k sector sizes and because it’s virtualizing storage, it too has a sector size associated with the virtual disk. Using powershell, we can see these sector sizes:

Get-PhysicalDisk | sort-object SlotNumber | select SlotNumber, FriendlyName, Manufacturer, Model, PhysicalSectorSize, LogicalSectorSize | ft

Any disk whose PhysicalSectorSize is 4k, but LogicalSectorSize is 512b is a 512e disk, a disk with a PhysicalSectorSize and LogicalSectorSize of 4k is a 4k native disk, and any disk with 512b for both PhysicalSectorSize and LogicalSectorSize is a standard HDD.

The problem with all of this is that the when creating a virtual disk with Storage Spaces, if you do not specify a LogicalSectorSize via the Powershell cmdlet, the system will create a virtual disk with a LogicalSectorSize equal to the greatest PhysicalSectorSize of any disk in the pool. This means if you have SSD’s in your pool and you created the virtual disk using the GUI, your virtual disk will have a 4k LogicalSectorSize. If a 512byte write is sent to a virtual disk with a 4k LogicalSectorSize, it will perform the RMW at the OS level – and if you’re physical disks are actually 512e, then they too will have to perform RMW at the disk controller for each 512-bytes of the 4k write it received from the OS. That’s a bit of a performance hit, and can cause you to see about 1/4th of the advertised write speeds and 8x the IO latency.

Why this matters with Hyper-V? Unless you’ve specifically formatted your VHDx files using 4k sectors, they are likely using 512-byte sectors, meaning every write to a VHDx storage on a Storage Spaces virtual disk is performing this RMW operation in memory at the OS and then again at the disk controller. The proof is in the IOMeter tests:

32K Request, 65% Read, 65% Random

Virtual Disk 4k LogicalSectorSize

Virtual Disk 512b LogicalSectorSize

Modifying IE Compatibility View Settings with Powershell

Posted on February 19, 2014 by jeff

I recently upgraded my workstation to Windows 8.1 and as such, am now using Internet Explorer 11. While there are some welcomed improvements, there are several changes that have made day-to-day administration activities a bit challenging. For instance, all of the Dell hardware we use has a Remote Access Controller installed that allows us to perform various remote administration tasks. Unfortunately, the current version of firmware for these DRACs is not compatible with IE 11. However, running in IE 7 compatibility mode allows the UI of the DRACs to function properly.

The problem is, we access all of these directly by private IP and adding hundreds of IP addresses to the IE compatibility view settings on multiple workstations is a bit of a pain. Thankfully, these compatibility view exceptions can be set with Group Policy, but the workstations I use are in a Workgroup and do not belong to a domain. I set out to find a way to programmatically add these exceptions using powershell.

First, it’s important to note that the registry keys that control this behavior changed in IE 11. In previous versions of IE, the setting was exclusively maintained under HKCU(HKLM)\Software\[Wow6432Node]\Policies\Microsoft\Internet Explorer\BrowserEmulation\PolicyList. Starting with IE 11, there’s an additional key under HKCU\Software\Microsoft\Internet Explorer\BrowserEmulation\ClearableListData\UserFilter that controls compatibility view settings. Unfortunately, this new key is stored in binary format and there’s not much information regarding it. I was able to find a stackoverflow post where a user attempted to decipher the data, but I found that some of the assumptions they made did not hold true. Via a process of trial-and-error, I was able to come up with a script that can set this value. However, because IE 11 still supports the previous registry key, I HIGHLY recommend using the other method described later in this post. While this seems to work, there are several values I was not able to decode.

The script will pipe the values in the $domains array into the UserFilter registry key. It accepts either top-level domains, IP addresses or Subnets in CIDR notation.

$key = "HKCU:\Software\Microsoft\Internet Explorer\BrowserEmulation\ClearableListData"
$item = "UserFilter"
 
. .\Get-IPrange.ps1
$cidr = "^((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\.){3}(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)/(3[0-2]|[1-2]?[0-9])$"
 
[byte[]] $regbinary = @()
 
#This seems constant
[byte[]] $header = 0x41,0x1F,0x00,0x00,0x53,0x08,0xAD,0xBA
 
#This appears to be some internal value delimeter
[byte[]] $delim_a = 0x01,0x00,0x00,0x00
 
#This appears to separate entries
[byte[]] $delim_b = 0x0C,0x00,0x00,0x00
 
#This is some sort of checksum, but this value seems to work
[byte[]] $checksum = 0xFF,0xFF,0xFF,0xFF
 
#This could be some sort of timestamp for each entry ending with 0x01, but setting to this value seems to work
[byte[]] $filler = 0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x01
 
#Examples: mydomain.com, 192.168.1.0/24
$domains = @("google.com","192.168.1.0/24")
 
function Get-DomainEntry($domain) {
[byte[]] $tmpbinary = @()
 
[byte[]] $length = [BitConverter]::GetBytes([int16]$domain.Length)
[byte[]] $data = [System.Text.Encoding]::Unicode.GetBytes($domain)
 
$tmpbinary += $delim_b
$tmpbinary += $filler
$tmpbinary += $delim_a
$tmpbinary += $length
$tmpbinary += $data
 
return $tmpbinary
}
 
if($domains.Length -gt 0) {
[int32] $count = $domains.Length
 
[byte[]] $entries = @()
 
foreach($domain in $domains) {
if($domain -match $cidr) {
$network = $domain.Split("/")[0]
$subnet = $domain.Split("/")[1]
$ips = Get-IPrange -ip $network -cidr $subnet
$ips | %{$entries += Get-DomainEntry $_}
$count = $count - 1 + $ips.Length
}
else {
$entries += Get-DomainEntry $domain
}
}
 
$regbinary = $header
$regbinary += [byte[]] [BitConverter]::GetBytes($count)
$regbinary += $checksum
$regbinary += $delim_a
$regbinary += [byte[]] [BitConverter]::GetBytes($count)
$regbinary += $entries
}
 
Set-ItemProperty -Path $key -Name $item -Value $regbinary

You’ll need the Get-IPrange.ps1 script from the technet gallery and you can download the above script here: IE11_CV

IE 11 still supports the older registry key, therefore it is the preferred method not only because the above is a hack, but also because the data is stored in the registry as strings and it supports specific hosts instead of only top-level domains. Again, this script supports hosts, domains, IP Addresses and Subnets in CIDR notation.

 
$key = "HKLM:\SOFTWARE\Wow6432Node\Policies\Microsoft"
 
if(-Not (Test-Path "$key\Internet Explorer")) {
New-Item -Path $key -Name "Internet Explorer" | Out-Null
}
 
if(-Not (Test-Path "$key\Internet Explorer\BrowserEmulation")) {
New-Item -Path "$key\Internet Explorer" -Name "BrowserEmulation" | Out-Null
}
 
if(-Not (Test-Path "$key\Internet Explorer\BrowserEmulation\PolicyList")) {
New-Item -Path "$key\Internet Explorer\BrowserEmulation" -Name "PolicyList" | Out-Null
}
 
. .\Get-IPrange.ps1
$cidr = "^((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\.){3}(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)/(3[0-2]|[1-2]?[0-9])$"
 
#Examples: mydomain.com, 192.168.1.0/24
$domains = @("google.com","192.168.1.0/24")
 
$regkey = "$key\Internet Explorer\BrowserEmulation\PolicyList"
 
foreach($domain in $domains) {
if($domain -match $cidr) {
$network = $domain.Split("/")[0]
$subnet = $domain.Split("/")[1]
$ips = Get-IPrange -ip $network -cidr $subnet
$ips | %{$val = New-ItemProperty -Path $regkey -Name $_ -Value $_ -PropertyType String | Out-Null}
$count = $count - 1 + $ips.Length
}
else {
New-ItemProperty -Path $regkey -Name $domain -Value $domain -PropertyType String | Out-Null
}
}

Again, you’ll need the Get-IPrange.ps1 script from the technet gallery and you can download the above script here: IE_CV

Roaming Profiles in a mixed OS environment

Posted on November 14, 2013 by jeff

**UPDATE: Microsoft recently released hotfixes for Windows 8/2012 and Windows 8.1/2012 R2 to address the roaming profile compatibility issue. The hotfix and regkey is outlined in Step 1 this technet article: http://technet.microsoft.com/en-us/library/jj649079.aspx. Note that there is no hotfix for Windows 7/2008 R2, so you’ll still need the procedure outlined below if you are trying to roam from Vista/2008 to newer versions.

Hotfixes
Windows 8/Server 2012 (KB 2887239)
Windows 8.1/Server 2012 R2 (KB 2887595)

I love mandatory roaming profiles. With the hundreds of servers I manage daily, it’s important I have to same settings on each system to maximize efficiency. Anything I do on one system is available on all systems – for the most part. Each version of Windows comes with a slightly different user profile and they’re not always compatible. For instance, the profile in use with Windows XP/2003 was vastly different from Windows Vista/2008 which was also different from Windows 7/2008 R2. This becomes an issue with roaming profiles since the path to the profile is set on the AD user account with no regard to the actual operating system you’ll be logging into. We support systems ranging from Windows Server 2003 to Windows Server 2012 R2, so I need the ability to roam on any platform.

Microsoft addressed this in the jump from XP to Vista by automatically appending a .V2 to profile names, so you actually had different profiles for XP and Vista. However, that’s as far as they went. Every version of Windows since Vista uses that .V2 profile, but they are not fully compatible. Make a change to the profile in Windows 8 and you’ll lose your Windows Vista desktop. Similarly, modify the start menu in Windows 8.1 and you won’t see the same layout on your Windows 8 systems.

The directory services team provided a workaround by creating multiple GPO’s with WMI filters that apply only to a specific operating system thereby allowing you to set an environment variable on every system to use the in the profile path of users. To me, that was too much clutter and too many GPO’s applying to all domain servers slowing down startup and login. A better solution is a single GPO with item level targeting. Item level targeting basically allows you to apply a preference only if certain criteria are met – in this case, the Operating System version.

To accomplish this, we’ll use a GPO to set an environment variable named PROFILEVER, and then use that variable in the user profile settings in AD. We’ll use the following values for each OS:

v1 = Windows XP/2003/2003 R2
v2 = Windows Vista/2008
v3 = Windows 7/2008 R2
v3.1 = Windows 8/2012
v3.2 = Windows 8.1/2012 R2

Create a new Group Policy Object.
Under Computer Configuration > Preferences > Windows Settings > Environment, create a new Environment Variable named PROFILEVER.
Set the Action to Update, the name to PROFILEVER, the Value to v1.
On the Common tab, select the check box for Item-Level targeting and click the Targeting button.
We want this to apply to multiple Operating Systems, so first add a collection.
Next, select New Item > Operating System, and then select Windows XP from the Product drop-down.*
Drag the Operating System object under the collection.
Repeat step 6 adding entries for Windows Server 2003 and Windows Server 2003 R2.
Repeat steps 2 through 8 for each profile version you wish to support using the proper value and Operating Systems.
As a fail-safe, I’ve added a v0 with no item-level targeting set. Because items are applied sequentially, PROFILEVER should be overwritten by one of the item-level targeted preferences.
Be sure to update AD Profile path to use the new environment variable:
\\server\share\profiles\%PROFILEVER%\%username%
You’ll need to logoff any existing sessions and run a gpupdate /force on systems (or reboot) for the setting to take effect.

*Note: In order to properly select Operating System versions, you need to perform this from the highest OS you wish to support. For instance, you’ll only be able to select Windows 8.1/Server 2012 R2 if you are editing the GPO on a Windows 8.1/Server 2012 R2 system.

Resolving error 0x8007007e Cannot cannect to wmi provider

Posted on October 22, 2013 by jeff

Recently, I had to troubleshoot a problem with SQL backups via Microsoft Data Protection Manager 2012 SP1 for a SQL Server 2008 system. DPM was alerting us that database auto-protection failed with error code ID 32511. The detailed errors showed that DPM could not enumerate SQL Server instances using Windows Management Instrumentation on the protected computer. This error was detailed in the DPMRACurr.errlog on the production server:

WARNING Failed: Hr: = [0x8007007e] : unable to execute the WQL query: SELECT * FROM ServerSettings

This pointed to a problem with the underlying WMI configuration for SQL, so I used wbemtest.exe from the remote DPM server to test WMI connectivity. If you are unsure of exactly what WMI namespaces are in use or what queries are being run, you can use WMI Tracing to see what’s happening under the hood.

Log Name: Microsoft-Windows-WMI-Activity/Trace
Source: Microsoft-Windows-WMI-Activity
Date: 10/22/2013 3:59:39 PM
Event ID: 1
Task Category: None
Level: Information
Keywords:
User: SYSTEM
Computer: SERVERNAME
Description:
GroupOperationId = 9283379; OperationId = 9300341; Operation = Start IWbemServices::ExecQuery – SELECT * FROM ServerSettings; ClientMachine = DPMSERVER; User = jeff; ClientProcessId = 2540; NamespaceName = \\.\root\Microsoft\SqlServer\ComputerManagement10

Once wbemtest is open, connect to the appropriate namespace:

SQL 2005
\\SERVERNAME\root\Microsoft\SqlServer\ComputerManagement

SQL 2008 & 2008 R2
\\SERVERNAME\root\Microsoft\SqlServer\ComputerManagement10

SQL 2012
\\SERVERNAME\root\Microsoft\SqlServer\ComputerManagement11

Once connected, try executing the WQL query that your application is using – in my case, it was SELECT * FROM ServerSettings. Doing this resulted in the error:

Number: 0x8007007e
Facility: Win32
Description: The specified module could not be found.

Some quick research shows this can most often be resolved by recompiling the WMI template for SQL with mofcomp:

http://support.microsoft.com/kb/956013

On 64-bit Windows with SQL 2008, the command is:

mofcomp “C:\Program Files (x86)\Microsoft SQL Server\100\Shared\sqlmgmproviderxpsp2up.mof”

You may need to adjust the command for bitness and version of SQL and then restart the WMI service for the changes to take effect. However, this did not resolve the issue on the specific system where I was encountering the problem. The same error was returned when trying to run a query in wbemtest after recompiling and restarting the service, the DPM console also displayed the same error when attempting to enumerate SQL instances. The 0x8007007e error typically means a DLL or registration is missing. Time to break out procmon and see what’s happening under the covers. Using filters to include only the wmiprvse.exe process and excluding entries with a SUCCESS result, I could see that there was a file it seemed to be looking for, but could not find:

It seemed to be scouring the path looking for sqlmgmprovider.dll and svrenumapi100.dll. I checked on disk, and sure enough, neither of those files existed under the path C:\Program Files\Microsoft SQL Server\100\Shared, however, their 32-bit counterparts were located under C:\Program Files\Microsoft SQL Server\100\Shared. Checking another 64-bit SQL 2008 server, I was able to find those files under that first path. After copying them from a known working system, the error was resolved. Also, the second file was only listed in procmon once I copied the first to the server and retested, so it make take several passes to completely resolve.

Note that this resolved this specific error for me, though it may not be the best solution. The reason those files were not on the server is because there was only a 32-bit instance of SQL Server on the system. By adding those two files and re-running wbemtest, an error was no longer returned, but the query also did not show any instances of SQL Server because it was querying for 64-bit instances.

Powershell Remoting

Posted on October 14, 2013 by jeff

One of the most powerful features of Powershell is the ability to issue commands to remote systems, better known as Powershell remoting. This further lessens the need for console or remote desktop access to administer systems. Powershell remoting was introduced in v2 and relies on the Windows Remote Management service (WinRM) to issue commands to remote systems.

Enabling PowerShell remoting is fairly simple. For a single system on a domain, you can run the Enable-PSRemoting -Force cmdlet which will perform necessary configuration steps. You can also ensure WinRM is configured properly using the quickconfig command:

winrm quickconfig

You can use Group Policy if you need to enable remoting on multiple systems. Do this by automatically configuring the WinRM service:

Computer Configuration\Policies\Administrative Templates\Windows Components\Windows Remote Management (WinRM)\WinRM Service\Allow automatic configuration of listeners

Enable this GPO setting and use * for IPv4 and IPv6 filters (unless you wish to limit WinRM requests to specific source IP ranges). In a non-domain environment, you will need to add the remote system to the TrustedHosts list on the client system:

winrm s winrm/config/client ‘@{TrustedHosts=”RemoteSystem”}’

Alternatively, for non-domain environments, you can configure WinRM to allow SSL connections. You will need a “Server Authentication” certificate installed in the local computer certificate store, with a CN matching the hostname that is not expired, revoked, or self-signed. WinRM then needs to be configured for HTTPS using the certificate:

winrm quickconfig -transport:https

To check the current WinRM configuration, use the get command:

winrm get winrm/config

Once WinRM is configured properly, ensure there is a firewall rule in the local Windows Firewall allowing inbound traffic on port 5985 (and port 5986 for SSL). If WinRM is not properly configured, or there is a firewall blocking traffic, you will likely see an error similar to this:

[RemoteSystem] Connecting to remote server RemoteSystem failed with the following error message : WinRM cannot complete the operation. Verify that the specified computer name is valid, that the computer is accessible over the network, and that a firewall exception for the WinRM service is enabled and allows access from this computer. By default, the WinRM firewall exception for public profiles limits access to remote computers within the same local subnet. For more information, see the about_Remote_Troubleshooting Help topic.
+ CategoryInfo : OpenError: (RemoteSystem:String) [], PSRemotingTransportException
+ FullyQualifiedErrorId : WinRMOperationTimeout,PSSessionStateBroken

Test Powershell remoting with the Test-WSMan cmdlet:

Test-WSMan -Computer RemoteSystem

Which will output something similar to this:

wsmid : http://schemas.dmtf.org/wbem/wsman/identity/1/wsmanidentity.xsd
ProtocolVersion : http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd
ProductVendor : Microsoft Corporation
ProductVersion : OS: 0.0.0 SP: 0.0 Stack: 2.0

Now that WinRM is configured properly, we can open a remote powershell session from the client system:

Enter-PSSession -ComputerName RemoteSystem -Credential username

The credential parameter is optional on domain-joined systems but can be used to open a remote Powershell session as a specific user. If the remote system cannot authenticate the client because it is not domain joined and you have not added it to the TrustedHosts list, you will receive an error similar to this:

Enter-PSSession : Connecting to remote server RemoteSystem failed with the following error message : The WinRM client cannot process the request. If the authentication scheme is different from Kerberos, or if the client computer is not joined to a domain, then HTTPS transport must be used or the destination machine must be added to the TrustedHosts configuration setting. Use winrm.cmd to configure TrustedHosts. Note that computers in the TrustedHosts list might not be authenticated. You can get more information about that by running the following command: winrm help config. For more information, see the about_Remote_Troubleshooting Help topic.
At line:1 char:1
+ Enter-PSSession -ComputerName RemoteSystem -Credential jeff
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidArgument: (RemoteSystem:String) [Enter-PSSession], PSRemotingTransportException
+ FullyQualifiedErrorId : CreateRemoteRunspaceFailed

You can also run commands on several remote systems simultaneously (this example will force a checkin with the WSUS server):

Invoke-Command -ComputerName RemoteSystem1, RemoteSystem2 -ScriptBlock {wuauclt /reportnow}

Happy Remoting!

Using CheckSUR to repair Windows file corruption

Posted on March 27, 2013 by jeff

Microsoft has developed a System Update Readiness tool which can identify and repair Windows system file corruption that could prevent other updates from being installed. The tool is available for all editions of Windows since Vista/Server 2008 and is published under KB947821. The utility installs as a MSU package on older platforms, and is baked in for Windows 8 and Server 2012 with the DISM utility. This utility is often referred to as CheckSUR – short for Check System Update Readiness.

Once installed, a log is generated under %windir%\Logs\CBS\CheckSUR.log. If CheckSUR is able to automatically repair files, it will do so and report this in the log file. For any corruption which cannot be automatically repaired, we can still use this utility to manually fix these items. You will find the KB article number of files that cannot be repaired in the CheckSUR log file:

=================================
Checking System Update Readiness.
Binary Version 6.1.7601.21645
Package Version 15.0
2012-07-06 13:57

Checking Windows Servicing Packages

Checking Package Manifests and Catalogs
(f) CBS MUM Corrupt 0x00000000 servicing\Packages\Package_2_for_KB2685939~31bf3856ad364e35~amd64~~6.1.1.2.mum Expected file name Microsoft-Windows-Foundation-Package~31bf3856ad364e35~amd64~~6.1.7600.16385.mum does not match the actual file name

Checking Package Watchlist
Checking Component Watchlist
Checking Packages
Checking Component Store

Summary:
Seconds executed: 109
Found 1 errors
CBS MUM Corrupt Total count: 1

Unavailable repair files: servicing\packages\Package_2_for_KB2685939~31bf3856ad364e35~amd64~~6.1.1.2.mum servicing\packages\Package_2_for_KB2685939~31bf3856ad364e35~amd64~~6.1.1.2.cat

From this log, we can see the corrupt files are part of KB2685939. To repair, you can follow these instructions.

1. Download the appropriate update package for KB2685939 for the target system from the Microsoft Download Center.

2. Expand the package using the expand command (this assumes the package was downloaded to C:\temp and that we’re expanding to C:\temp\KB2685939):

expand C:\temp\Windows6.1-KB2685939-x64.msu /f:* C:\temp\KB2685939

3. Expand the cab files to the same directory:

expand C:\temp\Windows6.1-KB2685939-x64.cab /f:* C:\temp\KB2685939

4. Copy the expanded *.mum and *.cat files to %windir%\Temp\CheckSUR\servicing\packages:

copy C:\temp\KB2685939\*.mum %windir%\Temp\CheckSUR\servicing\packages\
copy C:\temp\KB2685939\*.cat %windir%\Temp\CheckSUR\servicing\packages\

5. Re-run the System Update Readiness tool which will use the files in the %windir%\Temp\CheckSUR\servicing\packages folder to repair the corrupt or missing files.

Windows Server 2008 Firewall Block rule prevents RPC communication

Posted on February 10, 2013 by jeff

Recently opened a PSS case regarding on issue we discovered with the Windows Firewall with Advanced Security on Server 2008 SP2. As a web host, we have many customer web servers with various ports open to the Internet. From time to time, nefarious users will test the server’s security. Part of the standard response is to block all access to the server from the offending IP. This is realized by creating a Windows Firewall with Advanced Security rule that blocks traffic on all ports, for all services with the remote IP scope set to the IP in question.

The problem was uncovered when we noticed backups were failing. The backup program in use leverages a dynamic RPC endpoint for communication, and with the block rule in place, the communication between the customer’s server and the backup server was failing – even though the scope of the block rule was configured to use only the attacker’s IP address. Furthermore, there was a rule specifically allowing communication from the backup server’s on the dynamic RPC endpoint.

Block Rule:

Rule Name: Block Hacker
———————————————————————-
Enabled: Yes
Direction: In
Profiles: Domain,Private,Public
Grouping:
LocalIP: Any
RemoteIP: 1.2.3.4/255.255.255.255
Protocol: Any
Edge traversal: No
Action: Block

Backup server rule:

Rule Name: Allow Backups
———————————————————————-
Enabled: Yes
Direction: In
Profiles: Domain,Private,Public
Grouping:
LocalIP: Any
RemoteIP: 10.1.1.0/255.255.255.0
Protocol: TCP
LocalPort: RPC
RemotePort: Any
Edge traversal: No
Action: Allow

According to PSS, this is a known issue with Server 2008 SP2. It was fixed in 2008 R2 but this apparently will not be fixed in Server 2008. Luckily, there is a workaround. By creating a rule with action of Secure and allowing it to override block rules and selecting the computer account of the server in question, we can ensure proper communication:

Rule Name: Fix Server 2008 firewall bug
———————————————————————-
Enabled: Yes
Direction: In
Profiles: Domain,Private,Public
Grouping:
LocalIP: Any
RemoteIP: Any
Protocol: TCP
LocalPort: RPC
RemotePort: Any
Edge traversal: No
InterfaceTypes: Any
RemoteComputerGroup: D:(A;;CC;;;S-1-5-21-2041841331-1236329097-1724550537-522200)
Security: Authenticate
Action: Bypass

The reason this works has to do with the order in which Windows Firewall applies rules – that process is described in detail here: http://technet.microsoft.com/en-us/library/cc755191(v=ws.10).aspx. This also seems to be the reason the communication is blocked – block rules are processed before allow rules and rules with broader scope before those with a more narrow scope.

Jeff's Blog

Category Archives: Windows