Using same remote SQL 2012 SP1 instance for DPM 2012 SP1 and DPM 2012 R2

We recently began to deploy DPM 2012 R2 into our environment. For ease of management, we use a single remote SQL instance for all of our DPM installations. Naturally, we decided to use the same remote SQL 2012 SP1 instance for new DPM 2012 R2 installs.

One of the first steps requires that you run the DPM Remote SQL Prep on the SQL server. When we ran this from the DPM 2012 R2 installation media, it upgraded the existing DPM 2012 SP1 Remote SQL Prep files causing all of the existing jobs on the DPM 2012 SP1 servers to fail. The errors were not evident in the DPM console, rather they were logged to in the SQL Agent on the remote SQL instance:

Message
Executed as user: DOMAIN\sqlservice. The process could not be created for step 1 of job 0x8ADCFE6FE202F04F8C7A11C240E42059 (reason: The system cannot find the file specified). The step failed.

The resolution was to re-run the DPM Remote SQL Prep install from the DPM 2012 SP1 media AFTER the DPM Remote SQL Prep install was run from the DPM 2012 R2 media on the remote SQL server. This restored the necessary files on disk and jobs began running again immediately.

Resolving error 0x8007007e Cannot cannect to wmi provider

Recently, I had to troubleshoot a problem with SQL backups via Microsoft Data Protection Manager 2012 SP1 for a SQL Server 2008 system. DPM was alerting us that database auto-protection failed with error code ID 32511. The detailed errors showed that DPM could not enumerate SQL Server instances using Windows Management Instrumentation on the protected computer. This error was detailed in the DPMRACurr.errlog on the production server:

WARNING Failed: Hr: = [0x8007007e] : unable to execute the WQL query: SELECT * FROM ServerSettings

This pointed to a problem with the underlying WMI configuration for SQL, so I used wbemtest.exe from the remote DPM server to test WMI connectivity. If you are unsure of exactly what WMI namespaces are in use or what queries are being run, you can use WMI Tracing to see what’s happening under the hood.

Log Name: Microsoft-Windows-WMI-Activity/Trace
Source: Microsoft-Windows-WMI-Activity
Date: 10/22/2013 3:59:39 PM
Event ID: 1
Task Category: None
Level: Information
Keywords:
User: SYSTEM
Computer: SERVERNAME
Description:
GroupOperationId = 9283379; OperationId = 9300341; Operation = Start IWbemServices::ExecQuery – SELECT * FROM ServerSettings; ClientMachine = DPMSERVER; User = jeff; ClientProcessId = 2540; NamespaceName = \\.\root\Microsoft\SqlServer\ComputerManagement10

Once wbemtest is open, connect to the appropriate namespace:

SQL 2005
\\SERVERNAME\root\Microsoft\SqlServer\ComputerManagement

SQL 2008 & 2008 R2
\\SERVERNAME\root\Microsoft\SqlServer\ComputerManagement10

SQL 2012
\\SERVERNAME\root\Microsoft\SqlServer\ComputerManagement11

Once connected, try executing the WQL query that your application is using – in my case, it was SELECT * FROM ServerSettings. Doing this resulted in the error:

Number: 0x8007007e
Facility: Win32
Description: The specified module could not be found.

Some quick research shows this can most often be resolved by recompiling the WMI template for SQL with mofcomp:

http://support.microsoft.com/kb/956013

On 64-bit Windows with SQL 2008, the command is:

mofcomp “C:\Program Files (x86)\Microsoft SQL Server\100\Shared\sqlmgmproviderxpsp2up.mof”

You may need to adjust the command for bitness and version of SQL and then restart the WMI service for the changes to take effect. However, this did not resolve the issue on the specific system where I was encountering the problem. The same error was returned when trying to run a query in wbemtest after recompiling and restarting the service, the DPM console also displayed the same error when attempting to enumerate SQL instances. The 0x8007007e error typically means a DLL or registration is missing. Time to break out procmon and see what’s happening under the covers. Using filters to include only the wmiprvse.exe process and excluding entries with a SUCCESS result, I could see that there was a file it seemed to be looking for, but could not find:

Procmon WMI SQL

 

It seemed to be scouring the path looking for sqlmgmprovider.dll and svrenumapi100.dll. I checked on disk, and sure enough, neither of those files existed under the path C:\Program Files\Microsoft SQL Server\100\Shared, however, their 32-bit counterparts were located under C:\Program Files\Microsoft SQL Server\100\Shared. Checking another  64-bit SQL 2008 server, I was able to find those files under that first path. After copying them from a known working system, the error was resolved. Also, the second file was only listed in procmon once I copied the first to the server and retested, so it make take several passes to completely resolve.

Note that this resolved this specific error for me, though it may not be the best solution. The reason those files were not on the server is because there was only a 32-bit instance of SQL Server on the system. By adding those two files and re-running wbemtest, an error was no longer returned, but the query also did not show any instances of SQL Server because it was querying for 64-bit instances.

SQL Server Reporting Services error installing DPM 2012 SP1 with remote SQL 2012 database

We use Microsoft Data Protection Manager in our environment to protect our Windows workloads. Recently, DPM 2012 SP1 was released and we have begun the process of upgrading each of our DPM servers to this version, but encountered a problem with the latest server to be upgraded. Though the prerequisite check was successful, DPM would fail to install citing an error with SQL Server Reporting Services on our remote SQL 2012 server:

DPM Setup cannot query the SQL Server Reporting Services configuration

DPM Setup cannot query the SQL Server Reporting Services configuration

Viewing the error log, we can see the following error attempting to query the SSRS configuration via WMI:

[3/4/2013 12:05:44 PM] Information : Getting the reporting secure connection level for DPMSQL01/MSSQLSERVER
[3/4/2013 12:05:44 PM] Information : Querying WMI Namespace: \\DPMSQL01\root\Microsoft\SqlServer\ReportServer\RS_MSSQLSERVER\v10\admin for query: SELECT * FROM MSReportServer_ConfigurationSetting WHERE InstanceName=’MSSQLSERVER’
[3/4/2013 12:05:44 PM] * Exception : => System.Management.ManagementException: Provider load failure

DPM is using WMI to get information about the SSRS installation, and is getting a “Provider load failure” error message. The natural troubleshooting technique here is to attempt to run this query manually via wbemtest from the SQL server itself, and sure enough, we end up with a 0x80041013 “Provider Load Failure” error message:

0x80041013 Provider Load Failure

0x80041013 Provider Load Failure

The SQL Server was originally deployed as SQL 2008 R2 and then upgraded to SQL 2012 SP1. Though there is a KB article describing this issue, there is no update for SQL 2012 SP1. You’ll also notice that the path mentioned in the error code includes v10 – which refers to SQL 2008. So, it seems as though the underlying problem has to do with an issue with the upgrade from SQL 2008 R2 to SQL 2012 and the WMI namespace.

Rather than open a PSS case to find the root cause, we decided it was probably faster to uninstall SQL entirely, then install a fresh instance of SQL 2012 and restore the DPM databases. If you choose to go this route, be sure to take a backup of your SSRS encryption key, DPM databases, master db, msdb, and the SSRS databases. If you don’t, you’ll spend hours reconfiguring reports, setting up SQL security and you’ll have to run DPMSync to recreate the SQL jobs.

SQL Server Authentication Channnel Encryption

We had a customer recently inquire as to whether the authentication channel between a client and a SQL Server was encrypted by default. While we know that SSL is supported on SQL Server 2005, we did not have a certificate installed. However, it was rumored that the system would use a self-signed certificate. Also, we wanted to explore the differences between SQL Server 2000 and SQL Server 2005, as well as the differences between different providers.

I searched for documentation confirming that the authentication channel was indeed encrypted and was able to come up with the following from this MSDN article: http://msdn2.microsoft.com/en-us/library/ms189067.aspx

“Microsoft SQL Server 2005 can use Secure Sockets Layer (SSL) to encrypt data that is transmitted across a network between an instance of SQL Server and a client application. The SSL encryption is performed within the protocol layer and is available to all SQL Server clients except DB Library and MDAC 2.53 clients.”

“Credentials (in the login packet) that are transmitted when a client application connects to SQL Server 2005 are always encrypted. SQL Server will use a certificate from a trusted certification authority if available. If a trusted certificate is not installed, SQL Server will generate a self-signed certificate when the instance is started, and use the self-signed certificate to encrypt the credentials. This self-signed certificate helps increase security but it does not provide protection against identity spoofing by the server. If the self-signed certificate is used, and the value of the ForceEncryption option is set to Yes, all data transmitted across a network between SQL Server and the client application will be encrypted using the self-signed certificate.”

Well, this contradicted some posts I had read but did point us in the right direction. So, we decided to test this. I setup two Virtual Machines, one running a .NET web application and a .NET windows application, and one running SQL Server 2005 Express Edition. I then installed Network Monitor 3.1 and captured the traffic on the NIC as we tested the connection using the SQL Native Client (SQLNCLI) and OLEDB (System.Data.SqlClient).

We saw the server send a self-signed certificate to the client and after which, the authentication channel was encrypted. We also ran the same test on SQL Server 2000. While the authentication channel is not encrypted with SQL Server 2000, the password is not sent in clear text. Rather, it appears obfuscated – most likely using an offset of some kind. We did see the username come across in clear text.

To summarize, when using SQL Server 2005, the authentication channel is completely encrypted when using any clients except DB Library and MDAC 2.53 clients regardless of whether the server has a SSL certificate installed.