Pages

Thursday, September 16, 2010

NetApp Virtual Storage Console for VMware vSphere Setup Wizard ended prematurely

Ran into an issue installing NetApp’s Virtual Storage Console for VMware vSphere the other day on a Windows Server 2008 R2 64-bit server where the install would fail with:

NetApp Virtual Storage Console for VMware vSphere Setup Wizard ended prematurely

NetApp Virtual Storage Console for VMware vSphere Setup Wizard ended prematurely because of an error. Your system has not been modified. To install this program at a later time…

Click the Finish button to exit the Setup Wizard.

image

For those who don’t want to read and view all the screenshots I’m including, the problem is because the install wasn’t executing with “run as administrator”. Another problem with the package I was using was that it was an .msi package and right clicking on it does not provide me with the option so what I ended up doing was opening up a command prompt as administrator, navigate to the folder with the netapp_vsc_1.0.msi package, then executing it from there. The following shows the process:

image

image

image

image

image

image

image

image

image

So even thought we got the User Account Control (UAC) prompt, the install will still eventually fail:

image

image

Failed.

image

To get the package to install properly, open the command prompt running as an administrator and then execute the netapp_vsc_1.0.msi package:

image

image

image

image

image

image

image

image

image

image

image

image

Make sure you add the IP of the server with the NetApp Virtual Storage Console to the trusted sites of the IE so the tab within vCenter is displayed properly.

image

image

image

image

image

image

image

image

image

image

image

image

Wednesday, September 15, 2010

Troubleshooting SQL Backup Job Failure – Logs appear to be truncated

I don’t do quite as much SQL DBA tasks these days as the projects I’ve been working on at the current company aren’t as involved with SQL as the previous company I was with. With that being said, there are a few environments with SQL clusters with issues that are escalated to me every so often and one of them began having backup jobs from maintenance plans that I’ve set up fail once every week. As I began troubleshooting and reviewing the logs, I noticed that Job log error had an error logged but the information was somewhat truncated. The following is the flow of troubleshooting that I went through to ultimately figuring out why the jobs were failing.

The first logs I went to look at were the Error Logs section:

image

Unfortunately, the logs didn’t really tell me much (I also had the SQL Server Agent logs selected but did not see any errors) so I went ahead and checked the SQL Server Agent Jobs logs and was able to see an error with a bit more information but it was truncated:

image

Here’s a dump of the text from the error. In order for it to be displayed problem, it needs to be copied to notepad with the size of the window bigger:

Date 9/8/2010 2:00:01 AM
Log Job History (Temp backup to X.Subplan_1)

Step ID 1
Server xxxSQLCLUSTER
Job Name Temp backup to X.Subplan_1
Step Name Subplan_1
Duration 02:12:13
Sql Severity 0
Sql Message ID 0
Operator Emailed
Operator Net sent
Operator Paged
Retries Attempted 0

Message
Executed as user: XXXX\svc_sqlclusteragent. ...531.0 for 64-bit Copyright (C) Microsoft Corp 1984-2005. All rights reserved. Started: 2:00:01 AM Progress: 2010-09-08 02:00:10.94 Source: {7505B5D7-1DFB-4550-B691-085284227C50} Executing query "DECLARE @Guid UNIQUEIDENTIFIER EXECUTE msdb..sp...".: 100% complete End Progress Progress: 2010-09-08 02:00:13.58 Source: Maintenance Cleanup Task Executing query "EXECUTE master.dbo.xp_delete_file 0,N'X:\',N'bak',...".: 100% complete End Progress Progress: 2010-09-08 02:00:16.17 Source: Back Up Database (Full) Executing query "EXECUTE master.dbo.xp_create_subdir N'X:\\XXXX.xxx...".: 11% complete End Progress Progress: 2010-09-08 02:00:16.17 Source: Back Up Database (Full) Executing query "EXECUTE master.dbo.xp_create_subdir N'X:\\XXXX.xxx...".: 22% complete End Progress Progress: 2010-09-08 02:00:16.17 Source: Back Up Database (Full) Executing query "EXECUTE master.dbo.xp_create_subdir N'X:\\XXXX.xxx...".: 33% complete End Progress Progress: 2010-09-08 02:00:16.17 Source: Back Up Database (Full) Executing query "EXECUTE master.dbo.xp_create_subdir N'X:\\XXXX.xxx...".: 44% complete End Progress Progress: 2010-09-08 02:00:16.17 Source: Back Up Database (Full) Executing query "EXECUTE master.dbo.xp_create_subdir N'X:\\XXXX.xxx...".: 55% complete End Progress Progress: 2010-09-08 02:00:16.17 Source: Back Up Database (Full) Executing query "EXECUTE master.dbo.xp_create_subdir N'X:\\XXXX.xxx...".: 66% complete End Progress Progress: 2010-09-08 02:00:16.17 Source: Back Up Database (Full) Executing query "EXECUTE master.dbo.xp_create_subdir N'X:\\master' ...".: 77% complete End Progress Progress: 2010-09-08 02:00:16.17 Source: Back Up Database (Full) Executing query "EXECUTE master.dbo.xp_create_subdir N'X:\\model' ".: 88% complete End Progress Progress: 2010-09-08 02:00:16.19 Source: Back Up Database (Full) Executing query "EXECUTE master.dbo.xp_create_subdir N'X:\\msdb' ".: 100% complete End Progress Progress: 2010-09-08 02:14:24.05 Source: Back Up Database (Full) Executing query "BACKUP DATABASE [XXXX.xxx.Production] TO DISK = N...".: 50% complete End Progress Progress: 2010-09-08 02:27:30.57 Source: Back Up Database (Full) Executing query "declare @backupSetId as int select @backupSetId =...".: 100% complete End Progress Progress: 2010-09-08 02:58:42.92 Source: Back Up Database (Full) Executing query "BACKUP DATABASE [XXXX.xxx.Production.DataStore] TO...".: 50% complete End Progress Progress: 2010-09-08 04:02:53.63 Source: Back Up Database (Full) Executing query "declare @backupSetId as int select @backupSetId =...".: 100% complete End Progress Progress: 2010-09-08 04:03:14.19 Source: Back Up Database (Full) Executing query "BACKUP DATABASE [XXXX.xxx.Production.MemberShip] T...".: 50% complete End Progress Progress: 2010-09-08 04:03:23.57 Source: Back Up Database (Full) Executing query "declare @backupSetId as int select @backupSetId =...".: 100% complete End Progress Progress: 2010-09-08 04:05:09.59 Source: Back Up Database (Full) Executing query "BACKUP DATABASE [XXXX.xxx.Training] TO DISK = N'X...".: 50% complete End Progress Progress: 2010-09-08 04:05:56.55 Source: Back Up Database (Full) Executing query "declare @backupSetId as int select @backupSetId =...".: 100% complete End Progress Progress: 2010-09-08 04:07:41.12 Source: Back Up Database (Full) Executing query "BACKUP DATABASE [XXXX.xxx.Training.DataStore] TO ...".: 50% complete End Progress Progress: 2010-09-08 04:08:27.17 Source: Back Up Database (Full) Executing query "declare @backupSetId as int select @backupSetId =...".: 100% complete End Progress Error: 2010-09-08 04:08:53.25 Code: 0xC00291EC Source: Back Up Database (F... The package execution fa... The step failed.

The error is indicated at the bottom where it reads:

Error: 2010-09-08 04:08:53.25 Code: 0xC00291EC Source: Back Up Database (F... The package execution fa... The step failed.

The information’s not very helpful with the truncated text so I went back to SQL Server Business Management Studio to look around. I ended up at the Maintenance Plan node’s actual job:

image

Drilling down to the actual job and clicking history gave me more information:

image

The error number is: -1073573396

The error message is: Failed to acquire connection "Local server connection". Connection may not be configured correctly or you may not have the right permissions on this connection.

From here I was able to review the time the job failed, what else was going on and noticed that there was an overlap during the time of the backups and database re-indexing which probably lead to the backup not able to obtain the lock it needed on the database. What I ended up doing was adjust the timing of the two jobs.

Thursday, September 9, 2010

Problem executing “Generate Policy” after configuring UAG

As I mentioned in my previous 2 posts, the installation and configuration wasn’t really special as they were simply screenshots of pretty much the flow of a next… next… next… install. To add some value to the experience of this install, here’s the post of the problem I experienced. A few notes before I continue:

  1. I was stumped on the day of the install and configure so I figure I’d be lazy and post this error on our Microsoft partner forum which I would get an answer the next day. I do this a lot for certain deployments when finalizing the install is not immediately required because although I get a response the next day and may need to go back and forth with support, it frees up my time to work on other projects related to design (the Microsoft Partner Forums only support break/fix incidents).
  2. Unfortunately for me, I got a response from the partner forums that they currently don’t support UAG yet. While the engineer gave me a recommendation he had, it didn’t resolve my problem.
  3. Seeing how I wasn’t able to get help from our partner forums, I went ahead to the trusty isaserver.org forums to post and while an MVP got back to me, he wasn’t able to help me resolve the issue either.
  4. Google searches weren’t of any help either as the hits I got were way off.
  5. Knowing that I’m pretty much on my own, I went ahead and put aside the morning to troubleshoot this issue today and was pretty determined to figure this out.

I’d like to thank everyone who helped me (Jason Jones and Brandon Jiang) and I hope this will give back to the community helping anyone who may experience the same issue.

Ok, now with that out of the way, the following is the breakdown:

Problem

So once I completed the configuration of all the nodes for DirectAccess and proceeded to click on the Generate Policies button, then clicked on Apply Now to complete the configuration, I was presented with the following error:

image

> Executing policy script.

Unexpected token 'The' in expression or statement.

At C:\Users\tluk\AppData\Local\Temp\tmpFE62.tmp.ps1:13 char:80

+ if (-not ${UAGDA_CERT_MACHINE_AUTH}) { ${UAGDA_CERT_MACHINE_AUTH}="C=US, O="T

he <<<< Go Daddy Group, Inc.", OU=Go Daddy Class 2 Certification Authority" }

+ CategoryInfo : ParserError: (The:String) [], ParseException

+ FullyQualifiedErrorId : UnexpectedToken

After reviewing the error, I know for a fact that it had something to do with the certificate:

image

The root certificate to which remote clients chain is:

C=US, O=’The Go Daddy Group, Inc.”, OU=Go Daddy Class 2 Certification Authority

The certificate that the DirectAccess server uses for HTTPS is:

CN=someName.someDomain.com, OU=Domain Control Validated, O=someName.someDomain.com

Resolution

To dig a bit deeper into what the problem might be, I went ahead to export the script via the window where I was presented with the Apply Now button that generated this year:

image

I went ahead to export the script to the desktop:

image

Just to ensure that I’m not running into a problem caused by not using the Run as administrator, I opened up PowerShell as administrator, changed the directory to the desktop then executed the ps1 script:

image

File C:\users\tluk\Desktop\UAGDirectAccess_GroupPolicy.ps1 cannot be loaded because the execution of scripts is disabled on this system. Please see “get-help about_signing” for more details.

At line:1 char:34

+ .\UAGDirectAccess_GroupPolicy.ps1 <<<<

+ CategoryInfo : Not specified: (:) [], PSSecurityException

+ FullyQualifiedErrorID : RuntimeException

Hello Mr. Problem! I remember you from my first Exchange 2010 deployment! Let’s set the permissions to remotesigned.

image

image

image

Now to execute the ps1 script.

image

PS C:\users\tluk\Desktop> .\UAGDirectAccess_GroupPolicy-bad.ps1
Unexpected token 'The' in expression or statement.
At C:\users\tluk\Desktop\UAGDirectAccess_GroupPolicy-bad.ps1:12 char:80
+ if (-not ${UAGDA_CERT_MACHINE_AUTH}) { ${UAGDA_CERT_MACHINE_AUTH}="C=US, O="The <<<< Go Daddy Group, Inc.", OU=Go Da
ddy Class 2 Certification Authority" }
+ CategoryInfo : ParserError: (The:String) [], ParseException
+ FullyQualifiedErrorId : UnexpectedToken

Not good right? So I open up the UAGDirectoAccess_GroupPolicy.ps1 script and see the following:

# UAGDA Group Policy Configuration Script.
# Generated on Tuesday, 07 September 2010 18:43 UTC.
# Generator Version 4.0.0.0
# Runtime variables
Param(
[parameter(Mandatory=$False, ValueFromPipeline=$true, HelpMessage="A domain controller, in domain.com for faster operations.")][String] $DomainControllerComputerName,
[parameter(Mandatory=$False, ValueFromPipeline=$true, HelpMessage="Additional domains(seperated by '') to link the app-server's policy to.")][String] $AdditionalAppServerDomains,
[parameter(Mandatory=$False, ValueFromPipeline=$true, HelpMessage="Additional domains(seperated by '') to link the client's policy to.")][String] $AdditionalClientDomains
)
# Static variables
if (-not ${UAGDA_ACCESS_ENABLING_ADDRESSES_ALL}) { ${UAGDA_ACCESS_ENABLING_ADDRESSES_ALL}="2002:480e:ae7c:8001::ac14:105,2002:480e:ae7c:8000:0:5efe:172.20.1.5,2002:480e:ae7c:8001::ac14:10a,2002:480e:ae7c:8000:0:5efe:172.20.1.10,2002:480e:ae7c:8001::ac14:108,2002:480e:ae7c:8000:0:5efe:172.20.1.8,2002:480e:ae7c:8001::ac14:172,2002:480e:ae7c:8000:0:5efe:172.20.1.114,2002:480e:ae7d::480e:ae7d" }
if (-not ${UAGDA_CERT_MACHINE_AUTH}) { ${UAGDA_CERT_MACHINE_AUTH}="C=US, O="The Go Daddy Group, Inc.", OU=Go Daddy Class 2 Certification Authority" }
if (-not ${UAGDA_CERT_TYPE}) { ${UAGDA_CERT_TYPE}="root" }

What I suspect is causing the problem is this:

if (-not ${UAGDA_CERT_MACHINE_AUTH}) { ${UAGDA_CERT_MACHINE_AUTH}="C=US, O="The Go Daddy Group, Inc.", OU=Go Daddy Class 2 Certification Authority" }

It almost looks like the extra quotes: O="The Go Daddy Group, Inc." was terminated the parsing of the string too early. With that being said, whenever I do any ldifde or csvde exports, I remember having to put quotes for OUs that had names with spaces so this does make sense. However, why wasn’t there quotes for: OU=Go Daddy Class 2 Certification Authority ?

At this point, I knew it had something to do with the quotes seeing how the error pointed the C=US, O="The <<<< as the error so what I ended up trying anyways was to take out the extra quotes, execute it from PowerShell and while it did run, it failed at:

So now let’s remove the quote leaving that section as:

if (-not ${UAGDA_CERT_MACHINE_AUTH}) { ${UAGDA_CERT_MACHINE_AUTH}="C=US, O=The Go Daddy Group, Inc., OU=Go Daddy Class 2 Certification Authority" }

Looking good:

image

image

Sheesh… didn’t think I was that lucky. The red looks very bad:

image

Executing Set UAG DirectAccess Client - Clients Access Enabling Tunnel - All. ... failed. CA name not specified Usage: add rule name=<string> endpoint1=anylocalsubnetdnsdhcpwinsdefaultgatew ay <IPv4 address><IPv6 address><subnet><range><list> endpoint2=anylocalsubnetdnsdhcpwinsdefault gateway <IPv4 address><IPv6 address><subnet><range><list> action=requireinrequestoutrequestinreques tout requireinrequireoutrequireinclearoutnoauthentication [description=<string>] [mode=transport

Great, so now I’ve made it a bit further into the execution but did I really make any progress? Through tracing the script, it didn’t look like there was progress because the line where the script seemingly fails is here:

# Setting ipsec DA tunnel rule

PsNetshExec "Set UAG DirectAccess Gateway - Clients Access Enabling Tunnel - All." "localhost" "advfirewall consec add rule name=```"UAG DirectAccess Gateway - Clients Access Enabling Tunnel - All```" description=```"Policies to enable access granting resources(DC, DNS, NAP, etc.) over IPsec. Generated on Thursday, 09 September 2010 14:11 UTC.```" mode=tunnel action=requireinrequireout profile=private,public localtunnelendpoint=```"${UAGDA_DTE_ACCESS}```" remotetunnelendpoint=```"any```" endpoint1=```"${UAGDA_ACCESS_ENABLING_ADDRESSES_ALL}```" endpoint2=```"any```" auth1=computercert auth1ca=```"${UAGDA_CERT_MACHINE_AUTH} catype:${UAGDA_CERT_TYPE}```" auth2=userntlm qmsecmethods=```"${UAGDA_IPSEC_QM_SECMETHODS}```" "

This PsNetSHExec command is calling variables and one of the variables it was calling was UAGDA_CERT_MACHINE_AUTH which we can also see is the line that we’ve been having problems with earlier:

${UAGDA_CERT_MACHINE_AUTH}="C=US, O="The Go Daddy Group, Inc.", OU=Go Daddy Class 2 Certification Authority" }
if (-not ${UAGDA_CERT_TYPE}) { ${UAGDA_CERT_TYPE}="root" }

So now I’m back to where I was seeing how the error explicitly points out that there’s a problem with the CA specified:

Executing Set UAG DirectAccess Client - Clients Access Enabling Tunnel - All. ...
failed. CA name not specified Usage: add rule name=<string> endpoint1=anylocalsubnetdnsdhcpwinsdefaultgatew
ay <IPv4 address><IPv6 address><subnet><range><list> endpoint2=anylocalsubnetdnsdhcpwinsdefault
gateway <IPv4 address><IPv6 address><subnet><range><list> action=requireinrequestoutrequestinreques

This was when I went back to looking at the script and tried a few suggestions I got from the partner forum and isaserver.org forum here are the quotes:

James Jones:

You could try using double quotes or using a single quote before the double quotes as an escape character.
E.g.
if (-not ${UAGDA_CERT_MACHINE_AUTH}) { ${UAGDA_CERT_MACHINE_AUTH}="C=US, O=""The Go Daddy Group, Inc."", OU=Go Daddy Class 2 Certification Authority" }
or if (-not ${UAGDA_CERT_MACHINE_AUTH}) { ${UAGDA_CERT_MACHINE_AUTH}="C=US, O='"The Go Daddy Group, Inc."', OU=Go Daddy Class 2 Certification Authority" }
Cheers
JJ

---------------------------------------------------------------------------------------------

Brandon Jiang

Meanwhile, I performed some research on the information you posted, I agreed with you that it appears the issue was caused by the extra quotation mark (“ ”) in the subject name of this Go Daddy certificate.

if (-not ${UAGDA_CERT_MACHINE_AUTH}) { ${UAGDA_CERT_MACHINE_AUTH}="C=US, O="The Go Daddy Group, Inc.", OU=Go Daddy Class 2 Certification Authority" }

Sorry again for I don’t have any idea on UAG and this script. Is it possible for you to change this script and you may change “ to \” and have a try?

---------------------------------------------------------------------------------------------

Unfortunately, both didn’t work (thanks for the effort guys) and I continued to try every single variation I could think digging into my java programming and SQL query days in university:

image

So what do I know right now? I know that my colleague who originally ran into this problem earlier last week tried using our internal CA certificate and it worked but for some reason this Go Daddy certificate didn’t. I trust that this was true so opened up the Go Daddy and our internal Root certificate and compared them side by side. Unfortunately, the internal certificate we have does not have a space in the O= value (sorry, I can’t post a screenshot of it because internal IT wouldn’t like me doing it). This is when I went ahead and tried to use another certificate that had a space in the O= value as the certificate we’re using where the top is either the root or intermediate certificate you want to verify certificates sent to DirectAccess clients. The one I found was from Equifax:

image image

Coincidentally, and I’m sure the IT gods were helping, this certificate also had a period in the O= value which will trigger another hint as to what was causing the problem. For reference, here’s the Go Daddy certificate we’re having problems with:

image

So after assigning the Equifax certificate and running the wizard, I found that I did not run into the problem. I then went ahead to export the ps1 script, carefully look at the line and I can see that the format is exactly the same where there would be quotes to enclose the O= value. I then manually executed that script and the process completed successfully. YIKES!

I’m sure at this point many of you probably figured it out so long story short, the problem was the comma that was in the O= value for the Go Daddy certificate where we have:

O = The Go Daddy Group, Inc.

This was when I went ahead and changed the script to

# Static variables
if (-not ${UAGDA_ACCESS_ENABLING_ADDRESSES_ALL}) { ${UAGDA_ACCESS_ENABLING_ADDRESSES_ALL}="2002:480e:ae7a:8001::ac14:105,2002:480e:ae7a:8000:0:5efe:172.20.1.5,2002:480e:ae7a:8001::ac14:10a,2002:480e:ae7a:8000:0:5efe:172.20.1.10,2002:480e:ae7a:8001::ac14:108,2002:480e:ae7a:8000:0:5efe:172.20.1.8,2002:480e:ae7a:8001::ac14:172,2002:480e:ae7a:8000:0:5efe:172.20.1.114,2002:480e:ae7b::480e:ae7b" }
if (-not ${UAGDA_CERT_MACHINE_AUTH}) { ${UAGDA_CERT_MACHINE_AUTH}="C=US, O=The Go Daddy Group"," Inc., OU=Go Daddy Class 2 Certification Authority" }
if (-not ${UAGDA_CERT_TYPE}) { ${UAGDA_CERT_TYPE}="root" }

…which now encloses the comma with the quotes, executed the script and WOW, it finally completed! Here’s the short version:

O=The Go Daddy Group"," Inc.,

As simple as it was, this problem took me a total of 3 hours to figure out but I’m glad I got through it and it pretty much made my day in a good way!

UAG – 0

Terence – 1

:-)

Update:

I finally got extra cycles today to continue the UAG configuration and had a hard time finding the Activate button that the TechNet deployment guide keeps referring to. After clicking around the screen, the button is actually located on the top left hand corner:

image

Once you click on that button, you’ll see the following screen:

image

Once completed, you’ll see this window:

image