Experiencing slow boot up times during an update from ESX 3.5 to ESX/ESXi 4.0 or 4.1? Check your RDMs.

I ran into an interesting problem the other day while refreshing one of our client’s Virtual Infrastructure 3 (VI3) hosts to vSphere 4.1. The ESX cluster had clustered SQL servers configured with MSCS and raw device mappings from a NetApp SAN. Since we’re adding additional hosts to the cluster, the plan I had in place was to start off with installing ESXi 4.1.0 on the new hosts, test to ensure the hosts are configured properly (i.e. network, storage, etc) and stable, then schedule a window to migrate existing virtual machines over. After getting a few of the new hosts installed and included in the initiator groups for the existing LUNs, I noticed that the servers would take upwards of 15 to 20 minutes to boot. Through looking at the console during the boot up, I can see that it gets stuck at the LUN detection line.

After going through additional troubleshooting steps to isolate the problem (I didn’t know it was the RDMs), I went ahead and called VMware. What I was told was that the timeout for synchronous commands during boot up was changed in ESX and ESXi 4.x and therefore causes the host to take a very long time to boot when there are LUNs presented to the host but these LUNs had persistent reservations. In our case, it was the LUNs presented to the SQL cluster built on top of Microsoft Clustering Services (MSCS). The support engineer then proceeded to give me instructions documented in an internal KB that hasn’t been published yet to the public to change the timeout parameter to a lower value:

For vSphere 4.1 you need to modify the Scsi.CRTimeoutDuringBoot parameter form the GUI

1. Go to Host > Configuration > Advanced Settings.
2. Select SCSI.
3. Change the Scsi.CRTimeoutDuringBoot value to 10000.

Once I changed the value to 10000 which equals to 10 seconds), the host booted up properly.

The support engineer also had me verify the following which I wasn’t able to find ESXi 4.1.0:

To modify the Scsi.ConflictRetries parameter from the GUI:

1. Go to Host > Configuration > Advanced settings.
2. In the Advanced settings window, select SCSI.
3. Change the Scsi.UWConflictRetries value from the default 1000 to 80.

-------------------------------------------------------------------------------------------------------------------------------------

I was told that the retry value in ESX 3.5 is apparently set to a lower value which was why we never experienced this issue until we started upgrading the hosts.

Event ID log errors: 1017, 1022 logged after installing .NET 3.5 SP1

We have an environment with a Windows Server 2008 R1 64-Bit SP2 server for a web application we host and during a recent maintenance window, one of the developers requested that we install .NET 3.5 SP1 to fix an issue they’ve been experiencing with .NET 3.5.

So during the maintenance window, I went ahead to install .NET 3.5 SP1 which I downloaded from: http://support.microsoft.com/kb/959209. However, once I installed the service pack, the developers told me that the performance counter for ASP.NET State service no longer functions properly and because they use this to monitor the concurrent user to the system, they needed this fixed as soon as possible.

The troubleshooting began at the server’s event logs where I found Event ID 1017 and 1022 errors logged sequentially in the Application logs. The following are the errors:

Level: Error

Source: Perflib

Event ID: 1017

The following are the detailed event logs:

Log Name: Application

Source: Perflib

Event ID: 1017

Level: Error

Disabled performance counter data collection from the "aspnet_state" service because the performance counter library for that service has generated one or more errors. The errors that forced this action have been written to the application event log. Correct the errors before enabling the performance counters for this service.

------------------------------------------------------------------------------------------------------------------------------------------------

Log Name: Application

Source: Perflib

Event ID: 1022

Level: Error

Windows cannot open the 64-bit extensible counter DLL aspnet_state in a 32-bit environment. Contact the file vendor to obtain a 32-bit version. Alternatively if you are running a 64-bit native environment, you can open the 64-bit extensible counter DLL by using the 64-bit version of Performance Monitor. To use this tool, open the Windows folder, open the System32 folder, and then start Perfmon.exe.

------------------------------------------------------------------------------------------------------------------------------------------------

Log Name: Application

Source: Perflib

Event ID: 1017

Level: Error

Disabled performance counter data collection from the "ASP.NET_64_2.0.50727" service because the performance counter library for that service has generated one or more errors. The errors that forced this action have been written to the application event log. Correct the errors before enabling the performance counters for this service.

------------------------------------------------------------------------------------------------------------------------------------------------

Log Name: Application

Source: Perflib

Event ID: 1022

Level: Error

Windows cannot open the 64-bit extensible counter DLL ASP.NET_64_2.0.50727 in a 32-bit environment. Contact the file vendor to obtain a 32-bit version. Alternatively if you are running a 64-bit native environment, you can open the 64-bit extensible counter DLL by using the 64-bit version of Performance Monitor. To use this tool, open the Windows folder, open the System32 folder, and then start Perfmon.exe.

------------------------------------------------------------------------------------------------------------------------------------------------

Log Name: Application

Source: Perflib

Event ID: 1017

Level: Error

Disabled performance counter data collection from the "ASP.NET_64" service because the performance counter library for that service has generated one or more errors. The errors that forced this action have been written to the application event log. Correct the errors before enabling the performance counters for this service.

------------------------------------------------------------------------------------------------------------------------------------------------

Log Name: Application

Source: Perflib

Event ID: 1022

Level: Error

Windows cannot open the 64-bit extensible counter DLL ASP.NET_64 in a 32-bit environment. Contact the file vendor to obtain a 32-bit version. Alternatively if you are running a 64-bit native environment, you can open the 64-bit extensible counter DLL by using the 64-bit version of Performance Monitor. To use this tool, open the Windows folder, open the System32 folder, and then start Perfmon.exe.

------------------------------------------------------------------------------------------------------------------------------------------------

One of the application developer briefly looked at the problem earlier during the day and forwarded the following KB which does not map exactly to our problem but wanted to try it out:

http://support.microsoft.com/kb/969722

Due to the severity of the issue and sensitivity of the environment, I went ahead to post a question on our Partner Support forums and received the following response (summarized):

1. Install the hotfix the developer suggested—KB981201.
2. If that doesn’t fix it, remove .NET 3.5 SP1 and then reinstall.
3. If the problem still exists, install the additional hotfix KB959209.
4. If the problem still exists, try .NET 4.
5. If the problem still exists, completely remove .NET with cleanup tool and reinstall.

Unfortunately and not surprisingly, the developer did not want .NET 4.0 installed so we opted to try and install the hotfixes as suggested. The following were the results:

The hotfixes listed below that were found via the KB articles listed above:

1. Windows6.0-KB958481-x64.msu
2. Windows6.0-KB958483-x64.msu

…would not installed and the install states the following:

The hotfix that did end up installing and fixing the issue was:

NDP35SP1-KB958484-x64.exe

The server was also rebooted after this hotfix was installed and once the server was rebooted, the event ID 1017, 1022 errors were no longer logged in the application event logs.

The following are some extra screenshots I did while testing the perfmon counters after I installed the hotfix:

I hope this can provide help to any other professionals out there that may encounter the same problem.

OCS 2007 R2: “Some calls to and from people outside of your corporate network may not connect due to server connectivity problems… ” message is intermittently displayed to MOC users but calls continue to work

We had a client out West who have been undergoing network infrastructure changes and after completing the majority of the project, they began to notice that an error message intermittently gets displayed for users when they use MOC to call other remote users connected through the Edge server. The error message reads:

Some calls to and from people outside of your corporate network may not connect due to server connectivity problems. Try signing out and signing back in. If this problem continues, contact your system administrator with this information.

Since this appears to only affect calls from the internal network going out through the Edge server, I went straight to the front-end server and ran the following validation tests:

Front-End Server Validation:

Web Conferencing Server Validation:

A/V Conferencing Server Validation:

To eliminate a connectivity issue, I did a simple telnet from the front-end server to the edge server via the port 5062 and the connection establishes. I also checked the static persistent routes on the edge server to ensure the appropriate routes for the 172.x and 10.x subnets are in there.

All of these errors point to a mismatch of what the front-end server anticipates and what is presented. In this client’s case, it’s because the internal interface on the Edge server is currently using the public certificate with the name sip.companyName.com. When the internal front-end server initiates a connection over to the Edge server, it uses the name “xxx-edge-01.someDomain.local”.

To confirm that the edge server is indeed presenting the wrong interface, we can look at what certificate is assigned as shown here in the properties:

I went ahead and double checked the sip.companyName.com certificates and none of them have a SAN entry for the internal name. This means that they should use an internally issued certificate with the proper internal edge server name for the internal interface. The screenshot above shows that there’s actually such a certificate but it’s not assigned.

Resolution

The resolution is simple: all we need to do is assign the internal certificate. I’ve verified the expiry date for the certificate and it hasn’t expired yet.

Once I changed the certificate, the error in the validation went away and users stopped receiving the error message.

OCS 2007 R2 Voice Properties’ Location Profile unusable due to invalid translation rule

I ran into an interesting issue a few months back when a client with OCS 2007 R2 called our support services because all of the translation rules in his location profile simply stopped working. The problem was eventually escalated to me so I logged into the client’s network remotely and started looking around. It’s been more than 4 months so I couldn’t remember what I looked at before arrived to the solution so long story short, I eventually opened up Enterprise Voice Route Helper:

…and imported the routing data:

What I was basically looking for was to see if there was a normalization rule put at the top of the rules list that somehow caught all cases. I wasn’t able to spot anything out of the ordinary while scrolling down the list so I tried to perform and ad-hoc test with my cell number and this was where I got luck because I got the following message in the results:

The specified profile contains one or more rules that were invalid.

The profile cannot be used.

As shown in the screenshot above, there was a translation rule entered into the profile with an improper syntax and thus rendered the whole location profile unusable. This was an obvious typo the client made and I found it very interesting that the management console actually accepted it. Everything started working once I fixed the missing bracket.

How to forcefully reassign assigned disks on a NetApp Filer

We had to do some emergency maintenance with a new NetApp shelf a few days ago and found that because we were working with an older version of the firmware, there were commands that did not exist on the filer so we ended up using the GUI and CLI to work around the problem.

Note: Some of the commands may not be necessary but I wanted to list all the steps we had to take to get this to work so I’ve highlighted the steps that is possibly not needed in RED. Please also take note that we had a small window to work with so the instructions below may not abide by best practices.

Task: Disks in the new shelf has been assigned to 2 separate controllers (6 each).

Problem: We could not find the command to unassign the disks in this version of the firmware so we had to use the combination of the GUI and CLI to remove the disk from the controller and then reassign it to another.

NetApp Information

Filer: Some Name

Model: FAS2020

Version: 7.2.6.1

---------------------------------------------------------------------------------------------------------------------------------------------

Initiating a disk show shows the following:

login as: root
root@172.20.1.131's password:

login as: root
root@172.20.1.131's password:

FAC01> disk show
DISK OWNER POOL SERIAL NUMBER
------------ ------------- ----- -------------
0c.00.11 FAC02 (135048330) Pool0 4AD7XX3Q00009925DGCC
0c.00.4 FAC02 (135048330) Pool0 4AD7XX4800009928G98U
0c.00.2 FAC02 (135048330) Pool0 4AD7XX8G00009928PPLH
0c.00.6 FAC02 (135048330) Pool0 3QP0ZC2P00009926HBXK
0c.00.0 FAC02 (135048330) Pool0 4AD7XXC900009928PP3P
0c.00.7 FAC01 (135048291) Pool0 4AD7XXEW00009928GNHA
0c.00.10 FAC01 (135048291) Pool0 4NE20JG400009926HEZS
0c.00.3 FAC01 (135048291) Pool0 4NE20EC100009928G4P0
0c.00.1 FAC01 (135048291) Pool0 3QP0WZT400009926KCBN
0c.00.5 FAC01 (135048291) Pool0 4NE20FK5000099171W7V
0c.00.8 FAC01 (135048291) Pool0 3QP0ZRF100009926HEJT
0c.00.9 FAC01 (135048291) Pool0 4AD7XXHG00009928GDQW
0b.18 FAC01 (135048291) Pool0 XX-XXXXX8038093
0b.22 FAC01 (135048291) Pool0 XX-XXXXX8095587
0b.21 FAC01 (135048291) Pool0 XX-XXXXX7977856
0b.20 FAC01 (135048291) Pool0 XX-XXXXX8039600
0b.24 FAC01 (135048291) Pool0 XX-XXXXX8039114
0b.17 FAC01 (135048291) Pool0 XX-XXXXX8015493
0b.29 FAC02 (135048330) Pool0 XX-XXXXX8095049
0b.23 FAC02 (135048330) Pool0 XX-XXXXX8039807
0b.19 FAC01 (135048291) Pool0 XX-XXXXX7977641
0b.27 FAC02 (135048330) Pool0 XX-XXXXX8039645
0b.28 FAC01 (135048291) Pool0 XX-XXXXX7978194
0b.26 FAC01 (135048291) Pool0 XX-XXXXX8039342
0b.25 FAC02 (135048330) Pool0 XX-XXXXX8039954
0b.16 FAC01 (135048291) Pool0 XX-XXXXX7977866
FAC01>

What we want to do is move disks: ob.17, 0b.19 and 0b.21 off of FAC01 to FAC02.

A disk ? shows the following:

FAC01> disk ?
usage: disk <options>
Options are:
fail [-i] [-f] <disk_name> - fail a file system disk
remove [-w] <disk_name> - remove a spare disk
swap - prepare (quiet) bus for swap
unswap - undo disk swap and resume service
scrub { start stop } - start or stop disk scrubbing
assign {<disk_name> all -n <count> auto} [-p <pool>] [-o <ownername>] [-s <sysid>] [-c blockzoned] [-f] - assign a disk to a filer or all unowned disks by specifying "all" or <count> number of unowned disks
show [-o <ownername> -s <sysid> -n -v -a] - lists disks and owners
replace {start [-f] [-m] <disk_name> <spare_disk_name>} {stop <disk_name>} - replace a file system disk with a spare disk or stop replacing
zero spares - Zero all spare disks
checksum {<disk_name> all} [-c block zoned]
sanitize { start abort status release } - sanitize one or more disks
maint { start abort status list} - run maintenance tests on one or more disks
FAC01>

We tried the remove option as well as the replace but found that just as the description specifies, these commands expect you to be moving the spares around. As we were pressed for time to get the disks reassigned, we went into the GUI to offline these disks by setting them to remove:

After removing these 3 disks from the GUI, a disk show now shows the following:

FAC01> disk show
DISK OWNER POOL SERIAL NUMBER
------------ ------------- ----- -------------
0c.00.11 FAC02 (135048330) Pool0 4AD7XX3Q00009925DGCC
0c.00.4 FAC02 (135048330) Pool0 4AD7XX4800009928G98U
0c.00.2 FAC02 (135048330) Pool0 4AD7XX8G00009928PPLH
0c.00.6 FAC02 (135048330) Pool0 3QP0ZC2P00009926HBXK
0c.00.0 FAC02 (135048330) Pool0 4AD7XXC900009928PP3P
0c.00.7 FAC01 (135048291) Pool0 4AD7XXEW00009928GNHA
0c.00.10 FAC01 (135048291) Pool0 4NE20JG400009926HEZS
0c.00.3 FAC01 (135048291) Pool0 4NE20EC100009928G4P0
0c.00.1 FAC01 (135048291) Pool0 3QP0WZT400009926KCBN
0c.00.5 FAC01 (135048291) Pool0 4NE20FK5000099171W7V
0c.00.8 FAC01 (135048291) Pool0 3QP0ZRF100009926HEJT
0c.00.9 FAC01 (135048291) Pool0 4AD7XXHG00009928GDQW
0b.18 FAC01 (135048291) Pool0 XX-XXXXX8038093
0b.22 FAC01 (135048291) Pool0 XX-XXXXX8095587
0b.21 FAC01 (135048291) FAILED XX-XXXXX7977856
0b.20 FAC01 (135048291) Pool0 XX-XXXXX8039600
0b.24 FAC01 (135048291) Pool0 XX-XXXXX8039114
0b.17 FAC01 (135048291) FAILED XX-XXXXX8015493
0b.29 FAC02 (135048330) Pool0 XX-XXXXX8095049
0b.23 FAC02 (135048330) Pool0 XX-XXXXX8039807
0b.19 FAC01 (135048291) FAILED XX-XXXXX7977641
0b.27 FAC02 (135048330) Pool0 XX-XXXXX8039645
0b.28 FAC01 (135048291) Pool0 XX-XXXXX7978194
0b.26 FAC01 (135048291) Pool0 XX-XXXXX8039342
0b.25 FAC02 (135048330) Pool0 XX-XXXXX8039954
0b.16 FAC01 (135048291) Pool0 XX-XXXXX7977866
FAC01>

Since the disk reassign command can only be run in maintenance mode or during takeover in advanced mode, we executed the disk remove_ownership command but before we can execute that command, we needed to elevate our privileges to advanced:

FAC01> priv set advanced
Warning: These advanced commands are potentially dangerous; use
them only when directed to do so by Network Appliance
personnel.
FAC01*>

Then we executed:

FAC01*> disk remove_ownership 0b.17
Note: Disks may be automatically assigned to this node, since option disk.auto_assign is on.
Volumes must be taken offline. Are all impacted volumes offline(y/n)?? y

FAC01*> disk remove_ownership 0b.19
Note: Disks may be automatically assigned to this node, since option disk.auto_assign is on.
Volumes must be taken offline. Are all impacted volumes offline(y/n)?? y

FAC01*> disk remove_ownership 0b.21
Note: Disks may be automatically assigned to this node, since option disk.auto_assign is on.
Volumes must be taken offline. Are all impacted volumes offline(y/n)?? y
FAC01*>

The following is the output when we execute a disk show after the above commands were completed:

FAC01> disk show
DISK OWNER POOL SERIAL NUMBER
------------ ------------- ----- -------------
0c.00.11 FAC02 (135048330) Pool0 4AD7XX3Q00009925DGCC
0c.00.4 FAC02 (135048330) Pool0 4AD7XX4800009928G98U
0c.00.2 FAC02 (135048330) Pool0 4AD7XX8G00009928PPLH
0c.00.6 FAC02 (135048330) Pool0 3QP0ZC2P00009926HBXK
0c.00.0 FAC02 (135048330) Pool0 4AD7XXC900009928PP3P
0c.00.7 FAC01 (135048291) Pool0 4AD7XXEW00009928GNHA
0c.00.10 FAC01 (135048291) Pool0 4NE20JG400009926HEZS
0c.00.3 FAC01 (135048291) Pool0 4NE20EC100009928G4P0
0c.00.1 FAC01 (135048291) Pool0 3QP0WZT400009926KCBN
0c.00.5 FAC01 (135048291) Pool0 4NE20FK5000099171W7V
0c.00.8 FAC01 (135048291) Pool0 3QP0ZRF100009926HEJT
0c.00.9 FAC01 (135048291) Pool0 4AD7XXHG00009928GDQW
0b.18 FAC01 (135048291) Pool0 XX-XXXXX8038093
0b.22 FAC01 (135048291) Pool0 XX-XXXXX8095587
0b.21 FAC01 (135048291) FAILED XX-XXXXX7977856
0b.20 FAC01 (135048291) Pool0 XX-XXXXX8039600
0b.24 FAC01 (135048291) Pool0 XX-XXXXX8039114
0b.17 FAC01 (135048291) FAILED XX-XXXXX8015493
0b.29 FAC02 (135048330) Pool0 XX-XXXXX8095049
0b.23 FAC02 (135048330) Pool0 XX-XXXXX8039807
0b.19 FAC01 (135048291) FAILED XX-XXXXX7977641
0b.27 FAC02 (135048330) Pool0 XX-XXXXX8039645
0b.28 FAC01 (135048291) Pool0 XX-XXXXX7978194
0b.26 FAC01 (135048291) Pool0 XX-XXXXX8039342
0b.25 FAC02 (135048330) Pool0 XX-XXXXX8039954
0b.16 FAC01 (135048291) Pool0 XX-XXXXX7977866
FAC01>

It almost looks like nothing was changed and here’s why:

FAC01*> disk remove_ownership 0b.17
Note: Disks may be automatically assigned to this node, since option disk.auto_assign is on.
Volumes must be taken offline. Are all impacted volumes offline(y/n)?? y

Notice that the message indicates that disk.auto_assign is turned on so in order to have these disks remain unassigned, we need to execute the following:

FAC01*> options disk.auto_assign off
You are changing option disk.auto_assign which applies to both members of
the cluster in takeover mode.
This value must be the same in both cluster members prior to any takeover
or giveback, or that next takeover/giveback may not work correctly.
Sun Sep 26 21:25:53 EST [PHMSFAC01: reg.options.cf.change:warning]: Option disk.auto_assign changed on one cluster node.
FAC01*>

FAC02*> options disk.auto_assign off
You are changing option disk.auto_assign which applies to both members of
the cluster in takeover mode.
This value must be the same in both cluster members prior to any takeover
or giveback, or that next takeover/giveback may not work correctly.
Sun Sep 26 21:25:53 EST [PHMSFAC01: reg.options.cf.change:warning]: Option disk.auto_assign changed on one cluster node.
FAC02*>

Now in order to reassign these disks, we had to

unfail it with the command:

disk unfail <disk name>

disk unfail 0b.19

disk unfail 0b.17

disk unfail0b.21

Here’s what the SSH session looks like:

FAC01*> disk unfail 0b.17
disk unfail: unfailing disk 0b.17...
FAC01*> Sun Sep 26 21:28:16 EST [FAC01: raid.disk.unfail.reassim:info]: Disk 0b.17 Shelf 1 Bay 1 [WDC WD1002FBYS-05ASX NA01] S/N [WD-WMATV8015493] was unfailed, and is now being reassimilated
disk unfail 0b.19
disk unfail: unfailing disk 0b.19...
FAC01*> Sun Sep 26 21:28:23 EST [FAC01: raid.disk.unfail.reassim:info]: Disk 0b.19 Shelf 1 Bay 3 [WDC WD1002FBYS-05ASX NA01] S/N [WD-WMATV7977641] was unfailed, and is now being reassimilated
disk unfail 0b.21
disk unfail: unfailing disk 0b.21...
FAC01*> Sun Sep 26 21:28:27 EST [FAC01: raid.disk.unfail.reassim:info]: Disk 0b.21 Shelf 1 Bay 5 [WDC WD1002FBYS-05ASX NA01] S/N [WD-WMATV7977856] was unfailed, and is now being reassimilated

Now that we’ve unfailed the disks as well as turned off disk.auto_assign, we can execute the remove_ownership command again:

FAC01*> disk remove_ownership 0b.17
Volumes must be taken offline. Are all impacted volumes offline(y/n)?? y
FAC01*> disk remove_ownership 0b.19
Volumes must be taken offline. Are all impacted volumes offline(y/n)?? y
FAC01*> disk remove_ownership 0b.21
Volumes must be taken offline. Are all impacted volumes offline(y/n)?? y
FAC01*> disk show
DISK OWNER POOL SERIAL NUMBER
------------ ------------- ----- -------------
0c.00.11 FAC02 (135048330) Pool0 3QP0YV3Q00009925DGCC
0c.00.4 FAC02 (135048330) Pool0 3QP0YV4800009928G98U
0c.00.2 FAC02 (135048330) Pool0 3QP0YV8G00009928PPLH
0c.00.6 FAC02 (135048330) Pool0 3QP0ZC2P00009926HBXK
0c.00.0 FAC02 (135048330) Pool0 3QP0YVC900009928PP3P
0c.00.7 FAC01 (135048291) Pool0 3QP0YVEW00009928GNHA
0c.00.10 FAC01 (135048291) Pool0 3QP10JG400009926HEZS
0c.00.3 FAC01 (135048291) Pool0 3QP10EC100009928G4P0
0c.00.1 FAC01 (135048291) Pool0 3QP0WZT400009926KCBN
0c.00.5 FAC01 (135048291) Pool0 3QP10FK5000099171W7V
0c.00.8 FAC01 (135048291) Pool0 3QP0ZRF100009926HEJT
0c.00.9 FAC01 (135048291) Pool0 3QP0YVHG00009928GDQW
0b.18 FAC01 (135048291) Pool0 WD-WMATV8038093
0b.22 FAC01 (135048291) Pool0 WD-WMATV8095587
0b.20 FAC01 (135048291) Pool0 WD-WMATV8039600
0b.24 FAC01 (135048291) Pool0 WD-WMATV8039114
0b.29 FAC02 (135048330) Pool0 WD-WMATV8095049
0b.23 FAC02 (135048330) Pool0 WD-WMATV8039807
0b.27 FAC02 (135048330) Pool0 WD-WMATV8039645
0b.28 FAC01 (135048291) Pool0 WD-WMATV7978194
0b.26 FAC01 (135048291) Pool0 WD-WMATV8039342
0b.25 FAC02 (135048330) Pool0 WD-WMATV8039954
0b.16 FAC01 (135048291) Pool0 WD-WMATV7977866
NOTE: Currently 3 disks are unowned. Use 'disk show -n' for additional information.
FAC01*>

Notice how there are 3 disks stated as being unowned now. The final step is to hop over to the controller that you want to assign the disks and execute the following:

FAC02*> disk assign 0b.17
disk assign: Assign failed for one or more disks in the disk list.
FAC02*> disk assign 0b.17
Sun Sep 26 21:39:34 EST [FAC02: diskown.changingOwner:info]: changing ownership for disk 0b.17 (S/N WD-WMATV8015493) from unowned (ID -1) to FAC02 (ID 135048330)
FAC02*> disk assign 0b.19
Sun Sep 26 21:39:39 EST [FAC02: diskown.changingOwner:info]: changing ownership for disk 0b.19 (S/N WD-WMATV7977641) from unowned (ID -1) to FAC02 (ID 135048330)
FAC02*> disk assign 0b.21
Sun Sep 26 21:39:43 EST [FAC02: diskown.changingOwner:info]: changing ownership for disk 0b.21 (S/N WD-WMATV7977856) from unowned (ID -1) to FAC02 (ID 135048330)
FAC02*>

A disk show now shows the 3 disks being assigned to the other active controller.

Sorry about the extra steps I included so they may or may not be required to change assigned disks from one controller to the other.

Reminder: vCenter 4.1 uses 64-bit DSN | vCenter 4.0 uses 32-bit DSN

While performing some troubleshooting with a client a few weeks ago on their vCenter 4.1 server, I learned that vCenter 4.1 actually uses a 64-bit DSN. This meant that we no longer have to go into:

1. Install the 32-bit client via Microsoft’s downloads
2. Run 32-bit ODBC via the WoW64 folder
3. Create a 32-bit DSN

…which you had to do for a vCenter 4.0 install.

The following is the difference between the 2 version’s:

vCenter 4.0

Database Options

Select an ODBC data source for vCenter Server.

vCenter Server requires a database.

Use an existing supported database

Data Source Name (DSN): (Please create a 32-bit system (DSN)

vCenter 4.1

Database Options

Select an ODBC data source for vCenter Server.

vCenter Server requires a database.

Use an existing supported database

Data Source Name (DSN): (Please create a 64-bit system (DSN)

Now I’m just waiting for VUM (Update Manager) 4.1 to support a 64-bit ODBC DSN because it’s currently still a requirement for it to be installed on a 64-bit Operating System yet use a 32-bit DSN.

Update

The following might be helpful when configuring the 64-bit DSN:

Make sure you select SQL Server Native Client 10.0 for the System DSN and not the regular SQL Server or the vCenter installation wizard won’t detect it (you can’t just hit the back button then forward again to see the DSN).

Example with NetApp for realistic expectations of raw and usable capacity

Warning: I’m not a SAN expert but as I’ve gotten more opportunities to work in datacenter projects, I’m beginning to see more real world SAN implementations and while this doesn’t provide a complete breakdown of what to consider while calculating raw and usable storage, I hope this will at the very least provide some useful information to professionals out there looking for some real world numbers when provisioning SAN storage.

Configuration

Brand: NetApp

Model: FAS2020

Version: 7.2.6.1

RAID: RAID_DP (Double Parity)

RAID Size: 16

Number of Disks: 6

Disk Size: 300GB SAS

Actual usual disk size: 266GB

Total Aggregate Capacity: 908GB

As shown with the information listed above, configuring a FAS2020 with 6 x 300GB SAS drives realistically yields only 908GB for the aggregate. Working out the numbers we can see that:

Specifications on paper: 300GB x 6 disks = 1.8TB

Actual drive capacity: 266 x 6 disks = 1.598TB

Actual useable *aggregate* capacity after RAID_DP: 0.908GB

If we divide the numbers to get the amount of storage space you lose from the overhead such as RAID, we’re actually losing approximately: 51% of drive space. This 51% also does not include the spare disk you’ll need per controller (you need a disk for each controller so if you have an active/active setup, you’ll need 2 disks for each controller). Also don’t forget that the software for the controller also sits in aggregate 0 on the NetApp which means that will take up additional space. As of the year 2009, the NetApp technician told me that a minimum of 10GB is required for the root volume and 20GB is recommended for the FAS2020.

Lastly, as new volumes are created for LUNs, your volume needs more space than the actual LUN and the reason for this is because you will need extra space if you decide to use snapshots. Best practice as told by the NetApp engineer is that you should have 2x + delta (x being the size of the LUN) extra space because it covers the situation if a snapshot is taken of the LUN (let's say LUN was completely full), deleted all information on the LUN, filled it back up with different information but because the 2x + delta was followed, this means that your snapshot can hold all the information prior to deleting the original information. With that being said, as most companies don’t like to lose so much storage, another good practice is to use 1x + delta (x being the size of the LUN).

There are times when thinking about all the reasons that contribute to lost storage in exchange for redundancy often scares me so I find that it’s ever so much more important to communicate to customers all the variables and set their expectations appropriately.

-------------------------------------------------------------------------------------------------------------------------------------------------------

The following is another example similar to the configuration above but with 1TB drives:

Configuration

Brand: NetApp

Model: FAS2020

Version: 7.2.6.1

RAID: RAID_DP (Double Parity)

RAID Size: 6

Number of Disks: 6

Disk Size: 1TB SAS

Actual usual disk size: 828GB

Total Aggregate Capacity: 2.76TB

As shown with the information listed above, configuring a FAS2020 with 6 x 1TB SAS drives realistically yields only 2.76GB for the aggregate. Working out the numbers we can see that:

Specifications on paper: 1TB x 6 disks = 6TB

Actual drive capacity: 828 x 6 disks = 4.968TB

Actual useable *aggregate* capacity after RAID_DP: 2.76TB

If we divide the numbers to get the amount of storage space you lose from the overhead such as RAID, we’re actually losing approximately: 54% of drive space. As indicated in the example above, this 54% also does not include other variables that will contribute to more loss in storage space.

-------------------------------------------------------------------------------------------------------------------------------------------------------

Here’s another example similar to the first one but with 4 disks instead:

Configuration

Brand: NetApp

Model: FAS2020

Version: 7.2.6.1

RAID: RAID_DP (Double Parity)

RAID Size: 16

Number of Disks: 4

Disk Size: 300GB SAS

Actual usual disk size: 266GB

Total Aggregate Capacity: 454GB

As shown with the information listed above, configuring a FAS2020 with 4 x 300GB SAS drives realistically yields only 454GB for the aggregate. Working out the numbers we can see that:

Specifications on paper: 300GB x 4 disks = 1.2TB

Actual drive capacity: 266 x 4 disks = 1.02TB

Actual useable *aggregate* capacity after RAID_DP: 454GB

If we divide the numbers to get the amount of storage space you lose from the overhead such as RAID, we’re actually losing approximately: 63% of drive space. Again, this does not include other contributing factors that will decrease the amount of usable storage even more.

-------------------------------------------------------------------------------------------------------------------------------------------------------

Extras

Random notes I took during troubleshooting with NetApp engineer: There are ways to reclaim space such as snapshots for your aggregates, volumes, reducing factional reserve, reducing snapshot schedules’ frequency but they all contribute to reduced redundancy. Also, make sure a 1 LUN per volume mapping is followed in case a volume ever goes down, not all of your LUNs do. Lastly, make sure auto snap auto delete is turned on because if no space is left for snapshots, the NetApp will delete the old one and take the snapshot. If this was not turned on, the LUN will go offline if it fills up without space reservation.

My thoughts: Being as a consultant means we’re obligated to pass the truth about our knowledge to customers and while this may hard to digest for many clients, it’s important not to overlook why companies purchase SANs in the first place: because they want robust storage that provides redundancy, exceptional recovery time and performance and storage companies design their storage solutions with this as their number 1 priority. I’ve been fortunate enough to be at a training session delivered by Peter Henneberry from NetApp and it was quite the eye opener when he gave us real world statistics on how they can complete backups within the seconds or minutes rather than hours so while I can’t state all the benefits of a SAN, there are plenty of reasons.

I’m not much of storage consultant even though I’d like to get into it a bit more so please forgive any mistakes I have made in this post whether it’s calculations or information I have missed.