Powering on and off virtual machines via VMware ESX / ESXi 4.x console or SSH commands

An old colleague of mine reached out to me the other day when he ran into an issue with an vSphere environment where he was not able to connect to vCenter or the host via the vSphere Client.  The challenge he had was that he needed to get the virtual machines up as soon as possible but the only way he was able to access the hosts was through the console.  To make a long story short, the first thing that came to my mind was the “vim-cmd vmsvc” command so I went ahead and walked him through it.  Since I know this is bound to come up again sometime in the future, I thought I should write a blog post that will serve as a reference to people who ask me about performing operations in the console in the future.

The “vim-cmd vmsvc” command provides administrators the ability to configure and execute most operations that the vSphere Client provides.  The following are the available commands:

acquiremksticket          get.configoption          power.on
acquireticket             get.datastores            power.reboot
connect                   get.disabledmethods       power.reset
convert.toTemplate        get.environment           power.shutdown
convert.toVm              get.filelayout            power.suspend
createdummyvm             get.guest                 power.suspendResume
destroy                   get.guestheartbeatStatus  queryftcompat
device.connection         get.managedentitystatus   reload
device.connusbdev         get.networks              setscreenres
device.disconnusbdev      get.runtime               snapshot.create
device.diskadd            get.snapshotinfo          snapshot.dumpoption
device.diskaddexisting    get.summary               snapshot.get
device.diskremove         get.tasklist              snapshot.remove
device.getdevices         getallvms                 snapshot.removeall
device.toolsSyncSet       gethostconstraints        snapshot.revert
device.vmiadd             login                     snapshot.setoption
device.vmiremove          logout                    tools.cancelinstall
devices.createnic         message                   tools.install
get.capability            power.getstate            tools.upgrade
get.config                power.hibernate           unregister
get.config.cpuidmask      power.off                 upgrade

Since the example we’ll be using is to power on a virtual machine that is off, the commands we’ll be used are:

1. getallvms
2. power.getstate
3. power.on

The first information we’ll need to power on a virtual machine is to obtain the Vmid which is a unique identifier that the ESXi host uses to identify the virtual machine.  This information can be retrieved via the getallvms command:

~ # vim-cmd vmsvc/getallvms
Vmid     Name                       File                          Guest OS         Version
112    CONTOSOMNG06   [VMFS Volume 2] CONTOSOMNG06/CONTOSOMNG06.vmx   winLonghornGuest      vmx-07
128    CONTOSODB01    [VMFS Volume 2] CONTOSODB01/CONTOSODB01.vmx     winNetStandardGuest   vmx-07
16     CONTOSOCFL01   [VMFS Volume 1] CONTOSOCFL01/CONTOSOCFL01.vmx   winNetStandardGuest   vmx-07
192    CONTOSOVDR01   [VMFS Volume 4] CONTOSOVDR01/CONTOSOVDR01.vmx   rhel5_64Guest         vmx-07
208    CONTOSOMNG10   [VMFS Volume 4] CONTOSOMNG10/CONTOSOMNG10.vmx   winLonghorn64Guest    vmx-07
64     CONTOSOMX03    [VMFS Volume 1] CONTOSOMX03/CONTOSOMX03.vmx     winNetStandardGuest   vmx-07
80     CONTOSOMNG01   [VMFS Volume 2] CONTOSOMNG01/CONTOSOMNG01.vmx   winNetStandardGuest   vmx-07

Note the Vmid column above.  For this example, the virtual machine we would like to power on will be CONTOSOMX03 with Vmid 64.

The next command we should run which serves more as a sanity check is the power.getstate command to check whether the virtual machine is powered on or off:

~ # vim-cmd vmsvc/power.getstate 64
Retrieved runtime info
Powered off

Now that we know the virtual machine is indeed powered off, we can proceed to power on the virtual machine with the command power.on:

~ # vim-cmd vmsvc/power.on 64
Powering on VM:
~ #

As shown in the initial list of commands available, the vim-cmd vmsvc/ provides administrators with the ability to either manually or automatically (via scripts) to operate on virtual machines without having to vSphere Client over to the host.

Recovering an Exchange 2010 CAS and Hub Transport Server from a corrupted operating system

As mentioned in my previous post:

Recovering an Exchange 2010 Mailbox Server from a corrupted operating system

I had to recently redo an Exchange Server 2010 mailbox server’s operating system because the incorrect version was used to for the OS and in additional to this role, I also had to redo the server with the hub transport and CAS role as well.  More information about why can be found in the previous post so rather than reiterating the reasons, let’s get straight to the process.  The following example will be recovering a CAS server collocated with the hub transport server.

Note that you can find the following 2 TechNet articles is what I used for this operation:

Understanding Backup, Restore and Disaster Recovery – http://technet.microsoft.com/en-us/library/dd876874.aspx

Recover an Exchange Server – http://technet.microsoft.com/en-us/library/dd876880.aspx

Preparation

Prepare the new server’s operating system – the first step is to either create a new virtual machine or reconfigure a physical server and install the Windows Server 2008 R2 operating system.

Configure the IP address – It is important that you configure the new server’s IP address to be the same as the old server.

Reset the computer account of the old Exchange Server – Make sure you do not delete the computer account but rather reset the password with Active Directory Users and Computers.

Name new Exchange 2010 CAS / Hub Transport server – Make sure that you name the new CAS / Hub Transport server to be the same as the old one.

Join new Exchange 2010 CAS / Hub Transport server to the domain – If the new server is still in a workgroup, join it to the domain.

Prepare the new Exchange 2010 CAS / Hub Transport server’s roles and features – You can either manually install all of the prerequisites or use the script found here: http://terenceluk.blogspot.com/2010/07/exchange-2010-resource-domain-install.html.  Note that the Office Filter Pack found in the blog post is 1.0 and with Exchange Server 2010 SP1, you should install the Office Filter Pack 2.0 which can be downloaded from here:  http://go.microsoft.com/fwlink/?LinkID=191548.  The following is what the PowerShell script that installs the prerequisites look like:

 

Restart the server – Make sure you restart the server when you’ve completed the prerequisites install.

Export and import the public certificates – It’s important that you export an import the existing certificates you are using for your CAS server (i.e. OWA, Outlook Anywhere, ActiveSync, etc) into the new one.

Recovering the CAS and Hub Transport Server

Now that all of the prerequisites have been installed and configured, proceed with the following steps:

Exchange Server 2010 SP1 installation files –  Make sure the Exchange Server 2010 SP1 installation files are accessible by the new mailbox server whether it’s a DVD in the DVD-ROM drive or simply an extracted folder.

Open up a command prompt with administrator permissions to run setup – Recovering the Exchange Server 2010’s CAS and Hub Transport server is actually quite simple and the process is to simply execute the setup.exe executable from the installation binaries with the /m:RecoverServer switch as such:

Allow the recovery process to execute – Once you’ve ran setup.exe with the appropriate switch, you’ll see the setup go through the required prerequisite checks and then proceed to reinstall the hub transport and CAS role:

Reboot the server – Once the process completes, proceed with rebooting the server.

Check the services – Once the reboot of the server completes, proceed with checking the services and ensure that all of the automatic ones start up successfully.

Post Mailbox Server Recovery

Test Exchange services – Test all Exchange services (internal and external email, offline address book, public folders, free/busy, etc).

Review event logs – Ensure that you review the new mailbox server’s event logs and make sure there are no unexpected errors logged.

Reconfigure OWA services – Recovering the CAS server will require certain additional configuration to be made as settings such as the logon format for OWA which is provided by IIS is not recovered:

Reconfigure HTTP Redirect – If you’ve chosen to use IIS to redirect traffic going to https://webmail.domain.com to https://webmail.domain.com/owa then you’ll also need to reconfigure the setting:

Reconfigure OWA’s External URL field – The external URL field within the owa (Default Web Site) object in Exchange Management Console is also not carried over so be sure to reconfigure it.

Hope this helps anyone who may be looking for more information about what the recovery process looks like.  I’ll be writing another hub transport and CAS server recovery after this post.

Recovering an Exchange 2010 Mailbox Server from a corrupted operating system

I had to recently redo an Exchange Server 2010 mailbox server’s operating system because the incorrect version was used to for the OS and while I thought about simply using a swing approach where I’d installed another mailbox server, migrate mail, rebuild the operating system of the original server and migrate mail back (what we used to call a “swing migration”), I decided to simply treat the operation as a “disaster recovery” because:

1. An Exchange 2010 mailbox server in evaluation mode can only mount 5 information stores.
2. A swing migration like this would take too much time where as a disaster recovery would probably take me an hour to complete.
3. Since I already have the edb and log files intact, I did not need to do a backup and restore.

So while performing this recovery, I thought it it would be beneficial to document the process allowing me to write this blog post in case someone is looking for a walkthrough with screenshots.

Note that you can find the following 2 TechNet articles is what I used for this operation:

Understanding Backup, Restore and Disaster Recovery – http://technet.microsoft.com/en-us/library/dd876874.aspx

Recover an Exchange Server – http://technet.microsoft.com/en-us/library/dd876880.aspx

Preparation

Prepare the new server’s operating system – the first step is to either create a new virtual machine or reconfigure a physical server and install the Windows Server 2008 R2 operating system.

Configure the IP address – It is important that you configure the new server’s IP address to be the same as the old server.

Reset the computer account of the old Exchange Server – Make sure you do not delete the computer account but rather reset the password with Active Directory Users and Computers.

Name the new Exchange 2010 mailbox server – Make sure that you name the new mailbox server to be the same as the old one.

Join the new Exchange 2010 mailbox server to the domain – If the new server is still in a workgroup, join it to the domain.

Prepare the new Exchange 2010 mailbox server’s roles and features – You can either manually install all of the prerequisites or use the script found here: http://terenceluk.blogspot.com/2010/07/exchange-2010-resource-domain-install.html.  Note that the Office Filter Pack found in the blog post is 1.0 and with Exchange Server 2010 SP1, you should install the Office Filter Pack 2.0 which can be downloaded from here:  http://go.microsoft.com/fwlink/?LinkID=191548.  The following is what the PowerShell script that installs the prerequisites look like:

 

Restart the server – Make sure you restart the server when you’ve completed the prerequisites install.

Configure the Exchange 2010 mailbox information store and log drives – Make sure you configure the Exchange information store (.edb) and log (.log) drives with the same drive name and folder path as the previous mailbox server.  For this example, the information store is stored on the E drive and the logs are stored on the F drive.  Failure to preconfigure those drives will result in messages as such when you try to recover the server:

Recovering the Mailbox Server

Now that all of the prerequisites have been installed and configured, proceed with the following steps:

Exchange Server 2010 SP1 installation files –  Make sure the Exchange Server 2010 SP1 installation files are accessible by the new mailbox server whether it’s a DVD in the DVD-ROM drive or simply an extracted folder.

Open up a command prompt with administrator permissions to run setup – Recovering the Exchange Server 2010’s mailbox server is actually quite simple and the process is to simply execute the setup.exe executable from the installation binaries with the /m:RecoverServer switch as such:

Allow the recovery process to execute – Once you’ve ran setup.exe with the appropriate switch, you’ll see the setup go through the required prerequisite checks and then proceed to reinstall the mailbox role:

Reboot the server – Once the process completes, proceed with rebooting the server.

Check the services – Once the reboot of the server completes, proceed with checking the services and ensure that all of the automatic ones start up successfully.

Mount the information stores – Upon completion of the mailbox server recovery, Exchange will not mount the stores for you so start the Exchange Management Console and manually mount all of the stores.

Post Mailbox Server Recovery

Test Exchange services – Test all Exchange services (internal and external email, offline address book, public folders, free/busy, etc).

Review event logs – Ensure that you review the new mailbox server’s event logs and make sure there are no unexpected errors logged.

Hope this helps anyone who may be looking for more information about what the recovery process looks like.  I’ll be writing another hub transport and CAS server recovery after this post.

Setting up e-mail notifications for backing up Exchange Server 2010 with Windows Server Backup on Windows Server 2008 R2

It’s been awhile since I had to set up e-mail notifications with the feature that’s natively built into Windows Server 2008 R2 and as soon as I remember I haven’t written a blog post for it, I thought it would be a good idea to do so in case someone is searching for it on the internet.

What’s nice about Windows Server 2008, Vista, Windows 7 and Windows Server 2008 R2 is that you can now create triggers for events logged in the Event Viewer and if you had to use Windows Server Backup to backup Exchange Server 2010 because you don’t have a backup solution that can backup the information stores, you can use this feature to set tasks to send e-mail notifications out administrators when the backup completes.  The following example will use this situation to demonstrate how to set up such an alert that is triggered via an event log entry.

The first step in setting up this alert is to open up Server Manager, navigate to:

Diagnostics –> Event Viewer –> Applications and Services Logs –> Microsoft –> Windows –> Backup –> Operational.

Note that if you don’t see the Backup folder, it’s because you haven’t set up a scheduled backup job yet.

As shown in the screenshot above, you will need to locate the proper event entry that gets logged when the backup operation has completed and in this example, it’s the log with Event ID 14.

Now that you’ve located the log, right-click on the event and select: Attach Task To This Event:

A wizard will start up so continue to fill in the appropriate information:

Select the Send an e-mail option:

A summary of the task you’ve created will be presented so review the information and ensure it is correct:

A information window will pop up upon completing the wizard:

It may be a bit difficult to figure out where these event viewer triggers are stored so if you want to make any modifications to the tasks, simply open up the Task Scheduler and navigate to:

Task Scheduler Library –> Microsoft –> Event Viewer Tasks

Fairly simple setup but definitely very useful as you can set triggers for any event entries being logged.

Scheduling multiple backups with Task Scheduler and Windows Server Backup for Exchange Server 2010

I’ve been working on a project for a school that had limited budget this year for their infrastructure and was not able to upgrade their Microsoft DPM 2007 backup software to the latest version and since DPM 2007 does not support backing up the Exchange Server 2010 information stores, the only alternative solution for them was to use the build in Windows Server Backup application that is natively built into Windows Server 2008 R2.  The challenge of using Windows Server Backup was that you could only schedule one backup job from within the GUI and because I had to stagger the backup jobs to have them save the backups to different folders, what I needed to do was manually create backup jobs within Task Scheduler.

The first task was to figure out what the proper syntax was and since backup jobs in Windows Server Backup was essentially a task scheduled with the %windir%\System32\wbadmin.exe command, I simply looked up the switches available for the wbadmin.exe command.  Details and explanation for this command can be found here: http://technet.microsoft.com/en-us/library/cc754015(WS.10).aspx and all the switches to start a backup job can be found here: http://technet.microsoft.com/en-us/library/cc742083(WS.10).aspx.

As shown in the Parameters in the link above, the switches we’re interested are as follows:

Parameter	Description
-backupTarget	Specifies the storage location for this backup. Requires a hard disk drive letter (f:), a volume GUID-based path in the format of \\?\Volume{GUID}, or a Universal Naming Convention (UNC) path to a remote shared folder (\\<servername>\<sharename>\). By default, the backup will be saved at: \\<servername>\<sharename>\WindowsImageBackup\<ComputerBackedUp>\.
-include	For Windows°Vista and Windows Server 2008, specifies the comma-delimited list of volume drive letters, volume mount points, or GUID-based volume names to include in the backup. This parameter should be used only when the -backupTarget parameter is used. For Windows°7 and Windows Server 2008 R2, specifies the comma-delimited list of items to include in the backup. You can include multiple files, folders, or volumes. Volume paths can be specified using volume drive letters, volume mount points, or GUID-based volume names. If you use a GUID-based volume name, it should be terminated with a backslash (\). You can use the wildcard character (*) in the file name when specifying a path to a file. Should be used only when the -backupTarget parameter is used.
-vssFull	Performs a full back up using the Volume Shadow Copy Service (VSS). All files are backed up, each file's history is updated to reflect that it was backed up, and the logs of previous backups may be truncated. If this parameter is not used wbadmin start backup makes a copy backup, but the history of files being backed up is not updated.
-quiet	Runs the subcommand with no prompts to the user.

The schedule I worked out was to backup the information store in different folders on \\backupServer\f$ for different days as shown in the following:

Monday	Tuesday	Wednesday	Thursday	Friday	Saturday
Backup_Mon_Thurs	Backup_Tues_Fri	Backup_Wed_Sat	Backup_Mon_Thurs	Backup_Tues_Fri	Backup_Wed_Sat

Note that this is just an example and the folders are set up as such because of the limited drive space I had.

Once you’ve determined the amount of scheduled backups you need, you can proceed with setting up x amount of scheduled tasks in the Task Scheduler with the action setup as such:

start backup -backuptarget:\\backupServer\f$\Backup_Mon_Thurs -include:e:,f: -vssFull -quiet

start backup -backuptarget:\\backupServer\f$\Backup_Tues_Fri -include:e:,f: -vssFull -quiet

start backup -backuptarget:\\backupServer\f$\Backup_Wed_Sat -include:e:,f: -vssFull –quiet

In my example, the e: and f: drives are my database and log drives and therefore was included into the parameters.  The -vssFull switch is important for Exchange 2010 backups so that the job can flush the logs so make sure you include it into the parameters.

Hope this helps anyone who needs a temporary backup solution for Exchange Server 2010 and/or in some cases, need to flush the logs for an Exchange 2010 server because the log drive has filled up and turning on circular logging is not an option.

Downgrading or upgrading Windows Server 2008 R2 Datacenter, Enterprise or Standard Edition on an Exchange 2010 server

I ran into an interesting problem last week when I noticed that the new Exchange 2010 servers’ operating systems were all the Datacenter edition when they should have been Standard and the reason why this happened was because the template that was provided for the virtual machines was built with the Datacenter version.  The challenge I faced was that the servers were in production and getting downtime for mail wasn’t going to be easy and when I do get a window, I need to be sure that I fixed the operating systems as fast as possible.

The first idea I had was to simply install a new mailbox, cas and hub transport server and have them operate in parallel with the existing servers but I realized that this wasn’t going to work because the client only had 1 set of license keys for Exchange and when the mailbox server is in evaluation mode, it would only map 5 stores and in this environment, there was 7.  In addition to the mailbox store issue, the thought of building and migrating the services to temporary servers and then swinging it back to rebuilt servers simply wasn’t efficient enough.

The next idea I had was to simply mimic an operating system corruption and use the Setup /m:RecoverServer switch to reinstall the Exchange 2010 roles back onto the server.  I went through the TechNet article here: http://technet.microsoft.com/en-us/library/dd876880.aspx and figured it was going to work but before proceeding to schedule the window, I went ahead and posted a message on our Microsoft Partner forum to verify that it was going to work but ended up being disappointed with their answers.  I was initially told that it was possible when I used the “corrupted C drive” as an example but then was told that it wasn’t possible when I told them I just wanted to downgrade from Datacenter to Standard version of Windows Server 2008 R2.  The response I got from the Microsoft staff was to carefully ready the following:

• The server on which recovery is being performed must be running the same operating system as the lost server. For example, you can't recover a server that was running Exchange 2010 and Windows Server 2008 on a server running Windows Server 2008 R2, or vice versa.

My interpretation of the above was that we cannot replace an R2 operating system with R1 and vice versa which made sense because they are quite different but I was not convinced that cross versions would be a problem so I gave up on our Partner support and proceeded with cloning the virtual machines in the production environment and trying it out.  So if you’re looking for the answer to whether you can downgrade from Windows Server 2008 R2 Datacenter to either Enterprise or Standard for an Exchange 2010 server using the recover server option, the answer is YES you can.

Updating Firmware for Cisco UCS C Series Servers with Cisco UCS Host Upgrade Utility

As some of you may have come across one of my previous posts earlier in the year:

Updating Cisco UCS C-Series server’s firmware with Cisco UCS Server Standalone Utilities
http://terenceluk.blogspot.com/2011/01/updating-cisco-ucs-c-series-servers.html

… since it’s been quite some time since I’ve written another post for upgrading Cisco’s C series servers’ firmware, I took the time to do a few screenshots while upgrading 4 C460s a month ago so this post serves to show what the process looks like.

Proceed with downloading the Host Upgrade Utility ISO from the Cisco site:

hostUpgrade_v1_3_2i.iso
Release Date: 21/JUN/2011
Cisco UCS Host Upgrade Utility
Size: 109364.00 KB (111988736 bytes)

You can either burn the ISO onto a CD to boot with it or use the CIMC to mount the ISO so that the server can boot into the Linux OS that handles the firmware upgrade:

Once you’ve accepted the EULA, you’ll be presented with the following prompt with the options to:

1. Update CIMC Firmware
2. Update BIOS
3. Update LOM Firmware
4. Update LSI Firmware
5. Update UCS P81E VIC
6. All the above

Unless you have a reason to update only certain components, I would select option #6 to update all of the components:

The firmware update process will begin once you’ve entered your choice into the prompt and the first component the Host Upgrade Utility updates is the CIMC:

Then the BIOS:

Then the NICs:

Then the RAID controller:

… and finally the Palo card if it is installed:

Once the process finishes, continue by rebooting the server with option #7:

… and we’re done.  There’s not a whole lot to upgrading the firmware but I hope this post gives an idea of what it looks like.