Configure LDAPs an Active Directory Domain Controller for LDAP over SSL Connections

I recently had to configure a Directory Sync feature between a cloud based SPAM filtering service and a client’s Active Directory and came across the option of either syncing via regular LDAP port 389 (unecrypted) or LDAPS over SSL port 636.  While the service account the cloud based SPAM filtering service only requires a regular user account with no administrative permissions, I didn’t feel too comfortable with having the service sync via regular LDAP 389 over the internet with the login credentials being sent in clear text.  Yes, I’ve been told that we can lock the traffic down by source IP but I’m sure we all know how that isn’t exactly bulletproof.  So from here, I began configuring the domain controllers with an internal Microsoft Certificate Authority issued certificate to encrypt the traffic.  As I browsed through some old notes I had as I haven’t configured this in a while, I realized that I haven’t written a blog post about it so thought it would be a good idea to do so now so I have something to reference to in the future.  Note that I am configuring all of this on a Windows Server 2012 domain controller and certificate authority.  Both roles are installed onto the same server but I do not recommend doing so as I never liked installing CA services on a DC but this environment I configured this on recently only had 2 servers.

Step #1 – Create a new certificate template for LDAPS

Begin by creating a new certificate template on your internal Microsoft Certificate Authority to issue the certificate that will be used for LDAPS.  Launch the Certificate Authority management console, right-click on the Certificate Templates node and client on Manage:

In the Certificate Templates window, locate the Kerberos Authentication template, right click on it and click on Duplicate Template:

 

Click on the General tab and change the following fields:

Template display name: <Enter a name for the certificate>

Validity period: <Enter the number of years you want an issued certificate to be valid for>

Publish certificate in Active Directory: <I usually check this for convenience purposes so the certificate is displayed when a domain joined member is requesting a certificate>

In the Cryptography tab, enter a value for the Minimum key size. I usually enter at least 2048 as that seems to be the minimum size for public CAs these days and is a minimum requirement for Lync 2010/2013 deployments.

Navigate to the Subject Name tab and configure the following:

Build from this Active Directory information: Check

Subject name format: None

DNS Name: Check

Service principal name (SPN): Check

Lastly, navigate to the Request Handling tab and check the Allow private key to be exported option.  While this is optional, I usually enable it in case you ever need to export and reimport the certificate:

Click OK to create the new template and ensure it is now listed in the Certificates Templates:

Step #2 – Issue the new Certificate Template

With the new template created, navigate back to the Certificate Authority management console, right click on Certificate Templates, select New and click on Certificate Template to Issue:

Select the new certificate that was created and click OK:

Ensure that the new certificate is now listed in the Certificate Templates:

Step #3 – Request certificate for LDAPS over SSL on a Domain Controller

With the certificate created and published, proceed by navigating to a domain controller, open MMC and add the Certificates snap-in under the Computer account context:

Navigate to Certificates (Local Computer) –> Personal –> Certificates then right click on Certificates –> All Tasks –> Request New Certificate…:

Follow through the wizard:

Select Active Directory Enrollment Policy:

Check the new certificate template that was created:

Clicking on the Details button would show the following:

Click Enroll to request and retrieve the certificate:

Note that a new certificate should now be displayed with the following Intended Purposes properties:

• KDC Authentication
• Smart Card Logon
• Server Authentication
• Client Authentication

 

--------------------------------------------------------------------------------------------------------------------------------------------------------------------

**Note I’ve had a few colleagues ask me why they can’t use the default domain controller policy as shown here:

… and my response is that I was never able to get it to work even though most articles appear to suggest the Server Authentication is required for the Enhanced Key Usage properties.

--------------------------------------------------------------------------------------------------------------------------------------------------------------------

With the new certificate on the domain controller, hop onto another member server, launch LDP and try connecting to the DC via port 636 with SSL checked:

Hitting the OK button should show that you are now able to connect:

Repeat Step #3 for other domain controllers as necessary.

Hope this helps anyone looking for instructions on how to set this up.

Automating the process of removing / deleting orphaned and/or stale VDI (virtual desktops) in the VMware View Connection server’s ADAM database (VMware View Manager pools)

To follow up on one of my previous posts I wrote earlier in the year in February:

Manually deleting orphaned and/or stale virtual desktops in VMware View Manager pools
http://terenceluk.blogspot.com/2013/02/manually-deleting-orphaned-andor-stale.html

… I finally got fed up when I had to clean up more than 40 orphaned VDI objects at a client’s environment and took the time to use the native LDIFDE tool bundled with Windows for Active Directory export to automate the deletion of orphaned VDI objects in the ADAM database stored on VMware View’s 4.6 database.

As many View administrators already know, VMware View connection servers actually use an ADAM database to manage part of the VDI objects that are presented to us in the administration console.  Further information is stored in the View Composer database that is hosted on a database server such as Microsoft SQL.  My previous post demonstrated the process of removing orphaned objects via manually using adsiedit to edit the ADAM database on the View Connection servers then with a SQL query that went into all of the tables to remove the entries in a SQL database.  The manual process of using ADSIedit isn’t difficult but rather an extremely inefficient method that contains a bit too many clicks and cross referencing unique identifiers to my liking.  This process is tolerable for maybe 10 to 20 objects but as the number of objects increase, the manual labour becomes way too tedious.  So to make a long story short, one of the methods we can use to automate the process and make it a bit less painful is to use LDIFDE and Excel to organize the names in a format that you can pipe in as a list of VDIs to remove.

For those who are not familiar with the LDFIDE command, this tool basically allows you to export active directory objects into a csv file (or an .ldf file which is pretty much just text), and also allows you to import a list of LDFIDE allows you import a list of objects to remove them from the ADAM/Active Directory database.  So to achieved what I wanted to do, I needed to figure out the following:

1. Create a list of VDIs with their names that I would like to remove from the ADAM database
2. Use the LDFIDE command to export the list in #1
3. Edit the exported list (csv file) to set the objects for removal
4. Use the LDFIDE command to reimport the list with the objects flagged for removal

**Note: I rarely do this but have been told numerous times that I should state that you should be using this at your own risk and I cannot be held responsible for any damages made to your environment.  Use this at your own risk and back up your ADAM database before you proceed.

Step #1 - Create a list of VDIs with their names that I would like to remove from the ADAM database

This first step to obtain a list of orphaned VDIs is quite easy as it can all be done in the GUI by navigating to the Desktop Status window and clicking on the number beside the status that represents the desktops you would like to know:

In the screenshot above, I would like to list all of the Agent unreachable (missing) desktops so I would click on 38.

From here, I would use the little floppy symbol at the top right hand corner to save the list as a csv:

The only field we really need is the Desktop or DNS Name so choose one of those columns and delete the rest:

The next step is to add the line:

(pae-DisplayName=

… in the row before the name and a:

)

… in the row after the name as such:

From here, copy the fields and paste it into Notepad as such:

Then copy the space between the first two columns:

Then use the search and replace to remove these spaces:

The final product should look as such:

Now finally remove the line breaks from each of the rows as such:

The final output should look something like this:

(pae-DisplayName=VM-VIEW4-054)(pae-DisplayName=VM-VIEW4-060)(pae-DisplayName=VM-VIEW4-091)(pae-DisplayName=VM-VIEW4-098)(pae-DisplayName=VM-VIEW4-121)(pae-DisplayName=VM-VIEW4-010)(pae-DisplayName=VM-VIEW4-019)(pae-DisplayName=VM-VIEW4-144)(pae-DisplayName=VM-VIEW4-108)(pae-DisplayName=VM-VIEW4-011)(pae-DisplayName=VM-VIEW4-006)(pae-DisplayName=VM-VIEW4-007)(pae-DisplayName=VM-VIEW4-018)(pae-DisplayName=VM-VIEW4-026)(pae-DisplayName=VM-VIEW4-016)(pae-DisplayName=VM-VIEW4-021)(pae-DisplayName=VM-VIEW4-027)(pae-DisplayName=VM-VIEW4-041)(pae-DisplayName=VM-VIEW4-037)(pae-DisplayName=VM-VIEW4-068)(pae-DisplayName=VM-VIEW4-056)(pae-DisplayName=VM-VIEW4-061)(pae-DisplayName=VM-VIEW4-069)(pae-DisplayName=VM-VIEW4-081)(pae-DisplayName=VM-VIEW4-089)(pae-DisplayName=VM-VIEW4-084)(pae-DisplayName=VM-VIEW4-099)(pae-DisplayName=VM-VIEW4-095)(pae-DisplayName=VM-VIEW4-103)(pae-DisplayName=VM-VIEW4-105)(pae-DisplayName=VM-VIEW4-104)(pae-DisplayName=VM-VIEW4-107)(pae-DisplayName=VM-VIEW4-127)(pae-DisplayName=VM-VIEW4-140)(pae-DisplayName=VM-VIEW4-143)(pae-DisplayName=VM-VIEW4-139)(pae-DisplayName=VM-VIEW4-149)(pae-DisplayName=VM-VIEW4-119)

Step #2 - Use the LDFIDE command to export the list in #1

Just so for documenting purposes, the first LDFIDE command I came up with during the test was the following:

ldifde -f orphanedVDIs.txt -s vmv-01 -d "OU=Servers,dc=vdi,dc=vmware,dc=int" -r "(objectClass=pae-Server)"

While the command above allowed me to export all of the VDIs, the problem with the data included in the export was that it included all of the VDIs and all of the attributes.  The text file has way too much information to be easily edited and it doesn’t include a filter to only include the objects we want to remove.  While we can manually delete the information is not needed, it’s not very enjoyable and I would rather manually use ADSIedit than to go through this list.

With that out of the way, the proper command to use is something similar to the following:

ldifde -f orphanedVDIs.txt -s vmv-01 -d "OU=Servers,dc=vdi,dc=vmware,dc=int" -r "(|(pae-DisplayName=view45-006)(pae-DisplayName=view45-010))" -l "DN"

The command above allows us to only export the DN attribute of the objects as well as only the objects that we’ve included after the -r switch.  The | symbol actually represents OR which means we can include as many bracketed pae-DisplayName afterwards to have only those objects exported.

What needs to be done now is to paste the following at the beginning of the text file we edited in step #1:

ldifde -f orphanedVDIs.txt -s bps-vmv-01 -d "OU=Servers,dc=vdi,dc=vmware,dc=int" -r "(|

… and the following at the end:

)" -l "DN"

The cmdlet you should have now is the following:

ldifde -f orphanedVDIs.txt -s vmv-01 -d "OU=Servers,dc=vdi,dc=vmware,dc=int" -r "(|(pae-DisplayName=VM-VIEW4-054)(pae-DisplayName=VM-VIEW4-060)(pae-DisplayName=VM-VIEW4-091)(pae-DisplayName=VM-VIEW4-098)(pae-DisplayName=VM-VIEW4-121)(pae-DisplayName=VM-VIEW4-010)(pae-DisplayName=VM-VIEW4-019)(pae-DisplayName=VM-VIEW4-144)(pae-DisplayName=VM-VIEW4-108)(pae-DisplayName=VM-VIEW4-011)(pae-DisplayName=VM-VIEW4-006)(pae-DisplayName=VM-VIEW4-007)(pae-DisplayName=VM-VIEW4-018)(pae-DisplayName=VM-VIEW4-026)(pae-DisplayName=VM-VIEW4-016)(pae-DisplayName=VM-VIEW4-021)(pae-DisplayName=VM-VIEW4-027)(pae-DisplayName=VM-VIEW4-041)(pae-DisplayName=VM-VIEW4-037)(pae-DisplayName=VM-VIEW4-068)(pae-DisplayName=VM-VIEW4-056)(pae-DisplayName=VM-VIEW4-061)(pae-DisplayName=VM-VIEW4-069)(pae-DisplayName=VM-VIEW4-081)(pae-DisplayName=VM-VIEW4-089)(pae-DisplayName=VM-VIEW4-084)(pae-DisplayName=VM-VIEW4-099)(pae-DisplayName=VM-VIEW4-095)(pae-DisplayName=VM-VIEW4-103)(pae-DisplayName=VM-VIEW4-105)(pae-DisplayName=VM-VIEW4-104)(pae-DisplayName=VM-VIEW4-107)(pae-DisplayName=VM-VIEW4-127)(pae-DisplayName=VM-VIEW4-140)(pae-DisplayName=VM-VIEW4-143)(pae-DisplayName=VM-VIEW4-139)(pae-DisplayName=VM-VIEW4-149)(pae-DisplayName=VM-VIEW4-119))" -l "DN"

You can now use this cmdlet to export the orphaned VDIs you would like to delete into a list as such:

 

Note that you should see something similar to the following when you open up this exported list:

Step #3 - Edit the exported list (csv file) to set the objects for removal

What we need to do now is modify this list and change the add word to delete marking the objects to be deleted:

Step #4 - Use the LDFIDE command to reimport the list with the objects flagged for removal

With our list of VDIs to be deleted prepared, the final step is to simply execute the following:

ldifde -i -f orphanedVDIs.txt -s vmv-01

Please note that there is no undelete and as soon as you execute this command and remove the VDI objects in the ADAM database, it will get replicated to your other nodes as well so please proceed with caution.

From here on, you can proceed with using the SQL script I included in my previous blog post to remove the objects from the SQL database.  I have yet to find the time to automate that process even more but will try to do so sometime in the future (maybe when I have to do 400 desktops).

VMware View 4.6 no longer provisions, refreshes or sends any commands to vCenter 4.1 server even though all status’ are reported as being up

I ran into an issue last night after receiving a call from the same client who had an issue with their View 4.6 environment a few days ago that desktops are no longer being refreshed and all of desktops who have had users logged off of were now stuck in maintenance mode.  The first thought I had was that vCenter probably went down and therefore the commands View was setting over to vCenter were no longer being executed.  Logging into the environment and opening the View Administrator Console webpage showed that there was indeed a red down arrow for vCenter.  Hopping over to vCenter showed that the vpxd service was down and a quick restart got it back up.  Further reviewing the logs on the SQL cluster and the vCenter server showed that the cluster apparently crashed at around 3:49p.m. but recovered by failing over to the other node but vCenter eventually stopped at around 4:30p.m.  No problem, the services are all back up now and all arrows are now pointing upwards and green so I expected to see View actions popping up in the recent tasks window in vCenter.  10 minutes have passed and nothing appears to be happening so I go ahead and force refresh, still nothing. I proceed an go create a test pool, View creates the folder but then stops.  Reviewing the event logs, View Composer logs, View Connection logs showed no errors.  It’s as if either the commands aren’t being sent or there is something wrong with the composer service.  After trying several items such as reinstalling composer, re-entering service account credentials without any luck, I opened a call to VMware.

Fast forward to an hour later when I finally got an engineer on the line because of how busy they were, the engineer spent about an hour clicking around and checking everything I’ve reviewed and said nothing looks wrong.  Seeing how we’re going no where, he reached out to his senior engineer then came back after 30 minutes and told me to shut both of the View connection servers down (the environment had 2).  He then proceeded to wait 5 minutes, power up the first server, wait another 5 minutes after it was started, then boot the second one up.  Within 10 minutes of both servers being up, the desktops started getting refreshed.  His explanation was that this appears to happen to View 4 and 5 environments with more than 1 connection server.  He’s not sure why but it may be the ADAM database needing time to start up, queue commands then send it off which was why we waited.

Not exactly what I expected but I’m glad this worked and I hope this post with no screenshots may be able to help anyone who finds themselves in the same situation.

Upgrading Exchange Server 2010 to Service Pack 3 fails with: “The following error was generated when "$error.Clear(); …”

Problem

You have installed 2 new Exchange Server 2010 with SP1 mailbox servers (future DAG) and 2 hub transport / CAS servers an existing Exchange 2007 organization and proceed to install SP3 onto the servers but noticed that you are able to install it onto the HT/CAS servers and only 1 of the 2 mailbox servers.  The first mailbox server installs without an issue but the second one fails at the Mailbox Role stage with the error:

Summary: 6 item(s). 3 succeeded, 1 failed.
Elapsed time: 00:08:01

Language Files
Completed
Elapsed Time: 00:02:55

Restoring services
Completed
Elapsed Time: 00:00:01

Languages
Completed
Elapsed Time: 00:01:17

Mailbox Role
Failed
Error:
The following error was generated when "$error.Clear();
          $name = [Microsoft.Exchange.Management.RecipientTasks.EnableMailbox]::DiscoveryMailboxUniqueName;
          $dispname = [Microsoft.Exchange.Management.RecipientTasks.EnableMailbox]::DiscoveryMailboxDisplayName;
          $dismbx = get-mailbox -Filter {name -eq $name} -IgnoreDefaultScope -resultSize 1;
          if( $dismbx -ne $null)
          {
            $srvname = $dismbx.ServerName;
            if( $dismbx.Database -ne $null -and $RoleFqdnOrName -like "$srvname.*" )
            {
              Write-ExchangeSetupLog -info "Setup DiscoverySearchMailbox Permission.";
              $mountedMdb = get-mailboxdatabase $dismbx.Database -status | where { $_.Mounted -eq $true };
              if( $mountedMdb -eq $null )
              {
                Write-ExchangeSetupLog -info "Mounting database before stamp DiscoverySearchMailbox Permission...";
                mount-database $dismbx.Database;
              }
              $mountedMdb = get-mailboxdatabase $dismbx.Database -status | where { $_.Mounted -eq $true };
              if( $mountedMdb -ne $null )
              {
                $dmRoleGroupGuid = [Microsoft.Exchange.Data.Directory.Management.RoleGroup]::DiscoveryManagementWkGuid;
                $dmRoleGroup = Get-RoleGroup -Identity $dmRoleGroupGuid -DomainController $RoleDomainController -ErrorAction:SilentlyContinue;
                if( $dmRoleGroup -ne $null )
                {
                  Add-MailboxPermission $dismbx -User $dmRoleGroup.Identity -AccessRights FullAccess -DomainController $RoleDomainController -WarningAction SilentlyContinue;
                }
              }
            }
          }
        " was run: "Couldn't resolve the user or group "contoso.com/Microsoft Exchange Security Groups/Discovery Management." If the user or group is a foreign forest principal, you must have either a two-way trust or an outgoing trust.".
Couldn't resolve the user or group "contoso.com/Microsoft Exchange Security Groups/Discovery Management." If the user or group is a foreign forest principal, you must have either a two-way trust or an outgoing trust.
The trust relationship between the primary domain and the trusted domain failed.
Click here for help... http://technet.microsoft.com/en-US/library/ms.exch.err.default(EXCHG.141).aspx?v=14.3.123.3&e=ms.exch.err.Ex88D115&l=0&cl=cp
Elapsed Time: 00:03:46

Management Tools
Cancelled

Finalizing Setup
Cancelled

The error message includes a link but clicking on it brings you to a page indicating there’s no article written for this error:

Searching on the internet returns a lot of posts that suggest either disabling the Discovery Mailbox in the Exchange Management Console deleting the account completely, install SP3 then recreate it.  The challenge I had was that these suggestions did not resolve the issue because it would fail at the same stage and it also appears that the Discovery Mailbox also gets recreated during the process.

Another solution I read off of a forum post was to check the permissions was to check the Full Access permissions of the Discovery Mailbox through the EMC:

… and while it was missing a lot of permissions because it only had NT AUTHORITY\SELF:

 

… while the list should look more like this:

• DOMAIN\Discovery Management
• DOMAIN\Exchange Domain Servers
• DOMAIN\Exchange Servers
• DOMAIN\Exchange Services
• DOMAIN\Exchange Trusted Subsystem

… adding the above permissions and running the SP3 install continued to fail.

Solution

Having exhausted all of the available resources I could find and knowing I’d probably have to figure this one out myself, I went ahead and reviewed the error message line by line again to see if anything pop out at me but nothing did.  What I ended up doing was run through the following list of what I knew:

1. The discovery mailbox actually gets recreated by this second mailbox server during the SP3 install
2. The discovery mailbox is not available when the server that’s being upgraded to SP3 is down

Though probably not accurate at all, the error message appears to suggest some sort of access issue related to this mailbox and if the server was being upgraded, wouldn’t the store be down at some point?  This gave me the idea that since I had 2 mailbox servers with 2 mailbox databases, why not move this discovery mailbox off of the SP3 upgrade failing mailbox server’s database and onto the one that already has SP3 successfully installed:

 

Executed a move request:

 

Notice the mailbox is being moved:

The move completed:

With the discovery mailbox in a different store, I ran the SP3 upgrade again and this time it completed successfully:

A bit odd but I hope this helps anyone who might come across this issue as I did.

VMware View 4.6 linked clone pool provisioning fails with multiple errors

I was called by a client today asking me to to have a look at their VMware View 4.6 for their VDI infrastructure that was reporting vCenter was down.  Fixing vCenter wasn’t too difficult as a few reboot and re-initialization of VMware Heartbeat got the services back up.  With all of the arrows in VMware View pointing upwards and green:

… I went ahead check check on the pools only to notice that all them had a red x beside them.  What I noticed was that as soon as I re-enabled by selecting the pool, clicking on Status, then Enable Provisioning, I would see that View sends the commands:

1. Clone the virtual machine (you can see that the source is the replica with the name: replica-1fab7069…)
2. Add tag
3. Reconfigure virtual machine
4. Delete the virtual machine

After a deleting the created virtual machines, the pool would error out and stop provisioning.  Navigating to the Status tab of the pool would show various messages on the 3 pools I had.  Sorry about the lack of screenshots as I was rushing to troubleshooting this so didn’t actually capture any of them but the message would show up in the highlighted area below:

The 2 pools that already existed would error out with the following messages (their respective KB links are included):

Deploying a linked desktop pool fails with the error: Selected parent VM is not accessible (1024566)
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1024566

View Manager Admin console displays the error: Error during provisioning: Unexpected VC fault from View Composer (Unknown) (2014321)
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2014321

The new pool I created with a completely new master image with a new snapshot would fail with the following error:

Error during provisioning: Unable to find folder (1038077)
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1038077

In addition to the errors above, I would also see the error: sdd was not found in the recent tasks on vCenter before the pool errors out.

After spending hours reviewing logs, rebooting servers without any luck, I finally came across this forum thread:

https://communities.vmware.com/thread/304131?start=0&tstart=0

… and went ahead to uninstall then reinstall the same version of the View Composer with the Create a new SSL Certificate option (the only other option is use the existing one).  I noticed that I would still get the error:

View Manager Admin console displays the error: Error during provisioning: Unexpected VC fault from View Composer

… once Composer was reinstalled, so I proceeded to restart vCenter which was also where the Composer was installed and once the services came back up all of the pools began to provision but the status of it would end up being either of the following:

1. agent unreachable (missing)
2. unknown (missing)
3. provisioned (missing)

… then the pool would throw the same error as before:

Error during provisioning: 10/6/13 11:20:26 PM ADT: Unexpected VC fault from View Composer (Unknown): see event log for full error

Further troubleshooting of making sure the DHCP scope isn’t out of address leases, AD accounts are cleaned out, etc didn’t lead me any closer to a resolution so as I ran out of ideas, I figure I’ll just change the service account for vCenter and View Composer to my own login which has full permissions to everything:

I’m not sure why but this corrected the problem.

I hope this helps anyone out there who might come across this issue as it took quite a bit of time for me to figure it out because other than the error messages I found in the Status tab of the pools, I couldn’t find anything in the View Connection server logs and the Windows event logs.