Pages

Wednesday, December 29, 2021

Generating a network trace capture and analyzing with Microsoft Network Monitor

An ex-colleague recently reached out to me for assistance on how he could perform a network trace and analyze it for a particular Citrix Virtual Apps and Desktop environment and the most common tool I usually recommend is Wireshark. The challenge he had was that the Wireshark installation would error out during the NCAP install so attempting to use that tool was not a viable option.

My ex-colleague’s challenge lead me to remember another method I had used in the past (probably more than 5 years ago) where we could use the native netsh trace command to capture an ETL file without requiring any software installation and after successfully testing the process, I thought I’d write a blog post to demonstrate it.

Creating a network trace capture file on the virtual desktop

1. On the VDI, launch the command prompt in administrator mode and start a trace with the following command:

netsh trace start capture=yes tracefile=c:\net.etl persistent=yes maxsize=4096

image

2. Replicate issue, note the time stamp, and stop trace with the following command:

netsh trace stop

image

Analyzing the network trace

  1. Download and install Microsoft Network Monitor: https://www.microsoft.com/en-in/download/details.aspx?id=4865
  1. Launch Microsoft Network Monitor and open the ETL file:
  1. Click Tools > Options:

image

Navigate to Parser Profiles tab, right click on Windows and click Set as Active:

image

Drill down to the NDISPacCap node:

image

For the purpose of this demonstration, we’ll be searching for an SMB path that contains the string college.

Click on Load Filter > Standard Filters > SMB > SmbFileName:

image

Update the string to look up and click Apply:

image

Hope this helps anyone who may be looking for a alternative method for capturing network traffic and analyzing it in an environment that may not have Wireshark available.

Releasing NVIDIA RTX Virtual Workstation License(s) on the License Server Manager

One of the frequent questions I’ve been asked in the past for VDI deployments that are accelerated with NVIDIA GPU GRID cards is how we can release assigned licenses to VDIs that no longer exist. Scenarios that can cause this is if a set of virtual desktops were deployed but then had to get redeployed the same evening because of a required change in configuration (I had to do this once when I needed to change the GPU memory allocation for the desktops).

image

image

Failure to have sufficient licenses for the VDIs will display the following message upon logging into the virtual desktop:

Failed to acquire NVIDIA license.

Failed to acquire NVIDIA RTX Virtual Workstation license. Click here for more information.

image

The short answer is that there isn’t a way to do this via the on-premise license server, command line or the NVIDIA Application Hub, and the recommended method is to either reduce the lease time for the license or completely remove all the licenses allocated to the desktops.

The steps to modify the lease time are as follows:

  1. Log onto the VDI master image
  2. Open the registry editor and browse to HKEY_LOCAL_MACHINE\SOFTWARE\NVIDIA Corporation\Global\GridLicensing
  3. Edit the LicenseInterval DWord (REG_DWORD) and configure the interval time that represents how long the license lease is valid for

The integer configured should be within the range 10-10080 that specifies the period of time in minutes for which a license can be borrowed after it is checked out. After this period has elapsed, the client must obtain a new license from the server.
The default is 1440 minutes, which corresponds to a period of 1 day. The value can be reduce to an hour so a new license would be reissued.

The environment I was working with already had the lease time configured to be a day (default) and further reducing it was not ideal in case we ever had a license server failure so I opted to use the 2nd method, which was to completely remove all the licenses allocated to the desktops so new ones would be issued. It is worth noting that I was told by the NVIDIA support engineer that this does not adversely affect the VDIs currently in use so it can be performed during regular hours.

The following are the steps:

1. Begin by logging into the NVIDIA Application Hub via the URL: https://nvid.nvidia.com/dashboard/

image

2. Select NVIDIA LICENSING PORTAL:

image

3. Navigate to LICENSE SERVERS, expand the License Server node and click on Download:

image

4. The license file representing the licenses will be downloaded:

imageimage

5. On the on-premise NVIDIA licensing server, stop FlexNet License Server - nvidia service:

image

6. Navigate to the path: C:\Windows\ServiceProfiles\NetworkService\flexnetls\:

image

7. Rename the nvidia folder to nvidia-old:

image

8. Start FlexNet License Server - nvidia service:

image

9. Open the license portal on the on-premise NVIDIA license server, navigate to License Management, ensure that the server is up, and the following error message is NOT present:

Connection error: Please make sure the FNE server is up and running

image

10. Confirm that the nvidia folder previously renamed has been recreated:

image

11. Confirm that there are no licensed clients listed:

image

12. With the server services up, proceed to upload the previously downloaded license file (.bin):

image

13. Confirm that the message Successfully applied license file to the license server. is displayed:

image

14. Navigating back to the Licensed Clients window should initially show an empty list and then new clients being listed:

image

Enable Security Authentication for NVIDIA License Server Manager

Those who have worked with the NVIDIA License Server Manager that is deployed to provide virtual desktops with GPU licenses will quickly notice that the default install does not provide any security for the management console as navigating to the URL: http://localhost:8080/licserver/ will bring you straight into the console without authentication. Given that I’ve been asked many times in the past about securing this portal, this post serves to demonstrate the process.

Enabling the requirement for logging in as shown in the screenshot below cannot be done via the GUI:

image

To enable the authentication requirement, we’ll need to use the nvidialsadmin.bat via the command line. The nvidialsadmin.bat can be found in the directory C:\NVIDIA\LicenseServer\enterprise on the licensing server:

image

Enable Security for the NVIDIA License Server Manager

1. Begin by launching a command prompt as an administrator and navigating to the directory: C:\NVIDIA\LicenseServer\enterprise

2. Execute the following command to set the security flag as true:

nvidialsadmin.bat -server http://127.0.0.1:7070 -config -set security.enabled=true

image

3. Next, execute the following command with the default password for the admin account (Admin@123) and set the new password (Update the New-Password1 value to the password desired):

nvidialsadmin.bat -server http://127.0.0.1:7070 -authorize admin Admin@123 -users -edit admin New-Password1

image

4. Proceed to restart the Apache Tomcat 9.0 Tomcat9 service in the services console on the license server:

image

5. Wait for the license server to be fully started then try to navigate to the console at http://localhost:8080/licserver/ to verify that credentials are required:

image

Disable Security for the NVIDIA License Server Manager

1. To disable the security login requirement execute the following command with the configured password to authorize the session:

nvidialsadmin -server http://127.0.0.1:7070 -authorize admin New-Password1

image

2. Then set the security flag to false:

nvidialsadmin.bat -server http://127.0.0.1:7070 -authorize admin New-Password -config -set security.enabled=false

image

3. Proceed to restart the Apache Tomcat 9.0 Tomcat9 service in the services console on the license server:

image

4. Wait for the license server to be fully started then try to navigate to the console at http://localhost:8080/licserver/ to verify that credentials are no longer required.

Wednesday, December 1, 2021

Configuring Security Headers to secure Microsoft Active Directory Federation Services / AD FS for scoring an A on SecurityHeaders.com

I’ve recently been asked by a colleague who wanted to know how they can score an A+ on www.securityheaders.com with a Windows Server 2019 AD FS WAP server that is exposed to the internet. It has been a while since I’ve configured one so I had to dig up my old notes and thought it would be great to write this quick post with the headers that achieves an A score. The reason why an A+ is not possible because it would require the Content-Security-Policy header to exclude the values:

  1. 'unsafe-inline'
  2. 'unsafe-eval'

… and excluding these would throw the following error:

JavaScript required

JavaScript is required. This web browser does not support JavaScript or JavaScript in this web browser is not enabled.

To find out if your web browser supports JavaScript or to enable JavaScript, see web browser help.

image

The following are the configuration for headers that I’ve used in the past to score an A (these are executed on the internal AD FS server and not on the WAP):

Set-AdfsResponseHeaders -SetHeaderName "Content-Security-Policy" -SetHeaderValue "default-src 'self' 'unsafe-inline' 'unsafe-eval'; img-src 'self' "

Set-AdfsResponseHeaders -SetHeaderName "Strict-Transport-Security" -SetHeaderValue " max-age=157680000; includeSubDomains"

Set-AdfsResponseHeaders -SetHeaderName "X-XSS-Protection" -SetHeaderValue "1;mode=block"

Set-AdfsResponseHeaders -SetHeaderName "X-Content-Type-Options" -SetHeaderValue "nosniff"

Set-AdfsResponseHeaders -SetHeaderName "Referrer-Policy" -SetHeaderValue "no-referrer"

Set-AdfsResponseHeaders -SetHeaderName "Permissions-Policy" -SetHeaderValue "geolocation=(),microphone=(),fullscreen=(self), vibrate=(self)"

Note that X-Frame-Options is already set to DENY by the AD FS server so there is no need to configure it. Use the following cmdlet to review the settings:

Get-AdfsResponseHeaders | Select-Object -ExpandProperty ResponseHeaders

image

Hope this helps anyone who may be looking for these headers.

Thursday, September 30, 2021

Granting a CSP Foreign Principal the Reader or Owner role onto an Azure Subscription with PowerShell

Those who have worked at a Cloud Solution Provider (CSP) organization with access to the MSP Expert tool for migrating subscriptions from EA to CSP will know that one of the post migration steps is to grant the Foreign Principal representing the CSP as either an Owner or Reader on the migrated subscriptions. This step can be easily missed because you can see the subscriptions in the Partner portal with the billing details but will quickly notice that you cannot open tickets for the migrated subscriptions because they are not present when you’re logged into the client’s Azure portal as the CSP due to the lack of permissions.

Attempting to apply the permissions from within https://portal.azure.com will reveal that you cannot simply click into the IAM blade to assign the Owner or Reader role because the Foreign Principal will not be searchable. The only way to perform this operation is to use PowerShell and the present Microsoft documentation doesn’t appear to have instructions for this specific operation (this is the closest one I could find: https://docs.microsoft.com/en-us/partner-center/revoke-reinstate-csp?tabs=workspaces-view) so I would like to share the cmdlets and a script for it.

PowerShell Cmdlets

The following assumes that you have an existing subscription with the Foreign Principal assigned with a role to it as we’ll need to retrieve the ObjectID of the Foreign Principal from a subscription then use it to assign to another subscription. If there isn’t a subscription with the Foreign Principal already assigned with a role, you can temporarily provision a new subscription from the Partner Central portal in the Azure Plan.

1. Install and import the Az module if it is not present:

Install-Module -Name Az

Import-Module -Name Az

2. Connect to the Azure tenant:

Connect-AzAccount

3. Obtain the subscription ID of a subscription that already has the Foreign Principal ID assigned a role:

Get-AzSubscription

4. Create a variable to store the subscription ID that has the Foreign Principal ID assigned a role:

$subscriptionID = "/subscriptions/<replaceWithSubID>"

5. Retrieve the Foreign Principal object along with its attributes and values:

Get-AzRoleAssignment -Scope $subscriptionID | Where-Object {$_.DisplayName -match "Foreign Principal*"}

6. Create a variable to store the Foreign Principal’s ObjectID value:

$foreignPrincipalObjectID = Get-AzRoleAssignment -Scope $subscriptionID | Where-Object {$_.DisplayName -match "Foreign Principal*"} | Select -expand ObjectID

7. Retrieve the subscription ID of the subscription that you want to grant the Foreign Principal permissions:

Get-AzSubscription

8. Update the variable to store the subscription ID that you want to assign the Foreign Principal:

$subscriptionID = "/subscriptions/<replaceWithSubID>"

9. Grant the Foreign Principal either Reader or Owner permissions to the subscription:

New-AzRoleAssignment -ObjectId $foreignPrincipalObjectID -Scope $subscriptionID -RoleDefinitionName Reader -ObjectType "ForeignGroup"

**Note that I’ve had mixed results with the -ObjectType parameter. Some tenants appear to require it and some do not. If an error is thrown indicating the ObjectType is unknown, remove the -ObjectType "ForeignGroup"

**Note If you are granting Reader role to the Foreign Principal, you will also need to grant the "Support Request Contributor" role in order to open support tickets from the Azure portal.

New-AzRoleAssignment -ObjectId $foreignPrincipalObjectID -Scope $subscriptionID -RoleDefinitionName "Support Request Contributor" -ObjectType "ForeignGroup"

PowerShell Script

I’ve created a PowerShell script that will perform the following:

  1. Use the Connect-AzAccount to connect to Azure
  2. Retrieve the list of subscriptions and prompt the user to select one that already has the Foreign Principal assigned with a role
  3. Retrieve the list of Foreign Principals assigned to the subscription and prompt the user to select the one that will be used to assign a role for another subscription
  4. Retrieve the list of subscriptions again and prompt the user to select one that we are to assign the Foreign Principal assigned with a role
  5. Prompt the user to choose whether to assign the Foreign Principal the Owner or Reader role
  6. If Reader role is selected, prompt user if they want to also add the Support Request Contributor as well (tickets cannot be opened if the CSP Foreign Principal has Reader role but not Support Request Contributor role)
  7. Prompt the user to confirm the assignment with Y or N
  8. Output the result

This AssignSubscriptionRoleToForeignIdentity.ps1 script can be found on my GitHub account: https://github.com/terenceluk/Azure-CSP/tree/main/Role-Assigment

--------------------------------------------------------------------------------------------------------------------------

The following is a screenshot of a subscription that has the Tech Data Corporation Foreign Principal assigned the Owner role:

image

The following is a screenshot after using the script to assign the same Tech Data Corporation Foreign Principal the Reader role:

image

I hope this will help anyone who might be looking for a demonstration on how this.

Sunday, September 5, 2021

Started GitHub!

Many of my contacts have joked with me about how I need to modernize the way I provide scripts in my blog posts as I do not:

#1 - Use special code boxes with proper formatting and colour coding
#2 - Use GitHub to better manage the updates and enable collaboration

To everyone who has made the above comments to me: You are absolutely correct so I have created a GitHub account, uploaded my most recent Azure Site Recovery PowerShell Failover Scripts and will continue to add my old scripts to my account:

https://github.com/terenceluk

I want to thank everyone who has reached out to me over the years with suggestions or called out any mistakes I've made in my posts. Every message is appreciated and I hope to continue contributing to the community.

image

Performing the Initial Setup of the Citrix FAS Administration Console fails at Authorize this service with: "Failed to Issue certificate: CR_DISP_DENIED (code 2)"

Problem

You’re attempting to set up a Citrix Federated Authentication Service server to allow using Azure AD authentication with single sign-on but the configuration fails at the Authorize this service with the error:

The authorization request on <CertServerFQDN>\<CA Name> failed: Failed to Issue certificate: CR_DISP_DENIED (code 2).

image

Reviewing the Certification Authority management console’s Pending Requests does not show the expected pending request and reviewing the Failed Requests show the FAS server request being denied:

image

image

Request Status Code: The requested certificate template is not supported by this CA. 0x80094800 (-2146875392)

Request Disposition Message: Denied by Policy Module 0x80094800, The request was for a certificate template that is not supported by the Active Directory Certificate Services policy: Citrix_RegistratrionAuthority_ManualAuthorization.

Attempting to manually enroll from the certificates console for the certificate also fails:

image

Solution

One of the reasons why the authorization of the FAS server would fail is if the permissions for the Citrix_RegistrationAuthority_ManualAuthorization template is not configured properly. Begin by launching the Certificates Templates Console on the CA that the FAS server is attempting to be authorized and open the properties of the Citrix_RegistrationAuthority_ManualAuthorization template:

image

Navigate to the Security tab and verify that the Authenticated Users group has Read permissions:

image

Domain Computers has Read and Enroll:

image

With the required permissions in place, attempt to authorize the server again and the status should now display:

There is a pending authorization request on CertServerFQDN>\<CA Name>.

image

Navigate into the Certification Authority management console’s Pending Requests and you should now see the following pending request:

The operation completed successfully. 0x0 (WIN32:0)

image

Proceed to authorize the pending request and the Authorize this service step should now complete:

image

Friday, July 30, 2021

Automating Azure Site Recovery Recovery Plan Test Failover with PowerShell Script (on-premise VMs to Azure)

I’ve recently been asked by a colleague whether I had any PowerShell scripts that would automate the test failover and cleanup of Azure Site Recovery replicated VMs and my original thought was that there must be plenty of scripts available on the internet but quickly found that results from Google were either the official Microsoft documentation that goes through configuring ASR, replicate, and only failover over one VM (https://docs.microsoft.com/en-us/azure/site-recovery/azure-to-azure-powershell) or blog posts that provided bits and pieces of information and not a complete script.

Having been involved in Azure Site Recovery design, implementation and testing, I have created a PowerShell script to initiate the failover of a recovery plan and then perform the cleanup when the DR environment has been tested. This post serves to share the script that I use and I would encourage anyone who decides to use it to improve and customize the script as needed.

Environment

The environment this script will be used for will have the source as an on-premise and target in Azure’s East US region. The source environment are virtual machines hosted on VMware vSphere.

Requirements

  1. Account with appropriate permissions that will be used to connect to the tenant with the Connect-AzAccount PowerShell cmdlet
  2. Recovery Plan already configured (we’ll be initiating the Test failover on the Recovery Plan and not individual VMs).
  3. The Subscription ID containing the servers being repliated
  4. The name of the Recovery Services Vault containing the replicated VMs
  5. The Recovery Plan name that will be failed over
  6. The VNet name that will be used for the failover VMs

Script Process

  1. Connect to Azure with Connect-AzConnect
  2. Set the context to the subscription ID
  3. Initiates the Test Failover task for the recovery plan
  4. Wait until the Test Failover has completed
  5. Notify user that the Test Failover has completed
  6. Pause and prompt the user to cleanup the failover test VMs
  7. Proceed to clean up Test Failover
  8. End script

I have plans in the future to add additional improvements such as accepting a subscription ID upon execution, providing recovery plan selection for failover testing, or listing failed over VM details (I can’t seem to find a cmdlet that displays the list of VMs and its status in a specified Recovery Group).

Script Variables

$RSVaultName = <name of Recovery Services Group> - e.g. "rsv-us-eus-contoso-asr"

$ASRRecoveryPlanName = <name of Recovery Plan> - e.g. "Recover-Domain-Controllers"

$TestFailoverVNetName = <Name of VNet name in the failover site the VM is to be connected to> - e.g. "vnet-us-eus-dr"

The Script

The following is the script:

Connect-AzAccount

Set-AzContext -SubscriptionId "adae0952-xxxx-xxxx-xxxx-2b8ef42c9bbb"

$RSVaultName = "rsv-us-eus-contoso-asr"

$ASRRecoveryPlanName = "Recover-Domain-Controllers"

$TestFailoverVNetName = "vnet-us-eus-dr"

$vault = Get-AzRecoveryServicesVault -Name $RSVaultName

Set-AzRecoveryServicesAsrVaultContext -Vault $vault

$RecoveryPlan = Get-AzRecoveryServicesAsrRecoveryPlan -FriendlyName $ASRRecoveryPlanName

$TFOVnet = Get-AzVirtualNetwork -Name $TestFailoverVNetName

$TFONetwork= $TFOVnet.Id

#Start test failover of recovery plan

$Job_TFO = Start-AzRecoveryServicesAsrTestFailoverJob -RecoveryPlan $RecoveryPlan -Direction PrimaryToRecovery -AzureVMNetworkId $TFONetwork

do {

$Job_TFOState = Get-AzRecoveryServicesAsrJob -Job $Job_TFO | Select-Object State

Clear-Host

Write-Host "======== Monitoring Failover ========"

Write-Host "Status will refresh every 5 seconds."

try {

    }

catch {

Write-Host -ForegroundColor Red "ERROR - Unable to get status of Failover job"

Write-Host -ForegroundColor Red "ERROR - " + $_

        log "ERROR" "Unable to get status of Failover job"

        log "ERROR" $_

exit

    }

Write-Host "Failover status for $($Job_TFO.TargetObjectName) is $($Job_TFOState.state)"

Start-Sleep 5;

} while (($Job_TFOState.state -eq "InProgress") -or ($Job_TFOState.state -eq "NotStarted"))

if($Job_TFOState.state -eq "Failed"){

Write-host("The test failover job failed. Script terminating.")

Exit

}else {

Read-Host -Prompt "Test failover has completed. Please check ASR Portal, test VMs and press enter to perform cleanup..."

#Start test failover cleanup of recovery plan

$Job_TFOCleanup = Start-AzRecoveryServicesAsrTestFailoverCleanupJob -RecoveryPlan $RecoveryPlan -Comment "Testing Completed"

do {

$Job_TFOCleanupState = Get-AzRecoveryServicesAsrJob -Job $Job_TFOCleanup | Select-Object State

Clear-Host

Write-Host "======== Monitoring Cleanup ========"

Write-Host "Status will refresh every 5 seconds."

try {

    }

catch {

Write-Host -ForegroundColor Red "ERROR - Unable to get status of cleanup job"

Write-Host -ForegroundColor Red "ERROR - " + $_

        log "ERROR" "Unable to get status of cleanup job"

        log "ERROR" $_

exit

    }

Write-Host "Cleanup status for $($Job_TFO.TargetObjectName) is $($Job_TFOCleanupState.state)"

Start-Sleep 5;

} while (($Job_TFOCleanupState.state -eq "InProgress") -or ($Job_TFOCleanupState.state -eq "NotStarted"))

Write-Host "Test failover cleanup completed."

}

image

The following are screenshots of the PowerShell script output:

image

I hope this will help anyone out there who may be looking for a PowerShell script to automate ASR failover process.

One of the additions I wanted to add to this script was to list the Status VMs in the recovery group after the test failover has completed but I could not find a way to list the VMs that only belong to the recovery group. The cmdlets below lists all of the VMs that are protected but combing through the properties does not appear to contain any reference to what recovery plans they belong to. Please feel free to comment if you happen to know the solution.

$PrimaryFabric = Get-AzRecoveryServicesAsrFabric -FriendlyName svr-asr-01

#svr-asr-01 represents Configuration Servers

$PrimaryProtContainer = Get-AzRecoveryServicesAsrProtectionContainer -Fabric $PrimaryFabric

$ReplicationProtectedItem = Get-AzRecoveryServicesAsrReplicationProtectedItem -ProtectionContainer $PrimaryProtContainer

----------Update July 31, 2021---------

After reviewing some of my old notes, I managed to find another version of the PowerShell script that performed test failover for two plans and included steps to shutdown a VM, remove VNet peering between production and DR regions before the test failover, then recreate them afterwards. The following is a copy of the script:

Connect-AzAccount

Set-AzContext -SubscriptionId "53ea69af-xxx-xxxx-a020-xxxxea02f8b"

#Shutdown DC2

Write-Host "Shutting down DC2 VM in DR"

$DRDCName = "DC2"

$DRDCRG = "Canada-East-Prod"

Stop-AzVM -ResourceGroupName $DRDCRG -Name $DRDCName -force

#Declare variables for DR production VNet

$DRVNetName = "vnet-prod-canadaeast"

$DRVnetRG = "Canada-East-Prod"

$DRVNetPeerName = "DR-to-Prod"

$DRVNetObj = Get-AzVirtualNetwork -Name $DRVNetName

$DRVNetID = $DRVNetObj.ID

#Declare variables for Production VNet

$ProdVNetName = "Contoso-Prod-vnet"

$ProdVnetRG = "Contoso-Prod"

$ProdVNetPeerName = "Prod-to-DR"

$ProdVNetObj = Get-AzVirtualNetwork -Name $ProdVNetName

$ProdVNetID = $ProdVNetObj.ID

# Remove the DR VNet's peering to production

Write-Host "Removing VNet peering between Production and DR environment"

Remove-AzVirtualNetworkPeering -Name $DRVNetPeerName -VirtualNetworkName $DRVNetName -ResourceGroupName $DRVnetRG -force

Remove-AzVirtualNetworkPeering -Name $ProdVNetPeerName -VirtualNetworkName $ProdVNetName -ResourceGroupName $ProdVnetRG -force

#Failover Test for Domain Controller BREAZDC2

$RSVaultName = "rsv-asr-canada-east"

$ASRRecoveryPlanName = "Domain-Controller"

$TestFailoverVNetName = "vnet-prod-canadaeast"

$vault = Get-AzRecoveryServicesVault -Name $RSVaultName

Set-AzRecoveryServicesAsrVaultContext -Vault $vault

$RecoveryPlan = Get-AzRecoveryServicesAsrRecoveryPlan -FriendlyName $ASRRecoveryPlanName

$TFOVnet = Get-AzVirtualNetwork -Name $TestFailoverVNetName

$TFONetwork= $TFOVnet.Id

$Job_TFO = Start-AzRecoveryServicesAsrTestFailoverJob -RecoveryPlan $RecoveryPlan -Direction PrimaryToRecovery -AzureVMNetworkId $TFONetwork

do {

$Job_TFOState = Get-AzRecoveryServicesAsrJob -Job $Job_TFO | Select-Object State

Clear-Host

Write-Host "======== Monitoring Failover ========"

Write-Host "Status will refresh every 5 seconds."

try {

    }

catch {

Write-Host -ForegroundColor Red "ERROR - Unable to get status of Failover job"

Write-Host -ForegroundColor Red "ERROR - " + $_

        log "ERROR" "Unable to get status of Failover job"

        log "ERROR" $_

exit

    }

Write-Host "Failover status for $($Job_TFO.TargetObjectName) is $($Job_TFOState.state)"

Start-Sleep 5;

} while (($Job_TFOState.state -eq "InProgress") -or ($Job_TFOState.state -eq "NotStarted"))

if($Job_TFOState.state -eq "Failed"){

Write-host("The test failover job failed. Script terminating.")

Exit

}else {

#Failover Test for Remaining Servers

$ASRRecoveryPlanName = "DR-Servers"

$RecoveryPlan = Get-AzRecoveryServicesAsrRecoveryPlan -FriendlyName $ASRRecoveryPlanName

$Job_TFO = Start-AzRecoveryServicesAsrTestFailoverJob -RecoveryPlan $RecoveryPlan -Direction PrimaryToRecovery -AzureVMNetworkId $TFONetwork

do {

$Job_TFOState = Get-AzRecoveryServicesAsrJob -Job $Job_TFO | Select-Object State

Clear-Host

Write-Host "======== Monitoring Failover ========"

Write-Host "Status will refresh every 5 seconds."

try {

        }

catch {

Write-Host -ForegroundColor Red "ERROR - Unable to get status of Failover job"

Write-Host -ForegroundColor Red "ERROR - " + $_

            log "ERROR" "Unable to get status of Failover job"

            log "ERROR" $_

exit

        }

Write-Host "Failover status for $($Job_TFO.TargetObjectName) is $($Job_TFOState.state)"

Start-Sleep 5;

    } while (($Job_TFOState.state -eq "InProgress") -or ($Job_TFOState.state -eq "NotStarted"))

if($Job_TFOState.state -eq "Failed"){

Write-host("The test failover job failed. Script terminating.")

Exit

    }else {

Read-Host -Prompt "Test failover has completed. Please check ASR Portal, test VMs and press enter to perform cleanup..."

$Job_TFOCleanup = Start-AzRecoveryServicesAsrTestFailoverCleanupJob -RecoveryPlan $RecoveryPlan -Comment "Testing Completed"

do {

$Job_TFOCleanupState = Get-AzRecoveryServicesAsrJob -Job $Job_TFOCleanup | Select-Object State

Clear-Host

Write-Host "======== Monitoring Cleanup ========"

Write-Host "Status will refresh every 5 seconds."

try {

    }

catch {

Write-Host -ForegroundColor Red "ERROR - Unable to get status of cleanup job"

Write-Host -ForegroundColor Red "ERROR - " + $_

        log "ERROR" "Unable to get status of cleanup job"

        log "ERROR" $_

exit

    }

Write-Host "Cleanup status for $($Job_TFO.TargetObjectName) is $($Job_TFOCleanupState.state)"

Start-Sleep 5;

} while (($Job_TFOCleanupState.state -eq "InProgress") -or ($Job_TFOCleanupState.state -eq "NotStarted"))

$ASRRecoveryPlanName = "Domain-Controller"

$RecoveryPlan = Get-AzRecoveryServicesAsrRecoveryPlan -FriendlyName $ASRRecoveryPlanName

$Job_TFOCleanup = Start-AzRecoveryServicesAsrTestFailoverCleanupJob -RecoveryPlan $RecoveryPlan -Comment "Testing Completed"

do {

$Job_TFOCleanupState = Get-AzRecoveryServicesAsrJob -Job $Job_TFOCleanup | Select-Object State

Clear-Host

Write-Host "======== Monitoring Cleanup ========"

Write-Host "Status will refresh every 5 seconds."

try {

    }

catch {

Write-Host -ForegroundColor Red "ERROR - Unable to get status of cleanup job"

Write-Host -ForegroundColor Red "ERROR - " + $_

        log "ERROR" "Unable to get status of cleanup job"

        log "ERROR" $_

exit

    }

Write-Host "Cleanup status for $($ASRRecoveryPlanName) is $($Job_TFOCleanupState.state)"

Start-Sleep 5;

} while (($Job_TFOCleanupState.state -eq "InProgress") -or ($Job_TFOCleanupState.state -eq "NotStarted"))

Write-Host "Test failover cleanup completed."

}

}

#Create the DR VNet's peering to production

Write-Host "Recreating VNet peering between Production and DR environment after failover testing"

Add-AzVirtualNetworkPeering -Name $DRVNetPeerName -VirtualNetwork $DRVNetObj -RemoteVirtualNetworkId $ProdVNetID -AllowForwardedTraffic

Add-AzVirtualNetworkPeering -Name $ProdVNetPeerName -VirtualNetwork $ProdVNetObj -RemoteVirtualNetworkId $DRVNetID -AllowForwardedTraffic

#Power On DC2

Write-Host "Powering on DC2 VM in DR after testing"

Start-AzVM -ResourceGroupName $DRDCRG -Name $DRDCName