Pages

Thursday, May 31, 2018

Monitoring Citrix XenDesktop / XenApp 7.17 Delivery Controller services with PowerShell script and task scheduler

I recently wrote the following blog post:

Monitoring Microsoft Exchange Server 2010, 2013 and 2016 services with PowerShell script and task scheduler
http://terenceluk.blogspot.com/2018/03/monitoring-microsoft-exchange-server.html

… demonstrating how to use a script to check for hung or stopped services for Exchange and to restart and send an email notification as a reaction to these events.  A few people have reached out to me about using this for Citrix XenApp / XenDesktop environments and the short answer to the question is yes, you can indeed use this script because it universally monitors Windows services.

The following is the modified script that includes the Citrix XenDesktop / XenApp 7.17 Delivery Controller services:

#NAME: service_check.ps1 

#AUTHOR: Kevin Olson

#DATE: 4/29/2011

#Machine to be monitored

$Computer = "citrixServer"

#Create an array of all services running

$GetService = get-service -ComputerName $Computer

#Create a subset of the previous array for services you want to monitor

$ServiceArray = "CitrixADIdentityService","CitrixAnalytics","CitrixAppLibrary","CitrixBrokerService","CitrixConfigSyncService","CitrixConfigurationLogging","CitrixConfigurationService","CitrixConnector","CitrixDelegatedAdmin","CitrixEnvTest","CitrixHighAvailabilityService","CitrixHostService","CitrixMachineCreationService","CitrixMonitor","CitrixOrchestration","CitrixPrivilegedService","CitrixStorefront","CitrixTelemetryService","CitrixTrust","XaXdCloudProxy";

#Find any iWFM service that is stopped

foreach ($Service in $GetService)

{

     foreach ($srv in $ServiceArray)

     {

         if ($Service.name -eq $srv)

         {

             #check if a service is hung

             if ($Service.status -eq "StopPending")

             {

             $servicePID = (gwmi win32_Service | where { $_.Name -eq $srv}).ProcessID

             Stop-Process $ServicePID

             Start-Service -InputObject (get-Service -ComputerName $Computer -Name $srv)

             #email to notify if a service is down

             Send-Mailmessage -to administrator@someDomain.com -Subject "$srv is hung on $Computer" -from exchange@contoso.com -Body "The $srv service was found hung." -SmtpServer localhost

             }

             # check if a service is stopped

             elseif ($Service.status -eq "Stopped")

             {

             #automatically restart the service.

             Start-Service -InputObject (get-Service -ComputerName $Computer -Name $srv)

         #email to notify if a service is down

             Send-Mailmessage -to administrator@someDomain.com -Subject "$srv is stopped on $Computer" -from exchange@contoso.com -Body "The $srv service was found stopped." -SmtpServer localhost

             }

         }

     }

}

As the list of services change through the versions of XenDesktop / XenApp, I am including the following screenshot of the services for version 7.17:

clip_image002

Tuesday, May 29, 2018

Logging into SecurEnvoy 2fa enabled Exchange 2013/2016 OWA portal loops back to the login page

Problem

You’ve completed configuring your Exchange 2013/2016 OWA (Outlook Web App) for 2-factor authentication with SecurEnvoy and proceed to test the login portal:

image

You’re proceed to enter the 2nd factor passcode:

image

The authentication is successful with the following message displayed:

Authentication OK

Logging on to Exchange Please Wait …..

image

… but the screen refreshes and you are passed back to the login screen:

image

Solution

One of the reasons why this behavior would be exhibited is if you do not have a default domain configured and you’re logging in with only the user name and not domain\username. To test, try logging in with domain\username and if you are successful then you have the following 2 choices:

Option #1 – Configure a default domain within the SecurEnvoy configuration

Navigate to the Exchange server’s C:\windows directory and open the file seiis.ini:

image

Locate the DefaultDomain= line and add the domain name:

image

Option #2 – Configure a default domain within Exchange admin center

Navigate to servers > virtual directories, select the server hosting the OWA website, open the properties of the owa (Default Web Site), click on authentication, change the default Domain\user name option to User name only and select a default domain:

image

image

Alternatively, you could also use the following PowerShell cmdlets to review the OWA authentication configuration:

Get-OwaVirtualDirectory -Server tmrukexowa01 | FL LogonFormat,DefaultDomain

The following can configure the OWA authentication:

Set-OwaVirtualDirectory

The above should correct the issue and allow you to log into OWA successfully.

Thursday, May 24, 2018

Replacing ADFS Proxy Server's SSL certificate with the cmdlet Set-WebApplicationProxySslCertificate throws the error: "A positional parameter cannot be found that accepts argument..."

Problem

You need update the ADFS Proxy server’s certificate as per the following instructions:

Managing SSL Certificates in AD FS and WAP in Windows Server 2016
https://docs.microsoft.com/en-us/windows-server/identity/ad-fs/operations/manage-ssl-certificates-ad-fs-wap

image

… but executing the command throws the error:

PS C:\Users\Administrator> Set-WebApplicationProxySslCertificate "54d87e8e2acc62de2d8cff943f8f5ebdd1db330c"
Set-WebApplicationProxySslCertificate : A positional parameter cannot be found that accepts argument
'54d87e8e2acc62de2d8cff943f8f5ebdd1db330c'.
At line:1 char:1
+ Set-WebApplicationProxySslCertificate "54d87e8e2acc62de2d8cff943f8f5ebdd1db330c"
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidArgument: (:) [Set-WebApplicationProxySslCertificate], ParameterBindingException
    + FullyQualifiedErrorId : PositionalParameterNotFound,Microsoft.IdentityServer.Management.Proxy.Commands.SetSslCer
   tificateCommand

PS C:\Users\Administrator> Set-WebApplicationProxySslCertificate -thumbprint "54d87e8e2acc62de2d8cff943f8f5ebdd1db330c"

Message                                 Context                                                                  Status
-------                                 -------                                                                  ------
The configuration completed successf... DeploymentSucceeded                                                     Success

PS C:\Users\Administrator> Get-WebApplicationProxySslCertificate

HostName                           PortNumber  CertificateHash
--------                           ----------  ---------------
adfs.domain.com              443      54D87E8E2ACC62DE2D8CFF943F8F5EBDD1DB330C
adfs.domain.com             49443     54D87E8E2ACC62DE2D8CFF943F8F5EBDD1DB330C

PS C:\Users\Administrator>

image

Solution

The reason why this error is thrown is because the cmdlet shown on the website is missing the -Thumprint switch. To correct this, use the same syntax as the cmdlet for the internal ADFS server as shown below:

image

Wednesday, May 23, 2018

Running Test-ExchangeServerHealth.ps1 Exchange Server health check script returns services as being down

Problem

You’ve been using the Test-ExchangeServerHealth.ps1 Exchange Server health check script for sometime but noticed that it has began falsely reporting the Client Access Server Role, Hub Transport Server Role, Mailbox Server Role and Mailbox Server Role services as being in a Fail state:

image

Solution

One of the possible causes of the incorrect report of the status for the services is if WinRM is not started on the Exchange server.  A way of determining this is to execute the Test-ServiceHealth <serverName> then review the output:

image

Note how the last line for each service displays {ServicesNotRunning: WinRM}

WinRM will be reported as not running regardless of whether the Test-ServiceHealth cmdlet is executed remotely or locally on the server.

To correct the issue, log onto the affected Exchange server and ensure that the Windows Remote Management (WS-Management) service is started:

imageimage

Restarting the service and rerunning the health check PowerShell should now report the services as being up:

imageimage

Sunday, May 20, 2018

Configuring an on-premise Exchange 2016 OWA with SecurEnvoy for 2fa causes webpage to load with the error: "HTTP Error 403.18 - Forbidden"

Problem

You’ve downloaded the latest SecurEnvoy Version 9.1.501 package as of May 2018 from:

https://www.securenvoy.com/support/downloads.shtm

Then used the following guide to configure your on-premise Exchange 2016 OWA access for 2FA:

Microsoft Outlook Web Access 2013 - SecurEnvoy
https://www.securenvoy.com/IntegrationGuides/Microsoft/Outlook-Web-Access-2013.pdf

… but receive the following error when attempting to access the Outlook Web App page after enabling SecurEnvoy 2FA:

HTTP Error 403.18 - Forbidden
The specified request cannot be processed in the application pool that is configured for this resource on the Web server.
Most likely causes:

· An ISAPI filter or custom module changed the URL to run in a different application pool than the original URL.

· An ISAPI extension (or custom module) used ExecuteURL (or ExecuteRequest) to run in a different application pool than the original URL.

· You have a custom error page that is located in one application pool but is referenced by a Web site in another application pool. When the URL is processed, it is determined by IIS that that it should have been processed in the first application pool, not the other pool.

· The Web site has multiple applications configured. The application this request is configured to run in is set to run in an application pool that does not exist.

Things you can try:

· If you have an application that is trying to process a URL in another application pool (such as trying to process a custom error), ensure that they both run in the same application pool if appropriate.

· If you are trying to process a custom error URL that is located in another application pool, enable the custom errors Redirect feature.

· Verify that the application pool for the application exists.

· Create a tracing rule to track failed requests for this HTTP status code and see if ExecuteURL is being called. For more information about creating a tracing rule for failed requests, click here.

Detailed Error Information:

Module

   IIS Web Core

Notification

   BeginRequest

Handler

   SecurEnvoy MS Server Agent

Error Code

   0x00000000

Requested URL

   https://<webmailURL>:443/securenvoyauth/webauth.exe?action=auth&dir=WEBAUTHTEMPLATE&ip=7C91BFF7D8EBAB9B9879278A1F44F11D92&redirect=https://tmrbmexmb02/owa/

Physical Path

   C:\Program Files (x86)\SecurEnvoy\Microsoft Server Agent\WEB\webauth.exe

Logon Method

   Not yet determined

Logon User

   Not yet determined

More Information:

This error occurs if the application pool for the request does not exist, or if an ISAPI filter, ISAPI extension or HTTP module calls the ExecuteURL server support function (or ExecuteRequest) with a URL that is configured in a different application pool. Due to security reasons, a Web site in one application pool cannot make ExecuteURL requests against a URL in another application pool. If you have an application that is trying to process a URL in another application pool, ensure that they both run in the same application pool if appropriate.

View more information »

image

Server Error

403 - Forbidden: Access is denied.

You do not have permission to view this directory or page using the credentials that you supplied.

image

Solution

One of the possible causes of this error is if the MSExchangeOWAAppPool for the IIS server on the Exchange 2016 server is configured incorrectly. I’ve only configured SecurEnvoy 2FA with OWA 2016 once so I am unsure as to whether this is a common issue because the deployment guide (https://www.securenvoy.com/IntegrationGuides/Microsoft/Outlook-Web-Access-2013.pdf) does indicate this as a requirement but it is labeled as a note:

image

To verify that the parameter is configured correctly, launch the Internet Information (IIS) Manager on the Exchange server, navigate to the SecurEnvoyAuth virtual directory:

image

Right click on the SecurEnvoyAuth node, navigate to Manage Application and then select Advance Settings…:

image

If the Application Pool is configured as DefaultAppPool then change it to MSExchangeOWAAppPool:

image

imageimageimage

The page should now load with the SecurEnvoy customizations:

image

Note that the above screenshot shows that the images are missing, which is another issue I will blog about in another post.

Thursday, May 17, 2018

Attempting to remote desktop to Windows server fails with the error: "An authentication error has occurred. The function requested is not supported"

Problem

You attempt to use a Windows 10 workstation to remote desktop to a server but notice that the connection fails with the following error message:

An authentication error has occurred.

The function requested is not supported

Remote computer: <computerName>

This could be due to CredSSP encryption oracle remediation.

For more information, see https://go.microsoft.com/fwlink/?linkid=866660

image

Reviewing the System logs on the client will show that the following error is logged:

A CredSSP authentication to TERMSRV/server.fqdn.com failed to negotiate a common protocol version. The remote host offered version 3 which is not permitted by Encryption Oracle Remediation.

See https://go.microsoft.com/fwlink/?linkid=866660 for more information.

image

Using a Windows 7 workstation to perform the same operation would display the following error message:

An authentication error has occurred.

The function requested is not supported

Remote computer: <computerName>

image

Solution

The cause of this error is explained in detail in the following TechNet blog post: https://blogs.technet.microsoft.com/mckittrick/unable-to-rdp-to-virtual-machine-credssp-encryption-oracle-remediation/

The short answer is that a patch was released in May 2018 that addresses a vulnerability issue with the Credential Security Support Provider protocol (CredSSP) and if you have patched your workstation with this patch but have not done the same for the server then this error would be displayed. Note that patching the server but not patching the workstation would not cause this issue.

There are several ways to work around this and they are:

Workaround #1 – Disable NLA on Server

Disable Allow connections only from computers running Remote Desktop with Network Level Authentication (recommended) for the server:

image

Workaround #2 – Uninstall Patch

Another way is to uninstall the patch from the workstations. The patch to uninstall for Windows 10 is:

KB4103723

image

The patch to uninstall for Windows 7 is:

KB4103712

image

You can use the following cmdlets to search:

Get-HotFix | Where HotfixID -match "4103712"

… or with the following to uninstall the patch:

wusa.exe /uninstall /kb:<KB Number>

**Refer to this blog post for using PowerShell to search for installed hotfixes: http://terenceluk.blogspot.com/2014/10/handy-get-hotfix-windows-powershell.html

Workaround #3 – Adjust Encryption Oracle Remediation

The new configuration that causes this error can be located on the patched workstation’s Computer Configuration / Administrative Templates / System / Credentials Delegation:

image

You can temporarily disable this by changing Encryption Oracle Remediation policy to Enabled, and Protection Level to Vulnerable:

image

Recommended Solution

The three of these work arounds is *not* recommended as they are temporary measures and does not address the vulnerability. The recommended way of addressing this would be to install the corresponding update on the server:

KB4103725

imageimage

Wednesday, May 16, 2018

Skype for Business Peer-to-Peer Session Detail Report reports: "No media quality data is available." for the "Media Quality Report"

Problem

You attempt to retrieve information about a bad call reported by a user so you launch the Skype for Business Monitoring Reports, drill down to the Peer-to-Peer Session Detail Report reports, expand the Media Quality Report section but noticed that No media quality data is available. Is displayed and there is no data logged:

image

Reviewing the event logs on the Skype for Business Front-End server reveals that the following error is logged:

Log Name: Lync Server

Source: LS Data Collection

Event ID: 56407

Level: Error

Failed to execute a stored procedure on the back-end.

Component: QoE Adaptor

Stored Procedure: QoeInsertSessionReport2

Error: System.Data.SqlClient.SqlException (0x80131904): Trying to pass a table-valued parameter with 109 column(s) where the corresponding user-defined table type requires 101 column(s).

at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)

at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)

at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)

at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption)

at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)

at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)

at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)

at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()

at Microsoft.Rtc.Common.Data.DBCore.Execute(SprocContext sprocContext, SqlConnection sqlConnection, SqlTransaction sqlTransaction)

ClientConnectionId:26e8fcf1-c35d-4284-adf6-7bfa82b60d24

Error Number:500,State:1,Class:16

Cause: Configuration issues, an unreachable back-end or an unexpected condition has resulted in the error.

Resolution:

Verify the back-end is up and this Skype for Business Server has connectivity to it. If the problem persists, notify your organization's support team with the relevant details.

image

Solution

One of the possible causes to this would be if you have a version mismatch between the ExpectedVersion and InstalledVersion for the QoEMetrics database. To determine whether this is the cause, execute the following cmdlet:

Test-CsDatabase -ConfiguredDatabases -SqlServerFqdn <SQLserverHostingMonitoringDatabase>

Note the difference between the ExpectedVersion and InstalledVersion for the QoEMetrics database:

ExpectedVersion: 62.93.12

InstalledVersion: 62.93.8

image

In the event that the environment does have a mismatched version for SQL, execute the following cmdlet:

Install-CsDatabase -DatabaseType Monitoring -SqlServerFqdn <SQLserverHostingMonitoringDatabase> -DatabasePaths "Z:\Data\MonitoringStore\(default)\DbPath","Y:\Logs\MonitoringStore\(default)\LogPath"

**Replace the paths with the appropriate paths to the database and logs

A similar output will be displayed:

image

Execute the following cmdlet again to confirm that the database no longer has a mismatched version:

Test-CsDatabase -ConfiguredDatabases -SqlServerFqdn <SQLserverHostingMonitoringDatabase>

Note the matching versions between the ExpectedVersion and InstalledVersion for the QoEMetrics database:

ExpectedVersion: 62.93.12

InstalledVersion: 62.93.12

image

With the mismatched database version corrected, the Media Quality Report section will now have data recorded:

image

Monday, May 14, 2018

Dialing into Polycom hosted meeting with Skype for Business Server fails with: "Previous hop server component did not report diagnostic information";Domain-"

Problem

You’ve configured Skype for Business integration with Polycom RealPresence Collaboration Server 1800 to allow Skype for Business clients to join into scheduled Polycom meetings but notice that an attempt to dial into the meeting would ring but the Polycom does not answer. Performing a logging session reveals the following entries:

SIP/2.0 500 Server Internal Error

Previous hop server component did not report diagnostic information";Domain=”internalPolycomDomain.com”;PeerServer=”yourPolycomDMAserver.com”;source=”yourSfBFEserver.com”

image

image

Solution

I encountered this error after several components of the Polycom conferencing server were moved from one datacenter to another and the error messages provided by the trace did not help because it did not point me to the right direction. What did was logging into the RMX Manager and reviewing the Signaling Monitor status, which revealed the following:

Relay Server UDP Not Available

Relay Server TCP Not Available

image

What ended up causing this issue was that the relocated RMX server was no longer able to reach the Skype for Business Edge Server (not the front-end server) and reconfiguring the firewall to allow the required ports corrected the issue.

Friday, May 11, 2018

Attempting to move a mailbox from one mailbox database to another in Exchange 2016 stops with the StatusDetail: StalledDueToSource_MailboxCapacityExceeded

Problem

You’ve initiated a mailbox move request from one mailbox database to another (cross a WAN link in this example) but noticed that the migration stops / halts with the StatusDetail StalledDueToSource_MailboxCapacityExceeded:

image

image

Suspending and resuming the migration does not restart the process.

Solution

While there are various reasons why the mailbox migration would halt, one of the things to try to restart the process is to identify the source server that hosts the mailbox and restart it.  Once the source server has been restarted with the services back up for a while, review the status again and verify it is either displaying InitialSeeding or CopyingMessages:

image