Pages

Monday, March 21, 2022

Attempting to log onto a domain controller via RDP fails with the message: "An authentication error has occurred. The specified network password is not correct."

Problem

You attempt to use Remote Desktop to log into a domain controller but the attempt fails with the following message:

An authentication error has occurred.
The specified network password is not correct.

image

Attempting to log in via the console is not an option.

Solution

The workaround to successfully RDP to this domain controller is to disable the Allow connections only form computers running Remote Desktop with Network Level Authentication (recommended) in the RDP settings:

imageimage

If console access is available, try to log in via direct console access and disable the configuration. For situations where the domain controller is hosted in cloud providers such as Azure, console access will not an option. In scenarios where console access is not available, one of the potential workarounds is to use the Registry Editor to remotely connect to the domain controller and disable this setting via the registry.

Launch the Registry Editor, select the File tab and choose Connect Network Registry:

image

Enter the remote domain controller’s name:

image

The connection should succeed:

image

Navigate to the following registry path:

dc2\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp

Locate the REG_DWORD named UserAuthentication and change the value from 1 (enabled) to 0 (disabled):

imageimageimage

Changing this registry key should now have the NLA configuration disabled as such:

image

You should now be able to log into the domain controller:

image

Proceed to review the event logs of the domain controller and correct any health issues.

Wednesday, March 16, 2022

Configuring Azure Sentinel to capture Active Directory Domain Controller Security Events

I’ve recently been asked about why Log Analytics was not able to capture Security logs from a Windows server as it is not an option under the Log name list:

image

The short answer is because this is not a feature included natively within a Log Analytics Workspace as described in the following Microsoft documentation:

Configuring Windows Event logs
https://docs.microsoft.com/en-us/azure/azure-monitor/agents/data-sources-windows-events#configuring-windows-event-logs

You can't configure collection of security events from the workspace. You must use Microsoft Defender for Cloud or Microsoft Sentinel to collect security events.

image

In this post, I will demonstrate how to set up Azure Sentinel to capture a Windows Active Directory Domain Controller event logs and query them.

I will also write a separate post to demonstrate how to use Microsoft Defender for Cloud in the future.

Create Log Analytics Workspace

Begin by creating a Log Analytics Workspace that will store the Azure Sentinel streamed Active Directory Domain Controller logs:

image

Enable monitoring for Active Directory Domain Controller and attached it to the Log Analytics Workspace

Navigate to Azure Monitor > Virtual Machines > Not monitored and enable the domain controllers:

image

Enable Insights and select the Log Analytics Workspace that was created earlier to capture the logs when the following window is displayed:

image

The following agents will be installed:

image

Repeat the process for any additional domain controllers.

Add Microsoft Sentinel to the new Log Analytics Workspace

With the Domain Controllers added to the Log Analytics Workspace, proceed to create and add a new Microsoft Sentinel with the Log Analytics Workspace:

image

image

Configure Azure Sentinel Data Connector to collect Windows Security events

Navigate to Data Connectors, type in Security Events in the filter text field, select Windows Security Events via AMA then click on Open connector page:

image

In the opened connector page, navigate to the Configuration section and click on the button Create data collection rule button:

image

Fill in the field for the rule name, subscription and the resource group for the data collection rule:

image

Click on the +Add resource(s) button and note that This will also enable System Assigned Managed Identity on these machines, in addition to existing User Assigned Identities (if any).

image

Select the domain controller(s) to that this data collection rule will apply to then click Apply button:

image

The selected list of VMs will be listed as resources. Proceed to click on the Collect tab:

image

The following options are presented:

  • All Security Events
  • Common
  • Minimum
  • Custom
image

Hovering over the information button displays the following:

  • All events - All Windows Security and App Locker events.
  • Common - A standard set of events for auditing purposes.
  • Minimal - A small set of events that might indicate potential threats. By enabling this option, you won't be able to have a full audit trail.
  • Custom - Allows you to filter and select the security events to stream by using Xpath queries.

A more detailed breakdown of what is included in Common and Minimal can be found here:

https://docs.microsoft.com/en-us/azure/sentinel/windows-security-event-id-reference?WT.mc_id=AZ-MVP-5003408

  • All events - All Windows security and AppLocker events.
  • Common - A standard set of events for auditing purposes. A full user audit trail is included in this set. For example, it contains both user sign-in and user sign-out events (event IDs 4624, 4634). There are also auditing actions such as security group changes, key domain controller Kerberos operations, and other types of events in line with accepted best practices.

The Common event set may contain some types of events that aren't so common. This is because the main point of the Common set is to reduce the volume of events to a more manageable level, while still maintaining full audit trail capability.

  • Minimal - A small set of events that might indicate potential threats. This set does not contain a full audit trail. It covers only events that might indicate a successful breach, and other important events that have very low rates of occurrence. For example, it contains successful and failed user logons (event IDs 4624, 4625), but it doesn't contain sign-out information (4634) which, while important for auditing, is not meaningful for breach detection and has relatively high volume. Most of the data volume of this set is comprised of sign-in events and process creation events (event ID 4688).
  • Custom - A set of events determined by you, the user, and defined in a data collection rule using XPath queries. Learn more about data collection rules.

Minimal

1102, 4624, 4625, 4657, 4663, 4688, 4700, 4702, 4719, 4720, 4722, 4723, 4724, 4727, 4728, 4732, 4735, 4737, 4739, 4740, 4754, 4755, 4756, 4767, 4799, 4825, 4946, 4948, 4956, 5024, 5033, 8001, 8002, 8003, 8004, 8005, 8006, 8007, 8222

Common

1, 299, 300, 324, 340, 403, 404, 410, 411, 412, 413, 431, 500, 501, 1100, 1102, 1107, 1108, 4608, 4610, 4611, 4614, 4622, 4624, 4625, 4634, 4647, 4648, 4649, 4657, 4661, 4662, 4663, 4665, 4666, 4667, 4688, 4670, 4672, 4673, 4674, 4675, 4689, 4697, 4700, 4702, 4704, 4705, 4716, 4717, 4718, 4719, 4720, 4722, 4723, 4724, 4725, 4726, 4727, 4728, 4729, 4733, 4732, 4735, 4737, 4738, 4739, 4740, 4742, 4744, 4745, 4746, 4750, 4751, 4752, 4754, 4755, 4756, 4757, 4760, 4761, 4762, 4764, 4767, 4768, 4771, 4774, 4778, 4779, 4781, 4793, 4797, 4798, 4799, 4800, 4801, 4802, 4803, 4825, 4826, 4870, 4886, 4887, 4888, 4893, 4898, 4902, 4904, 4905, 4907, 4931, 4932, 4933, 4946, 4948, 4956, 4985, 5024, 5033, 5059, 5136, 5137, 5140, 5145, 5632, 6144, 6145, 6272, 6273, 6278, 6416, 6423, 6424, 8001, 8002, 8003, 8004, 8005, 8006, 8007, 8222, 26401, 30004

image

If the Custom option is desired, use the following documentation to fine tune the events collected:

Limit data collection with custom XPath queries
https://docs.microsoft.com/en-us/azure/azure-monitor/agents/data-collection-rule-azure-monitor-agent#limit-data-collection-with-custom-xpath-queries

An example of only collecting event ID 4625, which is an event logged for failed attempts is to use:

Security!*[System[(EventID=4625)]]

image

You can also configure to only capture successful login Event ID

Security!*[System[(EventID=4624)]]

Or perhaps both:

Security!*[System[(EventID=4624 or EventID=4625)]]

For the purpose of this demonstration, we’ll select Common and then create the rule:

image

image

The ingestion and availability of the data should be present within minutes.

To query for security events, navigate to the Logs blade:

image

We can start with a query as simple as the following, which would retrieve all events with the user name tluk:

SecurityEvent

| where TargetUserName == 'tluk'

image

Another more in depth query can be to use Event ID 4625, which is an event that is logged for any logon failure. With this in mind, we can reference the following Microsoft documentation:

https://docs.microsoft.com/en-us/windows/security/threat-protection/auditing/event-4625

… to create the following query which will query for event ID 4625, then map the SubStatus code with the reason so it is displayed in the results. Note that kusto queries are case sensitive and the hexadecimal code is in all lower case:

SecurityEvent

| where EventID == 4625

| extend Reason = case(

SubStatus == '0xc000005e', 'There are currently no logon servers available to service the logon request.',

SubStatus == '0xc0000064', 'User logon with misspelled or bad user account',

SubStatus == '0xc000006a', 'User logon with misspelled or bad password',

SubStatus == '0xc000006d', 'The cause is either a bad username or authentication information',

SubStatus == '0xc000006e', 'Indicates a referenced user name and authentication information are valid, but some user account restriction has prevented successful authentication (such as time-of-day restrictions).',

SubStatus == '0xc000006f', 'User logon outside authorized hours',

SubStatus == '0xc0000070', 'User logon from unauthorized workstation',

SubStatus == '0xc0000071', 'User logon with expired password',

SubStatus == '0xc0000072', 'User logon to account disabled by administrator',

SubStatus == '0xc00000dc', 'Indicates the Sam Server was in the wrong state to perform the desired operation.',

SubStatus == '0xc0000133', 'Clocks between DC and other computer too far out of sync',

SubStatus == '0xc000015b', 'The user has not been granted the requested logon type (also called the logon right) at this machine',

SubStatus == '0xc000018c', 'The logon request failed because the trust relationship between the primary domain and the trusted domain failed.',

SubStatus == '0xc0000192', 'An attempt was made to logon, but the Netlogon service was not started.',

SubStatus == '0xc0000193', 'User logon with expired account',

SubStatus == '0xc0000224', 'User is required to change password at next logon',

SubStatus == '0xc0000225', 'Evidently a bug in Windows and not a risk',

SubStatus == '0xc0000234', 'User logon with account locked',

SubStatus == '0xc00002ee', 'Failure Reason: An Error occurred during Logon',

SubStatus == '0xc0000413', 'Logon Failure: The machine you are logging on to is protected by an authentication firewall. The specified account is not allowed to authenticate to the machine.',

SubStatus == '0x0', 'Status OK',

strcat('Unknown reason substatus: ', SubStatus))

The following is an event I intentionally generated by typing in an incorrect password:

image

It is important to note that the amount of data collected can vary depending on the size of the environment so to avoid unexpected charges due to the amount of data ingested by Sentinel and into Log Analytics, start with minimum to get an idea of the amount of data per day and fine tune the configuration. Hope this helps anyone looking for a demonstration on how to set up security events collection in Azure. Kusto query is extremely powerful and will be able to provide all sorts of useful data for troubleshooting, auditing, and reporting.

Unable to remove Azure Virtual Machine from deleted Log Analytics Workspace

Problem

You’ve recently cleaned up Log Analytics Workspaces that had Virtual Machines connected to it and wanted to re-add those VMs to new workspaces but notice that you see the message:

Cannot enable – Workspace not found (Why?)

Clicking on the link provides the following explanation:

The workspace (e5a604e7-8908-4a30-9d15-5bc4aa81ca48) associated with this virtual machine could not be found. This usually happens if you do not have access to this workspace or the workspace is deleted. Please verify the existence of the workspace with your administrator. If you need to connect to a different workspace, please remove the monitoring agent extension from this virtual machine and try again.

image

You proceed to remove the monitoring agents from the Windows VM:

image

Then restart the OS but the previous message still persists.

Attempt to enabling Insights on the VM also fails with the following message:

The workspace (e5a604e7-8908-4a30-9d15-5bc4aa81ca48) associated with this virtual machine could not be found. This usually happens if you do not have access to this workspace or the workspace is deleted. Please verify the existence of the workspace with your administrator. If you need to connect to a different workspace, please remove the monitoring agent extension from this virtual machine and try again.

image

Solution

I’m not sure if the Azure portal requires more time to refresh but the error message was still present after 2 hours of uninstalling and restarting the VMs I worked on and what I found was to navigate into any Log Analytics Workspace, navigate to Virtual Machines and click into the VM in question:

image

Then click on the disconnect button:

imageimage

The process took about a minute and eventually displayed the VM from being Disconnecting to Not Connected:

image

Navigating back into Azure Monitor > Virtual Machines > Not monitored will now allow you to enable the VM:

image

Attempting to log into a Citrix ADC / NetScaler Gateway portal with Duo configured as MFA fails with: "Additional Information Required Please type your response below"

I recently had to perform routine patching of various Citrix ADC / NetScalers to ensure that they are up-to-date with the latest NS13.1 17.42.nc build (released Feb 23, 2022) and one HA pair in particular displayed the following message when the expected Duo MFA authentication prompt was supposed to be displayed:

Additional Information Required

Please type your response below

Attempting to enter your passcode or other information into the prompt leads to an authentication failure.

image

Searching on the internet displays the following Duo KB:

Why do I see the Citrix Gateway or Netscaler error "Additional Information Required" during Duo authentication?
https://help.duo.com/s/article/3506?language=en_US

… which did not apply for my situation and other forum posts suggest that the theme configured for the gateway was the culprit and that did not apply for this appliance either. What I ended up having to do was downgrade from the updated firmware:

NS13.1 17.42.nc

image

… back down to:

NS13.1 12.50.nc

image

Then I opened up a ticket with Duo and the response I received was the following:

Hi Terence,

Thanks for getting back to me.
As per the docs here:
https://duo.com/docs/citrix-netscaler
If you have Citrix Gateway build 12.1-51.16 or later with an "Advanced" or "Premium" edition license, or Citrix Gateway build 13.0-67.x or later with "Standard", "Advanced", or "Premium" edition license, try adding Duo 2FA with nFactor.
After updating past 13, you must use the nfactor version of the integration.
This involves one single change in the proxy.
the line that is:
[radius_server_iframe]
type=citrix_netscaler
will become:
[radius_server_iframe]
type=citrix_netscaler_rfwebui

The way the Citrix ADC / NetScaler is configured to integrate with Duo is as shown in the following Duo document:

Duo for Citrix Gateway Basic Primary Authentication
https://duo.com/docs/citrix-netscaler#:~:text=Log%20in%20to%20the%20Duo,information%20to%20complete%20your%20setup.

I’ve seen other methods such as configuring Duo as:

Duo for Citrix Gateway - Basic Secondary Authentication Instructions
https://duo.com/docs/citrix-netscaler-alt

… and the nFactor method as the support engineer provided but noticed that the nFactor article indicates that it suggests to “try” so I did not realize it was mandatory. I asked the support engineer to clarify this and the response I receive was:

Hi Terence,

Thanks for getting back to me.
You are not wrong, the wording is definitely ambiguous.
However in all the previous cases with this exact issue I have seen, I’m confident in the solution, given I know it works.
I suspect it was worded this way as there is some overlap where both will actually work for various versions/license editions, however given your experience you do not seem to be in the overlap given it is not working.
Please follow through and make the change and I expect the issue will be resolved.
Please let me know if this helps.
Have a great day!
Kind regards,

Raphael

I haven’t actually tried switching over the nFactor yet but will do so and update this post in the future. Hope this helps anyone who may encounter the same issue as I have.

Monday, February 28, 2022

Using an Automation Account to monitor a VM's Windows service and a Runbook to use Invoke-AzVMRunCommand to restart a stopped service

In my previous post:

Using Azure Change Tracking and Inventory to monitor Windows Services
http://terenceluk.blogspot.com/2022/02/using-azure-change-tracking-and.html

I demonstrated how to set up Change Tracking and Inventory in Azure Automation to monitor Windows services in a virtual machine and alert when the service was no longer in a running state. With monitoring and alerting in place, the next step is to incorporate automation so that issues can be immediately remediated without requiring manual intervention. Azure provides multiple methods for automation and for this post, I will demonstrate how to achieve this with the following:

To avoid recreating the same content that demonstrate how to set up monitoring for a virtual machine’s Windows service, let’s assume that we’ve gone through the same steps as we did for my previous post:

Using Azure Change Tracking and Inventory to monitor Windows Services
http://terenceluk.blogspot.com/2022/02/using-azure-change-tracking-and.html

Monitoring and alerting has already been setup and what needs to be done is to incorporate automation as shown in the following.

PowerShell cmdlet Invoke-AzVMRunCommand

The PowerShell cmdlet Invoke-AzVMRunCommand is a cmdlet that allows running PowerShell scripts or commands remotely on an Azure Virtual Machine and this will be the method we’ll be using to remotely restart a stopped Windows service in a automation runbook. For more information about this cmdlet, see the following documentation:

Invoke-AzVMRunCommand
https://docs.microsoft.com/en-us/powershell/module/az.compute/invoke-azvmruncommand?view=azps-7.2.0

Run scripts in your Windows VM by using action Run Commands
https://docs.microsoft.com/en-us/azure/virtual-machines/windows/run-command

PowerShell script that uses the Invoke-AzVMRunCommand cmdlet to restart a virtual machine’s Windows service

Next, we’ll incorporate the cmdlet that allows us to restart a virtual machine’s Windows service into the following script:

# This PowerShell using Invoke-AzVMRunCommand runs the PS cmdlet with a parameter directly that specifies the Windows Service name on the target VM

Connect-AzAccount

$resourceGroupName = "yourResourceGroupName"

$vmName ="ServerName"

$scriptToRun = "Start-Service -DisplayName 'Remote Registry'"

Out-File -InputObject $scriptToRun -FilePath ScriptToRun.ps1

Invoke-AzVMRunCommand -ResourceGroupName $resourceGroupName -Name $vmName -CommandId 'RunPowerShellScript' -ScriptPath ScriptToRun.ps1

Remove-Item -Path ScriptToRun.ps1

This script will need the Virtual Machine Contributor role to execute the Invoke-AzVMRunCommand cmdlet and it will be provided with the Managed Identity for the Automation Account (configured a bit later) so the final script will have the following lines inserted to execute Connect-AzAccount with the Managed Identity:

# Ensures you do not inherit an AzContext in your runbook

Disable-AzContextAutosave -Scope Process

# Connect to Azure with system-assigned managed identity

$AzureContext = (Connect-AzAccount -Identity).context

# set and store context

$AzureContext = Set-AzContext -SubscriptionName $AzureContext.Subscription -DefaultProfile $AzureContext

# This PowerShell using Invoke-AzVMRunCommand runs the PS cmdlet with a parameter directly that specifies the Windows Service name on the target VM

$resourceGroupName = "yourResourceGroupName"

$vmName ="ServerName"

$scriptToRun = "Start-Service -DisplayName 'Remote Registry'"

Out-File -InputObject $scriptToRun -FilePath ScriptToRun.ps1

Invoke-AzVMRunCommand -ResourceGroupName $resourceGroupName -Name $vmName -CommandId 'RunPowerShellScript' -ScriptPath ScriptToRun.ps1

Remove-Item -Path ScriptToRun.ps1

Automation Account Runbook

With the PowerShell script for restarting a Windows service prepared, proceed to create a runbook that will execute the script when an alert. Navigate to the Automation Account > Runbooks and Create a runbook:

image

The options available for the Runbook type are as follows:

  1. PowerShell
  2. Python
  3. PowerShell Workflow
  4. Graphical PowerShell
  5. Graphical PowerShell Workflow
image

This example will use a PowerShell script so we’ll select PowerShell with the runtime version as 5.1 and then create the runbook:

image

With the runbook created, navigate into the newly created runbook and click on the Edit button:

image

Insert the prepared script into the Runbook:

image

Proceed to click Save and then Publish to publish the PowerShell Runbook. Note that we won’t be testing this just yet because we haven’t configured the managed identity yet and therefore the runbook does not have the appropriate permissions to use the Invoke-AzVMRunCommand to start a Windows service on the VM.

Managed Identity

With the runbook created, we’ll need to configure a managed identity for the Automation account to run the PowerShell script. More documentation about managed identity can be found in the following documentation:

Using a system-assigned managed identity for an Azure Automation account
https://docs.microsoft.com/en-us/azure/automation/enable-managed-identity-for-automation

Navigate to the Automation Account > Identity > System assigned and switch the Status to On to enable a system assigned managed identity:

image

Next, click on the Azure role assignments and add the Virtual Machine Contributor role to the assignment:

image

The Automation Account now has the assigned role for the PowerShell script to execute with.

Test a Automation Account Runbook

With the managed identity configured, we can now proceed to test the PowerShell script and verify that it indeed starts the Windows service. Navigate into the runbook and click on the Edit button:

image

Click on the Test pane button:

image

Click on the Start button to execute the PowerShell script using the managed identity:

image

The runbook will now execute the script:

Queued..

Streams will display when the test completes.

image

Wait for the test complete and verify that the output indicates it succeeded without errors:

image

Proceed to verify that the Windows service on the VM has restarted:

image

Create an Alert to monitor for service down and create Action Group with Automation Account Runbook

With the Automation Account Runbook tested, let’s proceed to create an alert to detect when the Remote Registry service (or any service of your choice) has stopped. Navigate to the Log Analytics workspace that was created to monitor the service, click on Create > Alert rule:

image

Select Custom Log Search to provide a custom Kusto query:

image

Many posts, including a previous one I wrote, simply use the following query to look for when a service has stopped:

ConfigurationData

| where SvcName =~ "RemoteRegistry"

| project SvcName, SvcDisplayName, SvcState, TimeGenerated

| where SvcState != "Running"

I find that the issue with using this query is that it will return all records of when the service has not been running within the specified period. This means that if the service was restarted and is running afterwards then this query will not show it and therefore the alert would continue to be fired. After giving it bit of thought, what I wanted to was run a query to get the last time the service was running and the last time it was stopped then compare the two TimeGenerated. Given my lack of experience with Kusto query, I cannot figure out how I can take the time and compare them so I decided to capture the two queries in variables, join them together, then compare the time stamps as shown below:

let LastStopped =

ConfigurationData

| where SvcName =~ "RemoteRegistry"

and SvcState != "Running"

| project SvcName, SvcDisplayName, SvcState, TimeGenerated

| order by TimeGenerated desc

| limit 1;

let LastRunning =

ConfigurationData

| where SvcName =~ "RemoteRegistry"

and SvcState == "Running"

| project SvcName, SvcDisplayName, SvcState, TimeGenerated

| order by TimeGenerated desc

| limit 1;

LastRunning

| join LastStopped on SvcName

// LastRunning time is earlier than LastStopped time

| where TimeGenerated <= TimeGenerated1

The intention of this query is to only return a result if the most recent event was when the service was not running and to not return anything if the most recent event was when the service is running. I’m completely open to recommendations if anyone happen to read this and think there is a better way of doing it.

image

With the query in place, proceed to create the Alert by clicking on Continue Editing Alert to use the query, leave the rest of the conditions as default and click on Actions:

image

Create a new action group by clicking on Create action group:

image

Provide a name for this action group and click on Notifications:

image

Configuration the notification setting and then click on Actions:

image

Select Automation Runbook for the Action type:

image

Select the runbook that was created earlier and click OK:

image

Proceed to create the Action Group:

image

image

The newly created Action group should automatically be added to the Alert:

image

Fill in the Details tab for the alert:

image

Complete creating the Alert:

image

Test Alert and Automation Account Runbook

Proceed to test and confirm that an alert is fired when the service is down and the runbook has successfully executed to restart the service:

image

image

I hope this blog post is able to provide information on how to set up an Automation Account to monitor a virtual machine’s Windows service and a runbook that will execute a PowerShell script using the Invoke-AzVMRunCommand to restart a stopped service.