Monitoring, Alerting, Reporting Azure AD logins and login failures with Log Analytics and Logic Apps

Having monitoring and alerting set up for failed login attempts to any identity directory services (e.g. on-premise AD and Azure AD) have always been important yet often times neglected for many environments I’ve worked in. Those who have worked with on-premise Active Directory would know the pain of going through a domain controller’s security event logs and how difficult it is to obtain what you need as all security events are recorded. This is why there are third party monitoring and alerting products that have had so much success for years. The built-in Azure AD Sign-in Logs provides a display of sign-in attempts for troubleshooting but is also rarely reviewed and audited. If configuration has been made for the Azure AD logs, the retention can range from 7 days (Azure AD Free) to 30 days (Azure AD Premium P1 or P2) as stated here in the documentation: https://docs.microsoft.com/en-us/azure/active-directory/reports-monitoring/reference-reports-data-retention#how-long-does-azure-ad-store-the-data

I’ve always bundled in the step for increasing the retention of these logs in my projects and the action required to ensure logs are retained for a longer retention is to route them to an Azure storage account using monitor as described in the following Microsoft documentation: https://docs.microsoft.com/en-us/azure/active-directory/reports-monitoring/quickstart-azure-monitor-route-logs-to-storage-account. With the retention of the logs in place, the next critical component is to have monitoring and alerting in place to capture events that are of concern to the organization. A common use case is to monitor the first global admin account created for the tenant as that is usually one that isn’t associated with a particular user and may be used as the Emergency Break Glass Account that does not have MFA. Another common monitoring that should be set up is for potential malicious login attempts.

With the above in mind, what I would like to do in this post is demonstrate how to:

1. Set up monitoring of Azure AD with Log Analytics
2. Set up an alert using Kusto to query Azure AD Sign-In Logs
3. Set up reporting of Azure AD failed sign in attempts with Logic Apps

Configure Log Analytics Workspace for Azure AD

I won’t go into how to create a Log Analytics workspace so proceed to create one that will be used to ingest Azure AD logs as such:

Proceed to navigate into Azure Active Directory > Diagnostic settings and then click on Add diagnostic setting:

The following are the log category options:

• AuditLogs
• SignInLogs
• NonInteractiveUserSignInLogs
• ServicePrincipalSignInLogs
• ManagedIdentitySignInLogs
• ProvisioningLogs
• ADFSSignInLogs
• RiskyUsers
• UserRiskEvents
• NetworkAccessTrafficLogs
• RiskyServicePrincipals
• ServicePrincipalRiskEvents

For the purpose of this example, we’ll just capture the AuditLogs and SignInLogs and click on the Save button:

Note the following requirement:

In order to export Sign-in data, your organization needs Azure AD P1 or P2 license. If you don't have a P1 or P2, start a free trial.

With the AuditLogs and SignInLogs configured for collection, navigate to the corresponding Log Analytics workspace > Usage and estimated costs > Data Retention to configure the required retention:

Set up an alert using Kusto to query Azure AD Sign-In Logs

As mentioned earlier, a common use case is to monitor the first global admin account created for the tenant as that is usually one that isn’t associated with a particular user and may be used as the Emergency Break Glass Account that does not have MFA. To set this up, navigate to Monitor > Alerts > Create > Alert rule:

Select the log analytics workspace as the scope:

Select Custom log search for the Condition so we can define our own Kusto query:

The following query can be used to look for break glass account login:

SigninLogs
|where UserPrincipalName contains "cspadmin@contoso.onmicrosoft.com"

With the Condition defined, proceed to assign an Action Group that will alert the appropriate group an appropriate communication such as email.

Set up reporting of Azure AD failed sign in attempts with Logic Apps

Another common scenario I’ve come across is to provide a report of failed sign-in attempts to a group of administrators to review on either a hourly, daily or weekly basis. A frequent report can be very useful for an active attack while daily and weekly can be great for monitoring attacks. For the purpose of this example, I will use a directory with the following failed sign-in attempts from all sorts of Asia countries for a user:

The kusto query we’ll be using to look for failed sign-ins is the following:

SigninLogs

| where Status.errorCode != 0

| extend City=LocationDetails.city, State=LocationDetails.state, Country=LocationDetails.countryOrRegion, Error_Code=Status.errorCode, Failure_Reason=Status.failureReason

| project TimeGenerated, UserDisplayName, AppDisplayName, IPAddress, City, State, Country, AuthenticationRequirement, Failure_Reason, ConditionalAccessStatus, ConditionalAccessPolicies, Error_Code

You are free to adjust the query to include or exclude additional fields of the failed sign-in attempts.

With the kusto query created and tested, the next step is to create a Logic App that will generate and send a report to an email address for administrators to review. Create a Logic App as such:

**Note that the Enable log analytics option for the creation of the Logic App is to get richer debugging information about the logic apps during runtime.

Next, navigate into the Logic app and click on Logic app designer:

We’ll be creating 3 steps for this Logic App where:

1. Recurrence: This will configure a recurring schedule for this Logic App to execute
2. Run query and visualize results: This will allow us to run the Kusto query, set a Time Range and specify a Chart Type
3. Send an email (V2): This will allow us to send the Kusto query results via email

Recurrence:

I wanted to send this report every day at 5:00p.m. EST:

Run query and visualize results:

The query I wanted to execute is:

SigninLogs

| where Status.errorCode != 0

| extend City=LocationDetails.city, State=LocationDetails.state, Country=LocationDetails.countryOrRegion, Error_Code=Status.errorCode, Failure_Reason=Status.failureReason

| project TimeGenerated, UserDisplayName, AppDisplayName, IPAddress, City, State, Country, AuthenticationRequirement, Failure_Reason, ConditionalAccessStatus, ConditionalAccessPolicies, Error_Code

The time range I wanted to query for is the last 12 hours and the Chart Type I want is an HTML Table:

Send an email (V2):

The report will include the Attachment Content and Attachment Name derived from the query with the subject Failed Login Report. The email will look pretty barebone so you are free to add HTML code to pretty it up.

Proceed to save the Logic App:

Use the Run Trigger to test the Logic App and confirm that an email is sent:

Hope this helps anyone who may be looking for a way to monitor, alert, and report Azure AD logins with Log Analytics and Logic Apps.