Script to continuously run New-ComplianceSearchAction to purge emails discovered by eDiscovery

I recently had to assist with troubleshooting an issue where running the cmdlet New-ComplianceSearchAction against a Content Search result in Microsoft Purview (https://compliance.microsoft.com/) would not delete any email and discovered a few key behavioral items that I wanted to highlight and provide a script I used to delete the emails.

First off, attempting to use the cmdlet New-ComplianceSearchAction against a Content Search will not delete anything. The results from running:

Get-ComplianceSearchAction | FL SearchName,Results,Action,Name,Status

… will always display:

Results: Purge Type: SoftDelete; Item count: 0; Total size 0; Details: {}

What I discovered was that in order to receive results from the deletion is to create the search in eDiscovery, and use the cmdlet New-ComplianceSearchAction against that search:

Figuring this out then lead to another problem where the cmdlet would only delete 10 emails at a time so the final solution was to use a script that would continuously create a new job when the previous one has succeeded. This script can be found at my following GitHub repository: https://github.com/terenceluk/Azure/blob/main/PowerShell/New-ComplianceSearchAction-Continuously-Purge.ps1

Hope this helps anyone who may encounter the same issue as I had. The experience can be very frustrating given the changes to the cmdlets over the past few years.

Updated: Create an automated report for Office 365 / Microsoft 365 license usage with friendly names using Azure a Function App (using a system assigned managed identity) and Logic Apps

One of my colleagues who used my previous blog post:

Create an automated report for Office 365 / Microsoft 365 license usage with friendly names using Azure a Function App and Logic Apps
https://terenceluk.blogspot.com/2022/07/create-automated-report-for-office-365.html

… recently informed me that his Function App would fail to run every couple of months due to the issue I mentioned in my other blog post:

Azure Function App fails with: “ERROR: Assembly with same name is already loaded”
https://terenceluk.blogspot.com/2022/10/azure-function-app-fails-with-error.html

After giving the problem a bit of thought, what I recommended was to change the authentication for connecting to Microsoft Graph (Connect-MgGraph) in the function app from certificate authentication to using a Managed Identity instead. This post serves to provide the alternate steps for the setup.

Step Configurations that are no longer needed

The following are the changes to the steps in my previous blog post:

Create an automated report for Office 365 / Microsoft 365 license usage with friendly names using Azure a Function App and Logic Apps
https://terenceluk.blogspot.com/2022/07/create-automated-report-for-office-365.html

Step #1 - Creating a Service Principal for App-Only authentication for Microsoft Graph PowerShell SDK (Connect-MgGraph)
This step will no longer be required.

Step #3 – Create a Function App that will retrieve Office 365 / Microsoft 365 license usage with friendly names and return it in HTML format
As we will no longer be using certificate based authentication, we will not need to create the application settings appID and WEBSITE_LOAD_CERTIFICATES.

Step Configurations Additions and Changes

Step #2 – Create a Storage Account Container that will provide the CSV file containing the product friendly names
The reason why I opted to download the CSV file that Microsoft provides which contain the product friendly names is because the link may change over time but for this blog post, I will use the direct link to download.microsoft.com (https://download.microsoft.com/download/e/3/e/e3e9faf2-f28b-490a-9ada-c6089a1fc5b0/Product%20names%20and%20service%20plan%20identifiers%20for%20licensing.csv)

Step #3 – Create a Function App that will retrieve Office 365 / Microsoft 365 license usage with friendly names and return it in HTML format
Since we’ll be using Managed Identity for authentication, we’ll be adding the steps here.

The following are the steps for the new configuration.

Step #1 - Creating a Service Principal for App-Only authentication for Microsoft Graph PowerShell SDK (Connect-MgGraph)

Step #2 – Create a Storage Account Container that will provide the CSV file containing the product friendly names

Step #3 – Create a Function App that will retrieve Office 365 / Microsoft 365 license usage with friendly names and return it in HTML format

Proceed to create the new Function App as described in my previous post:

Create a Function App that will retrieve Office 365 / Microsoft 365 license usage with friendly names and return it in HTML format. This Function App collects the data that will in turn be call by a Logic App to generate an email and send the report off to an email address.

Proceed to create a Function App with the following parameters:

Publish: Code
Runtime stack: PowerShell Core
Version: 7.2
Operating System: Windows

Configure the rest of the parameters as required by the environment.

Step #4 – Create a System assigned managed identity for the Function App

With the Function App created, navigate to Settings > Identity and turn on the System assigned status to create a managed identity:

You should now see a Object (principal) ID created for the function app:

Make a note of the Object (principal) ID as well be using it to confirm an Enterprise application is created.

Next, navigate to Azure Active Directory > Enterprise applications, then search for the Object (principal) ID to look for the service principal created:

Step #5 – Use PowerShell to grant Graph API permissions to the System assigned managed identity for the Function App

Note the returned result of the service principal and the associated Application ID in the returned result. What normally happens now is when we navigate to the App Registration blade, locate the object with the Application ID then configure the API Permissions blade. However, given that this is a managed identity, you will not find the entry in the App Registration list and navigating to the Permissions blade of the Enterprise Application will not allow you to grant API permissions:

A quick search will return the following Microsoft blog post explaining the process:

Grant Graph API Permission to Managed Identity Object
https://techcommunity.microsoft.com/t5/integrations-on-azure-blog/grant-graph-api-permission-to-managed-identity-object/ba-p/2792127

As described in the blog post above, PowerShell is required to assign the permissions but while the instructions are fairly straight forward, the cmdlets did not work for me as of December 25, 2022 (the blog post was published on Sept 28, 2021). The problem appears to be due to line:

$AppRole = $GraphServicePrincipal.AppRoles | Where-Object {$_.Value -eq $PermissionName -and $_.AllowedMemberTypes -contains "Application"}

… because the Where-Object is not able to successfully filter the AppRole with the Value and AllowedMemberTypes parameters. This led me down the path to explore using Connect-MgGraph cmdlets that another user provided in the comments below the post but that did not work either because I was not able to use them to configure Managed Identities (they can configure other Enterprise Applications). Here are 2 additional useful sources I found while troubleshooting:

The complete list only shows applications configured in Azure AD as EnterpriseApplications, not Managed Identities:
https://github.com/Azure/azure-powershell/issues/18412

Using New-MgServicePrincipalAppRoleAssignment to configure permissions:
https://copyprogramming.com/howto/how-to-set-microsoft-graph-api-permissions-on-azure-managed-service-identity-with-powershell-7

So after trying numerous changes, what worked is to use both:

1. Get-AzureADServicePrincipal to retrieve the ObjectId of the Graph API
2. Get-AzADServicePrincipal to retrieve the App Roles of Graph API so we can assign them using the role’s ID

With the above challenges described, let’s proceed with how we can grant the following API permissions required for the Managed Identity:

• Directory.Read.All
• Directory.ReadWrite.All
• Organization.Read.All
• Organization.ReadWrite.All

If we were only granting one API permissions then the PowerShell script variables that are required to be defined are:

$TenantID="provide the tenant ID"

$GraphAppId = "00000003-0000-0000-c000-000000000000" <-- Note that this parameter is optional and the provided value here does not need to be changed because this corresponds to Graph API GUID.

$DisplayNameOfMSI="Provide the Function App name"

$PermissionName = "Directory.Read.All"

Since we have more than one permission to assign, we’ll be storing the required permissions in an array as such:

$TenantID="provide the tenant ID"

$GraphAppId = "00000003-0000-0000-c000-000000000000"

$DisplayNameOfMSI="Provide the Function App name"

$permissionsNames = @(

'Directory.Read.All'

'Directory.ReadWrite.All'

'Organization.Read.All'

'Organization.ReadWrite.All'

)

The script to configure the appropriate managed identity permissions can be found at my GitHub here: https://github.com/terenceluk/Azure/blob/main/PowerShell/Configure-Managed-Identity-API-Permissions.ps1

The following is a sample output once the permissions successfully granted via the PowerShell cmdlets:

The required permissions should now be listed in the Permissions blade of the Enterprise Application representing the Managed Identity:

Step #6 – Configure Function App to download required dependencies

Proceed to configure the requirements.psd1 file in the App files blade so the appropriate modules will be loaded for the PowerShell code in the function app. Note that I choose to import the specific modules required for the cmdlets Connect-MgGraph (Microsoft.Graph.Authentication) and Get-MgSubscribedSku (Microsoft.Graph.Identity.DirectoryManagement) because the Microsoft.Graph modules has 38 sub modules in it and I was not able to get the function app code to run by importing that.

'Az.Accounts' = '2.*'
'JoinModule' = '3.*'
'Microsoft.Graph.Identity.DirectoryManagement' = '1.*'
'Microsoft.Graph.Authentication' = '1.*'

Navigate to the profile.ps1 and verify that the following lines are uncommented:

if ($env:MSI_SECRET) {

Disable-AzContextAutosave -Scope Process | Out-Null

Connect-AzAccount -Identity

}

Step #7 – Create a HTTP trigger Function that will generate the report and turn on Application Insights

Application Insights is extremely useful for troubleshooting issues with the Function App so I would highly recommend turning it on:

With the prerequisites configured for the Function App, proceed to create the actual function trigger:

Select HTTP trigger as the template and provide a meaningful name:

With the trigger created, navigate to Code + Test and paste the following code from my GitHub repo into run.ps1:
https://github.com/terenceluk/Microsoft-365/blob/main/Administration/Get-M365-License-Report-Function-v2.ps1

The following are changes you’ll need to apply to the code:

<h2>Client: Contoso Limited</h2>

With the function app code in place, proceed to use the Test/Run feature to test it. Note that the function app expects the tenant ID to be passed to it so the Body of the test should include the following:

{

"tenant": "xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"

}

The following log entries will be displayed if Application Insights is turned on:

Confirm the HTTP response code of 200 OK and the HTTP response content results:

Note that I have experienced an odd behavior where running the Test/Run feature displays a 503 Service Unavailable HTTP response code as shown in the screenshot below:

Step #8 – Create a Logic App that is scheduled and will call the Azure Function App to retrieve the license report and then send it out

I won’t go into the details for the Logic App setup as it would be repeat from my previous post so please refer to it to set up the triggering of the Function App and send the report out via email.

Create an automated report for Office 365 / Microsoft 365 license usage with friendly names using Azure a Function App and Logic Apps
https://terenceluk.blogspot.com/2022/07/create-automated-report-for-office-365.html

Setting up BitTitan to migrate mailboxes from O365 to another O365 tenant fails source authentication with the error: "Your migration failed while checking source credentials. The request failed. The remote server returned an error: (401) Unauthorized."

It has been a while since I’ve been involved with Office 365 mail migrations but I was recently contacted by a colleague who was setting up BitTitan’s MigrationWiz for a migration and could not figure out why he was not able to authenticate so I hopped on to help. The search results from Google directed me to various documentation provided by BitTitan but none of them lead me to the right solution. I don’t usually use YouTube for troubleshooting as most of us probably feel it takes too long to watch a video as compared to reading a blog post like this but the following video is where I eventually found the answer:

How to solve BitTitan MigrationWiz Error 401 Unauthorized in 2022
https://www.youtube.com/watch?v=iI35AJrGYiw

This blog post serves to help anyone who might encounter the same problem quickly find the answer.

Problem

You attempt to use the Verify Credentials feature in MigrationWiz after setting up the source and destination tenants but receive the status: Failed (Verification)

Navigating into one of the accounts display the following error message:

Your migration failed while checking source credentials. The request failed. The remote server returned an error: (401) Unauthorized.

Solution

The reason why the environment I was troubleshooting in has this failure is because Microsoft had started disabling basic authentication for Office 365 that affects the EWS service that MigrationWiz relies on (3 minute mark in the video). This can be confusing because if you navigate to Settings > Org Settings > Modern Authentication, you’ll see that it states basic authentication is enabled for various services (including Exchange Web Services). The problem here is that this only applies to the modern Outlook client, which MigrationWiz isn’t.

To remediate this, click on the Help & support button at the bottom right corner of the administration console:

Then type in the following string to search:

diag: enable basic auth in exo

Proceed to click on the Run Tests button:

Assuming basic authentication is disabled, we should be provided with a drop down menu box to select a service to enable:

Select Exchange Web Services (EWS) to enable the MigrationWiz dependent service, then click on Update Settings:

The following message will be displayed:

Run diagnostics
Basic authentication has been re-enabled for the selected protocol.

The Basic authentication blocked applications setting has been updated. You should be able to use Basic authentication with the selected protocol within the next hour.

Proceed to try and verify the credentials in an hour or so and the process should complete successfully.

Hope this helps as it took me a bit of time to figure this out.

Create an automated report for Office 365 / Microsoft 365 license usage with friendly names using Azure a Function App and Logic Apps

I received quite a few requests about how to automate the process of my previous post where I described how to use Microsoft Graph PowerShell SDK to generate a Office 365 / Microsoft 365 license report that had friendly names:

Using Microsoft Graph PowerShell SDK to retrieve Office 365 / Microsoft 365 license usage with friendly names
http://terenceluk.blogspot.com/2022/07/using-microsoft-graph-powershell-sdk-to.html

I have to be honest that I haven’t actually converted my old reports with Msol online so I took the time over the weekend to create a new report and will now demonstrate the process.

Step #1 - Creating a Service Principal for App-Only authentication for Microsoft Graph PowerShell SDK (Connect-MgGraph)

The first step is to set up a App Registration / Service Principal so the Azure Function App can authenticate and sign into Microsoft Graph. The following documentation describes the process of registering the application:

Use app-only authentication with the Microsoft Graph PowerShell SDK
https://docs.microsoft.com/en-us/powershell/microsoftgraph/app-only?view=graph-powershell-1.0&tabs=azure-portal

… but does not include the process of creating a certificate so I will include a few other posts I’ve written which will demonstrate how to create a self-signed certificate (both the .cert and .pfx) for application authentication:

Step #3 – Generate a self-signed certificate for the application that will be authenticating
http://terenceluk.blogspot.com/2021/05/setting-up-app-only-authentication-for.html

Step #2 – Create a self-signed certificate on the local Windows desktop and export it to PFX with the private key
http://terenceluk.blogspot.com/2022/02/creating-service-principal-to-connect.html

The following are screenshots of the process:

Create a new App Registration as such:

Copying the following configuration once the App Registration has been created:

• Application (client) ID
• Directory (tenant) ID

Navigate to the Certificates & secrets blade, click on Upload certificate, select the .cer export of the certificate (this file does not contain the private key), add a description, and upload:

With the certificate uploaded, proceed to copy the Thumbprint:

The new App Registration will need to be configured with the required permissions by navigating to API permissions, click on App a permission, and select Microsoft Graph:

Select Application permissions:

Add the following Application permissions:

• Directory.Read.All
• Directory.ReadWrite.All
• Organization.Read.All
• Organization.ReadWrite.All

The configured permissions should be listed as shown in the screenshot below and before proceeding, make sure you click on Grant admin consent for organization or else the permissions will be come into effect.

**Note that User.Read is a Delegate permission and should already be added by default.

The App Registration / Service Principal that will be used to connect to Microsoft Graph should now be in place. You should be able to test authentication on a workstation that has the certificate along with its private key by using the following cmdlet:

Connect-MgGraph -ClientID "xxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxxx" -TenantId "xxxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxxxxx" -CertificateThumbprint "3548c8xxxxxxxxxxxxxxxxxxxxxb2f8affa214"

Step #2 – Create a Storage Account Container that will provide the CSV file containing the product friendly names

Microsoft provides a CSV file that contains the product friendly names from the following document: https://docs.microsoft.com/en-us/azure/active-directory/enterprise-users/licensing-service-plan-reference

Proceed to download the CSV file and place the file into a storage account as such:

The function app will retrieve this CSV file and use it to map the friendly names to the licenses.

Step #3 – Create a Function App that will retrieve Office 365 / Microsoft 365 license usage with friendly names and return it in HTML format

With the App Registration / Service Principal in place, the next step is to create a Function App that will retrieve Office 365 / Microsoft 365 license usage with friendly names and return it in HTML format. This Function App collects the data that will in turn be call by a Logic App to generate an email and send the report off to an email address.

Proceed to create a Function App with the following parameters:

Publish: Code
Runtime stack: PowerShell Core
Version: 7.2
Operating System: Windows

Configure the rest of the parameters as required by the environment.

With the Function App created, proceed to configure 2 new application settings that will contain the value of the previously copied Application (client) ID of the service principal and the thumbprint of the certificate we’ll be using to authentication to Microsoft Graph. The names I’ll use for these applications will be:

• appID
• WEBSITE_LOAD_CERTIFICATES

appId

WEBSITE_LOAD_CERTIFICATES

With the application settings configured, proceed to upload certificate file which contains the private key (.pfx) that the Function App will use to authenticate and sign into Microsoft Graph:

Confirm that the certificate has successfully uploaded and is in a healthy state:

Next proceed to configure the requirements.psd1 file in the App files blade so the appropriate modules will be loaded for the PowerShell code in the function app. Note that I choose to import the specific modules required for the cmdlets Connect-MgGraph (Microsoft.Graph.Authentication) and Get-MgSubscribedSku (Microsoft.Graph.Identity.DirectoryManagement) because the Microsoft.Graph modules has 38 sub modules in it and I was not able to get the function app code to run by importing that.

'JoinModule' = '3.*'
'Microsoft.Graph.Identity.DirectoryManagement' = '1.*'
'Microsoft.Graph.Authentication' = '1.*'

---------------------------------------------------------------------------------------------------------------------------

Update – Sept 5, 2022

The versions of the modules below use at the time of this original post was 1.10.0 and I realized that later versions such as 1.11.1 no longer worked with certificate authentication:

• Microsoft.Graph.Identity.DirectoryManagement
• Microsoft.Graph.Authentication

To correct the issue, please refer to this post:

Azure Function App using certificate authentication falls to authenticate when executing Connect-MgGraph
http://terenceluk.blogspot.com/2022/09/azure-function-app-using-certificate.html

---------------------------------------------------------------------------------------------------------------------------

Application Insights is extremely useful for troubleshooting issues with the Function App so I would highly recommend turning it on:

With the prerequisites configured for the Function App, proceed to create the actual function trigger:

Select HTTP trigger as the template and provide a meaningful name:

With the trigger created, navigate to Code + Test and paste the following code into run.ps1:

https://github.com/terenceluk/Microsoft-365/blob/main/Administration/Get-M365-License-Report-Function.ps1

The following are changes you’ll need to apply to the code:

The client name:

The storage account URI:

With the function app code in place, proceed to use the Test/Run feature to test it. Note that the function app expects the tenant ID to be passed to it so the Body of the test should include the following:

{

"tenant": "xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"

}

The following log entries will be displayed if Application Insights is turned on:

Confirm the HTTP response code of 200 OK and the HTTP response content results:

Step #4 – Create a Logic App that is scheduled and will call the Azure Function App to retrieve the license report and then send it out

With the Azure Function App created and tested, proceed to create the Logic App that will be scheduled, calls the Function App for the HTML license report and then email it out.

Navigate to the Logic app designer blade and begin to configure the steps for the Logic App. The following are the steps we’ll be configuring:

The first is the Recurrence step that will schedule this logic app to run on the last day of each month:

Note that the GUI doesn’t provide the required controls so we’ll be using the JSON code provided in this document for the configuration: https://docs.microsoft.com/en-us/azure/logic-apps/concepts-schedule-automated-recurring-tasks-workflows#run-once-at-last-day-of-the-month

Click on Code View:

Then edit the triggers section:

Create an additional step by clicking on the + button and select Add an action then type in Function:

Select the Function App that was created:

Select the trigger that was created:

Place the body containing the tenant ID into the Request Body:

{

"tenant": "xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"

}

Proceed to create two additional steps:

1. Initialize variable
2. Set variable

These two steps will place the retrieved HTML report into the body of the email:

Initialize variable

Name: EmailBody
Type: String
Value: <leave blank>

Set variable

Name: EmailBody
Value: Select the Body

Add the last step that will email this report to the email address required:

Proceed to use the Run Trigger feature to execute the Logic App and confirm that the report is generated and sent:

I hope this helps anyone who may be looking for instructions on how to configure automated reports. The task may seem trivial but it took me a few hours to troubleshoot the issues.