Using Azure Resource Graph Explorer to determine what resources are sending diagnostic data to a Log Analytics Workspace

One of the questions I am frequently asked is how we can effectively determine what resources are sending data to a particular Log Analytics Workspace. Those who are administrators of Azure will know that most subscriptions will eventually contain Log Analytics Workspaces as shown in the list and screenshot below:

• DefaultWorkspace-d3f0e229-2fcd-45df-a791-614ba183e648-canadaea
• DefaultWorkspace-d3f0e229-2fcd-45df-a791-614ba183e648-CCAN
• DefaultWorkspace-d3f0e229-2fcd-45df-a791-614ba183e648-EUS

This isn’t the fault of poor management as many resources such as Insights would automatically default to these types of workspaces when they are enabled.

Attempting to browse the blades in these Log Analytics Workspaces will not allow us to easily determine what resources in Azure are sending data to the Log Analytics Workspace:

While it is possible to review the type of tables created and if the schema and data stored is known, then we could possibly query the data for the resources but this can be prone to errors causing resources to be missed:

Trying to search for how to achieve this lead me to the PowerShell cmdlet: Get-AzOperationalInsightsDataSource (https://learn.microsoft.com/en-us/powershell/module/az.operationalinsights/get-azoperationalinsightsdatasource?view=azps-10.2.0) but this did not allow me to obtain the information I needed.

What I ended up thinking of was whether it was possible to use Resource Graph Explorer to retrieve this information and after viewing the properties of a resource that I need was sending logs to a Logs Analytics Workspace, I was able to confirm that it could be done.

The following the is properties of a Function App:

If we scroll down the properties of the resource, we will find the following name/value pair:

Name: "WorkspaceResourceId"
Value: "/subscriptions/dxxxxxx9-2fcd-xxxx-a791-xxxxxxxxe648/resourceGroups/DefaultResourceGroup-CCAN/providers/Microsoft.OperationalInsights/workspaces/DefaultWorkspace-d3f0e229-2fcd-45df-a791-614ba183e648-CCAN",

Validating that a resource would have the Log Analytics Workspace defined in its properties, we can use the following query to list all resources that contain this property:

resources
| where properties.WorkspaceResourceId == "/subscriptions/d3xxxxx-2fcd-xxxx-xxxx-6xxxxxe648/resourceGroups/DefaultResourceGroup-CCAN/providers/Microsoft.OperationalInsights/workspaces/DefaultWorkspace-d3f0e229-2fcd-45df-a791-614ba183e648-CCAN"
| project name

Note that if you do not know of at least one resource that uses the Log Analytics Workspace, we can retrieve the WorkspaceResourceId of the workspace by navigating to the Log Analytics Workspace in portal.azure.com and copying the string from the URL:

I hope this helps anyone who may be looking for this information as I did but unable to find an easy way to achieve this.

PowerShell script to bulk convert Azure Firewall logs in JSON Line format stored on a Storage Container to CSV format

This post serves as a follow up to my:

Converting Azure Firewall logs in JSON format created from Archive to a storage account diagnostic setting to CSV format
http://terenceluk.blogspot.com/2023/07/converting-azure-firewall-logs-in-json.html

… where I provided a script to convert a single JSON file that stored Azure Firewall Logs in a Storage Account container.

As noted in the previous post, I want to follow up with a script that would traverse through a folder reading JSON files in the sub directories and converting them to CSVs to avoid manually generating each CSV for every hour of the day.

Additional reasons for using this script are:

1. Allow the retrieval of archived Azure Firewall Logs that are no longer stored in Log Analytics
2. Bulk converting JSON Line files to CSVs with a specified start and end date
3. A method for working around the 30,000 records return limit when using Log Analytics Workspaces to query data

The entries for Azure Firewall Log activities can get fairly large so this script will read through each JSON file that are broken up in each hour of the day and convert them to CSV files. I originally thought about combining the files but working out the math for days of longs meant file sizes can get into the GBs and attempting to work with CSV files that large won’t be pleasant.

The JSON script can be found at GitHub repository here: https://github.com/terenceluk/Azure/blob/main/Azure%20Firewall/Bulk-Convert-Az-Firewall-Logs-JSON-to-CSV.ps1

The output file list would look as such:

Hope this helps anyone who may be looking for such a script that will save them the time required to manually convert the logs.

Creating Azure Firewall Policy Rule Collections in Network Collection Group with PowerShell and Excel reference files

Those who have configured Rule Collections for a Azure Firewall Policy whether via GUI or scripting will know how tedious the task can be due to the amount of time for any type of change to be applied and the non-parallel stream of updates you can push to the firewall. I’ve also noticed that attempting to use multiple browser windows to copy and apply changes can potentially overwrite changes to the configuration. Case in point, I had a negative experience where I had window #1 to copy similar rule collections to window #2, and everything went as planned as long as I only saved to window #2. However, if I were to make a change in window #1 where it had not been refreshed with the changes applied to window #2, the save operation would overwrite the changes I made in window #2. I lost quite a bit of configuration due to this scenario.

To minimize the mistakes and amount of time I spent staring at the Azure Firewall Policy window and slowly applying configuration updates one at a time, I decide to spend a bit of time to create PowerShell scripts to reference an Excel file with configuration parameters. The first script I created was one that read an Excel spreadsheet to create the list of Rule Collections that are placed under a predefined Rule Collection Group.

The PowerShell script can be found here in my GitHub repository: https://github.com/terenceluk/Azure/blob/main/Azure%20Firewall/Create-NetworkRuleCollection.ps1

The following is a sample spreadsheet for the PowerShell script to read from:

Here is a sample screenshot of the Rule Collections in the Azure management portal:

Hope this helps anyone who may be looking for such a script as the creation of Rule Collections can only be created one at a time.

Attempting to create a folder on an Azure Data Lake Storage Account with Private Endpoint fails with: "Failed to add directory 'Test'. Error: AuthorizationFailure: This request is not authorized to perform this operation."

Problem

A colleague of mine recently asked me to help troubleshoot an issue with an Azure Storage Account that has Hierarchical Namespace enabled, which is essentially an Azure Data Lake, where any attempts to create a folder would fail:

The error message presented was generic and appears to suggest that it is caused by a permissions issue:

Failed to add directory

Failed to add directory 'Test'. Error: AuthorizationFailure: This request is not authorized to perform this operation. RequestId:da720a90-c01f-0053-5d3f-c61ef5000000 Time:2023-08-03T19:22:01.2257950Z

Creating containers or uploading blobs (files) to the storage account did not have any issues as those operations were successful as shown in the following screenshot:

This error has been one that I’ve come across frequently in the past and it is usually because the storage account is locked down with only a private endpoint for the blob service and not for the data lake service created. The following Microsoft documentation explains the reason:

Use private endpoints for Azure Storage

https://learn.microsoft.com/en-us/azure/storage/common/storage-private-endpoints#creating-a-private-endpoint

If you create a private endpoint for the Data Lake Storage Gen2 storage resource, then you should also create one for the Blob Storage resource. That's because operations that target the Data Lake Storage Gen2 endpoint might be redirected to the Blob endpoint. Similarly, if you add a private endpoint for Blob Storage only, and not for Data Lake Storage Gen2, some operations (such as Manage ACL, Create Directory, Delete Directory, etc.) will fail since the Gen2 APIs require a DFS private endpoint. By creating a private endpoint for both resources, you ensure that all operations can complete successfully.

The following are screenshots confirming the missing configuration.

Note that Hierarchical Namespace is enabled:

Note that Public network access is set to Disabled:

Note that there is only 1 private endpoint configured for the storage account:

… and the Target sub-resource of the private endpoint is blob:

Solution

To correct the issue, we’ll need to create an additional private endpoint that has the Target sub-resource configured as DFS (Data Lake Storage Gen2). Begin by navigating to the Networking blade for the storage account and create a new Private Endpoint:

Proceed to fill in the details for the private endpoint:

Select dfs as the Target sub-resource:

Complete the creation of the private endpoint:

Folder creation should now succeed:

Hope this provides anyone who might have ran into this issue and is looking for a solution. I’ve found that searching for the error message does not always return results to this solution.