Python script that will asynchronously receive events from an Azure Event Hub and send it to a Log Analytics Workspace custom table

One of the key items I’ve been working on over the past week as a follow up to my previous post:

How to log the identity of a user using an Azure OpenAI service with API Management logging (Part 1 of 2)
https://terenceluk.blogspot.com/2023/11/how-to-log-identity-of-user-using-azure.html

… is to write a Python script that will read events as they arrive in an Event Hub, then send it over to a Log Analytics Workspace’s custom table for logging. The topology is as such:

The main reason why I decided to go with this method is because:

Tutorial: Ingest events from Azure Event Hubs into Azure Monitor Logs (Public Preview)
https://learn.microsoft.com/en-us/azure/azure-monitor/logs/ingest-logs-event-hub

… required the Log Analytics workspace to be linked to a dedicated cluster or to have a commitment tier. The lowest price for such a configuration would be cost prohibitive for me to deploy in a lab environment so I decided to build this simple ingestion method.

Log Analytics Pricing Tiers:

I used various documentation available to create the script, create the App Registration, configure the Data Collection Endpoint and Data Collection Rule for the Log Analytics ingestion. Here are a few for reference:

Send events to or receive events from event hubs by using Python
https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-python-get-started-send?tabs=passwordless%2Croles-azure-portal

Logs Ingestion API in Azure Monitor
https://learn.microsoft.com/en-us/azure/azure-monitor/logs/logs-ingestion-api-overview

Tutorial: Send data to Azure Monitor Logs with Logs ingestion API (Azure portal)
https://learn.microsoft.com/en-us/azure/azure-monitor/logs/tutorial-logs-ingestion-portal

The script can be found at my GitHub repository here: https://github.com/terenceluk/Azure/blob/main/Event%20Hub/Python/Receive-from-Event-Hub-with-checkpoint-store-async.py

The following are some screenshots of the execution and output:

OpenAI API Call from Postman to API Management:

Script Execution and Output:

Log Analytics Ingestion Results:

I hope this helps anyone who might be looking for a script for the processing of events and ingestion to Log Analytics as it took me quite a bit of time on and off to troubleshoot various issues encountered. With this script out of the way, I am no prepared to finish up the 2 of 2 post for an OpenAI logging end to end solution, which I will be writing shortly.