How to log the identity of a user using an Azure OpenAI service with API Management logging (Part 1 of 2)

The single question I’ve been asked the most over the past few months from colleagues, clients, other IT professionals is how can we identify exactly who is using the Azure OpenAI service so we can generate accurate consumption reports and allow proper charge back to a department? Those who have worked with the diagnostic settings for Azure OpenAI and API Management will know that logging is available but there are gaps that desperately needs to be addressed. A quick search over the internet will show that using API Management can log the caller’s IP address but that isn’t very useful for obvious reasons such as:

1. If it’s public traffic with a public inbound IP address, how would we be able to tell who the user is?
2. Even if we can tie a public IP address to an organization because that’s the outbound NAT, the identity of the user is not captured
3. Even if we authenticate the user so a JWT token is provided to call the API, having the public IP address in the logs alone wouldn’t identify the user
4. If these were private IP addresses, it would be a nightmare to try and match the inbound IP address with an internal workstation’s IP address that is likely DHCP

I believe the first time I was asked this question was 3 months ago and I’ve always thought that Microsoft will likely address this soon with a checkbox in the diagnostic settings or some other easy to configure offering but fast forward to today (November 2023), I haven’t seen a solution so I thought I’d do a bit of R&D over the weekend.

The closest solution I was able to find is from this DevRadio presentation:

Azure OpenAI scalability using API Management
https://www.youtube.com/watch?v=mdRu3GJm3zE&t=1s

… where the presenter used multiple instances of Azure OpenAI to separate prompts to the OpenAI service belonging to different business units. While this solution allowed costs to be separated between predefined business units, the thought of telling a client that I need multiple instances to serve this purpose didn’t seem like something they would be receptive. While the DevRadio solution did not meet the requirements I had, it did give me the idea that perhaps I can use the logging of events to event hubs feature of the Azure API Management to accomplish what I want in the solution.

I have to say that this blog post is probably one of the most exciting one I’ve written in a while because I was heads down focused on learning and testing the Azure API Management inbound processing capabilities over 3 days of my vacation time off and felt extremely fulfilled that I now have an answer to what I could not provide a solution to for months.

If you’re still reading this, you might be wondering why there is the label Part 1 of 2 and the reason is because I ran out of time and have gotten back to a busy work schedule so could not finish the last portion of this solution but don’t worry as what I will cover in Part 1 will at least capture the information to identify the calling user. Here is a summary of what I am able to cover in this blog post:

1. How to set up API Management to log events to Event Hub
2. What inbound processing code should be inserted to send the OAuth JWT token to event hub
3. What inbound processing code can be used to extract any values in the JWT token to event hub
4. How to view the logged entries in event hub

The following is what I will cover in Part 2 in a future post:

1. How to ingest events from Azure Event Hubs into Azure Monitor Logs
2. How to use KQL to join events logged by API Management’s diagnostic settings (containing token usage, prompt information) with Azure Event Hub ingested logs (containing user identification)

The following is a high level architecture diagram and the flow of the traffic:

I’m excited to get this post published so let’s get started.

Prerequisites

This solution will require us to place an Azure API Management service in front of the Azure OpenAI service so API calls are:

1. Logged by the APIM
2. Authorized with OAuth by the API Management

Please refer to my previous post for how to set this up:

Securing Azure OpenAI with API Management to only allow access for specified Azure AD users
https://terenceluk.blogspot.com/2023/11/securing-azure-openai-with-api.html

What is available today out-of-the-box: API Management Diagnostic Settings Logging Capabilities

Assuming you have configured the API Management service as I demonstrated in my prerequisite section and Diagnostics Logging is turned on:

… then a set of information for each API call would be logged in the configured Log Analytics. Let’s first review what is available out-of-the-box when for the API Management. The complain I hear repeatedly is that while the logs captured by the API Management provide all the following great information:

• TenantId
• TimeGenerated [UTC]
• OperationName
• Correlationid
• Region
• isRequestSuccess
• Category
• TotalTime
• CalleripAddress
• Method
• Url
• ClientProtocol
• ResponseCode
• BackendMethod
• BackendUrl
• BackendResponseCode
• BackendProtocol
• RequestSize
• ResponseSize
• Cache
• BackendTime
• Apid
• Operationid
• ApimSubscriptionid
• ApiRevision
• ClientTlsVersion
• RequestBody
• ResponseBody
• BackendRequestBody

None of these captured fields allow for identifying the caller. To address this gap, we can leverage the log-to-eventhub inbound processing feature of API Management and Event Hubs to send additional information about the inbound API call to an event hub, then process it according to our requirements.

Turning on the logging of events for the API Management to Event Hubs

The first step for this solution is to turn on the feature that has API Management log to an Event Hub. I won’t go into the usual detail I provide for setting up the components due to my limited time but begin by creating an Event Hub Instance and Event Hub as shown in the following screenshots to serve as a destination for the APIM to send its logs:

Once the Event Hub Instance and Event Hub is created, and the API Management’s System Managed Identity is granted, we will use the following instructions to turn on the feature in API Management and use the Event Hub:

Logging with Event Hub
https://azure.github.io/apim-lab/apim-lab/6-analytics-monitoring/analytics-monitoring-6-3-event-hub.html

More detail about how the API Management is configured is described here:

How to log events to Azure Event Hubs in Azure API Management
https://learn.microsoft.com/en-us/azure/api-management/api-management-howto-log-event-hubs?tabs=PowerShell

Configuring API Management’s Inbound Processing rule to log JWT token and its values

The API Management log-to-eventhub can send any type of information to the Event Hub. For this post, I am going to demonstrate how to send the following information:

• EventTime
• ServiceName
• RequestId
• RequestIp
• Operationname
• api-key
• request-body
• JWTToken
• AppId
• Oid
• Name

Let’s go through these fields in a bit more detail. The following list of fields:

• EventTime
• ServiceName
• RequestId
• RequestIp
• Operationname
• request-body

… are ones that can be retrieved from the out-of-the-box diagnostic settings logs. I haven’t looked into all the available fields but I suspect that we can send all the out-of-the-box diagnostic settings to event hub to recreate what we have and potentially allow us to turn off the built in logging. The advantage of such an approach is that all logs will be stored in a single log analytics workspace table. The disadvantage of such an approach is that if new fields are introduced into the built in logs then we would need to update our log-to-eventhub code to capture those fields.

The other fields:

• api-key
• JWTToken
• AppId
• Oid
• Name

… are ones that we’re looking for. The api-key probably isn’t as important, but I wanted to include this to show that it can be captured. The JWT Token that was passed to the API Management is captured and while it can be copied out, then decoded with https://jwt.io/, it isn’t very useful if we’re trying to use KQL to generate reports. The remaining fields, which is probably what everyone is looking for, AppId, Oid, Name are extracted from the fields in the JWT Token. These fields are just examples that I included in the demonstration, and it is possible to extract any other field you like by adding to the inbound processing XML code.

Navigate to the API Management service, APIs blade, Azure OpenAI Service API, All Operations, then click on the </> policy code editor icon under Inbound processing:

The following is the XML code insert that you’ll need so that the fields listed above will be captured and sent to the Event Hub:

<!--

    IMPORTANT:

    - Policy elements can appear only within the <inbound>, <outbound>, <backend> section elements.

    - To apply a policy to the incoming request (before it is forwarded to the backend service), place a corresponding policy element within the <inbound> section element.

    - To apply a policy to the outgoing response (before it is sent back to the caller), place a corresponding policy element within the <outbound> section element.

    - To add a policy, place the cursor at the desired insertion point and select a policy from the sidebar.

    - To remove a policy, delete the corresponding policy statement from the policy document.

    - Position the <base> element within a section element to inherit all policies from the corresponding section element in the enclosing scope.

    - Remove the <base> element to prevent inheriting policies from the corresponding section element in the enclosing scope.

    - Policies are applied in the order of their appearance, from the top down.

    - Comments within policy elements are not supported and may disappear. Place your comments between policy elements or at a higher level scope.

-->

<policies>

<inbound>

<base />

<set-header name="api-key" exists-action="append">

<value>{{dev-openai}}</value>

</set-header>

<validate-jwt header-name="Authorization" failed-validation-httpcode="403" failed-validation-error-message="Forbidden" output-token-variable-name="jwt-token">

<openid-config url=https://login.microsoftonline.com/{{Tenant-ID}}/v2.0/.well-known/openid-configuration />

<issuers>

<issuer>https://sts.windows.net/{{Tenant-ID}}/</issuer>

</issuers>

<required-claims>

<claim name="roles" match="any">

<value>APIM.Access</value>

</claim>

</required-claims>

</validate-jwt>

<set-variable name="request" value="@(context.Request.Body.As<JObject>(preserveContent: true))" />

<set-variable name="api-key" value="@(context.Request.Headers.GetValueOrDefault("api-key",""))" />

<set-variable name="jwttoken" value="@(context.Request.Headers.GetValueOrDefault("Authorization",""))" />

<log-to-eventhub logger-id="event-hub-logger">@{

        var jwt = context.Request.Headers.GetValueOrDefault("Authorization","").AsJwt();

        var appId = jwt.Claims.GetValueOrDefault("appid", string.Empty);

        var oid = jwt.Claims.GetValueOrDefault("oid", string.Empty);

        var name = jwt.Claims.GetValueOrDefault("name", string.Empty);

         return new JObject(

             new JProperty("EventTime", DateTime.UtcNow.ToString()),

             new JProperty("ServiceName", context.Deployment.ServiceName),

             new JProperty("RequestId", context.RequestId),

             new JProperty("RequestIp", context.Request.IpAddress),

             new JProperty("OperationName", context.Operation.Name),

             new JProperty("api-key", context.Variables["api-key"]),

             new JProperty("request-body", context.Variables["request"]),

             new JProperty("JWTToken", context.Variables["jwttoken"]),

             new JProperty("AppId", appId),

             new JProperty("Oid", oid),

             new JProperty("Name", name)

         ).ToString();

     }</log-to-eventhub>

</inbound>

<backend>

<base />

</backend>

<outbound>

<base />

</outbound>

<on-error>

<base />

</on-error>

</policies>

The XML code can be found at my GitHub Repository: https://github.com/terenceluk/Azure/blob/main/API%20Management/XML/Capture-APIM-Traffic-and-JWT-Token-Information.xml

Proceed to click on the Save button and additional set-variable and log-to-eventhub policies should be displayed under Inbound processing:

With the API Management’s inbound processing rule updated, initiating API calls to the APIM to generate request traffic and let it capture the information. Once a few requests have been made, navigate to the Event Hub then Process data:

Within the Process data blade, click on the Start button for Enable real time insights from events:

Click on the Test Query button to load the captured logs:

The logs typically take a minute or 2 to show up so if no logs are displayed then try executing the Test query again after a few minutes:

We can see that it is possible for us to edit the inbound processing policy to recreate the type of log entries the API Management out-of-the-box diagnostic settings but if that is not desired, it is possible to map the logs in the Event Hub to the logs in the diagnostic settings with the use of the RequestId from the Event Hub logs and the CorrelationId of the APIM diagnostic settings logs as shown in the screenshots below:

RequestID from Event Hub

CorrelationId from API Management Diagnostic Settings Logs

Note that there are different views available in the Event Hub logs. Below is a Raw view displayed as JSON:

As mentioned earlier, the JWT token passed for authorization is captured and it is possible to decode the value to view the full payload. If any additional fields are desired then the inbound processing policy can be modified to capture this information:

Now that we have the JWT token information captured, we can send the Azure Event Hubs logs into a Log Analytics Workspace and join the 2 tables together with KQL. I will be providing a walkthrough for how to accomplish this as outlined in this document:

Tutorial: Ingest events from Azure Event Hubs into Azure Monitor Logs (Public Preview)
https://learn.microsoft.com/en-us/azure/azure-monitor/logs/ingest-logs-event-hub

… in the part 2 of my future post.

I hope this helps anyone out there looking for a way to capture the identity of the user using the Azure OpenAI Service.