Attempting to create a folder on an Azure Data Lake Storage Account with Private Endpoint fails with: "Failed to add directory 'Test'. Error: AuthorizationFailure: This request is not authorized to perform this operation."

Problem

A colleague of mine recently asked me to help troubleshoot an issue with an Azure Storage Account that has Hierarchical Namespace enabled, which is essentially an Azure Data Lake, where any attempts to create a folder would fail:

The error message presented was generic and appears to suggest that it is caused by a permissions issue:

Failed to add directory

Failed to add directory 'Test'. Error: AuthorizationFailure: This request is not authorized to perform this operation. RequestId:da720a90-c01f-0053-5d3f-c61ef5000000 Time:2023-08-03T19:22:01.2257950Z

Creating containers or uploading blobs (files) to the storage account did not have any issues as those operations were successful as shown in the following screenshot:

This error has been one that I’ve come across frequently in the past and it is usually because the storage account is locked down with only a private endpoint for the blob service and not for the data lake service created. The following Microsoft documentation explains the reason:

Use private endpoints for Azure Storage

https://learn.microsoft.com/en-us/azure/storage/common/storage-private-endpoints#creating-a-private-endpoint

If you create a private endpoint for the Data Lake Storage Gen2 storage resource, then you should also create one for the Blob Storage resource. That's because operations that target the Data Lake Storage Gen2 endpoint might be redirected to the Blob endpoint. Similarly, if you add a private endpoint for Blob Storage only, and not for Data Lake Storage Gen2, some operations (such as Manage ACL, Create Directory, Delete Directory, etc.) will fail since the Gen2 APIs require a DFS private endpoint. By creating a private endpoint for both resources, you ensure that all operations can complete successfully.

The following are screenshots confirming the missing configuration.

Note that Hierarchical Namespace is enabled:

Note that Public network access is set to Disabled:

Note that there is only 1 private endpoint configured for the storage account:

… and the Target sub-resource of the private endpoint is blob:

Solution

To correct the issue, we’ll need to create an additional private endpoint that has the Target sub-resource configured as DFS (Data Lake Storage Gen2). Begin by navigating to the Networking blade for the storage account and create a new Private Endpoint:

Proceed to fill in the details for the private endpoint:

Select dfs as the Target sub-resource:

Complete the creation of the private endpoint:

Folder creation should now succeed:

Hope this provides anyone who might have ran into this issue and is looking for a solution. I’ve found that searching for the error message does not always return results to this solution.