additionally_allowed_tenants config not found in Spark session for Multi-tenant app

[soumikdas-oa]

We have a multi-tenant Azure app (Tenant A, Tenant B). We are using Tenant A's Service Principle credentials to access the ADLS files on Tenant B. We are using pySpark for this job.

Using the above multi-tenant app credential, we are getting 401 unauthorized error in spark session.

But if we are using Python library (azure.identity), then using the following 'additionally_allowed_tenants' options enables us to authorize the access Tenant B's ADLS containers using Tenant A's credentials.

default_credential = DefaultAzureCredential(additionally_allowed_tenants=['*'])

We want to achieve the same with pySpark, but we can't see any option or configuration available to do it through Spark session.

We are using:
- Apache Spark 3.5.0
- Databricks Notebook (Runtime Version 14.3 LTS)
- Azure Service Principle

Please guide us to resolve this issue.