azdls.sas-token.<account> support for vended credentials #27
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I'm adding support for adls.sas-token.xxx properties. As noted in this ticket: apache#1442, this is needed for vended credentials for Azure.
Also, the azure storage is a bit confusing. There's combination of [abfss/wasbs] x [.blob/.dfs] x [blob storage with / without hierarchical namespace] schemes. First is intended to be a driver, second to be an endpoint, while third is what is actually present on server side. However, because of its unclear semantics, nobody knows what these mean, and different systems implement different things.
For example, snowflake doesn't support writing to
.blobendpoints for externally managed iceberg tables. So when testing I create it withwasbs://...blob.... However, snowflake then keeps creating paths withabfss://...blob. Which is something that iceberg-rust doesn't allow, i.e. it matches driver to the endpoint. But that seems to be completely unnecessary restriction, so in this PR I'm removing it.Further, iceberg-rust delegates to OpenDAL. And before it does that, it replaces
.blobwith.dfs, in an https:// request that it forms. Because iceberg-rust usesAzdlsBackendOpenDAL backend, and neverAzblobBackend.So to conclude what my understanding is:
What would be proper fix though is that based on endpoint, we choose the proper OpenDAL backend. Then if it did, we could support Azurite for tests. However, I still think wasbs/abfss are just artifical limitations that don't matter much. But so is
scheme(I think we should auto-recognize scheme based on endpoint), so it may very well be that in the current architecture of iceberg-rust we split to different OpenDAL backends based on scheme, and only allow pairing abfss with .dfs endpoints. However, this would break for paths that snowflake produces in open catalog, so I don't think this is a good direction.