When obtaining the signature, the path must always use Path-Style.

[ChangxingJiang]

Apache Iceberg version

0.10.0 (latest release)

Please describe the bug 🐞

Problem

When using PyIceberg to connect to the Iceberg REST Catalog maintained by LakeKeeper, a persistent 403 error occurs, with the following error message:

Traceback (most recent call last):
  File "/home/*/miniconda3/envs/test/lib/python3.10/site-packages/s3fs/core.py", line 114, in _error_wrapper
    return await func(*args, **kwargs)
  File "/home/*/miniconda3/envs/test/lib/python3.10/site-packages/aiobotocore/context.py", line 36, in wrapper
    return await func(*args, **kwargs)
  File "/home/*/miniconda3/envs/test/lib/python3.10/site-packages/aiobotocore/client.py", line 424, in _make_api_call
    raise error_class(parsed_resonse, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

......

When requesting an S3-compatible object storage, the Virtual-Hosted-Style URL format is used, whereas the logs from PyIceberg show paths in Path-Style format. This inconsistency between the two path styles results in mismatched V4 signatures, which in turn causes V4 signature verification to fail with a 403 Forbidden error.

In LakeKeeper, signature generation defaults to the "Auto" mode, which determines the signature style based on the provided URL.

Expect

If I enable Virtual-Hosted-Style, then the signature should also be generated using the Virtual-Hosted-Style format.

Reproduce

• Use LakeKeeper as REST catalog, create warehouse with:
  • Disable path style access
  • Remote signing URL style: Auto
• Use KeyCloak to OAuth2

catalog = load_catalog(
    type="rest",
    uri="http://localhost:8181/catalog",
    warehouse="iceberg",
    credential=f"{CLIENT_ID}:{CLIENT_SECRET}",
    scope="lakekeeper",
    **{
        "oauth2-server-uri": "http://172.20.*.*:*/realms/master/protocol/openid-connect/token",
        "s3.force-virtual-addressing": "true",
        "s3.path-style-access": "false"
    }
)

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[ChangxingJiang]

I Find it is fix in the master branch now.

[kevinjqliu]

Thanks for reporting this.

s3.force-virtual-addressing for fsspec was added after the 0.10 release in #2517

[ChangxingJiang]

Thanks.