Skip to content

Iceberg: use sts for Iceberg credentials source with Glue + Cloud #1714

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kbatuigas merged 5 commits into from
May 28, 2026
Merged

Iceberg: use sts for Iceberg credentials source with Glue + Cloud #1714

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
a70ead6
DOC-2209: Recommend sts for Iceberg AWS Glue on Cloud
kbatuigas May 26, 2026
a168451
Merge branch 'main' into DOC-2209-update-iceberg-aws-glue-cloud-doc-t…
kbatuigas May 26, 2026
39226ff
Apply clarification per SME review
kbatuigas May 28, 2026
03619bb
Fix list item rendering
kbatuigas May 28, 2026
67b4496
Fix object storage property xrefs
kbatuigas May 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 9 additions & 7 deletions modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,10 +120,10 @@ You must configure credentials for the AWS Glue Data Catalog integration in eith

* Allow Redpanda to use the same `cloud_storage_*` credential properties configured for S3. This is the recommended approach.
* If you want to configure authentication to AWS Glue separately from authentication to S3, there are equivalent credential configuration properties named `iceberg_rest_catalog_aws_*` that override the object storage credentials. These properties only apply to REST catalog authentication, and never to S3 authentication:
** config_ref:iceberg_rest_catalog_credentials_source,true,properties/cluster-properties[`iceberg_rest_catalog_credentials_source`] overrides config_ref:cloud_storage_credentials_source,true,properties/cluster-properties[`cloud_storage_credentials_source`]
** config_ref:iceberg_rest_catalog_aws_access_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_access_key`] overrides config_ref:cloud_storage_access_key,true,properties/cluster-properties[`cloud_storage_access_key`]
** config_ref:iceberg_rest_catalog_aws_secret_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_secret_key`] overrides config_ref:cloud_storage_secret_key,true,properties/cluster-properties[`cloud_storage_secret_key`]
** config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] overrides config_ref:cloud_storage_region,true,properties/cluster-properties[`cloud_storage_region`]
** config_ref:iceberg_rest_catalog_credentials_source,true,properties/cluster-properties[`iceberg_rest_catalog_credentials_source`] overrides config_ref:cloud_storage_credentials_source,true,properties/object-storage-properties[`cloud_storage_credentials_source`]
** config_ref:iceberg_rest_catalog_aws_access_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_access_key`] overrides config_ref:cloud_storage_access_key,true,properties/object-storage-properties[`cloud_storage_access_key`]
** config_ref:iceberg_rest_catalog_aws_secret_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_secret_key`] overrides config_ref:cloud_storage_secret_key,true,properties/object-storage-properties[`cloud_storage_secret_key`]
** config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] overrides config_ref:cloud_storage_region,true,properties/object-storage-properties[`cloud_storage_region`]
endif::[]

ifdef::env-cloud[]
Expand All @@ -135,7 +135,9 @@ For an example cluster configuration that uses the same IAM credentials for both
* If you want to configure authentication to AWS Glue separately from authentication to S3, there are equivalent credential configuration properties named `iceberg_rest_catalog_aws_*` that override the object storage credentials. These properties only apply to REST catalog authentication, and never to S3 authentication:
+
--
** config_ref:iceberg_rest_catalog_credentials_source,true,properties/cluster-properties[`iceberg_rest_catalog_credentials_source`]. To use the cluster's IAM role, set the property to `aws_instance_metadata`. To use static credentials, set to `config_file`.
** config_ref:iceberg_rest_catalog_credentials_source,true,properties/cluster-properties[`iceberg_rest_catalog_credentials_source`]
*** Set the property to `sts` if you want to use the cluster's default IAM role.
*** Set to `config_file` if you want to scope Glue access through your own IAM user and policy instead of the cluster's default IAM role, or if you want to use static credentials.
** config_ref:iceberg_rest_catalog_aws_access_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_access_key`] (static credentials only)
** config_ref:iceberg_rest_catalog_aws_secret_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_secret_key`] (static credentials only), added as a secret value (see the <<update-cluster-configuration,next section>> for details)
** config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`]
Expand Down Expand Up @@ -185,7 +187,7 @@ Use your own values for the following placeholders:
+
--
* `<custom-namespace>`: A unique namespace for this cluster's Iceberg tables. Each Redpanda cluster that writes to the same Glue catalog must use a distinct namespace to avoid table name collisions. If omitted, the default namespace `redpanda` is used.
* `<glue-region>`: The AWS region where your Data Catalog is located. The region in the AWS Glue endpoint must match the region specified in either your config_ref:cloud_storage_region,true,properties/cluster-properties[`cloud_storage_region`] or config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] property.
* `<glue-region>`: The AWS region where your Data Catalog is located. The region in the AWS Glue endpoint must match the region specified in either your config_ref:cloud_storage_region,true,properties/object-storage-properties[`cloud_storage_region`] or config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] property.
* `<bucket-name>` and `<warehouse-path>`: AWS Glue requires you to specify the base location where Redpanda stores Iceberg data and metadata files. You must use an S3 URI; for example, `s3://<bucket-name>/iceberg`. This must be the same bucket used for object storage (your `cloud_storage_bucket`). You cannot specify a different bucket for Iceberg data.
+
`<warehouse-path>` is a name you choose (such as `iceberg`) as the logical name for the warehouse represented by all Redpanda Iceberg topic data in the cluster.
Expand Down Expand Up @@ -217,7 +219,7 @@ rpk cluster config set \
iceberg_catalog_type=rest \
iceberg_rest_catalog_endpoint=https://glue.<glue-region>.amazonaws.com/iceberg \
iceberg_rest_catalog_authentication_mode=aws_sigv4 \
iceberg_rest_catalog_credentials_source=aws_instance_metadata \
iceberg_rest_catalog_credentials_source=sts \
iceberg_rest_catalog_aws_region=<glue-region> \
iceberg_rest_catalog_base_location=s3://<cluster-storage-bucket-name>/<warehouse-path>
----
Expand Down
Loading