From 5dd4d4840ce6039d4cec1d8ae1b9aff807943232 Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Wed, 25 Mar 2026 15:53:53 -0700 Subject: [PATCH 01/22] Updates to IAM permissions for Glue in Cloud --- .../iceberg/iceberg-topics-aws-glue.adoc | 61 ++++++++++++++----- .../properties/cluster-properties.adoc | 2 + .../properties/object-storage-properties.adoc | 2 + 3 files changed, 50 insertions(+), 15 deletions(-) diff --git a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc index a530eeec06..f91ae5689a 100644 --- a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc +++ b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc @@ -69,6 +69,13 @@ When `iceberg_delete` or the topic override `redpanda.iceberg.delete` is set to == Authorize access to AWS Glue +ifdef::env-cloud[] +For BYOC clusters created in March 2026 or later, the required AWS Glue IAM policy is automatically provisioned and attached to the Redpanda broker role when Iceberg is enabled. You don't need to manually create IAM policies or roles for Glue access. + +For clusters created before March 2026, you must run `rpk byoc apply` to provision the Glue IAM policy before enabling Iceberg. This is a one-time operation that updates the broker role with the necessary Glue permissions. +endif::[] + +ifndef::env-cloud[] You must allow Redpanda access to AWS Glue services in your AWS account. You can use the same access credentials that you configured for S3 (IAM role, access keys, and KMS key), as long as you have also added read and write access to AWS Glue Data Catalog. For example, you could create a separate IAM policy that manages access to AWS Glue and attach it to the IAM role that Redpanda also uses to access S3. Add all AWS Glue API actions in the policy (`"glue:*"`) on the following resources: @@ -100,6 +107,7 @@ Your IAM policy should include a statement similar to the following: ---- For more information on configuring IAM permissions, see the https://docs.aws.amazon.com/glue/latest/dg/configure-iam-for-glue.html[AWS Glue documentation^]. +endif::[] == Configure authentication and credentials @@ -115,12 +123,14 @@ You must configure credentials for the AWS Glue Data Catalog integration in eith endif::[] ifdef::env-cloud[] -You must configure credentials for the AWS Glue Data Catalog integration using the following properties: +You can configure credentials for the AWS Glue Data Catalog integration in either of the following ways: -* config_ref:iceberg_rest_catalog_credentials_source,true,properties/cluster-properties[`iceberg_rest_catalog_credentials_source`] set to `config_file` -* config_ref:iceberg_rest_catalog_aws_access_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_access_key`] -* config_ref:iceberg_rest_catalog_aws_secret_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_secret_key`], added as a secret value (see the <> for details) -* config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] +* Allow Redpanda to use the same `cloud_storage_*` credential properties already configured for S3. If you do not configure the overrides listed below, Redpanda uses the same credentials for both S3 and AWS Glue. This is the recommended approach, especially in BYOC deployments where the broker's IAM role already includes the necessary Glue permissions. +* If you want to configure authentication to AWS Glue separately from authentication to S3, there are equivalent credential configuration properties named `iceberg_rest_catalog_aws_*` that override the object storage credentials. These properties only apply to REST catalog authentication, and never to S3 authentication: +** config_ref:iceberg_rest_catalog_credentials_source,true,properties/cluster-properties[`iceberg_rest_catalog_credentials_source`] overrides config_ref:cloud_storage_credentials_source,true,properties/cluster-properties[`cloud_storage_credentials_source`]. To use the broker's IAM role, set the property to `aws_instance_metadata`. To use static credentials, set to `config_file`. +** config_ref:iceberg_rest_catalog_aws_access_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_access_key`] (static credentials only) +** config_ref:iceberg_rest_catalog_aws_secret_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_secret_key`] (static credentials only), added as a secret value (see the <> for details) +** config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] endif::[] == Update cluster configuration @@ -167,37 +177,58 @@ Use your own values for the following placeholders: -- endif::[] ifdef::env-cloud[] -Use `rpk` like in the following example, or use the Cloud API to xref:manage:cluster-maintenance/config-cluster.adoc#set-cluster-configuration-properties[update these cluster properties]. The update might take several minutes to complete. +Use `rpk` like in the following examples, or use the Cloud API to xref:manage:cluster-maintenance/config-cluster.adoc#set-cluster-configuration-properties[update these cluster properties]. The update might take several minutes to complete. + [,bash] ---- rpk cloud login rpk profile create --from-cloud - +---- ++ +[tabs] +====== +IAM role (instance metadata):: ++ +-- +[,bash] +---- +rpk cluster config set \ + iceberg_enabled=true \ + iceberg_catalog_type=rest \ + iceberg_rest_catalog_endpoint=https://glue..amazonaws.com/iceberg \ + iceberg_rest_catalog_authentication_mode=aws_sigv4 \ + iceberg_rest_catalog_credentials_source=aws_instance_metadata \ + iceberg_rest_catalog_aws_region= \ + iceberg_rest_catalog_base_location=s3:/// +---- +-- +Static credentials:: ++ +-- +[,bash] +---- rpk cluster config set \ iceberg_enabled=true \ iceberg_catalog_type=rest \ iceberg_rest_catalog_endpoint=https://glue..amazonaws.com/iceberg \ iceberg_rest_catalog_authentication_mode=aws_sigv4 \ - # Because Tiered Storage does not support the use of distinct buckets for Iceberg, - # always place iceberg_rest_catalog_base_location in the same S3 bucket as cloud_storage_bucket - iceberg_rest_catalog_base_location=s3:/// \ - # Set credentials source to config_file if using - # iceberg_rest_catalog_aws_access_key and iceberg_rest_catalog_aws_secret_key iceberg_rest_catalog_credentials_source=config_file \ iceberg_rest_catalog_aws_region= \ iceberg_rest_catalog_aws_access_key= \ - iceberg_rest_catalog_aws_secret_key='${secrets.}' + iceberg_rest_catalog_aws_secret_key='${secrets.}' \ + iceberg_rest_catalog_base_location=s3:/// ---- +-- +====== + Use your own values for the following placeholders: + -- - ``: The AWS region where your Data Catalog is located. The region in the AWS Glue endpoint must match the region specified in your config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] property. - `` and ``: AWS Glue requires you to specify the base location where Redpanda stores Iceberg data and metadata files. You must use an S3 URI; for example, `s3:///iceberg`. As a security best practice, Redpanda Data recommends specifying a subfolder (using prefixes) rather than the root of the bucket. -- ``: The AWS access key ID for your Glue service account. -- ``: The name of the secret that stores the AWS secret access key for your Glue service account. To reference a secret in a cluster property, for example `iceberg_rest_catalog_aws_secret_key`, you must first xref:manage:iceberg/use-iceberg-catalogs.adoc#store-a-secret-for-rest-catalog-authentication[store the secret value]. +- `` (static credentials only): The AWS access key ID for your Glue service account. +- `` (static credentials only): The name of the secret that stores the AWS secret access key for your Glue service account. To reference a secret in a cluster property, for example `iceberg_rest_catalog_aws_secret_key`, you must first xref:manage:iceberg/use-iceberg-catalogs.adoc#store-a-secret-for-rest-catalog-authentication[store the secret value]. -- endif::[] + diff --git a/modules/reference/partials/properties/cluster-properties.adoc b/modules/reference/partials/properties/cluster-properties.adoc index a3ca43602a..f1fee64386 100644 --- a/modules/reference/partials/properties/cluster-properties.adoc +++ b/modules/reference/partials/properties/cluster-properties.adoc @@ -8015,6 +8015,7 @@ endif::[] // end::redpanda-cloud[] +// tag::redpanda-cloud[] === iceberg_rest_catalog_aws_credentials_source *Accepted values*: `aws_instance_metadata`, `azure_aks_oidc_federation`, `azure_vm_instance_metadata`, `config_file`, `gcp_instance_metadata`, `sts`. @@ -8075,6 +8076,7 @@ endif::[] |=== +// end::redpanda-cloud[] // tag::redpanda-cloud[] === iceberg_rest_catalog_aws_region diff --git a/modules/reference/partials/properties/object-storage-properties.adoc b/modules/reference/partials/properties/object-storage-properties.adoc index ed37c029f1..df623434e9 100644 --- a/modules/reference/partials/properties/object-storage-properties.adoc +++ b/modules/reference/partials/properties/object-storage-properties.adoc @@ -1519,6 +1519,7 @@ endif::[] |=== +// tag::redpanda-cloud[] === cloud_storage_credentials_source The source of credentials used to authenticate to object storage services. @@ -1574,6 +1575,7 @@ endif::[] |=== +// end::redpanda-cloud[] === cloud_storage_crl_file From 3eacad63d4ded3539b01ffc19775184aff4b8d1b Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Wed, 25 Mar 2026 16:00:40 -0700 Subject: [PATCH 02/22] Make SM and Cloud examples consistent for iceberg_delete --- .../pages/iceberg/iceberg-topics-aws-glue.adoc | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc index f91ae5689a..93fb1d9bc4 100644 --- a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc +++ b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc @@ -154,8 +154,8 @@ iceberg_delete: false iceberg_catalog_type: rest iceberg_rest_catalog_endpoint: https://glue..amazonaws.com/iceberg iceberg_rest_catalog_authentication_mode: aws_sigv4 - # Because Tiered Storage does not support the use of distinct buckets for Iceberg, - # always place iceberg_rest_catalog_base_location in the same S3 bucket as cloud_storage_bucket +# Because Tiered Storage does not support the use of distinct buckets for Iceberg, +# always place iceberg_rest_catalog_base_location in the same S3 bucket as cloud_storage_bucket iceberg_rest_catalog_base_location: s3:/// # Use the iceberg_rest_catalog_aws_* properties if you want to # use separate AWS credentials for the catalog, or omit these lines to reuse S3 @@ -189,12 +189,14 @@ rpk profile create --from-cloud [tabs] ====== IAM role (instance metadata):: -+ + -- [,bash] ---- rpk cluster config set \ iceberg_enabled=true \ + # Glue requires Redpanda Iceberg tables to be manually deleted + iceberg_delete=false \ iceberg_catalog_type=rest \ iceberg_rest_catalog_endpoint=https://glue..amazonaws.com/iceberg \ iceberg_rest_catalog_authentication_mode=aws_sigv4 \ @@ -204,12 +206,14 @@ rpk cluster config set \ ---- -- Static credentials:: -+ + -- [,bash] ---- rpk cluster config set \ iceberg_enabled=true \ + # Glue requires Redpanda Iceberg tables to be manually deleted + iceberg_delete=false \ iceberg_catalog_type=rest \ iceberg_rest_catalog_endpoint=https://glue..amazonaws.com/iceberg \ iceberg_rest_catalog_authentication_mode=aws_sigv4 \ From e60116073b748f039dcd7f6c776d550d8ef24f27 Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Wed, 25 Mar 2026 16:35:12 -0700 Subject: [PATCH 03/22] Fix tabs --- .../pages/iceberg/iceberg-topics-aws-glue.adoc | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc index 93fb1d9bc4..290212d081 100644 --- a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc +++ b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc @@ -179,20 +179,17 @@ endif::[] ifdef::env-cloud[] Use `rpk` like in the following examples, or use the Cloud API to xref:manage:cluster-maintenance/config-cluster.adoc#set-cluster-configuration-properties[update these cluster properties]. The update might take several minutes to complete. + -[,bash] ----- -rpk cloud login - -rpk profile create --from-cloud ----- -+ [tabs] ====== IAM role (instance metadata):: - ++ -- [,bash] ---- +rpk cloud login + +rpk profile create --from-cloud + rpk cluster config set \ iceberg_enabled=true \ # Glue requires Redpanda Iceberg tables to be manually deleted @@ -205,8 +202,9 @@ rpk cluster config set \ iceberg_rest_catalog_base_location=s3:/// ---- -- -Static credentials:: +Static credentials:: ++ -- [,bash] ---- From 4db2aaf95409d8687c3a9739c14292758acb0a99 Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Wed, 25 Mar 2026 18:32:18 -0700 Subject: [PATCH 04/22] Apply suggestion from auto review --- modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc index 290212d081..3b3c73aeac 100644 --- a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc +++ b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc @@ -186,13 +186,14 @@ IAM role (instance metadata):: -- [,bash] ---- +# Glue requires Redpanda Iceberg tables to be manually deleted +# so iceberg_delete is set to false. rpk cloud login rpk profile create --from-cloud rpk cluster config set \ iceberg_enabled=true \ - # Glue requires Redpanda Iceberg tables to be manually deleted iceberg_delete=false \ iceberg_catalog_type=rest \ iceberg_rest_catalog_endpoint=https://glue..amazonaws.com/iceberg \ @@ -208,9 +209,10 @@ Static credentials:: -- [,bash] ---- +# Glue requires Redpanda Iceberg tables to be manually deleted +# so iceberg_delete is set to false. rpk cluster config set \ iceberg_enabled=true \ - # Glue requires Redpanda Iceberg tables to be manually deleted iceberg_delete=false \ iceberg_catalog_type=rest \ iceberg_rest_catalog_endpoint=https://glue..amazonaws.com/iceberg \ From 477ce7b92090985a943f9a9afea0e9c213fc9391 Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Wed, 25 Mar 2026 18:57:15 -0700 Subject: [PATCH 05/22] Explain S3 bucket names --- modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc index 3b3c73aeac..b047cf79ce 100644 --- a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc +++ b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc @@ -230,7 +230,9 @@ Use your own values for the following placeholders: + -- - ``: The AWS region where your Data Catalog is located. The region in the AWS Glue endpoint must match the region specified in your config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] property. -- `` and ``: AWS Glue requires you to specify the base location where Redpanda stores Iceberg data and metadata files. You must use an S3 URI; for example, `s3:///iceberg`. As a security best practice, Redpanda Data recommends specifying a subfolder (using prefixes) rather than the root of the bucket. +- `` and ``: AWS Glue requires you to specify the base location where Redpanda stores Iceberg data and metadata files. You must use an S3 URI; for example, `s3:///iceberg`. For BYOC clusters, the bucket name is `redpanda-cloud-storage-`. For BYOVPC clusters, use the name of the bucket you created as a customer-managed resource. ++ +As a security best practice, Redpanda Data recommends specifying a subfolder (using prefixes) rather than the root of the bucket. - `` (static credentials only): The AWS access key ID for your Glue service account. - `` (static credentials only): The name of the secret that stores the AWS secret access key for your Glue service account. To reference a secret in a cluster property, for example `iceberg_rest_catalog_aws_secret_key`, you must first xref:manage:iceberg/use-iceberg-catalogs.adoc#store-a-secret-for-rest-catalog-authentication[store the secret value]. -- From f73bc50d48a7d6504d78128d357244cb25de4b1a Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Tue, 31 Mar 2026 18:15:58 -0700 Subject: [PATCH 06/22] Use partial to single source bucket name information --- .../manage/partials/iceberg-access-table.adoc | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 modules/manage/partials/iceberg-access-table.adoc diff --git a/modules/manage/partials/iceberg-access-table.adoc b/modules/manage/partials/iceberg-access-table.adoc new file mode 100644 index 0000000000..dc0f29707d --- /dev/null +++ b/modules/manage/partials/iceberg-access-table.adoc @@ -0,0 +1,20 @@ +For BYOC clusters, the bucket name and table location are as follows: + +|=== +| Cloud provider | Bucket or container name | Iceberg table location + +| AWS +| `redpanda-cloud-storage-` +.3+a| `redpanda-iceberg-catalog/redpanda/` + +| Azure +a| `` + +The Redpanda cluster ID is also used as the container name (ID) and the storage account ID. + +| GCP +| `redpanda-cloud-storage-` + +|=== + +For BYOVPC clusters, the bucket name is the name you chose when you created the Tiered Storage bucket as a customer-managed resource. From 4d46cb847a4f4c5b785c791062be3205c9d97385 Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Tue, 31 Mar 2026 18:22:56 -0700 Subject: [PATCH 07/22] Catalog responsbility --- modules/manage/pages/iceberg/use-iceberg-catalogs.adoc | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/modules/manage/pages/iceberg/use-iceberg-catalogs.adoc b/modules/manage/pages/iceberg/use-iceberg-catalogs.adoc index 4aa3f32a27..5268eda49b 100644 --- a/modules/manage/pages/iceberg/use-iceberg-catalogs.adoc +++ b/modules/manage/pages/iceberg/use-iceberg-catalogs.adoc @@ -14,7 +14,7 @@ include::shared:partial$enterprise-license.adoc[] ==== endif::[] -To read from the Redpanda-generated xref:{about-iceberg-doc}[Iceberg table], your Iceberg-compatible client or tool needs access to the catalog to retrieve the table metadata and know the current state of the table. The catalog provides the current table metadata, which includes locations for all the table's data files. You can configure Redpanda to either connect to a REST-based catalog, or use a filesystem-based catalog. +To read from the Redpanda-generated xref:{about-iceberg-doc}[Iceberg table], your Iceberg-compatible client or tool needs access to the catalog to retrieve the table metadata and know the current state of the table. The catalog provides the current table metadata, which includes locations for all the table's data files. You can configure Redpanda to either connect to a REST-based catalog, or use a filesystem-based catalog. For production deployments, Redpanda recommends <> to manage Iceberg metadata. This enables built-in table maintenance, safely handles multiple engines and tools accessing tables at the same time, facilitates data governance, and maximizes data discovery. However, if it is not possible to use a REST catalog, you can <> (`object_storage` catalog type), which does not require you to maintain a separate service to access the Iceberg data. @@ -25,6 +25,8 @@ After you have selected a catalog type at the cluster level and xref:{about-iceb [[rest]] == Connect to a REST catalog +NOTE: Redpanda connects to an Iceberg catalog that you provision and manage. Redpanda does not create or manage the catalog service, its databases, or any associated network configuration. + Connect to an Iceberg REST catalog using the standard https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml[REST API^] supported by many catalog providers. Use this catalog integration type with REST-enabled Iceberg catalog services, such as https://docs.databricks.com/en/data-governance/unity-catalog/index.html[Databricks Unity^] and https://other-docs.snowflake.com/en/opencatalog/overview[Snowflake Open Catalog^]. [TIP] From fb4fefd51cb6e59d69455cbc37c8dcb428a37234 Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Tue, 31 Mar 2026 18:26:31 -0700 Subject: [PATCH 08/22] BYOVPC prereq and bucket name clarification --- .../pages/iceberg/about-iceberg-topics.adoc | 23 +++---------------- .../iceberg-topics-databricks-unity.adoc | 2 +- 2 files changed, 4 insertions(+), 21 deletions(-) diff --git a/modules/manage/pages/iceberg/about-iceberg-topics.adoc b/modules/manage/pages/iceberg/about-iceberg-topics.adoc index 4f74450f63..ad771ef397 100644 --- a/modules/manage/pages/iceberg/about-iceberg-topics.adoc +++ b/modules/manage/pages/iceberg/about-iceberg-topics.adoc @@ -45,7 +45,7 @@ When you enable the Iceberg integration for a Redpanda topic, Redpanda brokers s To enable Iceberg for Redpanda topics, you must have the following: ifdef::env-cloud[] -* A running xref:get-started:cluster-types/byoc/index.adoc[BYOC] cluster on Redpanda version 25.1 or later. The Iceberg integration is supported only for BYOC, and the cluster properties to configure Iceberg are available with v25.1. +* A running xref:get-started:cluster-types/byoc/index.adoc[BYOC] or BYOVPC cluster on Redpanda version 25.1 or later. The Iceberg integration is supported only for BYOC and BYOVPC, and the cluster properties to configure Iceberg are available with v25.1. * rpk: See xref:get-started:rpk-install.adoc[]. * Familiarity with the Redpanda Cloud API. You must link:/api/doc/cloud-controlplane/authentication[authenticate] to the Cloud API and use the Control Plane API to update your cluster configuration. endif::[] @@ -189,26 +189,9 @@ ifdef::env-cloud[] === Access Iceberg data -To query the Iceberg table, you need access to the object storage bucket or container where the Iceberg data is stored. For BYOC clusters, the bucket name and table location are as follows: +To query the Iceberg table, you need access to the object storage bucket or container where the Iceberg data is stored. -|=== -| Cloud provider | Bucket or container name | Iceberg table location - -| AWS -| `redpanda-cloud-storage-` -.3+a| `redpanda-iceberg-catalog/redpanda/` - -| Azure -a| `` - -The Redpanda cluster ID is also used as the container name (ID) and the storage account ID. - - -| GCP -| `redpanda-cloud-storage-` - - -|=== +include::manage:partial$iceberg-access-table.adoc[] For Azure clusters, you must add the public IP addresses or ranges from the REST catalog service, or other clients requiring access to the Iceberg data, to your cluster's allow list. Alternatively, add subnet IDs to the allow list if the requests originate from the same Azure region. diff --git a/modules/manage/pages/iceberg/iceberg-topics-databricks-unity.adoc b/modules/manage/pages/iceberg/iceberg-topics-databricks-unity.adoc index 3fcef32425..8d4fa4267b 100644 --- a/modules/manage/pages/iceberg/iceberg-topics-databricks-unity.adoc +++ b/modules/manage/pages/iceberg/iceberg-topics-databricks-unity.adoc @@ -73,7 +73,7 @@ Follow the steps in the https://docs.databricks.com/aws/en/connect/unity-catalog The external location stores the Unity Catalog-managed Iceberg metadata, and the Iceberg data written by Redpanda. You must use the same bucket configured for glossterm:Tiered Storage[] for your Redpanda cluster. ifdef::env-cloud[] -For BYOC clusters, the bucket name is `redpanda-cloud-storage-`, where `` is the ID of your Redpanda cluster. +For BYOC clusters, the bucket name is `redpanda-cloud-storage-`, where `` is the ID of your Redpanda cluster. For BYOVPC clusters, the bucket name is the name you chose when you created the Tiered Storage bucket as a customer-managed resource. endif::[] Follow the steps in the https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/external-locations[Databricks documentation] to *manually* create an external location. You can create the external location in the Catalog Explorer or with SQL. You must create the external location manually because the location needs to be associated with the existing Tiered Storage bucket URL, `s3://`. From cfa83dcfa1a8b5166c0f8fce0d7d87c9e55fbf1b Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Tue, 31 Mar 2026 19:27:07 -0700 Subject: [PATCH 09/22] Clarify base location requirements --- .../iceberg/iceberg-topics-aws-glue.adoc | 32 +++++++++++++------ 1 file changed, 23 insertions(+), 9 deletions(-) diff --git a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc index b047cf79ce..c52aaa4162 100644 --- a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc +++ b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc @@ -67,6 +67,12 @@ endif::[] When `iceberg_delete` or the topic override `redpanda.iceberg.delete` is set to `false`, you can delete the Redpanda topic, and then delete the table in AWS Glue and the Iceberg data and metadata files in the S3 bucket. If you plan to re-create the topic after deleting it, you must delete the table data entirely before re-creating the topic. +ifdef::env-cloud[] +=== Same AWS account required + +For BYOC and BYOVPC deployments, the AWS Glue Data Catalog must be in the same AWS account as the cluster. Cross-account Glue access has not been tested or certified. If you need to use a Glue catalog in a different AWS account, contact https://support.redpanda.com[Redpanda support^]. +endif::[] + == Authorize access to AWS Glue ifdef::env-cloud[] @@ -172,8 +178,12 @@ iceberg_rest_catalog_base_location: s3:/// Use your own values for the following placeholders: + -- -- ``: The AWS region where your Data Catalog is located. The region in the AWS Glue endpoint must match the region specified in either your config_ref:cloud_storage_region,true,properties/cluster-properties[`cloud_storage_region`] or config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] property. -- `` and ``: AWS Glue requires you to specify the base location where Redpanda stores Iceberg data and metadata files. You must use an S3 URI; for example, `s3:///iceberg`. As a security best practice, Redpanda Data recommends specifying a subfolder (using prefixes) rather than the root of the bucket. +* ``: The AWS region where your Data Catalog is located. The region in the AWS Glue endpoint must match the region specified in either your config_ref:cloud_storage_region,true,properties/cluster-properties[`cloud_storage_region`] or config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] property. +* `` and ``: AWS Glue requires you to specify the base location where Redpanda stores Iceberg data and metadata files. You must use an S3 URI; for example, `s3:///iceberg`. This must be the same bucket used for Tiered Storage (your `cloud_storage_bucket`). You cannot specify a different bucket for Iceberg data. ++ +`` is a name you choose (such as `iceberg`) as the logical name for the warehouse represented by all Redpanda Iceberg topic data in the cluster. ++ +As a security best practice, do not use the bucket root for the base location. Always specify a subfolder to avoid interfering with Tiered Storage data. -- endif::[] ifdef::env-cloud[] @@ -200,7 +210,7 @@ rpk cluster config set \ iceberg_rest_catalog_authentication_mode=aws_sigv4 \ iceberg_rest_catalog_credentials_source=aws_instance_metadata \ iceberg_rest_catalog_aws_region= \ - iceberg_rest_catalog_base_location=s3:/// + iceberg_rest_catalog_base_location=s3:/// ---- -- @@ -221,7 +231,7 @@ rpk cluster config set \ iceberg_rest_catalog_aws_region= \ iceberg_rest_catalog_aws_access_key= \ iceberg_rest_catalog_aws_secret_key='${secrets.}' \ - iceberg_rest_catalog_base_location=s3:/// + iceberg_rest_catalog_base_location=s3:/// ---- -- ====== @@ -229,12 +239,16 @@ rpk cluster config set \ Use your own values for the following placeholders: + -- -- ``: The AWS region where your Data Catalog is located. The region in the AWS Glue endpoint must match the region specified in your config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] property. -- `` and ``: AWS Glue requires you to specify the base location where Redpanda stores Iceberg data and metadata files. You must use an S3 URI; for example, `s3:///iceberg`. For BYOC clusters, the bucket name is `redpanda-cloud-storage-`. For BYOVPC clusters, use the name of the bucket you created as a customer-managed resource. +* ``: The AWS region where your Data Catalog is located. The region in the AWS Glue endpoint must match the region specified in your config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] property. +* `` and ``: AWS Glue requires you to specify the base location where Redpanda stores Iceberg data and metadata files. You must use an S3 URI; for example, `s3:///iceberg`. +** Bucket name: For BYOC clusters, the bucket name is `redpanda-cloud-storage-`. For BYOVPC clusters, use the name of the bucket you created as a customer-managed resource. ++ +This must be the same bucket used for Tiered Storage. You cannot specify a different bucket for Iceberg data. +** Warehouse: This is a name you choose as the logical name (such as `iceberg`) for the warehouse represented by all Redpanda Iceberg topic data in the cluster. + -As a security best practice, Redpanda Data recommends specifying a subfolder (using prefixes) rather than the root of the bucket. -- `` (static credentials only): The AWS access key ID for your Glue service account. -- `` (static credentials only): The name of the secret that stores the AWS secret access key for your Glue service account. To reference a secret in a cluster property, for example `iceberg_rest_catalog_aws_secret_key`, you must first xref:manage:iceberg/use-iceberg-catalogs.adoc#store-a-secret-for-rest-catalog-authentication[store the secret value]. +As a security best practice, do not use the bucket root for the base location. Always specify a subfolder to avoid interfering with Tiered Storage data. +* `` (static credentials only): The AWS access key ID for your Glue service account. +* `` (static credentials only): The name of the secret that stores the AWS secret access key for your Glue service account. To reference a secret in a cluster property, for example `iceberg_rest_catalog_aws_secret_key`, you must first xref:manage:iceberg/use-iceberg-catalogs.adoc#store-a-secret-for-rest-catalog-authentication[store the secret value]. -- endif::[] + From d719306e75393136594d275a3bb2a4d2807a8cc6 Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Tue, 31 Mar 2026 19:36:52 -0700 Subject: [PATCH 10/22] Move grant access addition to new PR --- .../pages/iceberg/query-iceberg-topics.adoc | 19 +------------------ 1 file changed, 1 insertion(+), 18 deletions(-) diff --git a/modules/manage/pages/iceberg/query-iceberg-topics.adoc b/modules/manage/pages/iceberg/query-iceberg-topics.adoc index 939674dff5..7ae2356968 100644 --- a/modules/manage/pages/iceberg/query-iceberg-topics.adoc +++ b/modules/manage/pages/iceberg/query-iceberg-topics.adoc @@ -27,24 +27,7 @@ endif::[] ifdef::env-cloud[] Redpanda generates an Iceberg table with the same name as the topic. Depending on the processing engine and your Iceberg catalog implementation, you may also need to define the table (for example using `CREATE TABLE`) to point the data lakehouse to its location in the catalog. -For BYOC clusters, the bucket name and table location are as follows: - -|=== -| Cloud provider | Bucket or container name | Iceberg table location - -| AWS -| `redpanda-cloud-storage-` -.3+a| `redpanda-iceberg-catalog/redpanda/` - -| Azure -a| `` - -The Redpanda cluster ID is also used as the container name (ID) and the storage account ID. - -| GCP -| `redpanda-cloud-storage-` - -|=== +include::manage:partial$iceberg-access-table.adoc[] For Azure clusters, you must add the public IP addresses or ranges from the REST catalog service, or other clients requiring access to the Iceberg data, to your cluster's allow list. Alternatively, add subnet IDs to the allow list if the requests originate from the same Azure region. From d205bb2fdb12fb82f0418c5ce329156002b2a306 Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Wed, 1 Apr 2026 13:16:55 -0700 Subject: [PATCH 11/22] Same AWS account requirement --- .../manage/pages/iceberg/iceberg-topics-aws-glue.adoc | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc index c52aaa4162..a060f031d4 100644 --- a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc +++ b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc @@ -23,6 +23,10 @@ This guide walks you through querying Redpanda topics as Iceberg tables stored i == Prerequisites +* An AWS account with access to https://docs.aws.amazon.com/glue/latest/dg/what-is-glue.html[AWS Glue Data Catalog^]. +ifdef::env-cloud[] +** The AWS Glue Data Catalog must be in the same AWS account and region as the cluster. +endif::[] * Redpanda version {rp_version} or later. * xref:{rpk_install_doc}[`rpk`] installed or updated to the latest version. ifdef::env-cloud[] @@ -67,12 +71,6 @@ endif::[] When `iceberg_delete` or the topic override `redpanda.iceberg.delete` is set to `false`, you can delete the Redpanda topic, and then delete the table in AWS Glue and the Iceberg data and metadata files in the S3 bucket. If you plan to re-create the topic after deleting it, you must delete the table data entirely before re-creating the topic. -ifdef::env-cloud[] -=== Same AWS account required - -For BYOC and BYOVPC deployments, the AWS Glue Data Catalog must be in the same AWS account as the cluster. Cross-account Glue access has not been tested or certified. If you need to use a Glue catalog in a different AWS account, contact https://support.redpanda.com[Redpanda support^]. -endif::[] - == Authorize access to AWS Glue ifdef::env-cloud[] From 57e0a52c10ef9dcbfa4c954b52cada1dc3600891 Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Wed, 1 Apr 2026 13:22:08 -0700 Subject: [PATCH 12/22] Minor edit --- modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc index a060f031d4..5b3000fcda 100644 --- a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc +++ b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc @@ -25,7 +25,7 @@ This guide walks you through querying Redpanda topics as Iceberg tables stored i * An AWS account with access to https://docs.aws.amazon.com/glue/latest/dg/what-is-glue.html[AWS Glue Data Catalog^]. ifdef::env-cloud[] -** The AWS Glue Data Catalog must be in the same AWS account and region as the cluster. +** AWS Glue Data Catalog must be in the same AWS account and region as the cluster. endif::[] * Redpanda version {rp_version} or later. * xref:{rpk_install_doc}[`rpk`] installed or updated to the latest version. From 3112263450c61fa31f3c82d86cc48196fc86042f Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Wed, 1 Apr 2026 15:54:30 -0700 Subject: [PATCH 13/22] Apply suggestions from SME review --- .../pages/iceberg/iceberg-topics-aws-glue.adoc | 16 ++++++++-------- .../manage/partials/iceberg-access-table.adoc | 2 +- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc index 5b3000fcda..8163b6f068 100644 --- a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc +++ b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc @@ -129,9 +129,9 @@ endif::[] ifdef::env-cloud[] You can configure credentials for the AWS Glue Data Catalog integration in either of the following ways: -* Allow Redpanda to use the same `cloud_storage_*` credential properties already configured for S3. If you do not configure the overrides listed below, Redpanda uses the same credentials for both S3 and AWS Glue. This is the recommended approach, especially in BYOC deployments where the broker's IAM role already includes the necessary Glue permissions. +* Allow Redpanda to use the same object storage credential properties already configured for S3. If you do not configure the overrides listed below, Redpanda uses the same credentials for both S3 and AWS Glue. This is the recommended approach, especially in BYOC deployments where the cluster's existing AWS credentials already include the necessary Glue permissions. * If you want to configure authentication to AWS Glue separately from authentication to S3, there are equivalent credential configuration properties named `iceberg_rest_catalog_aws_*` that override the object storage credentials. These properties only apply to REST catalog authentication, and never to S3 authentication: -** config_ref:iceberg_rest_catalog_credentials_source,true,properties/cluster-properties[`iceberg_rest_catalog_credentials_source`] overrides config_ref:cloud_storage_credentials_source,true,properties/cluster-properties[`cloud_storage_credentials_source`]. To use the broker's IAM role, set the property to `aws_instance_metadata`. To use static credentials, set to `config_file`. +** config_ref:iceberg_rest_catalog_credentials_source,true,properties/cluster-properties[`iceberg_rest_catalog_credentials_source`]. To use the cluster's IAM role, set the property to `aws_instance_metadata`. To use static credentials, set to `config_file`. ** config_ref:iceberg_rest_catalog_aws_access_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_access_key`] (static credentials only) ** config_ref:iceberg_rest_catalog_aws_secret_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_secret_key`] (static credentials only), added as a secret value (see the <> for details) ** config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] @@ -158,7 +158,7 @@ iceberg_delete: false iceberg_catalog_type: rest iceberg_rest_catalog_endpoint: https://glue..amazonaws.com/iceberg iceberg_rest_catalog_authentication_mode: aws_sigv4 -# Because Tiered Storage does not support the use of distinct buckets for Iceberg, +# Because Redpanda does not support the use of distinct buckets for Iceberg, # always place iceberg_rest_catalog_base_location in the same S3 bucket as cloud_storage_bucket iceberg_rest_catalog_base_location: s3:/// # Use the iceberg_rest_catalog_aws_* properties if you want to @@ -177,11 +177,11 @@ Use your own values for the following placeholders: + -- * ``: The AWS region where your Data Catalog is located. The region in the AWS Glue endpoint must match the region specified in either your config_ref:cloud_storage_region,true,properties/cluster-properties[`cloud_storage_region`] or config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] property. -* `` and ``: AWS Glue requires you to specify the base location where Redpanda stores Iceberg data and metadata files. You must use an S3 URI; for example, `s3:///iceberg`. This must be the same bucket used for Tiered Storage (your `cloud_storage_bucket`). You cannot specify a different bucket for Iceberg data. +* `` and ``: AWS Glue requires you to specify the base location where Redpanda stores Iceberg data and metadata files. You must use an S3 URI; for example, `s3:///iceberg`. This must be the same bucket used for object storage (your `cloud_storage_bucket`). You cannot specify a different bucket for Iceberg data. + `` is a name you choose (such as `iceberg`) as the logical name for the warehouse represented by all Redpanda Iceberg topic data in the cluster. + -As a security best practice, do not use the bucket root for the base location. Always specify a subfolder to avoid interfering with Tiered Storage data. +As a security best practice, do not use the bucket root for the base location. Always specify a subfolder to avoid interfering with your cluster's data in object storage. -- endif::[] ifdef::env-cloud[] @@ -239,12 +239,12 @@ Use your own values for the following placeholders: -- * ``: The AWS region where your Data Catalog is located. The region in the AWS Glue endpoint must match the region specified in your config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] property. * `` and ``: AWS Glue requires you to specify the base location where Redpanda stores Iceberg data and metadata files. You must use an S3 URI; for example, `s3:///iceberg`. -** Bucket name: For BYOC clusters, the bucket name is `redpanda-cloud-storage-`. For BYOVPC clusters, use the name of the bucket you created as a customer-managed resource. +** Bucket name: For BYOC clusters, the bucket name is `redpanda-cloud-storage-`. For BYOVPC clusters, use the name of the object storage bucket you created as a xref:get-started:cluster-types/byoc/aws/vpc-byo-aws.adoc#configure-the-redpanda-network-and-cluster[customer-managed resource]. + -This must be the same bucket used for Tiered Storage. You cannot specify a different bucket for Iceberg data. +This must be the same bucket used for your cluster's object storage. You cannot specify a different bucket for Iceberg data. ** Warehouse: This is a name you choose as the logical name (such as `iceberg`) for the warehouse represented by all Redpanda Iceberg topic data in the cluster. + -As a security best practice, do not use the bucket root for the base location. Always specify a subfolder to avoid interfering with Tiered Storage data. +As a security best practice, do not use the bucket root for the base location. Always specify a subfolder to avoid interfering with the rest of your cluster's data in object storage. * `` (static credentials only): The AWS access key ID for your Glue service account. * `` (static credentials only): The name of the secret that stores the AWS secret access key for your Glue service account. To reference a secret in a cluster property, for example `iceberg_rest_catalog_aws_secret_key`, you must first xref:manage:iceberg/use-iceberg-catalogs.adoc#store-a-secret-for-rest-catalog-authentication[store the secret value]. -- diff --git a/modules/manage/partials/iceberg-access-table.adoc b/modules/manage/partials/iceberg-access-table.adoc index dc0f29707d..b9750111c0 100644 --- a/modules/manage/partials/iceberg-access-table.adoc +++ b/modules/manage/partials/iceberg-access-table.adoc @@ -17,4 +17,4 @@ The Redpanda cluster ID is also used as the container name (ID) and the storage |=== -For BYOVPC clusters, the bucket name is the name you chose when you created the Tiered Storage bucket as a customer-managed resource. +For BYOVPC clusters, the bucket name is the name you chose when you created the object storage bucket as a customer-managed resource. From 6a16d0ba562fd26bbefe1334a70d4f289396e225 Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Wed, 1 Apr 2026 16:15:16 -0700 Subject: [PATCH 14/22] Standardize to object storage bucket --- .../manage/pages/iceberg/iceberg-topics-databricks-unity.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/manage/pages/iceberg/iceberg-topics-databricks-unity.adoc b/modules/manage/pages/iceberg/iceberg-topics-databricks-unity.adoc index 8d4fa4267b..1969315bf4 100644 --- a/modules/manage/pages/iceberg/iceberg-topics-databricks-unity.adoc +++ b/modules/manage/pages/iceberg/iceberg-topics-databricks-unity.adoc @@ -73,7 +73,7 @@ Follow the steps in the https://docs.databricks.com/aws/en/connect/unity-catalog The external location stores the Unity Catalog-managed Iceberg metadata, and the Iceberg data written by Redpanda. You must use the same bucket configured for glossterm:Tiered Storage[] for your Redpanda cluster. ifdef::env-cloud[] -For BYOC clusters, the bucket name is `redpanda-cloud-storage-`, where `` is the ID of your Redpanda cluster. For BYOVPC clusters, the bucket name is the name you chose when you created the Tiered Storage bucket as a customer-managed resource. +For BYOC clusters, the bucket name is `redpanda-cloud-storage-`, where `` is the ID of your Redpanda cluster. For BYOVPC clusters, the bucket name is the name you chose when you created the object storage bucket as a customer-managed resource. endif::[] Follow the steps in the https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/external-locations[Databricks documentation] to *manually* create an external location. You can create the external location in the Catalog Explorer or with SQL. You must create the external location manually because the location needs to be associated with the existing Tiered Storage bucket URL, `s3://`. From ebcba9c18725da70dcc6404651d89986196d0a4b Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Thu, 2 Apr 2026 08:58:55 -0700 Subject: [PATCH 15/22] Only Iceberg credentials source is exposed in Cloud --- .../partials/properties/object-storage-properties.adoc | 2 -- 1 file changed, 2 deletions(-) diff --git a/modules/reference/partials/properties/object-storage-properties.adoc b/modules/reference/partials/properties/object-storage-properties.adoc index df623434e9..ed37c029f1 100644 --- a/modules/reference/partials/properties/object-storage-properties.adoc +++ b/modules/reference/partials/properties/object-storage-properties.adoc @@ -1519,7 +1519,6 @@ endif::[] |=== -// tag::redpanda-cloud[] === cloud_storage_credentials_source The source of credentials used to authenticate to object storage services. @@ -1575,7 +1574,6 @@ endif::[] |=== -// end::redpanda-cloud[] === cloud_storage_crl_file From 0eb981c4e310254a1b63c8b79744bfe5cf97ff37 Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Thu, 2 Apr 2026 10:24:28 -0700 Subject: [PATCH 16/22] Minor edits --- modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc index 8163b6f068..1be3927938 100644 --- a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc +++ b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc @@ -120,10 +120,10 @@ You must configure credentials for the AWS Glue Data Catalog integration in eith * Allow Redpanda to use the same `cloud_storage_*` credential properties configured for S3. If you do not configure the overrides listed below, Redpanda uses the same credentials for both S3 and AWS Glue. This is the recommended approach. * If you want to configure authentication to AWS Glue separately from authentication to S3, there are equivalent credential configuration properties named `iceberg_rest_catalog_aws_*` that override the object storage credentials. These properties only apply to REST catalog authentication, and never to S3 authentication: +** config_ref:iceberg_rest_catalog_credentials_source,true,properties/cluster-properties[`iceberg_rest_catalog_credentials_source`] overrides config_ref:cloud_storage_credentials_source,true,properties/cluster-properties[`cloud_storage_credentials_source`] ** config_ref:iceberg_rest_catalog_aws_access_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_access_key`] overrides config_ref:cloud_storage_access_key,true,properties/cluster-properties[`cloud_storage_access_key`] ** config_ref:iceberg_rest_catalog_aws_secret_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_secret_key`] overrides config_ref:cloud_storage_secret_key,true,properties/cluster-properties[`cloud_storage_secret_key`] ** config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] overrides config_ref:cloud_storage_region,true,properties/cluster-properties[`cloud_storage_region`] -** config_ref:iceberg_rest_catalog_credentials_source,true,properties/cluster-properties[`iceberg_rest_catalog_credentials_source`] overrides config_ref:cloud_storage_credentials_source,true,properties/cluster-properties[`cloud_storage_credentials_source`] endif::[] ifdef::env-cloud[] @@ -189,7 +189,7 @@ Use `rpk` like in the following examples, or use the Cloud API to xref:manage:cl + [tabs] ====== -IAM role (instance metadata):: +Use cluster's IAM credentials:: + -- [,bash] @@ -212,7 +212,7 @@ rpk cluster config set \ ---- -- -Static credentials:: +Use static credentials (override IAM):: + -- [,bash] From 1daa98faf5671b94f5c34de9bdc1b1a2b59cdfc4 Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Thu, 2 Apr 2026 10:59:25 -0700 Subject: [PATCH 17/22] Change broker role to cluster --- modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc index 1be3927938..4d150abf42 100644 --- a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc +++ b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc @@ -74,9 +74,9 @@ When `iceberg_delete` or the topic override `redpanda.iceberg.delete` is set to == Authorize access to AWS Glue ifdef::env-cloud[] -For BYOC clusters created in March 2026 or later, the required AWS Glue IAM policy is automatically provisioned and attached to the Redpanda broker role when Iceberg is enabled. You don't need to manually create IAM policies or roles for Glue access. +For BYOC clusters created in March 2026 or later, the required AWS Glue IAM policy is automatically provisioned and attached to the cluster's IAM role when Iceberg is enabled. You don't need to manually create IAM policies or roles for Glue access. -For clusters created before March 2026, you must run `rpk byoc apply` to provision the Glue IAM policy before enabling Iceberg. This is a one-time operation that updates the broker role with the necessary Glue permissions. +For clusters created before March 2026, you must run `rpk byoc apply` to provision the Glue IAM policy before enabling Iceberg. This is a one-time operation that updates the cluster's IAM role with the necessary Glue permissions. endif::[] ifndef::env-cloud[] From b99dc4ca94fb9b5aa8ebf8e1db382e03c44be3ee Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Thu, 2 Apr 2026 11:08:56 -0700 Subject: [PATCH 18/22] Specify rpk re-run for older cluster --- modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc index 4d150abf42..6bcbf6aabf 100644 --- a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc +++ b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc @@ -76,7 +76,7 @@ When `iceberg_delete` or the topic override `redpanda.iceberg.delete` is set to ifdef::env-cloud[] For BYOC clusters created in March 2026 or later, the required AWS Glue IAM policy is automatically provisioned and attached to the cluster's IAM role when Iceberg is enabled. You don't need to manually create IAM policies or roles for Glue access. -For clusters created before March 2026, you must run `rpk byoc apply` to provision the Glue IAM policy before enabling Iceberg. This is a one-time operation that updates the cluster's IAM role with the necessary Glue permissions. +For clusters created before March 2026, you must re-run `rpk byoc apply` to provision the Glue IAM policy before enabling Iceberg. This is a one-time operation that updates the cluster's IAM role with the necessary Glue permissions. endif::[] ifndef::env-cloud[] From a47f55f5ab68670ed0ca8dbf5f00e27bd7491845 Mon Sep 17 00:00:00 2001 From: Kat Batuigas <36839689+kbatuigas@users.noreply.github.com> Date: Mon, 6 Apr 2026 13:35:53 -0700 Subject: [PATCH 19/22] Update modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc Co-authored-by: Joyce Fee <102751339+Feediver1@users.noreply.github.com> --- modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc index 6bcbf6aabf..bb1351b3f0 100644 --- a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc +++ b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc @@ -185,7 +185,7 @@ As a security best practice, do not use the bucket root for the base location. A -- endif::[] ifdef::env-cloud[] -Use `rpk` like in the following examples, or use the Cloud API to xref:manage:cluster-maintenance/config-cluster.adoc#set-cluster-configuration-properties[update these cluster properties]. The update might take several minutes to complete. +Use `rpk` as shown in the following examples, or use the Cloud API to xref:manage:cluster-maintenance/config-cluster.adoc#set-cluster-configuration-properties[update these cluster properties]. The update might take several minutes to complete. + [tabs] ====== From ac1f633bc3f0a43ab08342ad780b8cfa4d612a72 Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Mon, 6 Apr 2026 13:46:27 -0700 Subject: [PATCH 20/22] Apply suggestion from doc review --- modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc index bb1351b3f0..940491fe7c 100644 --- a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc +++ b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc @@ -118,7 +118,7 @@ endif::[] ifndef::env-cloud[] You must configure credentials for the AWS Glue Data Catalog integration in either of the following ways: -* Allow Redpanda to use the same `cloud_storage_*` credential properties configured for S3. If you do not configure the overrides listed below, Redpanda uses the same credentials for both S3 and AWS Glue. This is the recommended approach. +* Allow Redpanda to use the same `cloud_storage_*` credential properties configured for S3. This is the recommended approach. * If you want to configure authentication to AWS Glue separately from authentication to S3, there are equivalent credential configuration properties named `iceberg_rest_catalog_aws_*` that override the object storage credentials. These properties only apply to REST catalog authentication, and never to S3 authentication: ** config_ref:iceberg_rest_catalog_credentials_source,true,properties/cluster-properties[`iceberg_rest_catalog_credentials_source`] overrides config_ref:cloud_storage_credentials_source,true,properties/cluster-properties[`cloud_storage_credentials_source`] ** config_ref:iceberg_rest_catalog_aws_access_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_access_key`] overrides config_ref:cloud_storage_access_key,true,properties/cluster-properties[`cloud_storage_access_key`] @@ -129,7 +129,7 @@ endif::[] ifdef::env-cloud[] You can configure credentials for the AWS Glue Data Catalog integration in either of the following ways: -* Allow Redpanda to use the same object storage credential properties already configured for S3. If you do not configure the overrides listed below, Redpanda uses the same credentials for both S3 and AWS Glue. This is the recommended approach, especially in BYOC deployments where the cluster's existing AWS credentials already include the necessary Glue permissions. +* Allow Redpanda to use the same object storage credential properties already configured for S3. This is the recommended approach, especially in BYOC deployments where the cluster's existing AWS credentials already include the necessary Glue permissions. * If you want to configure authentication to AWS Glue separately from authentication to S3, there are equivalent credential configuration properties named `iceberg_rest_catalog_aws_*` that override the object storage credentials. These properties only apply to REST catalog authentication, and never to S3 authentication: ** config_ref:iceberg_rest_catalog_credentials_source,true,properties/cluster-properties[`iceberg_rest_catalog_credentials_source`]. To use the cluster's IAM role, set the property to `aws_instance_metadata`. To use static credentials, set to `config_file`. ** config_ref:iceberg_rest_catalog_aws_access_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_access_key`] (static credentials only) From ae7f06465326944cf0b86afdd580a55d63cd9286 Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Mon, 6 Apr 2026 16:27:38 -0700 Subject: [PATCH 21/22] Use anchor links to point to cluster config examples --- modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc index 940491fe7c..6e47306d9a 100644 --- a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc +++ b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc @@ -130,11 +130,19 @@ ifdef::env-cloud[] You can configure credentials for the AWS Glue Data Catalog integration in either of the following ways: * Allow Redpanda to use the same object storage credential properties already configured for S3. This is the recommended approach, especially in BYOC deployments where the cluster's existing AWS credentials already include the necessary Glue permissions. ++ +For an example cluster configuration that uses the same IAM credentials for both S3 and AWS Glue, see the *Use cluster's IAM credentials* tab in the <>. * If you want to configure authentication to AWS Glue separately from authentication to S3, there are equivalent credential configuration properties named `iceberg_rest_catalog_aws_*` that override the object storage credentials. These properties only apply to REST catalog authentication, and never to S3 authentication: ++ +-- ** config_ref:iceberg_rest_catalog_credentials_source,true,properties/cluster-properties[`iceberg_rest_catalog_credentials_source`]. To use the cluster's IAM role, set the property to `aws_instance_metadata`. To use static credentials, set to `config_file`. ** config_ref:iceberg_rest_catalog_aws_access_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_access_key`] (static credentials only) ** config_ref:iceberg_rest_catalog_aws_secret_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_secret_key`] (static credentials only), added as a secret value (see the <> for details) ** config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] +-- ++ +For an example cluster configuration that uses separate access keys for AWS Glue, see the *Use static credentials (override IAM)* tab in the <>. + endif::[] == Update cluster configuration From c4f4ba961da4146f670db1552810386d4fdb769d Mon Sep 17 00:00:00 2001 From: Kat Batuigas Date: Mon, 6 Apr 2026 16:31:38 -0700 Subject: [PATCH 22/22] Minor edit --- modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc index 6e47306d9a..c0a58864e7 100644 --- a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc +++ b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc @@ -193,7 +193,7 @@ As a security best practice, do not use the bucket root for the base location. A -- endif::[] ifdef::env-cloud[] -Use `rpk` as shown in the following examples, or use the Cloud API to xref:manage:cluster-maintenance/config-cluster.adoc#set-cluster-configuration-properties[update these cluster properties]. The update might take several minutes to complete. +Use `rpk` as shown in the following examples, or xref:manage:cluster-maintenance/config-cluster.adoc#set-cluster-configuration-properties[use the Cloud API] to update these cluster properties. The update might take several minutes to complete. + [tabs] ======