Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 3 additions & 20 deletions modules/manage/pages/iceberg/about-iceberg-topics.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ When you enable the Iceberg integration for a Redpanda topic, Redpanda brokers s
To enable Iceberg for Redpanda topics, you must have the following:

ifdef::env-cloud[]
* A running xref:get-started:cluster-types/byoc/index.adoc[BYOC] cluster on Redpanda version 25.1 or later. The Iceberg integration is supported only for BYOC, and the cluster properties to configure Iceberg are available with v25.1.
* A running xref:get-started:cluster-types/byoc/index.adoc[BYOC] or BYOVPC cluster on Redpanda version 25.1 or later. The Iceberg integration is supported only for BYOC and BYOVPC, and the cluster properties to configure Iceberg are available with v25.1.
* rpk: See xref:get-started:rpk-install.adoc[].
* Familiarity with the Redpanda Cloud API. You must link:/api/doc/cloud-controlplane/authentication[authenticate] to the Cloud API and use the Control Plane API to update your cluster configuration.
endif::[]
Expand Down Expand Up @@ -189,26 +189,9 @@ ifdef::env-cloud[]

=== Access Iceberg data

To query the Iceberg table, you need access to the object storage bucket or container where the Iceberg data is stored. For BYOC clusters, the bucket name and table location are as follows:
To query the Iceberg table, you need access to the object storage bucket or container where the Iceberg data is stored.

|===
| Cloud provider | Bucket or container name | Iceberg table location

| AWS
| `redpanda-cloud-storage-<cluster-id>`
.3+a| `redpanda-iceberg-catalog/redpanda/<topic-name>`

| Azure
a| `<cluster-id>`

The Redpanda cluster ID is also used as the container name (ID) and the storage account ID.


| GCP
| `redpanda-cloud-storage-<cluster-id>`


|===
include::manage:partial$iceberg-access-table.adoc[]

For Azure clusters, you must add the public IP addresses or ranges from the REST catalog service, or other clients requiring access to the Iceberg data, to your cluster's allow list. Alternatively, add subnet IDs to the allow list if the requests originate from the same Azure region.

Expand Down
101 changes: 79 additions & 22 deletions modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@ This guide walks you through querying Redpanda topics as Iceberg tables stored i

== Prerequisites

* An AWS account with access to https://docs.aws.amazon.com/glue/latest/dg/what-is-glue.html[AWS Glue Data Catalog^].
ifdef::env-cloud[]
** AWS Glue Data Catalog must be in the same AWS account and region as the cluster.
endif::[]
* Redpanda version {rp_version} or later.
* xref:{rpk_install_doc}[`rpk`] installed or updated to the latest version.
ifdef::env-cloud[]
Expand Down Expand Up @@ -69,6 +73,13 @@ When `iceberg_delete` or the topic override `redpanda.iceberg.delete` is set to

== Authorize access to AWS Glue

ifdef::env-cloud[]
For BYOC clusters created in March 2026 or later, the required AWS Glue IAM policy is automatically provisioned and attached to the cluster's IAM role when Iceberg is enabled. You don't need to manually create IAM policies or roles for Glue access.

For clusters created before March 2026, you must re-run `rpk byoc apply` to provision the Glue IAM policy before enabling Iceberg. This is a one-time operation that updates the cluster's IAM role with the necessary Glue permissions.
endif::[]

ifndef::env-cloud[]
You must allow Redpanda access to AWS Glue services in your AWS account. You can use the same access credentials that you configured for S3 (IAM role, access keys, and KMS key), as long as you have also added read and write access to AWS Glue Data Catalog.

For example, you could create a separate IAM policy that manages access to AWS Glue and attach it to the IAM role that Redpanda also uses to access S3. Add all AWS Glue API actions in the policy (`"glue:*"`) on the following resources:
Expand Down Expand Up @@ -100,27 +111,38 @@ Your IAM policy should include a statement similar to the following:
----

For more information on configuring IAM permissions, see the https://docs.aws.amazon.com/glue/latest/dg/configure-iam-for-glue.html[AWS Glue documentation^].
endif::[]

== Configure authentication and credentials

ifndef::env-cloud[]
You must configure credentials for the AWS Glue Data Catalog integration in either of the following ways:

* Allow Redpanda to use the same `cloud_storage_*` credential properties configured for S3. If you do not configure the overrides listed below, Redpanda uses the same credentials for both S3 and AWS Glue. This is the recommended approach.
* Allow Redpanda to use the same `cloud_storage_*` credential properties configured for S3. This is the recommended approach.
* If you want to configure authentication to AWS Glue separately from authentication to S3, there are equivalent credential configuration properties named `iceberg_rest_catalog_aws_*` that override the object storage credentials. These properties only apply to REST catalog authentication, and never to S3 authentication:
** config_ref:iceberg_rest_catalog_credentials_source,true,properties/cluster-properties[`iceberg_rest_catalog_credentials_source`] overrides config_ref:cloud_storage_credentials_source,true,properties/cluster-properties[`cloud_storage_credentials_source`]
** config_ref:iceberg_rest_catalog_aws_access_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_access_key`] overrides config_ref:cloud_storage_access_key,true,properties/cluster-properties[`cloud_storage_access_key`]
** config_ref:iceberg_rest_catalog_aws_secret_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_secret_key`] overrides config_ref:cloud_storage_secret_key,true,properties/cluster-properties[`cloud_storage_secret_key`]
** config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] overrides config_ref:cloud_storage_region,true,properties/cluster-properties[`cloud_storage_region`]
** config_ref:iceberg_rest_catalog_credentials_source,true,properties/cluster-properties[`iceberg_rest_catalog_credentials_source`] overrides config_ref:cloud_storage_credentials_source,true,properties/cluster-properties[`cloud_storage_credentials_source`]
endif::[]

ifdef::env-cloud[]
You must configure credentials for the AWS Glue Data Catalog integration using the following properties:
You can configure credentials for the AWS Glue Data Catalog integration in either of the following ways:

* Allow Redpanda to use the same object storage credential properties already configured for S3. This is the recommended approach, especially in BYOC deployments where the cluster's existing AWS credentials already include the necessary Glue permissions.
+
For an example cluster configuration that uses the same IAM credentials for both S3 and AWS Glue, see the *Use cluster's IAM credentials* tab in the <<update-cluster-configuration,next section>>.
* If you want to configure authentication to AWS Glue separately from authentication to S3, there are equivalent credential configuration properties named `iceberg_rest_catalog_aws_*` that override the object storage credentials. These properties only apply to REST catalog authentication, and never to S3 authentication:
+
--
** config_ref:iceberg_rest_catalog_credentials_source,true,properties/cluster-properties[`iceberg_rest_catalog_credentials_source`]. To use the cluster's IAM role, set the property to `aws_instance_metadata`. To use static credentials, set to `config_file`.
** config_ref:iceberg_rest_catalog_aws_access_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_access_key`] (static credentials only)
** config_ref:iceberg_rest_catalog_aws_secret_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_secret_key`] (static credentials only), added as a secret value (see the <<update-cluster-configuration,next section>> for details)
** config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`]
--
+
For an example cluster configuration that uses separate access keys for AWS Glue, see the *Use static credentials (override IAM)* tab in the <<update-cluster-configuration,next section>>.

* config_ref:iceberg_rest_catalog_credentials_source,true,properties/cluster-properties[`iceberg_rest_catalog_credentials_source`] set to `config_file`
* config_ref:iceberg_rest_catalog_aws_access_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_access_key`]
* config_ref:iceberg_rest_catalog_aws_secret_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_secret_key`], added as a secret value (see the <<update-cluster-configuration,next section>> for details)
* config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`]
endif::[]

== Update cluster configuration
Expand All @@ -144,8 +166,8 @@ iceberg_delete: false
iceberg_catalog_type: rest
iceberg_rest_catalog_endpoint: https://glue.<glue-region>.amazonaws.com/iceberg
iceberg_rest_catalog_authentication_mode: aws_sigv4
# Because Tiered Storage does not support the use of distinct buckets for Iceberg,
# always place iceberg_rest_catalog_base_location in the same S3 bucket as cloud_storage_bucket
# Because Redpanda does not support the use of distinct buckets for Iceberg,
# always place iceberg_rest_catalog_base_location in the same S3 bucket as cloud_storage_bucket
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kbatuigas nobody knows what 'cloud_storage_bucket' means in Cloud. We need a pointer to docs with a screenshot of where they find the cluster bucket in the Cloud UI

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mattschumpert This is for the self-managed example, but I updated the descriptions in the Cloud version of the doc: https://deploy-preview-1633--redpanda-docs-preview.netlify.app/redpanda-cloud/manage/iceberg/iceberg-topics-aws-glue/#update-cluster-configuration If we can confirm whether the bucket name only shows up in the UI after Iceberg is enabled, I'm not sure that pointing them to the UI is helpful in this specific step

iceberg_rest_catalog_base_location: s3://<bucket-name>/<warehouse-path>
# Use the iceberg_rest_catalog_aws_* properties if you want to
# use separate AWS credentials for the catalog, or omit these lines to reuse S3
Expand All @@ -162,42 +184,77 @@ iceberg_rest_catalog_base_location: s3://<bucket-name>/<warehouse-path>
Use your own values for the following placeholders:
+
--
- `<glue-region>`: The AWS region where your Data Catalog is located. The region in the AWS Glue endpoint must match the region specified in either your config_ref:cloud_storage_region,true,properties/cluster-properties[`cloud_storage_region`] or config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] property.
- `<bucket-name>` and `<warehouse-path>`: AWS Glue requires you to specify the base location where Redpanda stores Iceberg data and metadata files. You must use an S3 URI; for example, `s3://<bucket-name>/iceberg`. As a security best practice, Redpanda Data recommends specifying a subfolder (using prefixes) rather than the root of the bucket.
* `<glue-region>`: The AWS region where your Data Catalog is located. The region in the AWS Glue endpoint must match the region specified in either your config_ref:cloud_storage_region,true,properties/cluster-properties[`cloud_storage_region`] or config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] property.
* `<bucket-name>` and `<warehouse-path>`: AWS Glue requires you to specify the base location where Redpanda stores Iceberg data and metadata files. You must use an S3 URI; for example, `s3://<bucket-name>/iceberg`. This must be the same bucket used for object storage (your `cloud_storage_bucket`). You cannot specify a different bucket for Iceberg data.
+
`<warehouse-path>` is a name you choose (such as `iceberg`) as the logical name for the warehouse represented by all Redpanda Iceberg topic data in the cluster.
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wdberkeley should we also clarify with a tip 'this path only affects the S3 folder used for table storage and not the catalog name used by Glue.' to select the catalog, you should configure the namespace.

@kbatuigas I assume we added thae custom namespace here too in Cloud docs ala (https://deploy-preview-1622--redpanda-docs-preview.netlify.app/current/manage/iceberg/iceberg-topics-aws-glue/#update-cluster-configuration)

I think it could be confusing as this name really isn't that important (just an implementation detail Glue forces them to set.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@simon0191 @wdberkeley is iceberg_default_catalog_namespace available yet in Cloud?

@mattschumpert It seems AWS also uses the "warehouse" terminology https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-format-iceberg.html not sure if we should change it to something else like <subdirectory> (I think the bucket-name and warehouse-path description later says to use a subfolder and not root)

+
As a security best practice, do not use the bucket root for the base location. Always specify a subfolder to avoid interfering with your cluster's data in object storage.
--
endif::[]
ifdef::env-cloud[]
Use `rpk` like in the following example, or use the Cloud API to xref:manage:cluster-maintenance/config-cluster.adoc#set-cluster-configuration-properties[update these cluster properties]. The update might take several minutes to complete.
Use `rpk` as shown in the following examples, or xref:manage:cluster-maintenance/config-cluster.adoc#set-cluster-configuration-properties[use the Cloud API] to update these cluster properties. The update might take several minutes to complete.
+
[tabs]
======
Use cluster's IAM credentials::
+
--
[,bash]
----
# Glue requires Redpanda Iceberg tables to be manually deleted
# so iceberg_delete is set to false.
rpk cloud login

rpk profile create --from-cloud <cluster-id>

rpk cluster config set \
iceberg_enabled=true \
iceberg_delete=false \
iceberg_catalog_type=rest \
iceberg_rest_catalog_endpoint=https://glue.<glue-region>.amazonaws.com/iceberg \
iceberg_rest_catalog_authentication_mode=aws_sigv4 \
iceberg_rest_catalog_credentials_source=aws_instance_metadata \
iceberg_rest_catalog_aws_region=<glue-region> \
iceberg_rest_catalog_base_location=s3://<cluster-storage-bucket-name>/<warehouse-path>
----
--

Use static credentials (override IAM)::
+
--
[,bash]
----
# Glue requires Redpanda Iceberg tables to be manually deleted
# so iceberg_delete is set to false.
rpk cluster config set \
iceberg_enabled=true \
iceberg_delete=false \
iceberg_catalog_type=rest \
iceberg_rest_catalog_endpoint=https://glue.<glue-region>.amazonaws.com/iceberg \
iceberg_rest_catalog_authentication_mode=aws_sigv4 \
# Because Tiered Storage does not support the use of distinct buckets for Iceberg,
# always place iceberg_rest_catalog_base_location in the same S3 bucket as cloud_storage_bucket
iceberg_rest_catalog_base_location=s3://<bucket-name>/<warehouse-path> \
# Set credentials source to config_file if using
# iceberg_rest_catalog_aws_access_key and iceberg_rest_catalog_aws_secret_key
iceberg_rest_catalog_credentials_source=config_file \
iceberg_rest_catalog_aws_region=<glue-region> \
iceberg_rest_catalog_aws_access_key=<glue-access-key> \
iceberg_rest_catalog_aws_secret_key='${secrets.<glue-secret-key-name>}'
iceberg_rest_catalog_aws_secret_key='${secrets.<glue-secret-key-name>}' \
iceberg_rest_catalog_base_location=s3://<cluster-storage-bucket-name>/<warehouse-path>
----
--
======
+
Use your own values for the following placeholders:
+
--
- `<glue-region>`: The AWS region where your Data Catalog is located. The region in the AWS Glue endpoint must match the region specified in your config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] property.
- `<bucket-name>` and `<warehouse-path>`: AWS Glue requires you to specify the base location where Redpanda stores Iceberg data and metadata files. You must use an S3 URI; for example, `s3://<bucket-name>/iceberg`. As a security best practice, Redpanda Data recommends specifying a subfolder (using prefixes) rather than the root of the bucket.
- `<glue-access-key>`: The AWS access key ID for your Glue service account.
- `<glue-secret-key-name>`: The name of the secret that stores the AWS secret access key for your Glue service account. To reference a secret in a cluster property, for example `iceberg_rest_catalog_aws_secret_key`, you must first xref:manage:iceberg/use-iceberg-catalogs.adoc#store-a-secret-for-rest-catalog-authentication[store the secret value].
* `<glue-region>`: The AWS region where your Data Catalog is located. The region in the AWS Glue endpoint must match the region specified in your config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] property.
* `<cluster-storage-bucket-name>` and `<warehouse-path>`: AWS Glue requires you to specify the base location where Redpanda stores Iceberg data and metadata files. You must use an S3 URI; for example, `s3://<cluster-storage-bucket-name>/iceberg`.
** Bucket name: For BYOC clusters, the bucket name is `redpanda-cloud-storage-<cluster-id>`. For BYOVPC clusters, use the name of the object storage bucket you created as a xref:get-started:cluster-types/byoc/aws/vpc-byo-aws.adoc#configure-the-redpanda-network-and-cluster[customer-managed resource].
+
This must be the same bucket used for your cluster's object storage. You cannot specify a different bucket for Iceberg data.
** Warehouse: This is a name you choose as the logical name (such as `iceberg`) for the warehouse represented by all Redpanda Iceberg topic data in the cluster.
+
As a security best practice, do not use the bucket root for the base location. Always specify a subfolder to avoid interfering with the rest of your cluster's data in object storage.
* `<glue-access-key>` (static credentials only): The AWS access key ID for your Glue service account.
* `<glue-secret-key-name>` (static credentials only): The name of the secret that stores the AWS secret access key for your Glue service account. To reference a secret in a cluster property, for example `iceberg_rest_catalog_aws_secret_key`, you must first xref:manage:iceberg/use-iceberg-catalogs.adoc#store-a-secret-for-rest-catalog-authentication[store the secret value].
--
endif::[]
+
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ Follow the steps in the https://docs.databricks.com/aws/en/connect/unity-catalog
The external location stores the Unity Catalog-managed Iceberg metadata, and the Iceberg data written by Redpanda. You must use the same bucket configured for glossterm:Tiered Storage[] for your Redpanda cluster.

ifdef::env-cloud[]
For BYOC clusters, the bucket name is `redpanda-cloud-storage-<cluster-id>`, where `<cluster-id>` is the ID of your Redpanda cluster.
For BYOC clusters, the bucket name is `redpanda-cloud-storage-<cluster-id>`, where `<cluster-id>` is the ID of your Redpanda cluster. For BYOVPC clusters, the bucket name is the name you chose when you created the object storage bucket as a customer-managed resource.
endif::[]

Follow the steps in the https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/external-locations[Databricks documentation] to *manually* create an external location. You can create the external location in the Catalog Explorer or with SQL. You must create the external location manually because the location needs to be associated with the existing Tiered Storage bucket URL, `s3://<bucket-name>`.
Expand Down
19 changes: 1 addition & 18 deletions modules/manage/pages/iceberg/query-iceberg-topics.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -27,24 +27,7 @@ endif::[]
ifdef::env-cloud[]
Redpanda generates an Iceberg table with the same name as the topic. Depending on the processing engine and your Iceberg catalog implementation, you may also need to define the table (for example using `CREATE TABLE`) to point the data lakehouse to its location in the catalog.

For BYOC clusters, the bucket name and table location are as follows:

|===
| Cloud provider | Bucket or container name | Iceberg table location

| AWS
| `redpanda-cloud-storage-<cluster-id>`
.3+a| `redpanda-iceberg-catalog/redpanda/<topic-name>`

| Azure
a| `<cluster-id>`

The Redpanda cluster ID is also used as the container name (ID) and the storage account ID.

| GCP
| `redpanda-cloud-storage-<cluster-id>`

|===
include::manage:partial$iceberg-access-table.adoc[]

For Azure clusters, you must add the public IP addresses or ranges from the REST catalog service, or other clients requiring access to the Iceberg data, to your cluster's allow list. Alternatively, add subnet IDs to the allow list if the requests originate from the same Azure region.

Expand Down
Loading
Loading