diff --git a/modules/manage/pages/iceberg/about-iceberg-topics.adoc b/modules/manage/pages/iceberg/about-iceberg-topics.adoc index 4f74450f63..ad771ef397 100644 --- a/modules/manage/pages/iceberg/about-iceberg-topics.adoc +++ b/modules/manage/pages/iceberg/about-iceberg-topics.adoc @@ -45,7 +45,7 @@ When you enable the Iceberg integration for a Redpanda topic, Redpanda brokers s To enable Iceberg for Redpanda topics, you must have the following: ifdef::env-cloud[] -* A running xref:get-started:cluster-types/byoc/index.adoc[BYOC] cluster on Redpanda version 25.1 or later. The Iceberg integration is supported only for BYOC, and the cluster properties to configure Iceberg are available with v25.1. +* A running xref:get-started:cluster-types/byoc/index.adoc[BYOC] or BYOVPC cluster on Redpanda version 25.1 or later. The Iceberg integration is supported only for BYOC and BYOVPC, and the cluster properties to configure Iceberg are available with v25.1. * rpk: See xref:get-started:rpk-install.adoc[]. * Familiarity with the Redpanda Cloud API. You must link:/api/doc/cloud-controlplane/authentication[authenticate] to the Cloud API and use the Control Plane API to update your cluster configuration. endif::[] @@ -189,26 +189,9 @@ ifdef::env-cloud[] === Access Iceberg data -To query the Iceberg table, you need access to the object storage bucket or container where the Iceberg data is stored. For BYOC clusters, the bucket name and table location are as follows: +To query the Iceberg table, you need access to the object storage bucket or container where the Iceberg data is stored. -|=== -| Cloud provider | Bucket or container name | Iceberg table location - -| AWS -| `redpanda-cloud-storage-` -.3+a| `redpanda-iceberg-catalog/redpanda/` - -| Azure -a| `` - -The Redpanda cluster ID is also used as the container name (ID) and the storage account ID. - - -| GCP -| `redpanda-cloud-storage-` - - -|=== +include::manage:partial$iceberg-access-table.adoc[] For Azure clusters, you must add the public IP addresses or ranges from the REST catalog service, or other clients requiring access to the Iceberg data, to your cluster's allow list. Alternatively, add subnet IDs to the allow list if the requests originate from the same Azure region. diff --git a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc index a530eeec06..c0a58864e7 100644 --- a/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc +++ b/modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc @@ -23,6 +23,10 @@ This guide walks you through querying Redpanda topics as Iceberg tables stored i == Prerequisites +* An AWS account with access to https://docs.aws.amazon.com/glue/latest/dg/what-is-glue.html[AWS Glue Data Catalog^]. +ifdef::env-cloud[] +** AWS Glue Data Catalog must be in the same AWS account and region as the cluster. +endif::[] * Redpanda version {rp_version} or later. * xref:{rpk_install_doc}[`rpk`] installed or updated to the latest version. ifdef::env-cloud[] @@ -69,6 +73,13 @@ When `iceberg_delete` or the topic override `redpanda.iceberg.delete` is set to == Authorize access to AWS Glue +ifdef::env-cloud[] +For BYOC clusters created in March 2026 or later, the required AWS Glue IAM policy is automatically provisioned and attached to the cluster's IAM role when Iceberg is enabled. You don't need to manually create IAM policies or roles for Glue access. + +For clusters created before March 2026, you must re-run `rpk byoc apply` to provision the Glue IAM policy before enabling Iceberg. This is a one-time operation that updates the cluster's IAM role with the necessary Glue permissions. +endif::[] + +ifndef::env-cloud[] You must allow Redpanda access to AWS Glue services in your AWS account. You can use the same access credentials that you configured for S3 (IAM role, access keys, and KMS key), as long as you have also added read and write access to AWS Glue Data Catalog. For example, you could create a separate IAM policy that manages access to AWS Glue and attach it to the IAM role that Redpanda also uses to access S3. Add all AWS Glue API actions in the policy (`"glue:*"`) on the following resources: @@ -100,27 +111,38 @@ Your IAM policy should include a statement similar to the following: ---- For more information on configuring IAM permissions, see the https://docs.aws.amazon.com/glue/latest/dg/configure-iam-for-glue.html[AWS Glue documentation^]. +endif::[] == Configure authentication and credentials ifndef::env-cloud[] You must configure credentials for the AWS Glue Data Catalog integration in either of the following ways: -* Allow Redpanda to use the same `cloud_storage_*` credential properties configured for S3. If you do not configure the overrides listed below, Redpanda uses the same credentials for both S3 and AWS Glue. This is the recommended approach. +* Allow Redpanda to use the same `cloud_storage_*` credential properties configured for S3. This is the recommended approach. * If you want to configure authentication to AWS Glue separately from authentication to S3, there are equivalent credential configuration properties named `iceberg_rest_catalog_aws_*` that override the object storage credentials. These properties only apply to REST catalog authentication, and never to S3 authentication: +** config_ref:iceberg_rest_catalog_credentials_source,true,properties/cluster-properties[`iceberg_rest_catalog_credentials_source`] overrides config_ref:cloud_storage_credentials_source,true,properties/cluster-properties[`cloud_storage_credentials_source`] ** config_ref:iceberg_rest_catalog_aws_access_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_access_key`] overrides config_ref:cloud_storage_access_key,true,properties/cluster-properties[`cloud_storage_access_key`] ** config_ref:iceberg_rest_catalog_aws_secret_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_secret_key`] overrides config_ref:cloud_storage_secret_key,true,properties/cluster-properties[`cloud_storage_secret_key`] ** config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] overrides config_ref:cloud_storage_region,true,properties/cluster-properties[`cloud_storage_region`] -** config_ref:iceberg_rest_catalog_credentials_source,true,properties/cluster-properties[`iceberg_rest_catalog_credentials_source`] overrides config_ref:cloud_storage_credentials_source,true,properties/cluster-properties[`cloud_storage_credentials_source`] endif::[] ifdef::env-cloud[] -You must configure credentials for the AWS Glue Data Catalog integration using the following properties: +You can configure credentials for the AWS Glue Data Catalog integration in either of the following ways: + +* Allow Redpanda to use the same object storage credential properties already configured for S3. This is the recommended approach, especially in BYOC deployments where the cluster's existing AWS credentials already include the necessary Glue permissions. ++ +For an example cluster configuration that uses the same IAM credentials for both S3 and AWS Glue, see the *Use cluster's IAM credentials* tab in the <>. +* If you want to configure authentication to AWS Glue separately from authentication to S3, there are equivalent credential configuration properties named `iceberg_rest_catalog_aws_*` that override the object storage credentials. These properties only apply to REST catalog authentication, and never to S3 authentication: ++ +-- +** config_ref:iceberg_rest_catalog_credentials_source,true,properties/cluster-properties[`iceberg_rest_catalog_credentials_source`]. To use the cluster's IAM role, set the property to `aws_instance_metadata`. To use static credentials, set to `config_file`. +** config_ref:iceberg_rest_catalog_aws_access_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_access_key`] (static credentials only) +** config_ref:iceberg_rest_catalog_aws_secret_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_secret_key`] (static credentials only), added as a secret value (see the <> for details) +** config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] +-- ++ +For an example cluster configuration that uses separate access keys for AWS Glue, see the *Use static credentials (override IAM)* tab in the <>. -* config_ref:iceberg_rest_catalog_credentials_source,true,properties/cluster-properties[`iceberg_rest_catalog_credentials_source`] set to `config_file` -* config_ref:iceberg_rest_catalog_aws_access_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_access_key`] -* config_ref:iceberg_rest_catalog_aws_secret_key,true,properties/cluster-properties[`iceberg_rest_catalog_aws_secret_key`], added as a secret value (see the <> for details) -* config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] endif::[] == Update cluster configuration @@ -144,8 +166,8 @@ iceberg_delete: false iceberg_catalog_type: rest iceberg_rest_catalog_endpoint: https://glue..amazonaws.com/iceberg iceberg_rest_catalog_authentication_mode: aws_sigv4 - # Because Tiered Storage does not support the use of distinct buckets for Iceberg, - # always place iceberg_rest_catalog_base_location in the same S3 bucket as cloud_storage_bucket +# Because Redpanda does not support the use of distinct buckets for Iceberg, +# always place iceberg_rest_catalog_base_location in the same S3 bucket as cloud_storage_bucket iceberg_rest_catalog_base_location: s3:/// # Use the iceberg_rest_catalog_aws_* properties if you want to # use separate AWS credentials for the catalog, or omit these lines to reuse S3 @@ -162,42 +184,77 @@ iceberg_rest_catalog_base_location: s3:/// Use your own values for the following placeholders: + -- -- ``: The AWS region where your Data Catalog is located. The region in the AWS Glue endpoint must match the region specified in either your config_ref:cloud_storage_region,true,properties/cluster-properties[`cloud_storage_region`] or config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] property. -- `` and ``: AWS Glue requires you to specify the base location where Redpanda stores Iceberg data and metadata files. You must use an S3 URI; for example, `s3:///iceberg`. As a security best practice, Redpanda Data recommends specifying a subfolder (using prefixes) rather than the root of the bucket. +* ``: The AWS region where your Data Catalog is located. The region in the AWS Glue endpoint must match the region specified in either your config_ref:cloud_storage_region,true,properties/cluster-properties[`cloud_storage_region`] or config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] property. +* `` and ``: AWS Glue requires you to specify the base location where Redpanda stores Iceberg data and metadata files. You must use an S3 URI; for example, `s3:///iceberg`. This must be the same bucket used for object storage (your `cloud_storage_bucket`). You cannot specify a different bucket for Iceberg data. ++ +`` is a name you choose (such as `iceberg`) as the logical name for the warehouse represented by all Redpanda Iceberg topic data in the cluster. ++ +As a security best practice, do not use the bucket root for the base location. Always specify a subfolder to avoid interfering with your cluster's data in object storage. -- endif::[] ifdef::env-cloud[] -Use `rpk` like in the following example, or use the Cloud API to xref:manage:cluster-maintenance/config-cluster.adoc#set-cluster-configuration-properties[update these cluster properties]. The update might take several minutes to complete. +Use `rpk` as shown in the following examples, or xref:manage:cluster-maintenance/config-cluster.adoc#set-cluster-configuration-properties[use the Cloud API] to update these cluster properties. The update might take several minutes to complete. ++ +[tabs] +====== +Use cluster's IAM credentials:: + +-- [,bash] ---- +# Glue requires Redpanda Iceberg tables to be manually deleted +# so iceberg_delete is set to false. rpk cloud login rpk profile create --from-cloud rpk cluster config set \ iceberg_enabled=true \ + iceberg_delete=false \ + iceberg_catalog_type=rest \ + iceberg_rest_catalog_endpoint=https://glue..amazonaws.com/iceberg \ + iceberg_rest_catalog_authentication_mode=aws_sigv4 \ + iceberg_rest_catalog_credentials_source=aws_instance_metadata \ + iceberg_rest_catalog_aws_region= \ + iceberg_rest_catalog_base_location=s3:/// +---- +-- + +Use static credentials (override IAM):: ++ +-- +[,bash] +---- +# Glue requires Redpanda Iceberg tables to be manually deleted +# so iceberg_delete is set to false. +rpk cluster config set \ + iceberg_enabled=true \ + iceberg_delete=false \ iceberg_catalog_type=rest \ iceberg_rest_catalog_endpoint=https://glue..amazonaws.com/iceberg \ iceberg_rest_catalog_authentication_mode=aws_sigv4 \ - # Because Tiered Storage does not support the use of distinct buckets for Iceberg, - # always place iceberg_rest_catalog_base_location in the same S3 bucket as cloud_storage_bucket - iceberg_rest_catalog_base_location=s3:/// \ - # Set credentials source to config_file if using - # iceberg_rest_catalog_aws_access_key and iceberg_rest_catalog_aws_secret_key iceberg_rest_catalog_credentials_source=config_file \ iceberg_rest_catalog_aws_region= \ iceberg_rest_catalog_aws_access_key= \ - iceberg_rest_catalog_aws_secret_key='${secrets.}' + iceberg_rest_catalog_aws_secret_key='${secrets.}' \ + iceberg_rest_catalog_base_location=s3:/// ---- +-- +====== + Use your own values for the following placeholders: + -- -- ``: The AWS region where your Data Catalog is located. The region in the AWS Glue endpoint must match the region specified in your config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] property. -- `` and ``: AWS Glue requires you to specify the base location where Redpanda stores Iceberg data and metadata files. You must use an S3 URI; for example, `s3:///iceberg`. As a security best practice, Redpanda Data recommends specifying a subfolder (using prefixes) rather than the root of the bucket. -- ``: The AWS access key ID for your Glue service account. -- ``: The name of the secret that stores the AWS secret access key for your Glue service account. To reference a secret in a cluster property, for example `iceberg_rest_catalog_aws_secret_key`, you must first xref:manage:iceberg/use-iceberg-catalogs.adoc#store-a-secret-for-rest-catalog-authentication[store the secret value]. +* ``: The AWS region where your Data Catalog is located. The region in the AWS Glue endpoint must match the region specified in your config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] property. +* `` and ``: AWS Glue requires you to specify the base location where Redpanda stores Iceberg data and metadata files. You must use an S3 URI; for example, `s3:///iceberg`. +** Bucket name: For BYOC clusters, the bucket name is `redpanda-cloud-storage-`. For BYOVPC clusters, use the name of the object storage bucket you created as a xref:get-started:cluster-types/byoc/aws/vpc-byo-aws.adoc#configure-the-redpanda-network-and-cluster[customer-managed resource]. ++ +This must be the same bucket used for your cluster's object storage. You cannot specify a different bucket for Iceberg data. +** Warehouse: This is a name you choose as the logical name (such as `iceberg`) for the warehouse represented by all Redpanda Iceberg topic data in the cluster. ++ +As a security best practice, do not use the bucket root for the base location. Always specify a subfolder to avoid interfering with the rest of your cluster's data in object storage. +* `` (static credentials only): The AWS access key ID for your Glue service account. +* `` (static credentials only): The name of the secret that stores the AWS secret access key for your Glue service account. To reference a secret in a cluster property, for example `iceberg_rest_catalog_aws_secret_key`, you must first xref:manage:iceberg/use-iceberg-catalogs.adoc#store-a-secret-for-rest-catalog-authentication[store the secret value]. -- endif::[] + diff --git a/modules/manage/pages/iceberg/iceberg-topics-databricks-unity.adoc b/modules/manage/pages/iceberg/iceberg-topics-databricks-unity.adoc index 3fcef32425..1969315bf4 100644 --- a/modules/manage/pages/iceberg/iceberg-topics-databricks-unity.adoc +++ b/modules/manage/pages/iceberg/iceberg-topics-databricks-unity.adoc @@ -73,7 +73,7 @@ Follow the steps in the https://docs.databricks.com/aws/en/connect/unity-catalog The external location stores the Unity Catalog-managed Iceberg metadata, and the Iceberg data written by Redpanda. You must use the same bucket configured for glossterm:Tiered Storage[] for your Redpanda cluster. ifdef::env-cloud[] -For BYOC clusters, the bucket name is `redpanda-cloud-storage-`, where `` is the ID of your Redpanda cluster. +For BYOC clusters, the bucket name is `redpanda-cloud-storage-`, where `` is the ID of your Redpanda cluster. For BYOVPC clusters, the bucket name is the name you chose when you created the object storage bucket as a customer-managed resource. endif::[] Follow the steps in the https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/external-locations[Databricks documentation] to *manually* create an external location. You can create the external location in the Catalog Explorer or with SQL. You must create the external location manually because the location needs to be associated with the existing Tiered Storage bucket URL, `s3://`. diff --git a/modules/manage/pages/iceberg/query-iceberg-topics.adoc b/modules/manage/pages/iceberg/query-iceberg-topics.adoc index 939674dff5..7ae2356968 100644 --- a/modules/manage/pages/iceberg/query-iceberg-topics.adoc +++ b/modules/manage/pages/iceberg/query-iceberg-topics.adoc @@ -27,24 +27,7 @@ endif::[] ifdef::env-cloud[] Redpanda generates an Iceberg table with the same name as the topic. Depending on the processing engine and your Iceberg catalog implementation, you may also need to define the table (for example using `CREATE TABLE`) to point the data lakehouse to its location in the catalog. -For BYOC clusters, the bucket name and table location are as follows: - -|=== -| Cloud provider | Bucket or container name | Iceberg table location - -| AWS -| `redpanda-cloud-storage-` -.3+a| `redpanda-iceberg-catalog/redpanda/` - -| Azure -a| `` - -The Redpanda cluster ID is also used as the container name (ID) and the storage account ID. - -| GCP -| `redpanda-cloud-storage-` - -|=== +include::manage:partial$iceberg-access-table.adoc[] For Azure clusters, you must add the public IP addresses or ranges from the REST catalog service, or other clients requiring access to the Iceberg data, to your cluster's allow list. Alternatively, add subnet IDs to the allow list if the requests originate from the same Azure region. diff --git a/modules/manage/pages/iceberg/use-iceberg-catalogs.adoc b/modules/manage/pages/iceberg/use-iceberg-catalogs.adoc index 4aa3f32a27..5268eda49b 100644 --- a/modules/manage/pages/iceberg/use-iceberg-catalogs.adoc +++ b/modules/manage/pages/iceberg/use-iceberg-catalogs.adoc @@ -14,7 +14,7 @@ include::shared:partial$enterprise-license.adoc[] ==== endif::[] -To read from the Redpanda-generated xref:{about-iceberg-doc}[Iceberg table], your Iceberg-compatible client or tool needs access to the catalog to retrieve the table metadata and know the current state of the table. The catalog provides the current table metadata, which includes locations for all the table's data files. You can configure Redpanda to either connect to a REST-based catalog, or use a filesystem-based catalog. +To read from the Redpanda-generated xref:{about-iceberg-doc}[Iceberg table], your Iceberg-compatible client or tool needs access to the catalog to retrieve the table metadata and know the current state of the table. The catalog provides the current table metadata, which includes locations for all the table's data files. You can configure Redpanda to either connect to a REST-based catalog, or use a filesystem-based catalog. For production deployments, Redpanda recommends <> to manage Iceberg metadata. This enables built-in table maintenance, safely handles multiple engines and tools accessing tables at the same time, facilitates data governance, and maximizes data discovery. However, if it is not possible to use a REST catalog, you can <> (`object_storage` catalog type), which does not require you to maintain a separate service to access the Iceberg data. @@ -25,6 +25,8 @@ After you have selected a catalog type at the cluster level and xref:{about-iceb [[rest]] == Connect to a REST catalog +NOTE: Redpanda connects to an Iceberg catalog that you provision and manage. Redpanda does not create or manage the catalog service, its databases, or any associated network configuration. + Connect to an Iceberg REST catalog using the standard https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml[REST API^] supported by many catalog providers. Use this catalog integration type with REST-enabled Iceberg catalog services, such as https://docs.databricks.com/en/data-governance/unity-catalog/index.html[Databricks Unity^] and https://other-docs.snowflake.com/en/opencatalog/overview[Snowflake Open Catalog^]. [TIP] diff --git a/modules/manage/partials/iceberg-access-table.adoc b/modules/manage/partials/iceberg-access-table.adoc new file mode 100644 index 0000000000..b9750111c0 --- /dev/null +++ b/modules/manage/partials/iceberg-access-table.adoc @@ -0,0 +1,20 @@ +For BYOC clusters, the bucket name and table location are as follows: + +|=== +| Cloud provider | Bucket or container name | Iceberg table location + +| AWS +| `redpanda-cloud-storage-` +.3+a| `redpanda-iceberg-catalog/redpanda/` + +| Azure +a| `` + +The Redpanda cluster ID is also used as the container name (ID) and the storage account ID. + +| GCP +| `redpanda-cloud-storage-` + +|=== + +For BYOVPC clusters, the bucket name is the name you chose when you created the object storage bucket as a customer-managed resource. diff --git a/modules/reference/partials/properties/cluster-properties.adoc b/modules/reference/partials/properties/cluster-properties.adoc index a3ca43602a..f1fee64386 100644 --- a/modules/reference/partials/properties/cluster-properties.adoc +++ b/modules/reference/partials/properties/cluster-properties.adoc @@ -8015,6 +8015,7 @@ endif::[] // end::redpanda-cloud[] +// tag::redpanda-cloud[] === iceberg_rest_catalog_aws_credentials_source *Accepted values*: `aws_instance_metadata`, `azure_aks_oidc_federation`, `azure_vm_instance_metadata`, `config_file`, `gcp_instance_metadata`, `sts`. @@ -8075,6 +8076,7 @@ endif::[] |=== +// end::redpanda-cloud[] // tag::redpanda-cloud[] === iceberg_rest_catalog_aws_region