Add iceberg_default_catalog_namespace to Cloud (#1680)

Co-authored-by: Joyce Fee <102751339+Feediver1@users.noreply.github.com>

Author: kbatuigas

modules/manage/pages/iceberg/about-iceberg-topics.adoc

@@ -78,6 +78,8 @@ To create an Iceberg table for a Redpanda topic, you must set the cluster config
 endif::[]
 
 . Set the `iceberg_enabled` configuration option on your cluster to `true`.
++
+When multiple clusters write to the same catalog, each cluster must use a distinct namespace to avoid table name collisions. This is especially critical for REST catalog providers that offer a single global catalog per account (such as AWS Glue), where there is no other isolation mechanism. By default, Redpanda creates Iceberg tables in a namespace called `redpanda`. To use a unique namespace for your cluster's REST catalog integration, also set config_ref:iceberg_default_catalog_namespace,true,properties/cluster-properties[`iceberg_default_catalog_namespace`] when you set `iceberg_enabled`. You cannot change this property after you enable Iceberg topics on the cluster.
 ifdef::env-cloud[]
 +
 [tabs]
@@ -90,6 +92,8 @@ rpk::
 rpk cloud login
 rpk profile create  --from-cloud <cluster-id>
 rpk cluster config set iceberg_enabled true
+# Optional: set a custom namespace (default is "redpanda")
+# rpk cluster config set iceberg_default_catalog_namespace '["<custom-namespace>"]'
 ----
 --
 
@@ -109,6 +113,8 @@ export RP_CLOUD_TOKEN=`curl -X POST "https://auth.prd.cloud.redpanda.com/oauth/t
     -d "client_secret=<client-secret>"`
 
 # Update cluster configuration to enable Iceberg topics
+# Optional: to set a custom namespace (default is "redpanda"),
+# add "iceberg_default_catalog_namespace":["<custom-namespace>"] to custom_properties
 curl -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" -X PATCH \
   "https://api.cloud.redpanda.com/v1/clusters/${RP_CLUSTER_ID}" \
  -H 'accept: application/json'\
@@ -122,8 +128,6 @@ The link:/api/doc/cloud-controlplane/operation/operation-clusterservice_updatecl
 endif::[]
 ifndef::env-cloud[]
 +
-When multiple clusters write to the same catalog, each cluster must use a distinct namespace to avoid table name collisions. This is especially critical for REST catalog providers that offer a single global catalog per account (such as AWS Glue), where there is no other isolation mechanism. By default, Redpanda creates Iceberg tables in a namespace called `redpanda`. To use a unique namespace for your cluster's REST catalog integration, also set config_ref:iceberg_default_catalog_namespace,true,properties/cluster-properties[`iceberg_default_catalog_namespace`] when you set `iceberg_enabled`. This property cannot be changed after you enable Iceberg topics on the cluster.
-+
 [,bash]
 ----
 rpk cluster config set iceberg_enabled true

modules/manage/pages/iceberg/iceberg-topics-aws-glue.adoc

@@ -150,9 +150,9 @@ endif::[]
 To configure your Redpanda cluster to enable Iceberg on a topic and integrate with the AWS Glue Data Catalog:
 
 . Edit your cluster configuration to set the `iceberg_enabled` property to `true`, and set the catalog integration properties listed in the example below.
-ifndef::env-cloud[]
 +
 By default, Redpanda creates Iceberg tables in a namespace called `redpanda`. Because AWS Glue provides a single catalog per account, each Redpanda cluster that writes to the same Glue catalog must use a distinct namespace to avoid table name collisions. To set a unique namespace, also set config_ref:iceberg_default_catalog_namespace,true,properties/cluster-properties[`iceberg_default_catalog_namespace`] when you set `iceberg_enabled`. This property cannot be changed after Iceberg is enabled.
+ifndef::env-cloud[]
 +
 Run `rpk cluster config edit` to update these properties:
 +
@@ -169,12 +169,12 @@ iceberg_rest_catalog_authentication_mode: aws_sigv4
 # Because Redpanda does not support the use of distinct buckets for Iceberg,
 # always place iceberg_rest_catalog_base_location in the same S3 bucket as cloud_storage_bucket
 iceberg_rest_catalog_base_location: s3://<bucket-name>/<warehouse-path>
-# Use the iceberg_rest_catalog_aws_* properties if you want to 
-# use separate AWS credentials for the catalog, or omit these lines to reuse S3 
+# Use the iceberg_rest_catalog_aws_* properties if you want to
+# use separate AWS credentials for the catalog, or omit these lines to reuse S3
 # (cloud_storage_*) credentials.
 # For access using access keys only, use iceberg_rest_catalog_aws_access_key
-# and iceberg_rest_catalog_aws_secret_key. For access with an IAM role, use 
-# iceberg_rest_catalog_credentials_source only. 
+# and iceberg_rest_catalog_aws_secret_key. For access with an IAM role, use
+# iceberg_rest_catalog_credentials_source only.
 # iceberg_rest_catalog_aws_region:
 # iceberg_rest_catalog_aws_access_key:
 # iceberg_rest_catalog_aws_secret_key:
@@ -184,6 +184,7 @@ iceberg_rest_catalog_base_location: s3://<bucket-name>/<warehouse-path>
 Use your own values for the following placeholders:
 +
 --
+* `<custom-namespace>`: A unique namespace for this cluster's Iceberg tables. Each Redpanda cluster that writes to the same Glue catalog must use a distinct namespace to avoid table name collisions. If omitted, the default namespace `redpanda` is used.
 * `<glue-region>`: The AWS region where your Data Catalog is located. The region in the AWS Glue endpoint must match the region specified in either your config_ref:cloud_storage_region,true,properties/cluster-properties[`cloud_storage_region`] or config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] property.
 * `<bucket-name>` and `<warehouse-path>`: AWS Glue requires you to specify the base location where Redpanda stores Iceberg data and metadata files. You must use an S3 URI; for example, `s3://<bucket-name>/iceberg`. This must be the same bucket used for object storage (your `cloud_storage_bucket`). You cannot specify a different bucket for Iceberg data.
 +
@@ -193,6 +194,7 @@ As a security best practice, do not use the bucket root for the base location. A
 --
 endif::[]
 ifdef::env-cloud[]
++
 Use `rpk` as shown in the following examples, or xref:manage:cluster-maintenance/config-cluster.adoc#set-cluster-configuration-properties[use the Cloud API] to update these cluster properties. The update might take several minutes to complete.
 +
 [tabs]
@@ -211,6 +213,7 @@ rpk profile create --from-cloud <cluster-id>
 rpk cluster config set \
   iceberg_enabled=true \
   iceberg_delete=false \
+  iceberg_default_catalog_namespace='["<custom-namespace>"]' \
   iceberg_catalog_type=rest \
   iceberg_rest_catalog_endpoint=https://glue.<glue-region>.amazonaws.com/iceberg \
   iceberg_rest_catalog_authentication_mode=aws_sigv4 \
@@ -230,6 +233,7 @@ Use static credentials (override IAM)::
 rpk cluster config set \
   iceberg_enabled=true \
   iceberg_delete=false \
+  iceberg_default_catalog_namespace='["<custom-namespace>"]' \
   iceberg_catalog_type=rest \
   iceberg_rest_catalog_endpoint=https://glue.<glue-region>.amazonaws.com/iceberg \
   iceberg_rest_catalog_authentication_mode=aws_sigv4 \
@@ -245,9 +249,10 @@ rpk cluster config set \
 Use your own values for the following placeholders:
 +
 --
+* `<custom-namespace>`: A unique namespace for this cluster's Iceberg tables. Each Redpanda cluster that writes to the same Glue catalog must use a distinct namespace to avoid table name collisions. If omitted, the default namespace `redpanda` is used.
 * `<glue-region>`: The AWS region where your Data Catalog is located. The region in the AWS Glue endpoint must match the region specified in your config_ref:iceberg_rest_catalog_aws_region,true,properties/cluster-properties[`iceberg_rest_catalog_aws_region`] property.
 * `<cluster-storage-bucket-name>` and `<warehouse-path>`: AWS Glue requires you to specify the base location where Redpanda stores Iceberg data and metadata files. You must use an S3 URI; for example, `s3://<cluster-storage-bucket-name>/iceberg`.
-** Bucket name: For BYOC clusters, the bucket name is `redpanda-cloud-storage-<cluster-id>`. For BYOVPC clusters, use the name of the object storage bucket you created as a xref:get-started:cluster-types/byoc/aws/vpc-byo-aws.adoc#configure-the-redpanda-network-and-cluster[customer-managed resource]. 
+** Bucket name: For BYOC clusters, the bucket name is `redpanda-cloud-storage-<cluster-id>`. For BYOVPC clusters, use the name of the object storage bucket you created as a xref:get-started:cluster-types/byoc/aws/vpc-byo-aws.adoc#configure-the-redpanda-network-and-cluster[customer-managed resource].
 +
 This must be the same bucket used for your cluster's object storage. You cannot specify a different bucket for Iceberg data.
 ** Warehouse: This is a name you choose as the logical name (such as `iceberg`) for the warehouse represented by all Redpanda Iceberg topic data in the cluster.

modules/manage/pages/iceberg/iceberg-topics-databricks-unity.adoc

@@ -191,12 +191,7 @@ echo "hello world\nfoo bar\nbaz qux" | rpk topic produce <topic-name> --format='
 
 You should see the topic as a table with data in Unity Catalog. The data may take some time to become visible, depending on your config_ref:iceberg_target_lag_ms,true,properties/cluster-properties[`iceberg_target_lag_ms`] setting.
 
-ifndef::env-cloud[]
 . In Catalog Explorer, open your catalog. You should see a `redpanda` schema (or the namespace you configured with config_ref:iceberg_default_catalog_namespace,true,properties/cluster-properties[`iceberg_default_catalog_namespace`]), in addition to `default` and `information_schema`.
-endif::[]
-ifdef::env-cloud[]
-. In Catalog Explorer, open your catalog. You should see a `redpanda` schema, in addition to `default` and `information_schema`.
-endif::[]
 . The schema and the table residing within it are automatically added for you. The table name is the same as the topic name.
 
 == Query Iceberg table using Databricks SQL

modules/manage/pages/iceberg/query-iceberg-topics.adoc

@@ -80,9 +80,7 @@ endif::[]
 {"user_id": 2324, "event_type": "BUTTON_CLICK", "ts": "2024-11-25T20:23:59.380Z"}
 ----
 
-ifndef::env-cloud[]
 NOTE: The query examples on this page use `redpanda` as the Iceberg namespace, which is the default. If you configured a different namespace using config_ref:iceberg_default_catalog_namespace,true,properties/cluster-properties[`iceberg_default_catalog_namespace`], replace `redpanda` with your configured namespace.
-endif::[]
 
 === Topic with schema (`value_schema_id_prefix` mode)
 

modules/manage/pages/iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc

@@ -169,12 +169,7 @@ echo "hello world\nfoo bar\nbaz qux" | rpk topic produce <topic-name> --format='
 You should see the topic as a table in Open Catalog.
 
 . In Open Catalog, select *Catalogs*, then open your catalog. 
-ifndef::env-cloud[]
 . Under your catalog, you should see the `redpanda` namespace (or the namespace you configured with config_ref:iceberg_default_catalog_namespace,true,properties/cluster-properties[`iceberg_default_catalog_namespace`]), and a table with the name of your topic. The namespace and the table are automatically added for you.
-endif::[]
-ifdef::env-cloud[]
-. Under your catalog, you should see the `redpanda` namespace and a table with the name of your topic. The namespace and the table are automatically added for you.
-endif::[]
 
 == Query Iceberg table in Snowflake
 

modules/manage/pages/iceberg/use-iceberg-catalogs.adoc

@@ -92,13 +92,11 @@ To connect to a REST catalog, set the following cluster configuration properties
 
 NOTE: You must set `iceberg_rest_catalog_endpoint` at the same time that you set `iceberg_catalog_type` to `rest`.
 
-ifndef::env-cloud[]
 ==== Configure table namespace
 
 Check if your REST catalog provider has specific requirements or recommendations for namespaces. For example, AWS Glue offers only a single global catalog per account, and each cluster that writes to the same Glue catalog must use a distinct namespace to avoid table name collisions.
 
 By default, Redpanda creates Iceberg tables in a namespace called `redpanda`. To use a unique namespace, configure the config_ref:iceberg_default_catalog_namespace,true,properties/cluster-properties[`iceberg_default_catalog_namespace`] cluster property. You must set this property before enabling the Iceberg integration or at the same time. After you have enabled Iceberg, do not change this property value.
-endif::[]
 
 ==== Configure authentication
 
@@ -283,9 +281,7 @@ SELECT * FROM streaming.redpanda.<table-name>;
 ----
 
 The Iceberg table name is the name of your Redpanda topic.
-ifndef::env-cloud[] 
 If you configured a different namespace using config_ref:iceberg_default_catalog_namespace,true,properties/cluster-properties[`iceberg_default_catalog_namespace`], replace `redpanda` with your configured namespace.
-endif::[]
 
 TIP: You may need to explicitly create a table for the Iceberg data in your query engine. For an example, see xref:manage:iceberg/redpanda-topics-iceberg-snowflake-catalog.adoc[].