|
| 1 | += Migrate Iceberg Catalogs |
| 2 | +:description: Switch the Iceberg catalog backend for an existing Redpanda cluster without losing untranslated topic data. |
| 3 | +:page-categories: Iceberg, Migration |
| 4 | + |
| 5 | +// tag::single-source[] |
| 6 | +:page-topic-type: how-to |
| 7 | +:personas: ops_admin, streaming_developer |
| 8 | +:learning-objective-1: Verify that a target Iceberg catalog supports your existing schemas and partition specs |
| 9 | +:learning-objective-2: Pause Iceberg translation and drain pending commits without losing untranslated data |
| 10 | +:learning-objective-3: Apply new catalog configuration and resume translation safely |
| 11 | + |
| 12 | +Use the procedures in this topic when moving from the filesystem-based `object_storage` catalog to a managed REST catalog, or when changing between REST catalogs. By doing so, you can switch your cluster from one Iceberg catalog backend to another without losing untranslated topic data. |
| 13 | + |
| 14 | +This procedure guides you through pausing Iceberg translation per topic, draining pending commits to the old catalog, applying the new catalog configuration, and restarting the cluster. |
| 15 | + |
| 16 | +After completing these steps, you will be able to: |
| 17 | + |
| 18 | +* [ ] {learning-objective-1} |
| 19 | +* [ ] {learning-objective-2} |
| 20 | +* [ ] {learning-objective-3} |
| 21 | +
|
| 22 | +[IMPORTANT] |
| 23 | +==== |
| 24 | +Do not change config_ref:iceberg_catalog_type,true,properties/cluster-properties[`iceberg_catalog_type`] or any other catalog cluster property in place without following this procedure. In-flight commits and untranslated data can be lost or stuck if the catalog changes mid-translation. |
| 25 | +==== |
| 26 | + |
| 27 | +== Prerequisites |
| 28 | + |
| 29 | +* Iceberg topics enabled and running on your Redpanda cluster. |
| 30 | +* Network connectivity from all brokers to the new catalog endpoint. |
| 31 | +* Credentials configured for the new catalog (REST endpoint, authentication mode, secret or token). For configuration guidance for each catalog type, see xref:manage:iceberg/use-iceberg-catalogs.adoc[]. |
| 32 | +* The new catalog must support the current schema and partition spec of every Iceberg topic. See <<verify-catalog-compatibility,Verify catalog compatibility>>. |
| 33 | + |
| 34 | +== Verify catalog compatibility |
| 35 | + |
| 36 | +Before starting the migration, verify that the new catalog can host every Iceberg topic's table with its existing schema and partition spec. If the new catalog is incompatible with your existing schemas or partition specs, already-translated Parquet files will fail to commit, and translation will stall in a state that is difficult to recover from. |
| 37 | + |
| 38 | +The simplest validation is to manually create a test table in the new catalog with the same schema and partition spec as one of your Iceberg topics. If the create call fails, fix the partition spec or schema before migrating. Delete the test table after validation. |
| 39 | + |
| 40 | +[CAUTION] |
| 41 | +==== |
| 42 | +AWS Glue does not support partitioning on a nested field, which is Redpanda's default partition spec for Iceberg topics. If you migrate to AWS Glue, you must change the partition spec to a Glue-compatible form before starting the migration procedure. |
| 43 | +==== |
| 44 | + |
| 45 | +== Run the migration |
| 46 | + |
| 47 | +. Save the current `retention.ms` and `retention.bytes` values for every Iceberg topic, then set both to `-1` (infinite retention): |
| 48 | ++ |
| 49 | +[,bash] |
| 50 | +---- |
| 51 | +rpk topic alter-config <topic-name> --set retention.ms=-1 --set retention.bytes=-1 |
| 52 | +---- |
| 53 | ++ |
| 54 | +While Iceberg translation is paused in the next step, the topic's retention anchor on the log is released. Without infinite retention, the cluster could delete untranslated data before the migration completes. |
| 55 | + |
| 56 | +. Pause Iceberg translation on every Iceberg topic by setting `redpanda.iceberg.mode` to `disabled`. Save each topic's previous mode value so you can restore it later. |
| 57 | ++ |
| 58 | +[,bash] |
| 59 | +---- |
| 60 | +rpk topic alter-config <topic-name> --set redpanda.iceberg.mode=disabled |
| 61 | +---- |
| 62 | ++ |
| 63 | +Setting the mode to `disabled` stops new translation while letting already-translated data finish committing to the old catalog. For more about Iceberg modes, see xref:manage:iceberg/specify-iceberg-schema.adoc[]. |
| 64 | ++ |
| 65 | +NOTE: Do not change config_ref:iceberg_enabled,true,properties/cluster-properties[`iceberg_enabled`] at the cluster level. The Iceberg integration must remain enabled at the cluster level so that pending commits can drain to the old catalog. |
| 66 | + |
| 67 | +. Wait for pending commits to drain. Monitor the `redpanda_iceberg_pending_commit_lag` metric until it reaches `0` for every Iceberg topic-partition. |
| 68 | ++ |
| 69 | +This metric reports the number of offsets pending a commit to the Iceberg catalog. While it is non-zero, Redpanda is still flushing translated data to the old catalog. A non-zero value here is expected while translation is paused, and reflects new records the cluster has not yet translated. |
| 70 | ++ |
| 71 | +[TIP] |
| 72 | +==== |
| 73 | +If you scrape Prometheus, the following expression returns `0` only when every Iceberg-topic partition has fully drained: |
| 74 | +
|
| 75 | +[,promql] |
| 76 | +---- |
| 77 | +sum(redpanda_iceberg_pending_commit_lag) |
| 78 | +---- |
| 79 | +==== |
| 80 | + |
| 81 | +. Apply the new catalog configuration. For example, to switch from `object_storage` to a REST catalog, update the catalog cluster properties: |
| 82 | ++ |
| 83 | +[,bash] |
| 84 | +---- |
| 85 | +rpk cluster config set iceberg_catalog_type rest |
| 86 | +rpk cluster config set iceberg_rest_catalog_endpoint <endpoint-url> |
| 87 | +rpk cluster config set iceberg_rest_catalog_authentication_mode oauth2 |
| 88 | +# Set additional credential properties for your chosen authentication mode. |
| 89 | +---- |
| 90 | ++ |
| 91 | +For full guidance on setting catalog cluster properties, see xref:manage:iceberg/use-iceberg-catalogs.adoc#rest[Connect to a REST catalog] and the individual xref:manage:iceberg/rest-catalog/index.adoc[REST catalog integration pages]. |
| 92 | + |
| 93 | +. Restart Redpanda. The catalog cluster properties require a restart to take effect. |
| 94 | ++ |
| 95 | +ifndef::env-cloud[] |
| 96 | +For instructions, see xref:manage:cluster-maintenance/rolling-restart.adoc[]. |
| 97 | +endif::[] |
| 98 | +ifdef::env-cloud[] |
| 99 | +Coordinate the restart with Redpanda Support. The restart must occur after `redpanda_iceberg_pending_commit_lag` has reached `0` and before you resume translation. |
| 100 | +endif::[] |
| 101 | + |
| 102 | +. After the cluster comes up, check broker logs for successful catalog requests and the absence of authentication errors to verify the new catalog connection. |
| 103 | + |
| 104 | +. Resume Iceberg translation by restoring `redpanda.iceberg.mode` on every Iceberg topic to its previous value: |
| 105 | ++ |
| 106 | +[,bash] |
| 107 | +---- |
| 108 | +rpk topic alter-config <topic-name> --set redpanda.iceberg.mode=<previous-mode> |
| 109 | +---- |
| 110 | + |
| 111 | +. Restore `retention.ms` and `retention.bytes` on every Iceberg topic to the values you saved before starting the migration. |
| 112 | + |
| 113 | +== Verify the migration |
| 114 | + |
| 115 | +After the migration completes, confirm that new data is reaching the new catalog: |
| 116 | + |
| 117 | +* Query an Iceberg table in your query engine using the new catalog and confirm that row counts continue to increase as your topic produces new records. |
| 118 | +* Check broker logs for any commit failures referencing the new catalog. Repeated failures often indicate a schema or partition spec mismatch. See <<troubleshooting,Troubleshooting>> for details. |
| 119 | + |
| 120 | +== Troubleshooting |
| 121 | + |
| 122 | +* Pending commits stuck after restart: A schema or partition spec mismatch between the original tables and the new catalog is the most common cause. See <<verify-catalog-compatibility,Verify catalog compatibility>>. If you cannot resolve the mismatch, contact https://support.redpanda.com/hc/en-us/requests/new[Redpanda Support^]. |
| 123 | +* Authentication errors against the new REST catalog: Verify that the credential cluster properties (for example, `iceberg_rest_catalog_client_id`, `iceberg_rest_catalog_client_secret`, `iceberg_rest_catalog_token`) match what the new catalog expects. For OAuth, also check `iceberg_rest_catalog_oauth2_server_uri`. |
| 124 | +* Translation does not resume after restoring `redpanda.iceberg.mode`: Check that `redpanda_iceberg_pending_translation_lag` is increasing as new records are produced. If it remains `0`, the cluster is not translating new records. Verify that your producer is still writing to the topic and that the topic's mode value is one of `key_value`, `value_schema_id_prefix`, or `value_schema_latest`. |
| 125 | + |
| 126 | +// end::single-source[] |
0 commit comments