aws_dynamodb_cdc: add checkpoint_namespace for checkpoint table sharing#4602
aws_dynamodb_cdc: add checkpoint_namespace for checkpoint table sharing#4602squiidz wants to merge 1 commit into
Conversation
Add an optional checkpoint_namespace field to the aws_dynamodb_cdc input so multiple independent pipelines can share a single checkpoint table without overwriting each other's checkpoints. When set, the namespace is prefixed to the checkpoint hash key value (StreamArn in default mode, TableId in global_table mode); shard checkpoints, snapshot progress rows, and the global-mode resume query are all scoped by it. The default (empty) leaves checkpoint keys byte-identical to previous behavior, so existing deployments and checkpoint tables are unaffected. Namespaces containing '#' are rejected at config validation time.
|
Commits Review LGTM |
6f1df05 to
529702d
Compare
|
Commits Review I verified that every hash-key construction site ( The CDC conformance gate at LGTM |
Summary
Adds an optional
checkpoint_namespacefield to theaws_dynamodb_cdcinput so multiple independent pipelines (e.g. one per developer or environment) can share a single checkpoint table without overwriting each other's checkpoints.Checkpoints were keyed only by
(StreamArn, ShardID)— or(TableId, ShardID)inglobal_tablemode — so concurrent consumers of the same stream pointing at the samecheckpoint_tableraced on the same rows, causing skipped/duplicated events. The workaround (a checkpoint table per consumer) means IAM and infrastructure sprawl.Behavior
checkpoint_namespaceis set, the namespace is prefixed to the hash key value with a#delimiter:StreamArn = "dev-alice#arn:aws:dynamodb:..."(default mode),TableId = "dev-alice#my-table"(global_tablemode). Each namespace occupies its own partition; shard checkpoints, snapshot progress rows, the snapshot completion sentinel, and the global-mode resume query are all scoped by it.start_from. Namespaces do not coordinate consumers — two pipelines sharing the same namespace still race, exactly like today.#is rejected at config validation time. Since DynamoDB table names and stream ARNs cannot contain#, the encoding is unambiguous.Testing
#rejected).