Iceberg stack 4/11: durable ingestion commit contracts

[karakanb]

Summary

Defines the durability and atomic-write contracts needed by the following Iceberg strategy/runtime layers.

It also adds an incarnation-fenced CDC truncate helper. When a managed CDC caller supplies the physical target incarnation, truncation fails closed if the table was dropped and recreated before the operation.

Before and after

Before this layer:

• record batches could not distinguish an exact replay-stable payload identity from a resumable checkpoint or a connection-local acknowledgement handle;
• strategies and destinations had no shared contracts for idempotent commits, atomic snapshot publication, atomic truncate+insert with schema evolution, direct merge, or owned cleanup;
• truncate-boundary handling had no generic way to require the originally bound physical table.

After this layer:

• exact payload identities and resumable checkpoints have separate, strongly typed fields and explicit semantics;
• destinations can opt into the new capabilities without changing existing destination behavior;
• managed CDC truncate callers can require a raw target incarnation and receive an error instead of truncating a replacement table.

Atomic snapshot attempts have an explicit retry contract: the attempt ID stays stable across retries, and begin/write/publish must be idempotent for that ID, including when publication committed but its success response was lost.

Kafka, NATS, and Redis cumulative watermarks intentionally remain connection-local acknowledgement tokens. They are not exposed as exact durable payload IDs because retry batch boundaries can change; their existing msg_id primary-key merge behavior remains the replay-safety mechanism.

Stack boundary

This PR intentionally introduces the contracts only. #963 consumes them in the strategy layer: it propagates raw target incarnations through streaming and merge paths, carries exact durable IDs into data and token-only commits, and adds crash/retry coverage. The later Iceberg runtime layer implements the destination-side capabilities.

Previous: #961 (merged) · Next: #963

Validation

• make format
• make lint
• make test
• go test -tags integration ./pkg/destination/postgres -run '^TestLateCDCTargetAtomicCreateAndConditionalTruncate$' -count=1 -v

[greptile-apps]

Prompt To Fix All With AI

Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 2
pkg/destination/destination.go:192
**`TableWriter` bool return value undocumented**

The new `TableWriter` field is a function type that returns `(bool, error)`, but nothing in the field declaration or surrounding comments explains what `true` vs `false` means. Every future caller (starting with #963) will have to reverse-engineer the intended contract from the implementation rather than the contract.

### Issue 2 of 2
pkg/source/source.go:158-172
**`DurableCommitID any` vs `DataBatchID string` type gap**

`RecordBatchResult.DurableCommitID` and `DurableCheckpointID` are declared `any`, while `CDCStateCommitToken.DataBatchID` — the durable serialized form — is `string`. The strategy layer in #963 will have to do an unchecked type assertion (e.g. `id.(string)`) each time it bridges the two; if a future source sets either field to a non-string, the assertion panics at runtime with no compile-time protection. A narrow helper or a typed alias (even `type DurableID = string`) at the `RecordBatchResult` level would close this gap.

_{Reviews (1): Last reviewed commit: "define durable ingestion commit contract..." | Re-trigger Greptile}

[greptile-apps]

_{Reviews (2): Last reviewed commit: "define durable ingestion commit contract..." | Re-trigger Greptile}

[greptile-apps]

_{Reviews (3): Last reviewed commit: "define durable ingestion commit contract..." | Re-trigger Greptile}