Skip to content

Commit 6aa78e9

Browse files
committed
feat: add source/cdc change-event codec
Decode the Debezium/OpenCDC JSON change-event envelope into a typed ChangeEvent (Operation enum, deferred before/after row images, decoded source metadata, commit timestamp) with BeforeAs/AfterAs projection and SourceHeaders. Tombstones decode to OpTombstone; malformed envelopes return *DecodeError (poison). Ships as a stdlib subpackage of the source module alongside retry/dlq, with a memsource consume test and a CDC-driven Drive example in source/statemachine. Native WAL connectors stay future work. Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
1 parent dc32523 commit 6aa78e9

9 files changed

Lines changed: 1141 additions & 8 deletions

File tree

README.md

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,7 @@ stability label.
6363
| `source/jetstream` | NATS JetStream Inlet over nats.go: pull consumer, ack/nak/term, MaxAckPending. | experimental |
6464
| `source/redis` | Redis Streams Inlet over go-redis: consumer group, XACK/pending-claim, DLQ. | experimental |
6565
| `source/cloudevents` | CloudEvents codec with structured and binary content modes. | experimental |
66+
| `source/cdc` | Change-data-capture codec: decode Debezium/OpenCDC change events, drive by key. | experimental |
6667
| `source/statemachine` | Bridge: an inbound message drives a transition, ack tied to the durable commit. | experimental |
6768

6869
source also ships composable reliability middleware as its own opt-in modules
@@ -83,8 +84,9 @@ snapshots; inspection; and JSON (de)serialization. It is backed by its `analysis
8384
until it reaches v1. The `telemetry` interface and its `slog`, `otel`, and
8485
`datadog` adapters are released. The `sink` egress seam and its destination
8586
adapters, and the `source` ingress seam with its Kafka, JetStream, and Redis
86-
Streams adapters, CloudEvents codec, reliability middleware, and state-machine
87-
bridge, are now available and documented; the `broker` module is planned.
87+
Streams adapters, CloudEvents and CDC codecs, reliability middleware, and
88+
state-machine bridge, are now available and documented; the `broker` module is
89+
planned.
8890

8991
## Roadmap: event-driven seams
9092

@@ -99,7 +101,10 @@ and forcing nothing third-party on the consumer:
99101
[Docs](https://stablekernel.github.io/crucible/sink/overview/).
100102
- [x] **`source`**: ingress. Subscribe external streams and drive machines, with
101103
the ack tied to a durable transition; the symmetric counterpart to `sink`.
102-
[Docs](https://stablekernel.github.io/crucible/source/overview/).
104+
[Docs](https://stablekernel.github.io/crucible/source/overview/). The
105+
`source/cdc` codec decodes Debezium/OpenCDC change-event topics into typed
106+
change events; a native database write-ahead-log connector (logical replication
107+
slot, binlog) remains future work.
103108
- [ ] **`bellows`** _(exploring)_: resilience seam. Circuit-breaking and
104109
backpressure around the IO edges.
105110

docs/src/content/docs/source/codecs.md

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,3 +46,58 @@ The decoded type then flows into your handler, and for the
4646
[state-machine binding](/crucible/source/with-state/) it becomes the event the
4747
router fires. A decode failure is a typed `*DecodeError` the engine classifies
4848
as poison and routes to the [DLQ](/crucible/source/reliability/#dlq).
49+
50+
## cdc
51+
52+
`crucible/source/cdc` is the change-data-capture codec. It decodes the standard
53+
Debezium JSON change-event envelope (also the de-facto OpenCDC normalized record
54+
shape) into a typed `ChangeEvent`:
55+
56+
- an `Operation` (create, snapshot read, update, delete, or tombstone),
57+
- the `before` and `after` row images, kept as deferred JSON so one codec serves
58+
every table on a topic without binding to a row type at decode time,
59+
- a decoded `Source` metadata block (connector, database, schema, table,
60+
snapshot marker, log position, transaction id), and
61+
- the commit `Timestamp` the connector reported.
62+
63+
Recover the value with `DecodeEvent`, project a row image into a concrete type
64+
with `BeforeAs[T]` / `AfterAs[T]`, and read the source metadata as typed
65+
`source.Headers` through `SourceHeaders`. A log-compaction tombstone (an empty
66+
payload) decodes to an `OpTombstone` event rather than a decode failure, so a
67+
handler routes it (a delete-and-forget for the key) or skips it. A malformed
68+
envelope is a typed `*DecodeError` the engine classifies as poison.
69+
70+
### Scope: envelope plus topic pattern, not a native connector
71+
72+
This codec covers the change-event **envelope** and the pattern for driving a
73+
statechart from a change-event **topic**. The intended shape is to let an
74+
existing connector (Debezium, or any producer emitting the same envelope) write
75+
row changes to a topic, consume that topic through a backend inlet such as
76+
[`source/kafka`](/crucible/source/inlets/), decode each message with this codec,
77+
and drive a statechart per primary key through the
78+
[state-machine binding](/crucible/source/with-state/). Because the codec is
79+
instance-scoped, a service can run a CDC consumer alongside a plain-JSON or
80+
CloudEvents consumer with no shared decode state.
81+
82+
A native database write-ahead-log connector (reading a Postgres logical
83+
replication slot or a MySQL binlog directly, without a broker in between) is
84+
deliberately out of scope and tracked as future work. The codec gives you the
85+
typed change event; the connector that produces those events stays a separate,
86+
operational concern.
87+
88+
```go
89+
// Decode a Debezium change event, then route its after-image into a transition.
90+
registry := source.NewRegistry().SetDefault(cdc.New())
91+
92+
router := func(m source.Message) (Key, Event, error) {
93+
change, err := cdc.DecodeEvent(registry, m)
94+
if err != nil {
95+
return zeroKey, zeroEvent, err
96+
}
97+
row, err := cdc.AfterAs[Row](change)
98+
if err != nil {
99+
return zeroKey, zeroEvent, err
100+
}
101+
return keyOf(row), eventOf(change.Operation, row), nil
102+
}
103+
```

0 commit comments

Comments
 (0)