Skip to content

Commit d0d184b

Browse files
authored
backup: Phase 0b M5-3 impl - SQS partitioned-FIFO reverse encoder (#929)
## Summary Phase 0b M5-3 implementation — SQS partitioned-FIFO reverse encoder. The design doc landed via PR #914 (merged 2026-06-03). This PR carries the implementation that the doc described, in 4 commits matching the slice plan: - **Slice A+B** (`235c13ba`): decoder partition plumbing — `parseSQSMessageDataKey` widens to `(enc, partition, isPartitioned, err)`, `sqsMessageRecord.Partition *uint32`, `HandleMessageData` wires the parsed partition. - **Slice C** (`6640674a`): mirror the four partitioned key constructors (`sqsPartitionedMsgDataKeyBytes`, `…Vis…`, `…ByAge…`, `…Dedup…`) plus `effectivePartitionCount` and the `"perQueue"` constant. M3b-3 circular-dep pattern — `internal/backup/` cannot import `adapter/`. - **Slice D** (`1037deba`): drop `ErrSQSEncodeUnsupportedPartitioned`, add four new sentinels (`ErrSQSEncodeMissingPartition`, `ErrSQSEncodeOutOfRangePartition`, `ErrSQSEncodePartitionRoutingMismatch`), add `validatePartitioning` + `sortMessagesForPartitionedEmit`, thread `partition *uint32` through `addMessage` and `addSideRecords`. - **Slice E** (`7603bf78`): 10 tests — the 8 design-doc rows plus `TestSQSEncodeGateUsesRawPartitionCount` (codex P2 v914 v7 regression) and `TestSQSEncodePartitionedSortStableAcrossPartitions`. ## Validation gates All four use **raw `meta.PartitionCount > 1`** as the partitioned-queue predicate, never `effectivePartitionCount`. Codex P2 v914 v7 demonstrated that effective-count gates would silently allow a `perQueue` + `PartitionCount=2` + `Partition==nil` dump to slip past validation, then `addMessage`'s `partition != nil` branch would emit classic-shape keys against a partitioned-keyspace queue (silent data loss on first read). | Gate | Predicate | Sentinel | |---|---|---| | missing partition | `PartitionCount > 1 && Partition == nil` | `ErrSQSEncodeMissingPartition` | | out-of-range | `PartitionCount > 1 && *Partition >= PartitionCount` | `ErrSQSEncodeOutOfRangePartition` | | classic + nonzero | `PartitionCount <= 1 && *Partition != 0` | `ErrSQSEncodeInvalidMessage` | | perQueue + nonzero | `PartitionCount > 1 && perQueue && *Partition != 0` | `ErrSQSEncodePartitionRoutingMismatch` | ## 5-lens self-review 1. **Data loss** — All four gates enforce key-shape ↔ predicate consistency. Classic-queue tests stay green (`Partition==nil` + `PartitionCount<=1` takes legacy path). 2. **Concurrency** — Offline encoder; no new goroutines / locks / shared state. 3. **Performance** — One extra O(N) pre-walk over `records`. No extra allocations. `effectivePartitionCount` not on the hot path. 4. **Data consistency** — Raw `PartitionCount > 1` selects key shape in both this encoder AND the live `adapter/sqs_keys_dispatch.go` — they agree. Partitioned dedup key includes `<group-seg>+|+<dedup-seg>` (CodeRabbit major PR #732 round 6). 5. **Test coverage** — 10 new tests; 4 are explicit gate regressions for prior review findings. ## Test plan - [x] `go test ./internal/backup/` passes - [x] `golangci-lint run --config=.golangci.yaml ./internal/backup/` 0 issues - [ ] Bot review cycle <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Full support for backing up and restoring partitioned SQS queues, preserving partition attribution and stable partitioned ordering. * **Behavior Changes** * Refined partition validation with distinct, actionable validation errors and stricter power-of-two checks. * Encoder now pre-sorts and stages partitioned messages to ensure stable FIFO sequence computation. * **Tests** * Extensive tests covering partitioned encoding, validation gates, dedup collisions, and round-trip semantics. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents c4924d9 + d36c398 commit d0d184b

6 files changed

Lines changed: 1376 additions & 61 deletions

File tree

0 commit comments

Comments
 (0)