Skip to content

Commit 797d571

Browse files
committed
docs(design): address Claude round-4 review on logical backup proposal
Important (verified against adapter/redis.go): - Stream TTLs were silently dropped. Streams can carry TTL via EXPIRE/PEXPIRE/PEXPIREAT, kept in !redis|ttl| by buildTTLElems (adapter/redis.go:2471). Added expire_at_ms to the _meta line so restore re-applies the original epoch. - HLL TTLs were silently dropped. HLL is reported as redisTypeString but stores TTL in the legacy scan index (adapter/redis.go:2319-2320 comment confirms it). Added hll_ttl.jsonl sidecar at the db_<n> root, same shape as strings_ttl.jsonl. Sidecar is omitted when no HLL key has a TTL. - Fixed XADD NOMKSTREAM semantic error in the skip-existing description. NOMKSTREAM only prevents stream creation; it does not skip entries by ID. Replaced with the correct mechanism: attempt XADD and treat the "ID equal or smaller" ERR as "already present, skip." Minor: - Added --begin-backup-deadline 5s to the elastickv-backup dump CLI block (the prose mentioned it but the flag list lagged). Tests added: TestRedisStreamTTLRoundTrip, TestRedisHLLTTLRoundTrip, TestRedisStreamSkipExistingHandlesERR.
1 parent e3b2936 commit 797d571

1 file changed

Lines changed: 38 additions & 8 deletions

File tree

docs/design/2026_04_29_proposed_logical_backup.md

Lines changed: 38 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -141,7 +141,8 @@ backup-<utc-timestamp>-<cluster-id>-<commit-ts>/
141141
│ ├── sets/<key>.json
142142
│ ├── zsets/<key>.json
143143
│ ├── streams/<key>.jsonl
144-
│ └── hll/<key>.bin # HyperLogLog opaque sketch (!redis|hll|<key>)
144+
│ ├── hll/<key>.bin # HyperLogLog opaque sketch (!redis|hll|<key>)
145+
│ └── hll_ttl.jsonl # HLL TTL sidecar (omitted when empty)
145146
└── sqs/
146147
└── <queue-name>/
147148
├── _queue.json
@@ -438,8 +439,9 @@ redis/
438439
│ └── leaderboard.json
439440
├── streams/
440441
│ └── events.jsonl
441-
└── hll/
442-
└── pfcount%3Auniques.bin
442+
├── hll/
443+
│ └── pfcount%3Auniques.bin
444+
└── hll_ttl.jsonl # one line per TTL'd HLL key
443445
```
444446

445447
Redis values are encoded so that `redis-cli --pipe` (or the equivalent in
@@ -486,14 +488,29 @@ any client library) can replay them without elastickv.
486488
```
487489
followed (optionally) by a trailing meta line:
488490
```
489-
{"_meta": true, "length": 2, "last_ms": 1714400000001, "last_seq": 0}
491+
{"_meta": true, "length": 2, "last_ms": 1714400000001, "last_seq": 0, "expire_at_ms": null}
490492
```
491493
The meta line lets a restore tool seed `XADD` IDs without re-deriving
492-
them from the entries.
494+
them from the entries. `expire_at_ms` is null when the stream has no
495+
TTL and is the absolute Unix-millis expiry otherwise — Redis streams
496+
can have TTLs set with `EXPIRE` / `PEXPIRE` / `PEXPIREAT`, kept in the
497+
`!redis|ttl|` scan index by `buildTTLElems` (`adapter/redis.go:2471`).
498+
Without `expire_at_ms` here, restoring a TTL'd stream would silently
499+
produce a permanent stream.
493500
- HyperLogLog (`!redis|hll|<key>`) is a binary opaque sketch; written under
494501
`hll/<key>.bin` byte-for-byte. A non-elastickv consumer that does not
495502
know HLL can still copy the bytes; a restore back into elastickv (or a
496-
Redis-compatible HLL implementation) reads them as-is.
503+
Redis-compatible HLL implementation) reads them as-is. HLL keys can
504+
also carry a TTL — the adapter reports HLL as `redisTypeString` but
505+
stores the TTL in `!redis|ttl|` rather than inline (see comment at
506+
`adapter/redis.go:2319-2320`). The TTL therefore lives in a
507+
`hll_ttl.jsonl` sidecar at the `db_<n>` root, in the same shape as
508+
`strings_ttl.jsonl`:
509+
```
510+
{"key":"pfcount%3Auniques","expire_at_ms":1735689600000}
511+
```
512+
A key with no TTL has no entry in the sidecar; the sidecar file is
513+
absent entirely when no HLL key in the database has a TTL.
497514

498515
Bitmaps and binary strings flow through the `strings/` path and remain
499516
raw bytes.
@@ -843,6 +860,7 @@ elastickv-backup dump \
843860
[--include-sqs-side-records] \
844861
[--checksums sha256] \
845862
[--ttl-ms 1800000] \
863+
[--begin-backup-deadline 5s] \
846864
[--scan-page-size 1024] \
847865
[--dynamodb-bundle-mode per-item|jsonl] \
848866
[--dynamodb-bundle-size 64MiB] \
@@ -971,8 +989,17 @@ parser, the format has failed its goal.
971989
The restore tool's stream behavior:
972990
- `--mode replace` — `DEL` the stream, then `XADD` every entry with
973991
the original ID (matches the dump's `_meta` line).
974-
- `--mode skip-existing` — `XADD NOMKSTREAM` only entries whose ID
975-
is not already present.
992+
- `--mode skip-existing` — for each entry, attempt
993+
`XADD <key> <id> ...` and treat the
994+
`ERR The ID specified in XADD is equal or smaller than the target
995+
stream top item` response as "already present, skip." `NOMKSTREAM`
996+
is **not** a skip-existing mechanism — it only suppresses stream
997+
creation; it does not skip entries by ID. A `XRANGE <key> <id>
998+
<id> COUNT 1` probe per entry is an alternative implementation,
999+
but is the same number of round-trips for the worst case (full
1000+
overlap) and is slower for the common case (small overlap), so
1001+
the `XADD`-and-handle-error form is preferred. The cost is O(N)
1002+
in the entry count regardless.
9761003
- `--mode merge` — refuses to operate on streams unless the target
9771004
is empty; emits an error pointing at the conflict. There is no
9781005
sound way to splice mid-stream without losing the original IDs.
@@ -1088,6 +1115,9 @@ Scope: out of this proposal; mentioned only to draw the boundary.
10881115
| `TestSQSPreserveVisibilityFlag` | Default leaves messages immediately visible on restore; `--preserve-visibility` retains in-flight receipts |
10891116
| `TestRedisTTLExpiredKeySkippedByDefault` | A key whose `expire_at_ms` is in the past at restore time is not re-applied without `--preserve-ttl`; with the flag it is re-applied with the original epoch |
10901117
| `TestRedisStreamMergeRejectsNonEmpty` | `--mode merge` on a non-empty target stream errors out; `--stream-merge-strategy=auto-id` falls back to `XADD *` and writes `_id_remap.jsonl` |
1118+
| `TestRedisStreamTTLRoundTrip` | A stream with `PEXPIREAT` round-trips through dump and restore: `expire_at_ms` is captured in the `_meta` line and re-applied so the restored stream expires at the original epoch |
1119+
| `TestRedisHLLTTLRoundTrip` | A TTL'd HLL key surfaces in `hll_ttl.jsonl` and is re-applied on restore via the same `EXPIREAT` path used for strings; no-TTL HLLs leave `hll_ttl.jsonl` absent |
1120+
| `TestRedisStreamSkipExistingHandlesERR` | The restore tool's `skip-existing` path tolerates `ERR The ID specified in XADD is equal or smaller` without aborting the dump; entries with non-conflicting IDs are still applied |
10911121

10921122
### P2
10931123

0 commit comments

Comments
 (0)