26 enhanced kv store key value experience#80
Conversation
|
/crush_fast AI review started. |
|
/review AI review started. |
Advanced AI Review
Click to expand reviewI'll review this PR thoroughly. Let me start by exploring the codebase structure and reading the key files.Now let me look at the constants, types, and errors files:Now let me look at the types file and the chelonia.ts kv/set and kv/get changes:Now let me look at the chelonia.ts kv/set implementation in detail:Let me look at the events.ts, index.ts and the chelonia.ts connect function in more detail:Now let me look at the pubsub index to check the PUBSUB_RECONNECTION_SUCCEEDED constant and the kvFilter:Let me search for some key patterns to understand the code better:Let me check a few more specific things:Now let me check one more specific area — the duplicated validation logic in Code Review: KV Slots API1. 🔴 DRY violation: duplicated reducer-output validation in
|
|
/review AI review started. |
Advanced AI Review
Click to expand reviewI'll start by examining the key files in this PR to provide a thorough review. Let me look at the new Review: PR #26 — Enhanced KV Store Key-Value ExperienceIssue 1 🔴
Star-export ambiguity makes
export * from './kv-constants.js'
export * from './kv.js'And export {
KV_AUTO_LOAD,
KV_DEFAULT_ENCRYPTION_KEY_NAME,
KV_DEFAULT_SIGNING_KEY_NAME,
KV_LOAD_STATUS,
KV_NOOP,
KV_UPDATE_REASON,
KV_VALIDATION_REASON_REVALIDATE
} from './kv-constants.js'
export type { KvNoop } from './kv-constants.js'Under the ES module spec (which TypeScript follows for import { KV_NOOP } from '@chelonia/lib'will get a compile error: Fix — remove the redundant // src/index.ts — remove this line:
// export * from './kv-constants.js'Internal-only constants ( Issue 2 🟡
Three typos in comments inside the rewritten
// All of these situations should trigger parsing the respinse and
// conlict resolution"respinse" → "response", "conlict" → "conflict".
// for subsequent iterations that require conflic resolution."conflic" → "conflict". Issue 3 🟡
Both should be added to the error table: | `ChelErrorInvalidMessageHeight` | `parseEncryptedOrUnencryptedMessage` received a message whose height is outside `[0, currentHeight]`. |
| `ChelErrorKvReentrant` | A KV write selector (`update`/`clear`/`sync`) was called synchronously from within the same contract's `onUpdate` callback, which would deadlock the per-contract queue lane. |Issue 4 🟡
DRY violation: reducer-output validation logic is duplicated between the first-attempt path and the conflict-retry The following validation sequence appears twice in This is ~50 lines of near-identical code, differing only in variable names, error message suffixes ("threw" vs "threw on retry"), and the Suggestion — extract a shared helper: function validateReducerOutput (
slot: SlotDefinition,
contractID: string,
key: string,
output: unknown,
context: 'first-attempt' | 'conflict-retry'
): { noop: true } | { noop: false; value: JSONType } {
if (typeof output === 'symbol') {
if (output === KV_NOOP) return { noop: true }
throw new ChelErrorKvUpdateInvalid(
`[chelonia/kv] update: ${contractID}::${key} reducer returned ` +
`an unexpected symbol${context === 'conflict-retry' ? ' on retry' : ''}; use KV_NOOP to abort`
)
}
if (output === null || output === undefined) {
throw new ChelErrorKvUpdateInvalid(
`[chelonia/kv] update: ${contractID}::${key} reducer returned ` +
`${String(output)}${context === 'conflict-retry' ? ' on retry' : ''}; use chelonia/kv/clear or KV_NOOP instead`
)
}
const label = context === 'conflict-retry'
? `update onconflict retry ${contractID}::${key}`
: `update ${contractID}::${key}`
const value = slot.schema
? parseSyncSlotValue(slot, output, label)
: assertJsonShape(output, label)
return { noop: false, value }
}Then each call site becomes: // First attempt
const result = validateReducerOutput(slot, contractID, key, reducerOut, 'first-attempt')
if (result.noop) return undefined
let nextValue = result.value
// Conflict retry
const result = validateReducerOutput(slot, contractID, key, retried, 'conflict-retry')
if (result.noop) throw new KvNoopAbort()
nextValue = result.valueIssue 5 🟡
The basis-selection logic in
Suggestion — extract a shared function resolveReducerBasis (
slot: SlotDefinition,
rawValue: JSONType | undefined | null,
contractID: string,
key: string,
context: 'seed' | 'conflict',
clone: boolean
): JSONType | undefined {
if (rawValue === undefined || rawValue === null) {
return slot.resolvedDefault !== undefined
? cloneDeep(slot.resolvedDefault)
: undefined
}
const label = context === 'conflict'
? `update onconflict currentData ${contractID}::${key}`
: `update seed ${contractID}::${key}`
const parsed = slot.schema
? parseSyncSlotValue(slot, rawValue, label)
: assertJsonShape(rawValue, label)
return clone ? cloneDeep(parsed) : parsed
}Issue 6 🟡
| `ChelErrorKvValidation` | Reducer output failed `schema.parse`. |This error is thrown in seven distinct paths, not just reducer output validation:
The description should be broadened to match: | `ChelErrorKvValidation` | A slot value failed `schema.parse` or `assertJsonShape`. Thrown by `update` (reducer output or server `currentData` on conflict), `_loadSlotNow` (GET response decode/shape failure), and `revalidateMirrorEntry`. Companion event: `CHELONIA_KV_VALIDATION_ERROR` for load/remote/re-validate failures (local reducer-output failures reject the `update` promise instead). |Issue 7 ⚪️
No Unlike "./kv-constants": {
"import": {
"types": "./dist/esm/kv-constants.d.mts",
"default": "./dist/esm/kv-constants.mjs"
},
"require": {
"types": "./dist/cjs/kv-constants.d.cts",
"default": "./dist/cjs/kv-constants.cjs"
}
}Issue 8 ⚪️
Type mismatch:
const onconflict = async (conflictArgs: {
currentData?: JSONType;
etag?: string | null;
}): Promise<[JSONType, string | undefined]> => {But currentData: JSONType | undefined;(required key, though the value may be Issue 9 ⚪️
this.kvContractsModifiedListener = (
_contracts: Set<string>,
payload: { added: string[]; removed: string[] }
) => {But all _contracts: string[],Review generated using |
|
/review AI review started. |
Advanced AI Review
Click to expand reviewI'll review this large KV slots PR thoroughly. Let me start by examining the key files.Now let me read the constants and types files in parallel:Now let me check the chelonia.ts KV-related changes (the kv/set, kv/get, and the connect/reset paths):Let me run the tests to verify the implementation works:Now let me investigate specific concerns more deeply. Let me check the I've thoroughly reviewed the diff — Issue 1 — 🟡 DRY: repeated validation-failure pattern in
|
|
Fable review ready! DetailsCode ReviewBase: Verification performed
Compatibility assessment (changes outside
|
| Change | Verdict |
|---|---|
chelonia/kv/set resolves to { etag: string | null } (was void); accepts signal, forwards maxAttempts |
Additive. All existing callers ignore the return value. |
chelonia/kv/set conflict exhaustion now throws ChelErrorKvMaxAttempts (internal class) instead of plain Error |
Same message string ('kv/set conflict setting KV value'); only .name/prototype differ. Documented in internal-errors.ts. Callers matching on message are unaffected. |
chelonia/kv/set recovery GET on body-less 409/412 (chelonia.ts, recoveryGetAttempted) |
New behavior, strictly better for direct callers: onconflict now receives a recovered etag/currentData instead of undefined, bounded to one GET per call (locked in by chelonia-kv-set.test.ts). |
Conflict with no onconflict handler now throws before the 0–1500 ms random delay instead of after |
Same error, faster failure. Not observable except in timing. |
chelonia/kv/get returns ChelKvGetResult (parsed message + lazily attached etag) |
Additive; defineProperty on the (unfrozen) result object, laziness of the data accessor preserved. |
chelonia/kv/queuedSet forwards signal/maxAttempts, returns { etag } |
Additive. |
ChelKvOnConflictCallback return type widened to [JSONType, string | undefined] | false |
Type-level only. The runtime if (!result) return false falsy-abort path already existed on main; the widened type documents pre-existing behavior (the "behaviour change" note in types.ts slightly overstates this — see issue 6). |
NOTIFICATION_TYPE.KV message handler restructured (chelonia.ts:1170+) |
Verified: a consumer-provided raw KV handler is still invoked with [key, parsed] inside the same per-contract queueInvocation lane; the slot layer runs in addition. When neither a raw handler nor the kv module exists, the handler returns before parsing. No regression. |
parseEncryptedOrUnencryptedMessage throws ChelErrorInvalidMessageHeight (was plain Error) |
Name change only; exported and documented in docs/api.md. |
chelonia/externalStateSetup now uses sbp(stateSelector) in the CONTRACTS_MODIFIED handler (was hardcoded state/vuex/state) |
Bug fix. Behavior changes only for consumers that passed a stateSelector different from 'state/vuex/state' — previously their contract projections silently went to Vuex. Worth a line in release notes. |
chelonia/externalStateSetup returns a teardown function |
Additive (previously returned undefined). |
chelonia/reset: abort moved before postCleanupFn, KV writes drained via _waitInFlight |
In-flight fetches are now aborted earlier (before postCleanupFn instead of after). All contract waits still precede it; low risk. |
chelonia/contract/fullState gains optional key param and kvState/kvEntry result fields |
Additive. |
chelonia/defineContract throws if contract.kv is present and the kv module isn't loaded |
Only affects new-style contracts; correct fail-fast. |
createKvMessage gains optional cid; NOTIFICATION_TYPE.KV msg type gains cid? |
Additive. Old servers that emit frames without cid degrade gracefully (authoritative-GET path, documented in docs/api.md). |
index.ts import reordering (chelonia.ts last, because it locks the chelonia SBP domain) |
Consistent with the pre-existing constraint that already applied to ./journal, ./files, etc. Deep-importing ./chelonia before ./kv silently drops the kv selector registration (SBP warns) — pre-existing pattern, now explicitly documented in index.test.ts. |
Cross-cutting wiring verified specifically:
- Filter restoration across connect/reconnect:
chelonia/connectpre-seedsclient.kvFiltersynchronously before the socket can open, and the pubsub client readskvFilterlazily at SUB-send time (with a KV_FILTER re-send on SUB-ack mismatch), so filters survive first connect and every reconnect. The direct-map-mutation idiom atchelonia.ts:1244is only safe because the socket is guaranteed not-yet-open there (it does not transmit) — fine as used, but do not reuse that idiom against a connected client. CONTRACTS_MODIFIED: all four emit sites passArray.from(this.subscriptionSet)(matches the new listener typing), mutatesubscriptionSetbefore emitting (the design's ordering claim holds), and the direct_cleanupContractRuntimecall inremoveImmediatelyplus the listener-driven call are correctly idempotent.- Self-echo suppression / conflict-marker state machine (
_handleRemote): traced both arrival orderings (echo-first and competing-frame-first), the demote-don't-delete rule, TTL expiry, and the eviction cap; no mirror-regression path found. Suppression is recorded before the abort/staleness checks inupdate/clear, so a committed write's echo can never re-validate against a replaced slot. - Lane serialization: mirror reads, reducers,
kv/set,_handleRemote, andonUpdateall run inside the per-contractqueueInvocationlane;_waitInFlight's three-source drain (index, echo CIDs, pending-write counters) covers the queued-but-not-started window.
The issues found follow, in priority order.
1. 🟡 Filter flush enters an infinite 2-second retry loop if slots reconcile before chelonia/connect
- Addressed
- Dismissed
chelonia/kv/setFilter dereferences this.pubsub unguarded (src/chelonia.ts:3023-3025):
'chelonia/kv/setFilter': function (this: CheloniaContext, contractID: string, filter?: string[]) {
this.pubsub.setKvFilter(contractID, filter)
},this.pubsub is only assigned inside chelonia/connect (src/chelonia.ts:1086). Before this PR that was harmless — only consumers called setFilter, and they did so after connecting. Now the slot layer calls it autonomously: defineSlot reconciles against already-synced contracts and queues a filter flush (src/kv.ts:1398-1408 → queueFilterFlush → flushDirtyFilters). If an app syncs contracts (HTTP-only, works without a websocket) and registers slots before calling chelonia/connect — or never connects at all (SSR / CLI / tooling contexts) — the flush throws TypeError: Cannot read properties of undefined (reading 'setKvFilter'), which is caught at src/kv.ts:918-928 and routed to scheduleFilterRetry:
} catch (e) {
console.warn(`[chelonia/kv] setFilter flush failed for ${cID}`, e)
...
scheduleFilterRetry(ctx, cID)
}scheduleFilterRetry (src/kv.ts:942-956) re-queues the contract and re-flushes after KV_FILTER_RETRY_MS (2 s), fails identically, and re-enters itself — an unbounded retry loop that (a) spams one console.warn per contract every 2 seconds and (b) keeps a live setTimeout chain that prevents a Node process from ever exiting. Nothing distinguishes this permanent precondition failure from the transient server errors the retry was designed for.
It self-heals only if chelonia/connect is eventually called (the retry then succeeds), which is why tests (which stub setFilter) and Group Income (which connects at boot) never hit it.
Suggested fix — treat "no pubsub yet" as "nothing to flush" rather than a failure, since chelonia/connect already re-applies the complete filter set from kvActiveFilters (src/chelonia.ts:1242-1246):
// in flushDirtyFilters, before the setFilter call (or at loop entry):
if (!ctx.pubsub) {
// chelonia/connect re-seeds client.kvFilter from kvActiveFilters,
// so pre-connect flushes are safely dropped.
ctx.kvFilterDirty.clear()
return
}Alternatively guard inside chelonia/kv/setFilter itself (if (!this.pubsub) return). Also consider .unref?.() on kvFilterRetryTimer so a genuinely-retrying timer cannot pin a Node process open.
2. 🟡 docs/api.md contradicts the implemented 'error'-status seeding behavior of chelonia/kv/update
- Addressed
- Dismissed
docs/api.md (added in this PR, "rejection taxonomy" section) states:
When a slot is in
'error'status,updateseeds the reducer the same wayreaddoes: from the declared default, not from the retained mirror value. The mirror may still hold the last valid value and etag after a validation failure, but the reducer does not observe that retained value until a successful load, sync, remote update, or local write transitions the slot out of'error'.
This describes an earlier design. The implementation (and AGENTS.md's "Data-loss-guard reload (status quiet)" section) do the opposite for the retained-value case: chelonia/kv/update on an 'error' slot that still holds a value performs a silent authoritative reload and then seeds the reducer from the mirror — the live server value on reload success, or the retained value on reload failure (src/kv.ts:2492-2555, computeSeedValue at src/kv.ts:365-381):
if (mirrorEntry && mirrorEntry.status === KV_LOAD_STATUS.ERROR && mirrorEntry.value !== undefined) {
reloadRan = true
... // silent _loadSlotNow; on failure, seed from retained value
}
const seedValue = computeSeedValue(slot, mirrorEntry, reloadRan)Only the 'error'-with-no-retained-value case seeds from the default. The implemented behavior is the right one (the doc's described behavior is exactly the silent-data-loss scenario the reload guard exists to prevent), so the fix is to rewrite this api.md paragraph to match AGENTS.md — e.g. "When a slot is in 'error' status but still holds a retained value, update first performs one silent authoritative reload and seeds the reducer from the refreshed (or retained) mirror value; only an 'error' slot with no retained value seeds from the declared default."
While editing docs: AGENTS.md's config table says autoLoad: 'on-demand' "waits for read/sync" — chelonia/kv/read never triggers a fetch (it synchronously returns the default); only sync or a successful update materializes the value. docs/api.md's selector table has the same implication for read. Both should say "waits for sync (or a successful update)".
3. ⚪️ Released contracts leave a stale empty kvFilter entry on the pubsub client, silencing raw-API KV frames after re-sync
- Addressed
- Dismissed
chelonia/kv/_cleanupContractRuntime (src/kv.ts:1910-1936) queues an empty-filter flush before dropping per-contract state, so the flush emits setFilter(cID, []). On the release path, chelonia/pubsub/update unsubscribes the channel synchronously (during the CONTRACTS_MODIFIED emit) before the flush's microtask runs, so setKvFilter (src/pubsub/index.ts:977-1000) takes the not-subscribed branch: it does not send a frame but it does record client.kvFilter.set(cID, []) — and an empty array means "deliver no KV keys". That map entry is never removed.
If the same contract is later re-synced, the SUB payload carries the stale kvFilter: [] (pubPayloadFactory, src/pubsub/index.ts:272-275). When active slots reconcile, the subsequent flush corrects it (via the SUB-ack mismatch path), so the slot API is unaffected. But if no slot matches the re-synced contract anymore — slot definitions removed via defineContract replacement, or every match returns false — the stale [] persists and the server delivers no KV frames for that contract, whereas a never-slotted contract receives all of them. A raw-API consumer (the legacy NOTIFICATION_TYPE.KV callback) would silently observe nothing.
The "Filter ownership: don't mix slots with raw setFilter" section of docs/api.md partially covers this, but the ownership claim there is per-contract-with-declared-slots, while this leak outlives the slots. Suggested fix: on the release path, remove the client-side filter entry instead of setting it empty — e.g. have _cleanupContractRuntime skip the flush-and-empty dance when the contract is leaving the subscription set entirely, and instead call chelonia/kv/setFilter(cID) (the documented remove-filters form) or delete from client.kvFilter directly, restoring the "no filter = all keys" default for a future unslotted re-subscribe.
4. ⚪️ update/clear aborted after the server write commits leave the mirror stale indefinitely (by design, but undocumented for consumers)
- Addressed
- Dismissed
When the caller's signal aborts in the window after chelonia/kv/set resolves, update records the echo CID first and then rejects with AbortError (src/kv.ts:2726-2733; same pattern in clear at src/kv.ts:3016-3031). The write has committed server-side, its pubsub echo is deliberately suppressed, and the §4.2 abort contract ("mirror unchanged, no event") is honored — with the consequence that this client's mirror (value and etag) stays stale until the next remote frame, local write (whose stale ifMatch then round-trips through a 412), or explicit sync. Other clients see the new value; the aborting client does not, potentially forever on a quiet slot.
The in-code rationale (src/kv.ts:2711-2725) is sound and I found no better alternative that keeps the abort contract, but neither AGENTS.md's abort discussion nor docs/api.md's taxonomy row ("AbortError — mirror is unchanged; no event fires") tells the consumer that the write may nonetheless have been persisted and the mirror may diverge from the server afterwards. Since aborting near the commit boundary is precisely when a caller most needs to know this, add a sentence to both: "an abort that lands after the server accepted the write leaves the write committed; the mirror intentionally does not reflect it until the next load/remote/write — call chelonia/kv/sync after aborting if you need the mirror reconciled."
5. ⚪️ Dead branch in assertJsonShape
- Addressed
- Dismissed
src/kv.ts:317-323:
if (value === null || value === undefined) {
throw new ChelErrorKvValidation(...)
}
const t = typeof value
if (t === 'string' || t === 'boolean' || t === 'undefined') returnt === 'undefined' is unreachable — value === undefined already threw four lines above. Harmless, but the dead condition reads as if undefined were accepted, which is the opposite of the invariant this function enforces. Drop || t === 'undefined'.
6. ⚪️ types.ts doc comment overstates the onconflict falsy-abort as a behavior change
- Addressed
- Dismissed
The new doc block on ChelKvOnConflictCallback (src/types.ts, "NOTE: this is a behaviour change vs. pre-KV-revamp consumers, who could only return a tuple… will now silently no-op") is inaccurate: the runtime if (!result) return false abort path already existed on main (it is unchanged context in this diff, resolveData in src/chelonia.ts). Only the type was narrowed before; a falsy return already silently aborted the write pre-revamp rather than "historically threw downstream". The same claim is repeated in docs/api.md's migration notes ("Pre-revamp consumers that accidentally returned a non-tuple falsy value would have crashed downstream; they now silently no-op"). Both should say the type now documents pre-existing runtime behavior — otherwise a direct caller auditing per these notes will hunt for a runtime change that never happened.
Notes (no action required)
src/kv.tsoverall: the implementation is unusually defensive and internally consistent — staleness guards on every await boundary, the §11.2 index invariant enforced by a dev-mode checker, non-destructive mirror reads to avoid orphaned records, deep-clone discipline on every value crossing the mirror boundary, and a sound synchronous-window re-entrancy guard foronUpdate. The conflict-marker demote-don't-delete logic in_handleRemotecorrectly handles both echo orderings.- The design doc's §11.6 asks for
_assertIndexConsistentin every test'safterEach; the suite instead calls it at targeted points (5 call sites). Acceptable, but a globalafterEachwould be cheap insurance. chelonia/kv/getstill treats HTTP 410 as an error whilechelonia/kv/settreats 404/410 alike as "no data" — a pre-existing asymmetry this PR neither introduced nor fixed; a released-then-GC'd contract's slot load will surface'error'rather than'non-init'.- The
cidfield on pubsub KV frames requires a matching server-side change to be emitted; until servers send it, every remote frame on an etag-bearing slot takes the authoritative-GET path (one extra GET per frame). This is documented indocs/api.mdand degrades gracefully, but is worth remembering when sequencing the server deploy.
|
Ready |
Closes #26
KV-REVAMPED.mdKV-REVAMPED.mdKV-REVAMPED.md contains the spec used (which this PR should conform to).
AI disclosure
GPT-5.5, Opus 4.7 and GLM 5.1 were used in generating code for this PR. GPT-5.5, Opus 4.7 & GLM 5.2 were used for reviews.