Skip to content

Commit e0df07e

Browse files
samwilliskevin-dpclaudeautofix-ci[bot]
authored
feat: add persisted sync metadata support (#1380)
* feat(examples): add wa-sqlite OPFS persistence demo to offline-transactions Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(examples): add Vite alias for db-sqlite-persisted-collection-core The browser wa-sqlite package source re-exports from the core package, so resolving from source requires both aliases for the build to succeed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix some type errors * ci: apply automated fixes * feat: add persisted sync metadata RFC Document the transactional metadata model for persisted collections, including row and collection metadata, query retention, and Electric resume state. Made-with: Cursor * feat: add persisted sync metadata implementation plan Break the persisted sync metadata RFC into phased implementation docs covering the core API, SQLite integration, query collection, Electric collection, and required invariants tests. Made-with: Cursor * docs: refine persisted sync metadata design docs Tighten the RFC and phased plan around startup metadata reads, query-owned reconciliation, cold-row retention cleanup, replay fallback behavior, and Electric reset semantics. Made-with: Cursor * feat: implement persisted sync metadata support Add transactional row and collection metadata plumbing across core sync state, SQLite persistence, query collections, and Electric resume state so persisted ownership and resume metadata survive restarts. Made-with: Cursor * ci: apply automated fixes * fix: align persisted metadata writes with sync startup Buffer persisted metadata writes within wrapper transactions and dedupe concurrent collection setup so warm starts no longer trip missing sync transaction errors or collection registry races. Made-with: Cursor * chore: remove persisted sync metadata docs from branch Drop the RFC and phased implementation plan from the branch while leaving the local working copies in place. Made-with: Cursor * fix: tighten persisted sync metadata restart behavior Finish the core persisted metadata follow-through so reloads, retained query ownership, and Electric resume/reset state behave correctly across startup and recovery while clarifying metadata semantics around inserts and cleanup. Made-with: Cursor * feat: complete persisted sync metadata coverage Finish the remaining persisted metadata work by adding cold-row retained query cleanup, runtime TTL expiry, stronger Electric resume identity checks, and metadata delta replay for follower recovery while keeping reload fallback for reset-like cases. Made-with: Cursor * ci: apply automated fixes * fix: encode replay deltas with persisted serializer Use the persisted JSON encoder for replay payloads so bigint and date values survive applied_tx serialization and package-level SQLite adapter tests pass under the CLI runtime. Made-with: Cursor * ci: apply automated fixes * fix: harden persisted startup and resume metadata handling Restore markReady fallback behavior on persisted startup errors, make load cancellation deterministic, and tighten migration/error handling and resume identity stability so persisted sync state survives edge cases without hanging or false reset fallback. Made-with: Cursor * chore: add changeset for persisted metadata follow-ups Add patch release notes for db, sqlite persisted core, and electric collection to cover startup readiness fallback, deterministic resume identity handling, and migration/truncate metadata hardening. Made-with: Cursor * ci: apply automated fixes * fix(electron-persistence): bridge metadata RPC methods across IPC Forward collection metadata and row scan adapter calls through the Electron protocol so sqlite adapter contract behavior matches direct-core persistence semantics. Made-with: Cursor * ci: apply automated fixes * chore(expo-e2e): add metro-runtime lockfile entry Record the Expo runtime dependency update in the workspace lockfile so Expo emulator e2e installs are reproducible in CI. Made-with: Cursor * fix(lockfile): drop stale metro-runtime specifier from expo e2e app Regenerate pnpm-lock.yaml to match the current expo runtime app package spec so frozen-lockfile installs succeed in CI. Made-with: Cursor --------- Co-authored-by: Kevin De Porre <kevin@electric-sql.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com>
1 parent ee81d15 commit e0df07e

17 files changed

Lines changed: 4380 additions & 161 deletions

File tree

‎.changeset/persisted-metadata-followups.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
---
2+
'@tanstack/db': patch
3+
'@tanstack/db-sqlite-persisted-collection-core': patch
4+
'@tanstack/electric-db-collection': patch
5+
---
6+
7+
fix(persistence): harden persisted startup, truncate metadata semantics, and resume identity matching
8+
9+
- Restore persisted wrapper `markReady` fallback behavior so startup failures do not leave collections stuck in loading state
10+
- Replace load cancellation reference identity tracking with deterministic load keys for `loadSubset` / `unloadSubset`
11+
- Document intentional truncate behavior where collection-scoped metadata writes are preserved across truncate transactions
12+
- Tighten SQLite `applied_tx` migration handling to only ignore duplicate-column add errors
13+
- Stabilize Electric shape identity serialization so persisted resume compatibility does not depend on object key insertion order

‎packages/db-electron-sqlite-persisted-collection/src/main.ts‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,19 @@ type ElectronMainPersistenceAdapter = PersistenceAdapter<
2020
ElectronPersistedRow,
2121
ElectronPersistedKey
2222
> & {
23+
loadCollectionMetadata?: (
24+
collectionId: string,
25+
) => Promise<Array<{ key: string; value: unknown }>>
26+
scanRows?: (
27+
collectionId: string,
28+
options?: { metadataOnly?: boolean },
29+
) => Promise<
30+
Array<{
31+
key: ElectronPersistedKey
32+
value: ElectronPersistedRow
33+
metadata?: unknown
34+
}>
35+
>
2336
pullSince?: (
2437
collectionId: string,
2538
fromRowVersion: number,
@@ -110,6 +123,41 @@ async function executeRequestAgainstAdapter(
110123
}
111124
}
112125

126+
case `loadCollectionMetadata`: {
127+
if (!adapter.loadCollectionMetadata) {
128+
throw new InvalidPersistedCollectionConfigError(
129+
`loadCollectionMetadata is not supported by the configured electron persistence adapter`,
130+
)
131+
}
132+
const result = await adapter.loadCollectionMetadata(request.collectionId)
133+
return {
134+
v: ELECTRON_PERSISTENCE_PROTOCOL_VERSION,
135+
requestId: request.requestId,
136+
method: request.method,
137+
ok: true,
138+
result,
139+
}
140+
}
141+
142+
case `scanRows`: {
143+
if (!adapter.scanRows) {
144+
throw new InvalidPersistedCollectionConfigError(
145+
`scanRows is not supported by the configured electron persistence adapter`,
146+
)
147+
}
148+
const result = await adapter.scanRows(
149+
request.collectionId,
150+
request.payload.options,
151+
)
152+
return {
153+
v: ELECTRON_PERSISTENCE_PROTOCOL_VERSION,
154+
requestId: request.requestId,
155+
method: request.method,
156+
ok: true,
157+
result,
158+
}
159+
}
160+
113161
case `applyCommittedTx`: {
114162
await adapter.applyCommittedTx(request.collectionId, request.payload.tx)
115163
return {

‎packages/db-electron-sqlite-persisted-collection/src/protocol.ts‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,8 @@ export type ElectronPersistenceResolution = {
1919

2020
export type ElectronPersistenceMethod =
2121
| `loadSubset`
22+
| `loadCollectionMetadata`
23+
| `scanRows`
2224
| `applyCommittedTx`
2325
| `ensureIndex`
2426
| `markIndexRemoved`
@@ -30,6 +32,12 @@ export type ElectronPersistencePayloadMap = {
3032
options: LoadSubsetOptions
3133
ctx?: { requiredIndexSignatures?: ReadonlyArray<string> }
3234
}
35+
loadCollectionMetadata: {}
36+
scanRows: {
37+
options?: {
38+
metadataOnly?: boolean
39+
}
40+
}
3341
applyCommittedTx: {
3442
tx: PersistedTx<ElectronPersistedRow, ElectronPersistedKey>
3543
}
@@ -48,6 +56,12 @@ export type ElectronPersistencePayloadMap = {
4856

4957
export type ElectronPersistenceResultMap = {
5058
loadSubset: Array<{ key: ElectronPersistedKey; value: ElectronPersistedRow }>
59+
loadCollectionMetadata: Array<{ key: string; value: unknown }>
60+
scanRows: Array<{
61+
key: ElectronPersistedKey
62+
value: ElectronPersistedRow
63+
metadata?: unknown
64+
}>
5165
applyCommittedTx: null
5266
ensureIndex: null
5367
markIndexRemoved: null

‎packages/db-electron-sqlite-persisted-collection/src/renderer.ts‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -153,6 +153,13 @@ type ElectronRendererResolvedAdapter<
153153
T extends object,
154154
TKey extends string | number = string | number,
155155
> = PersistedCollectionPersistence<T, TKey>[`adapter`] & {
156+
loadCollectionMetadata: (
157+
collectionId: string,
158+
) => Promise<Array<{ key: string; value: unknown }>>
159+
scanRows: (
160+
collectionId: string,
161+
options?: { metadataOnly?: boolean },
162+
) => Promise<Array<{ key: TKey; value: T; metadata?: unknown }>>
156163
pullSince: (
157164
collectionId: string,
158165
fromRowVersion: number,
@@ -202,6 +209,28 @@ function createResolvedRendererAdapter<
202209
resolution,
203210
)
204211
},
212+
loadCollectionMetadata: async (
213+
collectionId: string,
214+
): Promise<Array<{ key: string; value: unknown }>> => {
215+
return executeRequest(
216+
`loadCollectionMetadata`,
217+
collectionId,
218+
{},
219+
resolution,
220+
)
221+
},
222+
scanRows: async (
223+
collectionId: string,
224+
options?: { metadataOnly?: boolean },
225+
): Promise<Array<{ key: TKey; value: T; metadata?: unknown }>> => {
226+
const result = await executeRequest(
227+
`scanRows`,
228+
collectionId,
229+
{ options },
230+
resolution,
231+
)
232+
return result as Array<{ key: TKey; value: T; metadata?: unknown }>
233+
},
205234
ensureIndex: async (
206235
collectionId: string,
207236
signature: string,

‎packages/db-electron-sqlite-persisted-collection/tests/electron-ipc.test-d.ts‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,22 @@ test(`renderer persistence requires invoke transport`, () => {
3636
latestRowVersion: 0,
3737
},
3838
})
39+
case `loadCollectionMetadata`:
40+
return Promise.resolve({
41+
v: 1,
42+
requestId: request.requestId,
43+
method: request.method,
44+
ok: true,
45+
result: [],
46+
})
47+
case `scanRows`:
48+
return Promise.resolve({
49+
v: 1,
50+
requestId: request.requestId,
51+
method: request.method,
52+
ok: true,
53+
result: [],
54+
})
3955
default:
4056
return Promise.resolve({
4157
v: 1,

0 commit comments

Comments
 (0)