feat: preserve stores on schema version or rollup address change (#24631)

Changes store management so that changes in rollup addresses and schema
versions cause different stores to be used. Previously we only supported
one store to be present at a time, which meant any version or rollup
change wiped pre-existing stores (note this includes network changes).

An underlying design decision in this PR is to strip the kv-store
package from responsibility over location on disk, knowledge about
rollup addresses, etc, at least as as regards PXE and wallet storage.
Other users of LMDB-v2 should not be impacted by this change.

In consonance, IndexedDB and SQLite backends drop their `createStore`
functions, which shoehorned wallet and PXE store creation to an
homogeneous interface that made it hard to let them independently
evolve.

Since we're changing this, I decided to also include the chain id as a
component of the store id, in addition to the already present schema
version and rollup address. It's not clear that we'll ever work on a
testnet or a different L1, but doing so is trivial and removes the need
to deal with this in the future.

Closes F-809

Author: mverzilli

docs/docs-developers/docs/resources/migration_notes.md

@@ -9,11 +9,20 @@ Aztec is in active development. Each version may introduce breaking changes that
 
 ## TBD
 
-### [PXE] Local PXE database is reset on upgrade
+### [PXE] Stores are now selected by `(l1ChainId, rollupAddress, schemaVersion)` instead of being wiped on mismatch
 
-The persisted tagging stores now key every entry by the self-describing `<kind>:<secret>:<app>` form of `AppTaggingSecret`; unconstrained secrets previously used a two-part `<secret>:<app>` key. This bumps the PXE data schema version, and there is no forward migration for the old keys: on first open the PXE clears any database whose stored schema version differs from the current one. The wipe resets the entire PXE store, not just the tagging data, because all of it shares one backing database.
+Previously, connecting a PXE or embedded wallet to a different or redeployed rollup, or bumping the store schema version, wiped the existing on-disk store in place. That meant master account keys could be destroyed simply by pointing a wallet at a different network. PXE data stores now exist per `(l1ChainId, rollupAddress, schemaVersion)` triple, and switching networks (or upgrading) selects or creates the matching store instead of overwriting previous ones. The embedded wallet's `wallet_data` store is partitioned the same way, so accounts and aliases are per network: switching networks starts with an empty account list until accounts are re-imported, and switching back finds the originals intact.
 
-**Impact**: On upgrade your local PXE state is reset. You must re-register accounts and re-sync from genesis. Wallets should surface a "your local state was reset, please re-register accounts and re-sync" path.
+**Impact**: The first start after upgrading to this version begins with a fresh, empty store; the pre-upgrade data is not deleted. On Node.js environments (lmdb-v2) pre-upgrade data stays at `<dataDirectory>/<name>` while new per-identity `pxe_data` stores live under `<dataDirectory>/<name>-stores/`. The embedded Node.js wallet previously stored data in cwd-relative, rollup-address-suffixed directories instead (`pxe_data_<rollupAddress>/pxe_data` for the PXE store, `wallet_data_<rollupAddress>/wallet_data` for the wallet store): if you used it before this release, that is where the old data lives. The embedded wallet now defaults its data root to `aztec-wallet-data/`, with per-identity wallet stores under `<dataDirectory>/wallet_data-stores/` on Node.js and OPFS store names prefixed `wallet_data_` in the browser. Browser apps can enumerate and clean up `pxe_data` and `wallet_data` stores for networks no longer in use with the new `listStores()` / `deleteStore()` utilities:
+
+```ts
+import { deleteStore, listStores } from '@aztec/kv-store/sqlite-opfs';
+
+const names = await listStores();
+await deleteStore(names[0]);
+```
+
+This change also removes `createStore` from `@aztec/kv-store/sqlite-opfs` and `@aztec/kv-store/deprecated/indexeddb`: stores are now opened by name with `AztecSQLiteOPFSStore.open` / `AztecIndexedDBStore.open`.
 
 ### [Aztec.nr] `TestEnvironmentOptions::with_tagging_secret_strategy` replaced
 

playground/src/components/navbar/components/NetworkSelector.tsx

@@ -7,7 +7,7 @@ import AddIcon from '@mui/icons-material/Add';
 import { AddNetworksDialog } from './AddNetworkDialog';
 import CircularProgress from '@mui/material/CircularProgress';
 import KeyboardArrowDownIcon from '@mui/icons-material/KeyboardArrowDown';
-import { createStore } from '@aztec/kv-store/sqlite-opfs';
+import { AztecSQLiteOPFSStore, storePoolDirectory } from '@aztec/kv-store/sqlite-opfs';
 import { AztecContext } from '../../../aztecContext';
 import { navbarButtonStyle, navbarSelect } from '../../../styles/common';
 import { NETWORKS } from '../../../utils/networks';
@@ -50,10 +50,12 @@ export function NetworkSelector() {
       }
       setIsContextInitialized(true);
       WebLogger.create(setLogs, setTotalLogCount);
-      const store = await createStore('playground_data', {
-        dataDirectory: 'playground',
-        dataStoreMapSizeKb: 1e6,
-      });
+      const store = await AztecSQLiteOPFSStore.open(
+        WebLogger.getInstance().createLogger('playground_data'),
+        'playground_data',
+        false,
+        storePoolDirectory('playground_data'),
+      );
       const playgroundDB = PlaygroundDB.getInstance();
       playgroundDB.init(store, WebLogger.getInstance().createLogger('playground_db').info);
       setPlaygroundDB(PlaygroundDB.getInstance());

yarn-project/kv-store/src/deprecated/indexeddb/index.ts

@@ -1,34 +1,9 @@
-import { type Logger, createLogger } from '@aztec/foundation/log';
-import type { DataStoreConfig } from '@aztec/stdlib/kv-store';
+import { createLogger } from '@aztec/foundation/log';
 
-import { initStoreForRollupAndSchemaVersion } from '../../utils.js';
 import { AztecIndexedDBStore } from './store.js';
 
 export { AztecIndexedDBStore } from './store.js';
 
-/**
- * @deprecated The IndexedDB backend is being retired. Use `@aztec/kv-store/sqlite-opfs` instead.
- */
-export async function createStore(
-  name: string,
-  config: DataStoreConfig,
-  schemaVersion: number | undefined = undefined,
-  log: Logger = createLogger('kv-store'),
-) {
-  let { dataDirectory } = config;
-  if (typeof dataDirectory !== 'undefined') {
-    dataDirectory = `${dataDirectory}/${name}`;
-  }
-
-  log.info(
-    dataDirectory
-      ? `Creating ${name} data store at directory ${dataDirectory} with map size ${config.dataStoreMapSizeKb} KB`
-      : `Creating ${name} ephemeral data store with map size ${config.dataStoreMapSizeKb} KB`,
-  );
-  const store = await AztecIndexedDBStore.open(createLogger('kv-store:indexeddb'), dataDirectory ?? '', false);
-  return initStoreForRollupAndSchemaVersion(store, schemaVersion, config.rollupAddress, log);
-}
-
 /**
  * @deprecated The IndexedDB backend is being retired. Use `@aztec/kv-store/sqlite-opfs` instead.
  */

yarn-project/kv-store/src/sqlite-opfs/index.ts

@@ -1,28 +1,11 @@
-import { type Logger, createLogger } from '@aztec/foundation/log';
-import type { DataStoreConfig } from '@aztec/stdlib/kv-store';
+import { createLogger } from '@aztec/foundation/log';
 
-import { initStoreForRollupAndSchemaVersion } from '../utils.js';
 import { AztecSQLiteOPFSStore } from './store.js';
 
 export { AztecSQLiteOPFSStore } from './store.js';
 export { SqliteEncryptionError } from './errors.js';
 export type { SqliteEncryptionErrorCode } from './errors.js';
-
-export async function createStore(
-  name: string,
-  config: DataStoreConfig,
-  schemaVersion: number | undefined = undefined,
-  log: Logger = createLogger('kv-store'),
-) {
-  const { dataDirectory } = config;
-  log.info(
-    dataDirectory
-      ? `Creating ${name} SQLite-OPFS data store with map size ${config.dataStoreMapSizeKb} KB`
-      : `Creating ${name} ephemeral SQLite-OPFS data store with map size ${config.dataStoreMapSizeKb} KB`,
-  );
-  const store = await AztecSQLiteOPFSStore.open(createLogger('kv-store:sqlite-opfs'), name, false);
-  return initStoreForRollupAndSchemaVersion(store, schemaVersion, config.rollupAddress, log);
-}
+export { OPFS_POOL_DIR_PREFIX, deleteStore, listStores, storePoolDirectory } from './manage.js';
 
 export function openTmpStore(ephemeral: boolean = false): Promise<AztecSQLiteOPFSStore> {
   return AztecSQLiteOPFSStore.open(createLogger('kv-store:sqlite-opfs'), undefined, ephemeral);

yarn-project/kv-store/src/sqlite-opfs/manage.ts

@@ -0,0 +1,36 @@
+/** Prefix for the per-store OPFS SAH pool directories owned by this package. */
+export const OPFS_POOL_DIR_PREFIX = '.aztec-kv-';
+
+/**
+ * OPFS directory holding a store's SAH pool. One directory per store: the SAH-pool VFS allows only one
+ * concurrent instance per directory and its capacity does not grow automatically, so sharing a pool across
+ * stores would make concurrently opened stores contend for locks and orphaned stores exhaust pool slots.
+ */
+export function storePoolDirectory(effectiveName: string): string {
+  return `${OPFS_POOL_DIR_PREFIX}${effectiveName}`;
+}
+
+/**
+ * Lists the names of every persistent sqlite-opfs store in this origin, by enumerating the per-store pool
+ * directories. Includes stores other than the current one, so wallets can surface and clean up data for
+ * networks/versions no longer in use.
+ */
+export async function listStores(): Promise<string[]> {
+  const root = await navigator.storage.getDirectory();
+  const names: string[] = [];
+  for await (const [entryName, handle] of root.entries()) {
+    if (handle.kind === 'directory' && entryName.startsWith(OPFS_POOL_DIR_PREFIX)) {
+      names.push(entryName.slice(OPFS_POOL_DIR_PREFIX.length));
+    }
+  }
+  return names;
+}
+
+/**
+ * Permanently deletes a store by effective name (as returned by {@link listStores}). The store must be closed:
+ * an open store's SAH pool holds locks on the directory and the removal will reject.
+ */
+export async function deleteStore(effectiveName: string): Promise<void> {
+  const root = await navigator.storage.getDirectory();
+  await root.removeEntry(storePoolDirectory(effectiveName), { recursive: true });
+}

yarn-project/kv-store/src/sqlite-opfs/store_management.test.ts

@@ -0,0 +1,56 @@
+import { mockLogger } from '../interfaces/utils.js';
+import { AztecSQLiteOPFSStore } from './index.js';
+import { deleteStore, listStores, storePoolDirectory } from './manage.js';
+
+const openByName = (name: string) => AztecSQLiteOPFSStore.open(mockLogger, name, false, storePoolDirectory(name));
+
+describe('sqlite-opfs store management', () => {
+  it('round-trips data for a store reopened by name', async () => {
+    const store = await openByName('mech_roundtrip');
+    await store.openSingleton<string>('payload').set('data');
+    await store.close();
+
+    const reopened = await openByName('mech_roundtrip');
+    expect(await reopened.openSingleton<string>('payload').getAsync()).toEqual('data');
+    await reopened.close();
+    await deleteStore('mech_roundtrip');
+  });
+
+  it('opens two different stores concurrently in the same tab', async () => {
+    const a = await openByName('mech_concurrent_a');
+    const b = await openByName('mech_concurrent_b');
+
+    await a.openSingleton<string>('k').set('a');
+    await b.openSingleton<string>('k').set('b');
+    expect(await a.openSingleton<string>('k').getAsync()).toEqual('a');
+    expect(await b.openSingleton<string>('k').getAsync()).toEqual('b');
+
+    await a.close();
+    await b.close();
+    await deleteStore('mech_concurrent_a');
+    await deleteStore('mech_concurrent_b');
+  });
+
+  it('lists created stores and deletes them', async () => {
+    const store = await openByName('mech_managed');
+    await store.openSingleton<string>('k').set('v');
+    await store.close();
+
+    expect(await listStores()).toContain('mech_managed');
+    await deleteStore('mech_managed');
+    expect(await listStores()).not.toContain('mech_managed');
+
+    // Recreating after deletion starts empty.
+    const fresh = await openByName('mech_managed');
+    expect(await fresh.openSingleton<string>('k').getAsync()).toBeUndefined();
+    await fresh.close();
+    await deleteStore('mech_managed');
+  });
+
+  it('refuses to delete a store that is currently open', async () => {
+    const store = await openByName('mech_locked');
+    await expect(deleteStore('mech_locked')).rejects.toThrow();
+    await store.close();
+    await deleteStore('mech_locked');
+  });
+});

yarn-project/kv-store/tsconfig.json

@@ -5,7 +5,8 @@
     "rootDir": "src",
     "tsBuildInfoFile": ".tsbuildinfo",
     "allowJs": true,
-    "checkJs": false
+    "checkJs": false,
+    "lib": ["dom", "dom.asynciterable", "esnext", "es2017.object"]
   },
   "references": [
     {

yarn-project/pxe/src/entrypoints/client/bundle/index.ts

@@ -6,3 +6,4 @@ export * from '../../../contract_logging.js';
 export * from '../../../storage/index.js';
 export * from './utils.js';
 export type { PXECreationOptions } from '../../pxe_creation_options.js';
+export { openBrowserStore } from '../store.js';

yarn-project/pxe/src/entrypoints/client/bundle/utils.ts

@@ -1,6 +1,5 @@
 import { BBBundlePrivateKernelProver } from '@aztec/bb-prover/client/bundle';
 import { createLogger } from '@aztec/foundation/log';
-import { createStore } from '@aztec/kv-store/sqlite-opfs';
 import { BundledProtocolContractsProvider } from '@aztec/protocol-contracts/providers/bundle';
 import { WASMSimulator } from '@aztec/simulator/client';
 import { getStandardAuthRegistry } from '@aztec/standard-contracts/auth-registry';
@@ -12,6 +11,7 @@ import type { PXEConfig } from '../../../config/index.js';
 import { PXE } from '../../../pxe.js';
 import { PXE_DATA_SCHEMA_VERSION } from '../../../storage/metadata.js';
 import { type PXECreationOptions, isPrivateKernelProver } from '../../pxe_creation_options.js';
+import { openBrowserStore } from '../store.js';
 
 /**
  * Create and start an PXE instance with the given AztecNode.
@@ -31,16 +31,28 @@ export async function createPXE(
   const actor = options.loggerActorLabel;
   const loggers = options.loggers ?? {};
 
-  const l1ContractAddresses = await aztecNode.getL1ContractAddresses();
+  const { l1ChainId, l1ContractAddresses, rollupVersion } = await aztecNode.getNodeInfo();
   const configWithContracts = {
     ...config,
     ...l1ContractAddresses,
+    l1ChainId,
+    rollupVersion,
   } as PXEConfig;
 
   const storeLogger = loggers.store ?? createLogger('pxe:data', { actor });
 
   const store =
-    options.store ?? (await createStore('pxe_data', configWithContracts, PXE_DATA_SCHEMA_VERSION, storeLogger));
+    options.store ??
+    (await openBrowserStore(
+      'pxe_data',
+      PXE_DATA_SCHEMA_VERSION,
+      {
+        l1ChainId,
+        rollupAddress: l1ContractAddresses.rollupAddress,
+        dataStoreMapSizeKb: configWithContracts.dataStoreMapSizeKb,
+      },
+      storeLogger,
+    ));
 
   const simulator = options.simulator ?? new WASMSimulator();
   const proverLogger = loggers.prover ?? createLogger('pxe:bb:wasm:bundle', { actor });

yarn-project/pxe/src/entrypoints/client/lazy/index.ts

@@ -6,3 +6,4 @@ export * from '../../../error_enriching.js';
 export * from '../../../contract_logging.js';
 export * from './utils.js';
 export { type PXECreationOptions } from '../../pxe_creation_options.js';
+export { openBrowserStore } from '../store.js';