Feat/hypercore11 migration

[RangerMauve]

Closes #1042

[gmaclennan]

I used a bunch of tokens on Claude Fable (available until July 7th on the monthly usage plan) to review this on "xhigh", which includes confirmation of issues found. The issues marked CONFIRMED should have been tested and confirmed. "Plausible" could be wrong.

Critical — sync correctness

1. forceBitfieldExchange is silently a no-op on hypercore 11 — src/utils.js:356 (also test/helpers/create-core.js:46) · CONFIRMED
peer.core is hypercore 11's internal Core object, which has no .length property (the real length is peer.core.state.length). So maxPage = Math.ceil(undefined / 131072) is NaN and the want-send loop never executes, and bitfield.want(0, undefined) yields nothing. The entire eager bitfield exchange degrades to sending only the contiguous range. With sparse data (blob cores with download filters, partially synced cores), neither side learns the other's real bitfield — sync progress appears stuck, and older clients never get the want that prompts them to send their bitfield. Verified at runtime by the finder. Notably, the unit test covering this ('eagerly updates remote bitfields') was skipped in this branch, so nothing catches it.

High — migration module crashes and misbehavior

2. checkShouldMigrate throws ENOENT on fresh installs — src/migration.js:25 · CONFIRMED
listProjectsFromStorage calls readdir(storagePath) unguarded. On a fresh install where the storage folder doesn't exist yet, the migration pre-check rejects with a raw filesystem error instead of returning { shouldUpgrade: false }.

3. Free-space check contradicts its own contract — src/migration.js:254 · CONFIRMED
The JSDoc says available space must be at least 2.5× the largest core, but AVAILABLE_SPACE_MULTIPLIER is 1.05. A device with 1.1× space passes the check, then hits ENOSPC mid-migration and is left with a half-migrated corestore — the exact situation the check exists to route to the fallback path.

4. Broken published types for the new @comapeo/core/migration.js entry point — package.json:20 · CONFIRMED
dist/migration.d.ts ends with import CorestoreStorage from 'hypercore-storage', but the only declaration for that module lives in the unpublished types/ folder. Verified with tsc -p tsconfig.npm.json: consumers get TS2307 with skipLibCheck: false, or silent any for the migration API's key types with it on.

5. Migration progress counts can disagree — src/migration.js:196 · CONFIRMED
The total is pre-counted from a filesystem heuristic (dirs containing an oplog file) while doneSoFar increments per entry of storage.createCoreStream(). When the sets differ, a progress UI shows >100% or never reaches 100%.

6. entry.path ?? entry.parentPath isn't portable — src/migration.js:73 · PLAUSIBLE
dirent.path (deprecated, slated for removal) and dirent.parentPath (added v20.12) don't both exist on all runtimes. On a runtime with neither (e.g. Node 18-era nodejs-mobile), path.join(undefined, entry.name) throws and the whole migration pre-check crashes. Worth checking what nodejs-mobile ships before relying on either.

High — latent data-corruption and resource risks

7. PretendCorestore.get lets keyPair overwrite an explicit key — src/blob-store/hyperdrive-index.js:111 · PLAUSIBLE
The keyPair branch runs after the key branch and unconditionally overwrites opts.publicKey. Hyperdrive 13's _openBlobsFromHeader passes both (key: blobsKey, keyPair: db.core.keyPair), which would resolve the blobIndex core instead of the blobs core — blob reads decode hyperbee blocks as blob bytes, and writes corrupt the replicated drive index. Currently latent because comapeo cores open in compat mode, but it fires for any non-compat blobIndex core. The fix is cheap: only apply the keyPair fallback when no explicit key is given.

8. File-descriptor cap removed with no replacement — src/mapeo-manager.js:252 · CONFIRMED
The 768-fd RandomAccessFilePool (added for Android's 1024-fd process limit) is gone, and no equivalent limit is configured on the RocksDB-backed CorestoreStorage stack. The covering test was skipped rather than replaced. Multi-project users on Android risk EMFILE — the field failure the pool existed to prevent.

Medium

9. PeerState.want() off-by-one vs wantWord() — src/sync/core-sync-state.js:445 · CONFIRMED
want() uses index > contiguousLength where wantWord() uses >=, so the first block after a peer's contiguous range is reported as not wanted by one path and wanted by the other.

10. kCreateOrUpdateWithDocId no-op detection misses field removal — src/datatype/index.js:202 · PLAUSIBLE
hasChanges iterates only Object.keys(value), so an update whose purpose is dropping a field (omitting the key) is treated as no change and skipped — the stale field persists while the caller sees success.

Test-coverage regressions

11–12. The two tests covering exactly the rewritten mechanisms were skipped — test/core-manager.js:512 (deleteOthersData) and test/core-manager.js:53 (eager bitfield exchange) · both CONFIRMED
deleteOthersData() was the only test asserting other members' data is actually removed from disk; its replacement (purgeCore() poking private storage internals, pending upstream hypercore PR #788) is now covered only by an e2e check that disk usage decreased by some amount. The bitfield test skip is what let finding #1 land unnoticed. If either mechanism drifts in a hypercore patch release, no test catches it.

Cleanup

13. Stale JSDoc: coreStorage no longer accepts a factory function — src/mapeo-manager.js:163 (also src/core-manager/index.js:74) · CONFIRMED — a consumer following the docs gets an opaque path.join TypeError deep inside MapeoManager.

14. migrateStorage JSDoc claims a dry-run — src/migration.js:143 · CONFIRMED — it actually runs with dryRun: false, gc: true, irreversibly migrating; the misleading doc ships in the published .d.ts.

15. Leftover console.log(error) — src/migration.js:208 · CONFIRMED — dumps raw errors to stdout from library code; the caller already gets the same error via results[projectId].error.

Six candidates were refuted in verification — among them "purgeCore leaves dangling references" (guarded by hypercore 11's closed-session checks) and "Downloader swallows legitimate cancellations" (the consumer already ignored those errors on main).

The standouts are #1 (breaks the sync behavior this branch exists to preserve, with its test skipped) and the migration-module cluster (#2–#6), since that code runs exactly once per device with no second chance. Happy to fix any of these — say the word (or re-run with --fix).

[RangerMauve]

1. Got the test passing, passed in proper reference to the core

[RangerMauve]

2. added check
3. My docs were messed, in my tests we just needed like 3% extra storage + the size of the core. So 2.5 is overkill. 1.05 is enough, but maybe I should go just a bit higher, 1.5
4. I think downstream should include the types folder in their tsconfig if that is an issue. The argument is optional and only used for testing anyway.
5. I don't think it'll cause problems but just in case I added a thing that will ensure the cb is called with the total at the end.
6. entry.path ?? entry.parentPath is portable. I added this specifically so the tests would pass in node 18/20/24
7. Not an issue we will face since we won't have drives open without a content key in their header. Really not sure why they even have that fallback
8. non issue since we won't be approaching that limit unless we open hundreds of projects.

[RangerMauve]

9. was an issue, added tests to catch and fix it
10. I think this is working as intended?

[gmaclennan]

I pushed two small commits to fix the exported types (which I think will be helpful for frontend implementation) and addresses an issue where the tests weren't actually testing the eager want-sending.

I think it would be good to explicitly add hypercore-storage as a dependency before merging - it's imported in the code, and only works because it's a transitive dependency

[cimigree]

For the designs in Figma, https://www.figma.com/design/iUeC0Qzhb4H0unuPfxQAsM/-Mobile--CoMapeo?node-id=11371-4308&t=nKpXn1KBC1KIdSK8-0, we need a value for the amount of space needed. This will most likely help the user know how much space to free up...