Draft: In-memory writeahead file system

[simolus3]

As a follow-up to #1029 (see in particular the discussion in #1029 (comment)), this adds a multi-threaded variant of the in-memory file system with support for concurrent reads / writes.

To use this, import { InMemoryWriteAheadLogPool } from '@powersync/web/in-memory-wal-experiment'; and open PowerSync databases with the opened: new InMemoryWriteAheadLogPool({ numWorkers: 2 }) option.

This implementation avoids navigator locks and and doesn't give databases a name, each instance is a fresh and completely independent database instance. For the most part, this VFS works like the OPFS write-ahead VFS, except that:

1. The write-ahead overlay is not serialized, we literally just append blobs to a WAL and track offsets in a JavaScript Map we can send between workers.
2. Checkpointing doesn't rely on navigator locks, since all shared array buffers are bound to a single tab we can manage the checkpointing process with a local Semaphore instead.
3. This doesn't use a WAL2 approach, a checkpointer blocks everything else instead. I believe this should be fine since checkpointing is essentially a memcpy.

There are a few things still to do here. For example, the current checkpointing logic triggers from very small WAL sizes and could likely use some fine-tuning. This also doesn't support batched statements yet and doesn't expose options to configure page cache or prepared statement cache sizes.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 817d0df

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR