Docs: Is the browser storage performance chart mis-labeled?

[nickchomey]

It shows the OPFS Worker being slower than OPFS main thread except for with Find 50 docs by ID (serial), which doesn't seem like it should be the case given what the docs say about createSyncAccessHandle

Perhaps they're mis-labeled? Or is there an issue with the benchmarking?

[nickchomey]

Find 3000 docs by query (parallel) is also slower for both OPFS. Is that expected?

[nickchomey]

I think it would also be useful, if possible, to give some indication of how much the benchmarks improved from v16 to v17

[pubkey]

No these are the actual results. The performance test run function is exported from RxDB Core so you can run it yourself.

/// <reference path="../node_modules/@types/mocha/index.d.ts" />
/**
 * Performance tests for OPFS storage in browser.
 * These run WITHOUT RxDBDevModePlugin so that runPerformanceTests
 * can create databases without needing a schema validator wrapper.
 */
import {
    getRxStorageWorker
} from '../plugins/storage-worker';
import {
    getRxStorageOPFSMainThread
} from '../plugins/storage-opfs';
import {
    clearAllLocalIndexedDB,
    clearAllLocalStorage,
    runPerformanceTests
} from 'rxdb/plugins/test-utils';

const workerPath = '/workers/opfs.worker.js';

/**
 * clearAllLocalOPFS() from test-utils calls removeEntry on the
 * native FileSystemDirectoryHandle directly, so transient
 * NoModificationAllowedError from still-locked worker handles
 * is not retried.  This wrapper retries each entry individually.
 */
async function clearOPFSWithRetry(maxRetries = 20, delayMs = 200): Promise<void> {
    if (
        typeof navigator === 'undefined' ||
        !navigator.storage ||
        typeof navigator.storage.getDirectory !== 'function'
    ) {
        return;
    }
    const root = await navigator.storage.getDirectory();
    // @ts-expect-error entries() is not in all TS lib definitions
    for await (const [name] of root.entries()) {
        for (let attempt = 0; attempt < maxRetries; attempt++) {
            try {
                await root.removeEntry(name, { recursive: true });
                break;
            } catch (err: any) {
                if (err?.name === 'NoModificationAllowedError' && attempt < maxRetries - 1) {
                    await new Promise(resolve => setTimeout(resolve, delayMs));
                } else {
                    throw err;
                }
            }
        }
    }
}

describe('performance-opfs.test.ts', () => {

    it('clear local storage before tests', async () => {
        await clearAllLocalIndexedDB();
        await clearOPFSWithRetry();
        await clearAllLocalStorage();
    });

    it('run performance test (worker)', async function () {
        const isCI = process.env.GITHUB_ACTIONS;
        this.timeout(isCI ? 60000 : 500 * 1000);
        const result = await runPerformanceTests(
            getRxStorageWorker({
                workerInput: workerPath
            }),
            'OPFS (Worker)',
            isCI ? { runs: 1, docsAmount: 600, waitBetweenTests: 0, log: true } : {}
        );
        console.log(JSON.stringify(result, null, 4));
    });

    it('run performance test (main-thread)', async function () {
        const isCI = process.env.GITHUB_ACTIONS;
        this.timeout(isCI ? 60000 : 500 * 1000);
        const result = await runPerformanceTests(
            getRxStorageOPFSMainThread({}),
            'OFPS (Main-Thread)',
            isCI ? { runs: 1, docsAmount: 600, waitBetweenTests: 0, log: true } : {}
        );
        console.log(JSON.stringify(result, null, 4));
    });
    it('clear local storage after tests', async () => {
        await clearAllLocalIndexedDB();
        await clearOPFSWithRetry();
        await clearAllLocalStorage();
    });
});

[nickchomey]

Ok, but the OPFS docs say many times that the worker version is faster than the main thread version. But the benchmarks say otherwise. So, either there is:

1. an issue with the benchmark,
2. an issue with the OPFS Worker implementation, or
3. the Docs should be updated to say that main thread is actually (somehow, inexplicably) faster

And, again, the Find 3000 docs by query (parallel) test is slower than non-parallel for both OPFS worker and mainthread. That doesn't seem like it should be the case...

Another thought: is it possible that the various RxStorage tests themselves are running in parallel (eg it('run performance test (worker)' and it('run performance test (main-thread)') and thus there's some contention? I assume not, but you never know.

[nickchomey]

Also, the previous benchmarks better-reflected what is described in the docs.

Perhaps the tests themselves changed, so the absolute values can be ignored. But the relative values seem to have changed quite suspiciously...

[pubkey]

Good point. I will check.

[pubkey]

Perhaps the tests themselves changed, so the absolute values can be ignored

In the past we tested performance without setting up a user profile in the browser which turns out to make the storage be non-persistend and have a different performance pattern. So comparing it to the previous performance chart is difficult. Also the chrome browser itself was updated and the V8 engine etc..

[nickchomey]

Fair enough! But it still remains that OPFS main thread is (unexpectedly) faster than worker. This was not the case with v16. Something is wrong, but I don't know what. Could even just be that the faster synchronous access in worker is negated by main<->worker messaging?

And parallel queries are slower for OPFS in both v16 and v16. Doesn't seem correct? Or is the anomaly the indexeddb, dexie etc where parallel queries are faster?

It seems to me that its all worth investigating... It's likely that some bugs will be revealed

[pubkey]

faster synchronous access in worker is negated by main<->worker messaging

Yes that might be one reason, sending data from and to the worker is very expensive. Source

I just ran the performance tests again, so the results are correct (see results below).
I think it is very likely that chrome has just improved here. In the end any disc-writes/reads are Async IO and from my understanding there is no real reason why it should be faster on a worker then the js-process is blocked (non-async) compared to having an async operation on the main thread.
I will updated the docs so people know.

{
    "description": "OPFS (Worker)",
    "collectionsAmount": 4,
    "docsAmount": 3000,
    "time-to-first-insert": 25.05,
    "insert-documents-500": 9.89,
    "find-by-ids-3000": 24.7,
    "serial-inserts-50": 7,
    "serial-find-by-id-50": 6.95,
    "find-by-query": 21.1,
    "find-by-query-parallel-4": 36.65,
    "4x-count": 2.8,
    "property-access": 5.5
}'


{
    "description": "OFPS (Main-Thread)",
    "collectionsAmount": 4,
    "docsAmount": 3000,
    "time-to-first-insert": 4.4,
    "insert-documents-500": 6.82,
    "find-by-ids-3000": 21.55,
    "serial-inserts-50": 1.8,
    "serial-find-by-id-50": 26.5,
    "find-by-query": 18.35,
    "find-by-query-parallel-4": 23.4,
    "4x-count": 1.95,
    "property-access": 5.15
}

[nickchomey]

I just did a quick search of the rxdb src and rxdb-premium minified code. It looks like you only ever use postMessage(message), which means you are not taking advantage of transferable objects, which can be literally transfered between main<->worker without copy.

https://developer.mozilla.org/en-US/docs/Web/API/Window/postMessage#transfer

A related excerpt from the article you shared

Binary formats
Another way to deal with the performance impact of structured cloning is to not use it at all. Apart from structured cloning objects, postMessage() can also transfer certain types. ArrayBuffer is one of these transferable types. As the name implies, transferring an ArrayBuffer does not involve copying. The sending realm actually loses access to the buffer and it is now owned by the receiving realm. Transferring an ArrayBuffer is extremely fast and independent of the size of the ArrayBuffer.

I see some use of ArrayBuffer and things like Uint8Array through the OPFS and other code. The transferrable objects doc says

Note: Typed arrays like Int32Array and Uint8Array, are serializable, but not transferable. However their underlying buffer is an ArrayBuffer, which is a transferable object. We could have sent uInt8Array.buffer in the data parameter, but not uInt8Array in the transfer array.

So, perhaps you can make use of transferable objects to speed things up between workers?

[nickchomey]

I will updated the docs so people know.

Also, while you're at it, I wonder if you could clarify some things in https://rxdb.info/rx-storage.html

In the Browser: Use the LocalStorage storage for simple setup and small build size. For bigger datasets, use either the dexie.js storage (free) or the IndexedDB RxStorage if you have 👑 premium access which is a bit faster and has a smaller build size.

Why do you generally recommend the IndexedDB premium storage rather than OPFS, which seems to have better perf. Are there things that OPFS can't do?

Also, related to the current issue, you even say this there:

We do not use workers because sending data from the main thread to workers and backwards would increase the latency.

Likewise in https://rxdb.info/rx-storage-performance.html#browser-based-storages-performance-comparison

The performance patterns of the browser based storages are very diverse. The IndexedDB storage is recommended for mostly all use cases so you should start with that one. Later you can do performance testings and switch to another storage like OPFS or memory-mapped.

Its not clear at all why we should use IndexedDB if OPFS is faster. If there's tradeoffs to using it vs indexeddb, they should be made clearer

[pubkey]

I tried to use transferable objects (this was on the todo-list of the v17 changes) but I could not find a way to use them to actuall improve performance. In the end we need JSON and transforming stuff into an ArrayBuffer to send it faster makes it slower in most cases.

Why do you generally recommend the IndexedDB premium storage rather than OPFS

It is way more battle tested. It can be debugged easier with chrome devtools...

I think in the long run browsers will optimize the performance of IndexedDB which in total couuld be faster than anything you can do in javascript/wasm because the browser can run stuff directly in C++/Rust.
But this is only a prediction. The OPFS storage has much potential and the whole logic is even reused in other stuff like node-filesystem and the new expo-storage.

[nickchomey]

Thanks, makes sense. Sounds like all that's needed is a bit of tweaks to the OPFS, Storage and Performance docs to clarify things a bit more.

I'll probably use mainthread OPFS in development, just to see if I hit any issues. It appears to be a smaller bundle and, apparently, faster. (though, I'll use opfs mainthread in a sharedworker, but without using the getRxStorageSharedWorker plugin - im just running a single instance of rxdb in the sharedworker without running rxdb in the main thread, as i dont need reactivity etc... its just a browser db/search engine for me).

[nickchomey]

I think i missed the commit or PR that closed this with some docs changes to address the issues raised here