Streaming file transfer support in @powersync/common attachments: RemoteStorageAdapter forces files through the JS heap

[Traviskn]

Summary

The AttachmentQueue (@powersync/common@1.52.0, @powersync/attachments-storage-react-native@0.0.2) is well-suited for small attachments but unusable for large files without a workaround, because the RemoteStorageAdapter and LocalStorageAdapter contracts both round-trip the file body through an ArrayBuffer in JavaScript memory.

I'd love to use the queue as the single file-handling pipeline in my React Native app (uploads, downloads, retries, archival, etc.) The framework already does almost everything I need; the one blocker is the buffer-shaped adapter API. I wanted to start a conversation about whether a streaming-friendly variant of the adapter interface could land upstream.

Problem detail

Looking at SyncingService in @powersync/common@1.52.0:

Upload path (SyncingService.uploadAttachment):

  const fileBlob = await this.localStorage.readFile(attachment.localUri); // ArrayBuffer
  await this.remoteStorage.uploadFile(fileBlob, attachment);

Download path (SyncingService.downloadAttachment):

  const fileData = await this.remoteStorage.downloadFile(attachment); // ArrayBuffer
  const localUri = this.localStorage.getLocalUri(attachment.filename);
  await this.localStorage.saveFile(localUri, fileData);

In both directions, the entire file body is materialized in JS heap memory as an ArrayBuffer and then handed to the next adapter. On lower-end phones with large files, this is enough to trigger memory pressure or OOM.

expo-file-system provides FileSystem.uploadAsync(url, fileUri, { uploadType: BINARY_CONTENT }) and FileSystem.downloadAsync(url, fileUri), both of which stream the file body native-to-network (or vice versa) without ever materializing it in JS. For any file >~5 MB this is the right tool.

I currently work around the framework by making my two adapters cooperate behind the existing API:

1. Custom LocalStorageAdapter:
   - readFile() returns new ArrayBuffer(0) (the framework only forwards this value to uploadFile, and my uploadFile ignores it).
   - saveFile(filePath, data) treats data of length 0 as an "already on disk" sentinel, looks up the existing file's size and returns it without writing.
2. Custom RemoteStorageAdapter:
   - uploadFile(_data, attachment) ignores the buffer and calls FileSystem.uploadAsync(presignedUrl, attachment.localUri, …).
   - downloadFile(attachment) pre-computes the destination URI via localStorage.getLocalUri(attachment.filename), streams to it with FileSystem.downloadAsync, and returns new ArrayBuffer(0) as the sentinel for the framework's saveFile call.

This works today, but feels like a bit of a hack and has some downsids:

• It depends on framework internals that aren't part of the public contract. Specifically, that SyncingService only uses the buffer as a pass-through between adapter calls, no hashing, integrity checks, or telemetry on the bytes between read and upload. The library is @experimental @Alpha, so this is explicitly subject to change.
• It silently breaks AttachmentQueue.saveFile({ data: ArrayBuffer }) for any consumer that doesn't know about the sentinel. Our subclass's overridden saveFile distinguishes by byteLength === 0, but that's brittle if anyone legitimately wants to save an empty file.
• It makes on-disc files awkward to register. attachmentQueue.saveFile requires data: AttachmentData, so we have a queueVoiceRecordingForUpload helper that moves the finalized file into the attachment dir, then calls saveFile with new ArrayBuffer(0) plus a known id. A first-class "register existing file" API would be cleaner.

What a more ergonomic API might look like

A few non-exclusive ideas (happy to discuss tradeoffs):

1. Add file-URI-based variants alongside the buffer-based methods:

  interface RemoteStorageAdapter {
    // Existing
    uploadFile?(fileData: ArrayBuffer, attachment: AttachmentRecord): Promise<void>;
    downloadFile?(attachment: AttachmentRecord): Promise<ArrayBuffer>;

    // New
    uploadFileFromUri?(localUri: string, attachment: AttachmentRecord): Promise<void>;
    downloadFileToUri?(localUri: string, attachment: AttachmentRecord): Promise<void>;
  }

SyncingService would prefer the URI variants when present and skip the localStorage.readFile / localStorage.saveFile round-trip in that case. Buffer methods remain for adapters that already work that way.

2. Allow AttachmentQueue.saveFile to register an existing file:

  saveFile({
    existingFileUri?: string;  // alternative to `data`
    data?: AttachmentData;
    fileExtension: string;
    …
  });

When existingFileUri is provided, the queue moves the file into its storage directory and creates the attachment record. Avoids the "load into ArrayBuffer just to write it back to disk" anti-pattern for app-originated saves.

3. Document the framework's use of readFile/downloadFile return values.

Even without an API change, a docs-level contract that says "the buffer returned by readFile is only passed to remoteStorage.uploadFile, the framework does not inspect it" would make the workaround above a deliberate, supported pattern rather than a fragile dependency on implementation details.

Context

• Stack: React Native + Expo SDK 55, @powersync/react-native@…, @powersync/common@1.52.0, @powersync/attachments-storage-react-native@0.0.2.

[stevensJourney]

Thanks for the detailed message!

I think the core issue here is that, in some cases, we should avoid forcing large attachment data through JS as a single ArrayBuffer.

In the Kotlin SDK we avoid this by using Flows of binary chunks, so the full file does not need to be loaded into memory at once. In JS, the closest universal abstraction would probably be something like a ReadableStream or AsyncIterable<Uint8Array>. That could work well in some environments, especially web/node, but I am not sure it is the best abstraction for React Native.

For React Native / Expo specifically, even if the JS API is stream-based, the data may still need to cross the native bridge in chunks. That avoids one huge allocation, but it may still be inefficient or unsupported by the underlying storage APIs. This is probably why APIs like Expo's FileSystem.uploadAsync do both the file reading and network streaming on the native side.

Because of that, I think this proposed shape:

interface RemoteStorageAdapter {
  uploadFile?(fileData: ArrayBuffer, attachment: AttachmentRecord): Promise<void>;
  downloadFile?(attachment: AttachmentRecord): Promise<ArrayBuffer>;

  uploadFileFromUri?(localUri: string, attachment: AttachmentRecord): Promise<void>;
  downloadFileToUri?(localUri: string, attachment: AttachmentRecord): Promise<void>;
}

could become a bit tricky to manage internally. The syncing service would need to understand which combination of methods is supported, then choose between buffered, URI-based, and possibly streaming paths for every transfer.

It may be a little cleaner to model the higher-level operation the syncing service actually needs. Something like:

Note the details below are very rough: I'm just sharing these thoughts as potential conversation pieces. I have not entirely investigated the feasibility.

The AttachmentTransferAdapter (A very bad name, could be AttachmentStorageAdapter) can be a somewhat combination of the current Local/Remote storage adapters.

interface AttachmentTransferAdapter {
  /**
   * Uploads the attachment's local file to remote storage.
   *
   * Implementations can choose the best available transfer mechanism:
   * ArrayBuffer, stream/chunks, local file URI, or a platform-native upload API.
   */
  uploadAttachment(attachment: AttachmentRecord): Promise<void>;

  /**
   * Downloads the remote file into `attachment.localUri`.
   *
   * The syncing service assigns `localUri` before calling this method so the
   * transfer adapter has both the attachment metadata and destination URI.
   */
  downloadAttachment(attachment: AttachmentRecord & { localUri: string }): Promise<void>;
}

The default implementation could preserve the current behavior by composing the existing local and remote storage adapters:

class BufferedAttachmentTransferAdapter implements AttachmentTransferAdapter {
  constructor(
    private localStorage: LocalStorageAdapter,
    private remoteStorage: RemoteStorageAdapter
  ) {}

  async uploadAttachment(attachment: AttachmentRecord): Promise<void> {
    if (attachment.localUri == null) {
      throw new Error(`No localUri for attachment ${attachment.id}`);
    }

    const fileData = await this.localStorage.readFile(attachment.localUri);
    await this.remoteStorage.uploadFile(fileData, attachment);
  }

  async downloadAttachment(attachment: AttachmentRecord & { localUri: string }): Promise<void> {
    const fileData = await this.remoteStorage.downloadFile(attachment);
    await this.localStorage.saveFile(attachment.localUri, fileData);
  }
}

Then the syncing service branches on attachment actions, not storage capabilities:

async uploadAttachment(attachment: AttachmentRecord): Promise<AttachmentRecord> {
  await this.transferAdapter.uploadAttachment(attachment);

  return {
    ...attachment,
    state: AttachmentState.SYNCED,
    hasSynced: true
  };
}

async downloadAttachment(attachment: AttachmentRecord): Promise<AttachmentRecord> {
  const localUri = this.localStorage.getLocalUri(attachment.filename);
  const attachmentWithLocalUri = { ...attachment, localUri };

  await this.transferAdapter.downloadAttachment(attachmentWithLocalUri);

  return {
    ...attachmentWithLocalUri,
    state: AttachmentState.SYNCED,
    hasSynced: true
  };
}

That gives us a backwards-compatible path for existing web/node-style adapters, while leaving room for React Native / Expo implementations to keep the actual transfer out of JS:

• upload: local file URI -> native upload API / signed URL
• download: signed URL / native download API -> local file URI

Streams or async iterators could still be useful as an implementation detail, or as a capability for environments that support them well. But I think the public abstraction may be more robust if it represents the whole attachment transfer operation rather than individual read/write/upload/download primitives.

We could potentially overload the AttachmentQueue constructor to support either options for storage adapters. However, as mentioned, since this feature is in Alpha, we are free to also change the default parameters.