Skip to content

STREAMS.md

Latest commit

 

History

History
138 lines (100 loc) · 9.4 KB

STREAMS.md

File metadata and controls

138 lines (100 loc) · 9.4 KB
Raw

Streaming Data in SSH Term

One of the primary challenges in a browser-based application like SSH Term is efficiently moving data between the user's local machine and the sandboxed WebAssembly (WASM) application. This is especially critical for file transfers (SFTP), where files can be too large to fit into memory. SSH Term uses two distinct, stream-based approaches for uploads and downloads.

This document explains the technical implementation of these streaming mechanisms, focusing on the interaction between the Go code in go/internal/jsutil/streams.go and the Service Worker in docroot/stream-helper.js.

Streaming Uploads (Local Machine to Remote Server)

Streaming uploads are relatively straightforward and do not involve the Service Worker. The process relies on the browser's ability to provide a readable stream for a local file selected by the user.

Process Flow

  1. User Action: The user initiates an upload via the put command in an SFTP session or by dragging and dropping a file onto the terminal.

  2. File Selection: The application uses the browser's standard file input mechanism (<input type="file"> or the DataTransfer API for drag-and-drop).

  3. Stream Creation (JavaScript): When the user selects a file, the browser provides a File object. Crucially, this object has a .stream() method that returns a ReadableStream. This allows the file's content to be read in chunks without loading the entire file into memory.

  4. Go-side Wrapper (streams.go):

    • The JavaScript ReadableStream is passed to the Go WASM module.
    • The jsutil.NewStreamReader(stream) function creates a Go io.ReadCloser that wraps the JavaScript stream.
    • The StreamReader struct holds a reference to the stream's ReadableStreamDefaultReader.
    • Its Read() method is a Go-native method that, under the hood, calls the JavaScript reader's .read() method asynchronously, waits for the promise to resolve, and copies the resulting chunk of data into the Go byte slice.

  5. SFTP Upload:

    • The SFTP client in go/internal/app/sftp.go receives this io.ReadCloser.
    • It treats it like any standard Go io.Reader and reads from it, sending the data over the SSH connection to the remote server.

This "pull" model is efficient for memory, as only small chunks of the file are held in memory at any given time during the transfer.

Upload Visualization

sequenceDiagram
    participant User
    participant Browser (JS Frontend)
    participant Go (WASM)
    participant SFTP Server

    User->>Go (WASM): Enters "put" command
    Go (WASM)->>Browser (JS Frontend): Request file import
    Browser (JS Frontend)->>User: Show file picker
    User->>Browser (JS Frontend): Selects local file
    Browser (JS Frontend)->>Go (WASM): Provide ReadableStream for the file
    Note over Go (WASM): Wraps JS stream in a Go io.Reader
    loop Read file in chunks
        Go (WASM)->>Browser (JS Frontend): Read chunk from stream
        Browser (JS Frontend)-->>Go (WASM): Return chunk
        Go (WASM)->>SFTP Server: Send chunk over SSH
    end
    SFTP Server-->>Go (WASM): Acknowledge transfer
    Go (WASM)-->>User: Show completion
Loading

Streaming Downloads (Remote Server to Local Machine)

Streaming downloads are more complex due to the browser sandbox. The WASM application cannot directly write a file to the user's disk. To solve this, SSH Term uses a Service Worker (stream-helper.js) to act as a proxy between the Go application and the browser's download manager. This allows the application to serve a file to the browser as if it were a regular web download, without buffering the entire file in the main application's memory.

Components

  • go/internal/jsutil/streams.go: Contains the Go-side logic (StreamHelper) to manage the stream and communicate with the Service Worker.
  • docroot/stream-helper.js: A Service Worker script that intercepts specific network requests and serves responses generated by the Go application.

Process Flow

  1. User Action: The user initiates a download via the get command in an SFTP session. The Go SFTP client gets an io.ReadCloser for the remote file.

  2. Stream Registration (streams.go):

    • The StreamHelper.Download() function is called with the io.ReadCloser for the remote file.
    • It generates a unique, random streamId.
    • It stores the io.ReadCloser in a map, keyed by the streamId. This map holds the live data streams waiting to be consumed.

  3. Download Trigger:

    • A special URL is constructed: ./stream/<streamId>.
    • The Go code dynamically creates an <a> HTML element with its href set to this URL and programmatically clicks it. This tells the browser to "download" the URL.

  4. Service Worker Interception (stream-helper.js):

    • The Service Worker is registered to control the page and has an onfetch event listener.
    • It intercepts the browser's GET request for ./stream/<streamId>.
    • It extracts the streamId from the URL.

  5. Stream Request Handshake:

    • The Service Worker sends a message containing the streamId back to the main application page using client.postMessage(). This is a request for the actual data stream.
    • The onmessage handler in streams.go (set up during NewStreamHelper initialization) receives this request.

  6. Stream Provisioning (streams.go):

    • The Go code looks up the streamId in its map to find the corresponding io.ReadCloser.
    • It then creates a JavaScript ReadableStream that wraps the Go io.ReadCloser using the NewReadableStream function.
    • NewReadableStream has a pull method. When the browser is ready for more data, it calls pull, which in turn reads a chunk from the Go io.Reader and enqueues it into the JavaScript stream. This bridges the Go and JavaScript streaming worlds.

  7. Response Generation (stream-helper.js):

    • The Go code sends the newly created ReadableStream back to the Service Worker via postMessage.
    • The Service Worker receives the stream and uses it to construct a Response object. It also sets appropriate headers like Content-Disposition to give the file a name.
    • The fetch event's respondWith() method is called with this Response.

  8. File Download: The browser receives the Response from the Service Worker and treats it as a standard file download, saving the streamed data to the user's local disk. The main application page never has to hold the entire file in memory.

This solution allows the sandboxed Go application to provide a data stream that the browser can consume natively for downloads, providing an efficient and user-friendly experience for large files.

Download Visualization

sequenceDiagram
    participant User
    participant Go (WASM)
    participant Browser (JS Frontend)
    participant Service Worker
    participant SFTP Server

    User->>Go (WASM): Enters "get" command
    Go (WASM)->>SFTP Server: Request remote file
    SFTP Server-->>Go (WASM): Provide file stream (io.Reader)
    Note over Go (WASM): Generates stream ID, stores reader in map
    Go (WASM)->>Browser (JS Frontend): 'Trigger download of /stream/<id>'
    Browser (JS Frontend)->>Service Worker: 'fetch("/stream/<id>")'
    Service Worker->>Browser (JS Frontend): 'postMessage({streamId: <id>})'
    Note right of Browser (JS Frontend): Message is received by the<br/>main page's JS context.
    Browser (JS Frontend)->>Go (WASM): Forward stream request
    Note over Go (WASM): Finds io.Reader by ID,<br/>wraps it in a JS ReadableStream.
    Go (WASM)->>Browser (JS Frontend): Return ReadableStream
    Browser (JS Frontend)->>Service Worker: postMessage({body: ReadableStream})
    Service Worker->>Browser (JS Frontend): respondWith(new Response(ReadableStream))
    Browser (JS Frontend)->>User: Show "Save As" dialog
Loading

Service Worker Trade-offs

Using a Service Worker for streaming downloads is a powerful technique, but it comes with certain trade-offs.

  • Complexity: The Service Worker introduces significant architectural complexity. It runs in a separate thread from the main application, manages its own lifecycle (install, activate, fetch), and communicates with the main page via asynchronous messaging (postMessage). This is more complex than a standard, single-threaded web application.

  • Debugging: Debugging interactions between the main application and the Service Worker can be challenging. Developers need to use the browser's developer tools to inspect the Service Worker's state, view its console logs separately, and trace messages passed between the two contexts.

  • Browser and Security Requirements: Service Workers are only available in secure contexts. This means the application must be served over HTTPS for them to function in a production environment (localhost is considered a secure context for development). While this enforces good security practices, it is a strict prerequisite.

  • Browser Compatibility: While Service Workers are now widely supported across modern browsers, there can be subtle implementation differences or bugs that may require browser-specific workarounds.

Despite these challenges, the Service Worker approach is the most effective solution for enabling large, memory-efficient file downloads from a sandboxed WASM application, providing a seamless experience for the end-user.