You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Update playground-how-it-works for on-demand WASM module loading
The playground now loads only the coreutils multicall binary eagerly and
fetches the standalone modules (grep, find/locate/updatedb, diff/cmp, sed)
on demand. Update the diagrams, components table, dispatch notes and add an
on-demand loading section to match.
The [uutils playground](/playground) lets you run real Rust coreutils directly in your browser, with no server, no installation, and no network round-trips after the initial page load. This page explains the architecture behind it.
9
+
The [uutils playground](/playground) lets you run real Rust coreutils directly in your browser, with no serverand no installation. This page explains the architecture behind it.
B -->|"2. Fetch WASM binary"| C["uutils.wasm<br/>(~10 MB)"]
16
+
B -->|"2. Fetch core binary"| C["uutils.wasm<br/>(coreutils multicall)"]
17
17
A -->|"3. User types command"| D["JavaScript shell"]
18
-
D -->|"4. Execute via WASI"| E["WebAssembly runtime"]
18
+
D -->|"4a. On-demand fetch<br/>(grep, find, diff, sed…)"| F["Standalone<br/>WASM modules"]
19
+
D -->|"4b. Execute via WASI"| E["WebAssembly runtime"]
20
+
F --> E
19
21
E -->|"5. Output"| A
20
22
</pre>
21
23
22
-
Everything runs **client-side**. Once the WASM binary is downloaded, the playground works entirely offline.
24
+
Everything runs **client-side**. The page loads only the **coreutils multicall binary** up front; the optional standalone modules (`grep`, `find`/`locate`/`updatedb`, `diff`/`cmp`, `sed`) are fetched **on demand**the first time you use them — either by clicking their "Load" button or simply by running the command. Once a module is downloaded, it works entirely offline.
23
25
24
26
## Architecture
25
27
@@ -50,7 +52,8 @@ flowchart TB
50
52
|**[xterm.js](https://xtermjs.org/)**| Terminal emulator rendered in the browser. Handles cursor, colors, input, scrollback. |
51
53
|**JavaScript shell**| Parses command lines, manages pipes, handles builtins (`help`, `clear`, `cd`, `locale`), and dispatches to WASM. |
52
54
|**[browser_wasi_shim](https://github.com/bjorn3/browser_wasi_shim)**| Implements the WASI (WebAssembly System Interface) in JavaScript so that uutils can perform I/O operations. |
53
-
|**uutils.wasm**| The actual Rust coreutils, compiled with the `feat_wasm` feature to a single WASM binary containing 60+ commands. |
55
+
|**uutils.wasm**| The Rust coreutils, compiled with the `feat_wasm` feature to a single multicall WASM binary containing 60+ commands. This is the only binary loaded eagerly at page load. |
56
+
|**Standalone modules**| Separate uutils projects shipped as their own WASM binaries, **loaded on demand**: `grep.wasm` ([uutils/grep](https://github.com/uutils/grep)), `find.wasm`/`locate.wasm`/`updatedb.wasm` ([uutils/findutils](https://github.com/uutils/findutils)), `diffutils.wasm` providing `diff`/`cmp` ([uutils/diffutils](https://github.com/uutils/diffutils)), and `sed.wasm` ([uutils/sed](https://github.com/uutils/sed)). |
54
57
|**Virtual filesystem**| An in-memory filesystem backed by WASI shim `PreopenDirectory`, pre-populated with sample files. Persists across commands within a session. |
55
58
56
59
## Lifecycle of a Command
@@ -95,11 +98,14 @@ sequenceDiagram
95
98
Key details:
96
99
97
100
-**Pipeline execution**: each pipe stage is a fresh WASM instantiation. The stdout of one stage becomes the stdin of the next.
98
-
-**Command dispatch**: every command goes through `["coreutils", command, ...args]` - the WASM binary is a multicall binary, similar to BusyBox.
101
+
-**Command dispatch**: coreutils commands go through `["coreutils", command, ...args]` - the core WASM binary is a multicall binary, similar to BusyBox. Standalone modules (`grep`, `find`, `diff`, `sed`…) are invoked **directly by their own name** as `argv[0]`, since each is its own binary rather than a multicall entry.
102
+
-**On-demand loading**: if a command lives in a standalone module that hasn't been fetched yet, the shell loads that module first (printing a `loading <module>… done` notice), then runs the command.
99
103
-**Path resolution**: relative paths are resolved against a virtual `cwd` maintained by the JS shell.
100
104
101
105
## WASM Loading & Initialization
102
106
107
+
The playground splits its WASM into one eagerly-loaded core binary and several optional modules fetched lazily, keeping the initial page download small.
- The WASM binary is compiled with `WebAssembly.compileStreaming()` for best performance, with a fallback to `arrayBuffer()` if the server doesn't set `application/wasm` content-type.
122
-
- Commands are disabled until the WASM binary finishes loading. The terminal shows a loading message and a prompt appears once it's ready.
127
+
- Only the **coreutils multicall binary** loads eagerly. The standalone modules (`grep`, `find`/`locate`/`updatedb`, `diffutils`, `sed`) are **not** part of this startup fetch.
128
+
- WASM binaries are compiled with `WebAssembly.compileStreaming()` for best performance, with a fallback to `arrayBuffer()` if the server doesn't set the `application/wasm` content-type.
129
+
- Commands are disabled until the core binary finishes loading. The terminal shows a loading message and a prompt appears once it's ready.
123
130
- The `SharedArrayBuffer` polyfill stub prevents `ReferenceError` in browsers without cross-origin isolation headers.
124
131
132
+
### On-Demand Loading of Standalone Modules
133
+
134
+
<preclass="mermaid">
135
+
flowchart TB
136
+
A["User runs grep/find/diff/sed<br/>(or clicks its Load button)"] --> B{"Module already<br/>compiled?"}
137
+
B -->|Yes| F["Run command"]
138
+
B -->|No| C{"Fetch in flight?"}
139
+
C -->|Yes| D["Share the existing<br/>in-flight fetch"]
140
+
C -->|No| E["fetch + compileStreaming<br/>/wasm/<module>.wasm"]
141
+
D --> G["Dispatch<br/>uutils:program-loaded"]
142
+
E --> G
143
+
G --> F
144
+
</pre>
145
+
146
+
- Each module is fetched **once**, the first time it's needed. Concurrent callers share a single in-flight fetch, and the compiled module is cached for the rest of the session.
147
+
- A single module can back several commands: `diffutils.wasm` provides both `diff` and `cmp`, and the **"Load" button for `find`** brings in `find`, `locate` and `updatedb` together.
148
+
- On success the page dispatches a `uutils:program-loaded` event, which the "Load" buttons listen for so their label flips to `✓ <name> loaded` — whether the module was loaded by the button or by running the command.
149
+
- If a module's binary isn't present (e.g. local dev without a CI build), the command reports that it's unavailable instead of breaking the terminal.
150
+
125
151
## Command Parsing & Pipes
126
152
127
153
The shell implements a simple but functional parser:
@@ -192,6 +218,8 @@ flowchart LR
192
218
193
219
Utilities are excluded when they depend on OS-level syscalls not available in WASI - for example, `df` needs filesystem stats, `du` needs directory traversal with metadata, and `chown`/`chcon` need permission and SELinux APIs.
194
220
221
+
> **Note:**`grep`, `find`/`locate`/`updatedb`, `diff`/`cmp` and `sed` are **not** part of the coreutils `feat_wasm` set — they live in separate uutils projects ([grep](https://github.com/uutils/grep), [findutils](https://github.com/uutils/findutils), [diffutils](https://github.com/uutils/diffutils), [sed](https://github.com/uutils/sed)) and are compiled to their own WASM modules, loaded on demand as described above. (`xargs` is intentionally absent: it must spawn child processes, which the browser WASI sandbox can't do.)
0 commit comments