docs(bench): profile daytona/modal filesystem trade-offs

[signalrush]

Summary

Adds benchmark scripts and three writeups characterising the remote-FS
surfaces an agent-sdk session can land on:

• Daytona sandbox FS — Sandbox.fs.{upload,download,delete}_file
• Modal sandbox FS — Sandbox.filesystem.{copy_from_local,copy_to_local,remove}
• Modal volume (client API) — Volume.{batch_upload, read_file, remove_file}
• Modal volume (mounted in a sandbox) — /vol/<sub> accessed via in-sandbox exec

Doc-only PR; benchmark/fs/ is a sibling of the existing benchmark/load/ and benchmark/micro/ and is not imported by the server.

What's in the PR

• benchmark/fs/bench_fs.py — single-backend bench. Workloads: small (N×S, concurrent), 200m, 1g, 2g (opt-in). Reports aggregate throughput + per-op p50/p95/p99 latency.
• benchmark/fs/bench_contention.py — N parallel workers each on a disjoint subpath of the same backend; surfaces per-prefix vs per-volume contention.
• benchmark/fs/bench_mount.py — in-sandbox dd / cp -P16 / pip install pandas against /tmp (ephemeral) and /vol/<sub> (mounted Modal volume) inside one Modal sandbox.
• benchmark/fs/README.md, RESULTS.md, RESULTS_CONTENTION.md, RESULTS_MOUNT.md — usage + numbers + analysis for each.

All scripts share a Backend abstraction, asyncio.to_thread over the blocking SDK calls, and a bumped default executor (128 workers) so concurrency knobs aren't capped by the asyncio default thread pool. Each provider sandbox is provisioned with 4 vCPU / 4 GiB RAM.

Headline findings

bench_fs.py — small (200×64 KiB, conc=64), 200 MiB, 1 GiB

op	daytona_sb	modal_sb	modal_vol
200 MiB write	10.2 MiB/s	3.8 MiB/s	9.0 MiB/s
1 GiB read	48 MiB/s	51 MiB/s	99 MiB/s (CDN)
small writes p99	1.73 s	1.20 s	38.9 s ← batch_upload commits queue when many target the same prefix
small reads p99	1.58 s	5.65 s ← per-task RPC HoL	1.08 s

bench_contention.py — same total ops, spread across 1/5/10 disjoint subpaths

Modal volume parallelises near-ideally when prefixes are disjoint:

backend	10-worker write throughput	speedup vs 1 worker
modal_vol	10.1 MiB/s	25.5×
modal_sb	2.7 MiB/s	5.9×
daytona_sb	3.9 MiB/s	4.9×

Per-op p50 stays flat for modal_vol (0.37 s across 1/5/10 workers) but grows linearly for modal_sb (0.69 → 0.94 → 1.58 s) — the sandbox FS RPC channel queues per task. Daytona is mostly flat to 5 workers, then degrades.

So the earlier "modal_vol writes serialise" headline is wrong in the general case: the contention is per-prefix, not per-volume. Hammering one path serialises, fanning out across subpaths scales.

bench_mount.py — agent doing work inside the sandbox

workload	/tmp	/vol/<sub>	delta
dd 200 MiB write	460 MiB/s	463 MiB/s	parity
dd 1 GiB read	2887 MiB/s	2709 MiB/s	-6%
1000 × 4 KiB cp (P=16)	8.02 s	7.52 s	parity
pip install pandas	6.71 s	24.39 s	3.6× slower
rm -rf of pip tree	0.24 s	1.22 s	5× slower

Bulk I/O through the mount is essentially free; metadata-heavy workloads (pip install, rm -rf of deep trees) pay 3–5× because each mkdir/rename/fsync/unlink round-trips to the volume control plane while data writes stay in the local cache.

Practical guidance the report lands on

• Bulk artifacts (model weights, datasets, tarballs): put on the mounted volume; no penalty.
• Package installs / source builds / git clone: install to /tmp then bulk-copy onto /vol.
• Modal volume + many writers: fan out across subpaths. Avoid hammering a single prefix.
• Modal sandbox FS: scale by number of sandboxes, not concurrency-per-sandbox.
• Daytona sandbox FS: ~5 in-flight writers per sandbox is the sweet spot.

Caveats called out in the docs

• All numbers are single-trial; cold-call spikes (especially modal_vol/1/write) move ±15 % run-to-run.
• The modal_vol/1g/write 0.52 s outlier is a content-addressed-storage dedup artefact (the 1 GiB file repeats the same 1 MiB urandom seed used by the 200 MiB phase). Flagged in RESULTS.md.
• High dd-read rates (~2.7 GiB/s) on bench_mount.py are page-cache reads.

Test plan

This PR adds new files only; no existing code changes.

• Reviewer reviews findings + writeups for clarity / accuracy
• Optional: re-run benchmark/fs/bench_fs.py and benchmark/fs/bench_mount.py to confirm numbers are reproducible on another workspace (creds load from ~/.env)

🤖 Generated with Claude Code