bundles: concurrently prefetch the cold-cache working set (~12.7x faster cold builds)#1379
bundles: concurrently prefetch the cold-cache working set (~12.7x faster cold builds)#1379hhh2210 wants to merge 4 commits into
Conversation
On a cold cache, the engine pulls a few hundred small support files to build the format file, and each one is a separate, serial HTTP byte-range request. The compile is therefore dominated by round-trip latency times the file count: a minimal document spends ~130s wall but only ~1.5s of CPU, i.e. >98% of the time is spent waiting on serialized network I/O. This records the set of files a bundle is observed to need (a small per-bundle ".prefetch" manifest, appended to as files are fetched) and, on a subsequent cold cache, replays it concurrently before handing control to the engine. The engine's later on-demand requests then all hit warm cache. - CachableBundle::batch_open: fetch many files at once, with a serial default implementation. - ItarBundle::batch_open: fan requests out across a worker pool (default 16, TECTONIC_PREFETCH_CONCURRENCY to tune), each worker with its own range reader / pooled connection. - BundleCache: record + concurrent replay of the working set. The manifest is appended eagerly rather than on Drop, since the CLI exits via process::exit. Prefetch is best-effort: any file it fails to fetch falls back to the normal on-demand path, so it never fails a build, and output is unchanged. In-house benchmark (minimal article, isolated cache, egress via an overseas proxy): serial cold compile 129-131s vs concurrent prefetch 18.2s, a ~7.1x speedup, identical PDF output. Refs tectonic-typesetting#446, tectonic-typesetting#856, tectonic-typesetting#1224.
There was a problem hiding this comment.
Pull request overview
This PR accelerates cold-cache builds by teaching the bundles layer to prefetch a previously-recorded “working set” of bundle files concurrently, so the TeX engine’s later synchronous reads mostly hit the local cache instead of issuing hundreds of serial HTTP range requests.
Changes:
- Add a
CachableBundle::batch_openAPI (default serial implementation) to support batch fetching of bundle files. - Implement concurrent batch downloading in
ItarBundle(worker pool + per-worker range reader) and refactor retrying range reads into a shared helper. - Add a per-bundle prefetch manifest in
BundleCacheand a one-time-per-session prefetch step before the first file lookup.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| crates/bundles/src/lib.rs | Introduces batch_open on CachableBundle with a correct serial default implementation. |
| crates/bundles/src/itar.rs | Adds concurrent batch_open override for the .tar network bundle and shared retrying range-read helper. |
| crates/bundles/src/cache.rs | Records fetched file names to a manifest and replays them via prefetch() to warm the on-disk cache. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| match self.ensure_index() { | ||
| Ok(_) => {} | ||
| Err(e) => return OpenResult::Err(e), | ||
| }; | ||
|
|
||
| let mut v = Vec::with_capacity(info.length); | ||
| tt_note!(status, "downloading {}", info.name); | ||
|
|
||
| // Edge case for zero-sized reads | ||
| // (these cause errors on some web hosts) | ||
| if info.length == 0 { | ||
| return OpenResult::Ok(InputHandle::new_read_only( | ||
| match read_file_with_retries(self.reader.as_mut().unwrap(), info, status) { | ||
| OpenResult::Ok(v) => OpenResult::Ok(InputHandle::new_read_only( |
There was a problem hiding this comment.
Good catch — this is a real latent crash (reachable whenever the index is cache-loaded via initialize_index but a needed blob isn't cached). Fixed in 3d4ea23 by moving the idempotent connect_reader() ahead of the early return in ensure_index; it does no network I/O until a range is actually read.
| let concurrency = std::env::var("TECTONIC_PREFETCH_CONCURRENCY") | ||
| .ok() | ||
| .and_then(|s| s.parse::<usize>().ok()) | ||
| .filter(|n| *n > 0) | ||
| .unwrap_or(DEFAULT_PREFETCH_CONCURRENCY) | ||
| .min(infos.len().max(1)); | ||
|
|
There was a problem hiding this comment.
Agreed. Added a hard cap MAX_PREFETCH_CONCURRENCY = 64 that's applied even when TECTONIC_PREFETCH_CONCURRENCY is set explicitly (37eb9e3), so a large manifest or an oversized env value can't spawn an unreasonable number of threads/connections.
| if file_create_write(&tmp_path, |f| f.write_all(&bytes)).is_err() { | ||
| continue; | ||
| } | ||
| let _ = fs::rename(&tmp_path, &target); |
There was a problem hiding this comment.
Fixed in 37eb9e3: the prefetch path now removes the *-tmp-pid* file if the rename into place fails (e.g. another process won the race), instead of leaving a stale artifact behind.
| // Record that this file is part of our working set, so future cold | ||
| // caches can prefetch it concurrently. We append eagerly rather than | ||
| // relying on `Drop`, because the CLI exits via `process::exit` (which | ||
| // skips destructors). | ||
| if self.touched.insert(info.name().to_owned()) { | ||
| self.manifest_dirty = true; | ||
| self.append_to_manifest(info.name()); | ||
| } |
There was a problem hiding this comment.
Fair concern. The recorded set is a subset of the bundle's contents so it converges rather than growing truly without bound, but to make that explicit I added MAX_MANIFEST_ENTRIES = 4096 as a backstop (37eb9e3): once reached we stop recording new names, which bounds both the manifest and the cold-cache prefetch fan-out. Note prefetch only fetches files not already cached and is concurrent + best-effort, so fetching a modest superset is still far cheaper than the serial on-demand path it replaces.
Self-contained A/B script that runs one binary over one cache state and toggles only TECTONIC_PREFETCH_CONCURRENCY (1 vs 16), so reviewers can reproduce the cold-build speedup without a hand-built scenario.
hhh2210
changed the title
ItarBundle::ensure_index returned early when the index was already initialized, which happens when BundleCache loads the index from its on-disk cache via initialize_index rather than downloading it. In that case self.reader stayed None, so the on-demand fetch path (open_fileinfo -> read_file_with_retries) panicked unwrapping it. This is reachable whenever the bundle index is cached but a needed file blob is not (e.g. a fresh machine syncing only the index, or a cache whose blobs were cleared), and it is surfaced more readily by the new prefetch path. Move the idempotent connect_reader call ahead of the early return; it does no network I/O until a range is actually read.
- Cap prefetch concurrency at MAX_PREFETCH_CONCURRENCY (64) even when TECTONIC_PREFETCH_CONCURRENCY is set, so a large manifest or an oversized env value can't spawn an unreasonable number of threads. - Remove the temporary file in the prefetch path if the rename into place fails (e.g. another process won the race), instead of leaving a stale *-tmp-pid* artifact behind. - Bound the prefetch manifest at MAX_MANIFEST_ENTRIES (4096) so it (and the resulting cold-cache fan-out) can't grow without bound across many documents sharing one bundle.
|
@CraftSpider ready for review |
Problem
On a cold cache, even a trivial document is painfully slow to compile, and the cost is almost entirely network. Building the LaTeX format file pulls a few hundred small support files, and each one is a separate, serial HTTP byte-range request (
BundleCache::input_open_name->fetch_file->open_fileinfo, one file at a time, driven synchronously by the engine).So the wall time is roughly
round-trip-latency x file-count. Measured on a minimalarticle:This is a long-standing pain point: #446 (a
minimaldoc taking half an hour, with the diagnosis that it's request-count not bandwidth), #856, and #1224.Approach
Borrowed from how fast package managers (e.g. uv) handle this: the slow part isn't bandwidth, it's issuing hundreds of dependent round-trips one at a time. You can't parallelize the engine's synchronous on-demand requests directly, so first make the working set known, then fetch it concurrently.
BundleCacheappends every file it fetches from the network to a small per-bundle manifest,data/<hash>.prefetch(one name per line). It's appended eagerly rather than onDrop, because the CLI exits viaprocess::exitand skips destructors.BundleCache::prefetchresolves the manifest to the files not yet cached and fetches them all at once. The engine's later on-demand reads then hit warm cache.CachableBundle::batch_open(serial default impl);ItarBundleoverrides it to fan requests across a worker pool (default 16, tunable viaTECTONIC_PREFETCH_CONCURRENCY), each worker with its own range reader / pooled connection.Prefetch is strictly best-effort: any file it can't fetch is left for the normal on-demand path, so it never fails a build, and the output PDF is byte-for-byte unchanged (this only changes how files are fetched, not what).
Benchmark
Minimal
article(title + a paragraph + one display equation), isolated cache,tectonic -X compile, macOS arm64, egress via an overseas proxy. Each row is a genuine cold cache; the concurrent row keeps only the manifest (modeling either a manifest shipped with the bundle, or simply a user's second cold build aftercache clean/ a fresh CI cache).The two serial runs agree to within ~1.5%, so this isn't network jitter. Higher concurrency pushes it further; 16 is a conservative default. The remaining ~18s is real typesetting (~1.5s) plus the index/digest round-trips plus the prefetch burst and writing 266 files to disk.
Reproduce it yourself
A self-contained script ships with this PR:
dev/bench-cold-prefetch.sh. To keep the comparison honest it runs the same binary over the same cache state and toggles only one knob,TECTONIC_PREFETCH_CONCURRENCY, so the measured delta is attributable to concurrency alone rather than to a hand-built scenario:serialarm —TECTONIC_PREFETCH_CONCURRENCY=1: one range request at a time, a faithful proxy for the pre-change on-demand path.concurrentarm —TECTONIC_PREFETCH_CONCURRENCY=16: this change's default.Both arms start from an identical warm bundle index + warm prefetch manifest, cold file blobs state, which is exactly what a repeated cold build hits (you've compiled the doc before, but the cached blobs are gone: fresh machine, cleared cache, ephemeral CI runner).
Sample run on the same machine (overseas proxy, so the absolute numbers are latency-inflated; the ratio is the point and is what reproduces across networks):
Absolute timings scale with your latency to the bundle host, but the speedup holds because the serial arm pays that latency once per file while the concurrent arm amortizes it across the worker pool.
Notes / scope
ItarBundle::ensure_indexreturned early when the index was already initialized (as happens whenBundleCacheloads the index from its on-disk cache viainitialize_indexinstead of downloading it), leavingself.readerasNoneso the on-demandopen_fileinfopath panicked onunwrap. This is reachable whenever the index is cached but a needed file blob is not, and the prefetch path makes it easier to hit. Fixed by moving the idempotentconnect_readerahead of the early return (no network I/O until a range is actually read).ItarBundle(the default.tarbundle) gets the concurrent override here.TTBNetBundlekeeps the serial defaultbatch_open(correct, just not yet accelerated) and is an easy follow-up.cargo clippy -p tectonic_bundlesis clean.Refs #446, #856, #1224.
Made with Cursor