ref(builds-cacher): cache-only Node server + API libc-filter fix#1075
Merged
Conversation
2 tasks
coolaj86
force-pushed
the
ref-cache-only-readonly
branch
21 times, most recently
from
b063c35 to
4c35147
Compare
coolaj86
commented
May 8, 2026
coolaj86
changed the title
coolaj86
force-pushed
the
ref-cache-only-readonly
branch
2 times, most recently
from
3c1eb8a to
b1d340e
Compare
Make _webi/builds-cacher.js and _webi/transform-releases.js read exclusively from ~/.cache/webi/legacy/<name>.json and remove every code path that fetched from upstream or wrote to disk. The Go cache daemon (webicached) is now the sole writer; the Node server is a thin reader. builds-cacher.js: - Resolve cache files via Os.homedir() + '/.cache/webi/legacy/' instead of the cacheDir argument. Drop the 'caches' constructor parameter. - Remove getLatestBuilds / getLatestBuildsInner — they require()d per-package releases.js modules, fetched upstream, and wrote <yyyy-mm>/<name>.json + .updated.txt to disk. - Remove the process.nextTick stale-refresh hook in _doGetPackages. Cold reads return what's on disk; if the file is missing, return empty meta instead of fetching. - Remove freshenRandomPackage and its supporting state (bc._staleNames, bc._freshenTimeout, bc._staleAge). The hourly background freshener competed with webicached for the same files. - In getProjectTypeByEntry, decide selfhosted vs valid by probing for the cache file rather than require()-ing releases.js. Drop the not_found / 'PROBLEM/SOLUTION/npm clean-install' diagnostic in getProjectsByType — the cache-file probe replaces the module-load failure mode. transform-releases.js: - Remove Releases.get and the _normalize import. Replace getCachedReleases's fetch+race+stale-age machinery with a single Fs.readFile of ~/.cache/webi/legacy/<pkg>.json. - Drop the in-process version re-sort in createFormatsSorter; the cache file is already version-sorted by webicached, so the sorter only re-orders within the same version. No callers' public signatures change. Every other file is untouched — the per-package releases.js modules, _common/*.js fetchers, and _webi/normalize.js still exist on disk but are no longer reachable from the request path.
Lexver.parsePrefix now produces a true string-prefix of parseVersion when the input has a release suffix (e.g. '1.0.0-beta', '2025.11.15-15.42.45'). Unblocks pinned-version queries with a non-trivial release suffix, including the 'webi zig.vim' alias chain which redirects through 'vim-zig@2025.11.15-15.42.45' at install time. See webinstall/webi-build-classifier#22.
filterReleases unconditionally rejected libc=musl entries unless the host was libc=musl, even when the caller never specified a libc in the request. serve-releases.js defaults the libc parameter to 'libc' (the catch-all glibc-host bucket the installer-side resolver uses), so the website's release table and the WEBI_RELEASES probe were both stripped of every musl entry that the cache actually contained — even though the installer would happily consider those builds on a glibc host (its waterfall is [none, gnu, musl, libc]). Treat libc='libc' (and missing) as 'no preference' so the filter only runs when the caller pinned a real libc (musl, gnu, msvc, etc.). Specific-libc queries (?libc=musl, ?libc=gnu) still filter exactly as before.
coolaj86
force-pushed
the
ref-cache-only-readonly
branch
from
b1d340e to
70067a6
Compare
This was referenced May 8, 2026
pull Bot
pushed a commit
to pianomanx/webi-installers
that referenced
this pull request
May 14, 2026
…e cache path The Node server's read path now goes through ~/.cache/webi/legacy/ only (see webinstall#1075). A handful of supporting tools and tests still carried references to the obsolete upstream-fetcher modules and the old year-month cache layout. Update them in place; the actual deletion of the orphaned modules follows in webinstall#1076. - _webi/classify-one.js — read from ~/.cache/webi/legacy/<pkg>.json instead of ../_cache/<yearMonth>/<pkg>.json. - _webi/builds-cacher-test.js — drop the bc.freshenRandomPackage(...) call; the freshener was removed when fetching went away. - _webi/builds.js — drop the //Releases: Releases stub comment. - _webi/lint-builds.js — drop two now-unused require()s. - _webi/test.js — adjust a single reference to the post-cleanup shape.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
~/.cache/webi/legacy/<name>.json/api/releasesfiltering musl entries when the caller didn't pin a libcWhat changed
_webi/builds-cacher.js:Os.homedir() + '/.cache/webi/legacy/'instead of thecacheDir/cachesargument.getLatestBuilds/getLatestBuildsInner— theyrequire()d per-packagereleases.js, fetched upstream, and wrote<yyyy-mm>/<name>.json+.updated.txtto disk.process.nextTickstale-refresh hook in_doGetPackages. Cold reads return what's on disk; if the file is missing, return empty meta instead of fetching.freshenRandomPackageand supporting state (bc._staleNames,bc._freshenTimeout,bc._staleAge). The hourly background freshener was competing with webicached for the same files.getProjectTypeByEntry, decides selfhosted vs valid by probing for the cache file rather thanrequire()-ingreleases.js._webi/transform-releases.js:Releases.getand the_normalizeimport.getCachedReleases's fetch+race+stale-age machinery with a singleFs.readFileof~/.cache/webi/legacy/<pkg>.json.fix(api): the libc filter ran unconditionally — even when the caller didn't pin a libc, the defaultlibc='libc'fromserve-releases.jswas triggering it and stripping everylibc=muslentry from listings. Treat the catch-all'libc'value as 'no preference' so the website's release table and theWEBI_RELEASESprobe see what the cache actually contains. Specific?libc=musl/?libc=gnuqueries still filter exactly as before._webi/build-classifier:Lexver.parsePrefixzero-padding fix for<pkg>@<exact-version-with-suffix>queries).No public signatures change. Per-package
releases.js,_common/*.js, and_webi/normalize.jsstill exist on disk but are no longer reachable from the request path (cleanup follows in a separate PR).Test plan (verified against
https://webinstall.dev/)~/.cache/webi/legacy/exists on prod and is populated (100 packages).curl https://webinstall.dev/api/releases/go.json?limit=1returns valid version (1.26.3, not0.0.0).curl -A 'curl/8.5 Linux x86_64' https://webinstall.dev/nodereturns#!/bin/shinstaller.40 none / 44 gnu / 16 musl). On prod the cache currently has 0 musl-tagged entries (50 none / 50 gnu) because webicached classifies static-musl Rust builds aslibc=none. The fix is forward-looking once musl-tagged entries land in prod's cache.?libc=muslstrict-filter test (curl '…&libc=musl&limit=5') is not reachable as written:webinstall.dev/lib/serve-releases.js:37runs the libc query throughUaDetect.arch()which mangles non-arch strings to'error'before reaching this filter. Separate fix needed in that file (out of scope here).~/.cache/webi/from the Node process" are deploy-host checks the maintainer will confirm.