Summary
Publish the already-completed NVD years (2002 … last year) as an immutable,
project-native cache (e.g. attached to a GitHub Release), and let vulnz cve --cache
bootstrap from that base, then fetch only the recent delta from the NVD API.
This makes the first --cache run fast and reliable out of the box, without changing the
"build your own mirror" philosophy — most of the machinery is already in the tool.
Problem
Since NVD retired the legacy JSON data feeds (2023-12-15), the only way to seed a cache is
to crawl the full history through the throttled 2.0 API. In practice the initial
cve --cache (and the docker mirror) routinely takes hours and dies on 503 /
H2StreamResetException storms — the biggest onboarding pain. Every user, in every org,
repeats this same full crawl.
Proposal
- As part of the release/mirror pipeline, publish the completed years in the native cache
format (nvdcve-<year>.json.gz + .meta), plus cache.properties. Completed years are
immutable, so each refresh only re-uploads the current-year file + properties — small.
- On a first run,
cve --cache seeds from that base, then queries the NVD API only for the
recent window to close the gap — bounded to roughly the current year instead of the
entire history.
- (Optional) A
--base-url (default: the official release; overridable / disable-able) to
auto-download and verify the base — the piece that makes it truly out-of-the-box.
How this differs from #56 and #257
It's your own canonical export, in this tool's native 2.0 format,
so users stay on your data and your code path. It's not a re-mirror; it's a published seed.
Most of this already works today
Reading NvdMirrorService (v9.0.x), the hard parts are already implemented:
- Integrity — the tool already writes a
.meta per file with sha256, size,
gzSize, lastModifiedDate. A published base is self-verifying; nothing new needed.
- Merge —
writeJsonMirror already reads and merges existing nvdcve-<year>.json.gz
files if present.
- Delta catch-up —
process() already reads cache.properties:lastModifiedDate and
issues a date-range API query for the gap.
So dropping a base into --directory and topping it up is essentially the existing code
path (we run exactly this against an internal mirror). The only genuinely new pieces are
publishing the base and (optionally) fetching it via --base-url.
One constraint worth calling out
The catch-up uses a single 120-day window and falls back to a full crawl if the cache is
>120 days old. So the published base needs refreshing within that window (monthly is
safe and cheap), not once a year. Optionally, looping consecutive 120-day windows would
let an arbitrarily-old base catch up without a full crawl — removing that cadence
requirement.
Payoff
- First
cve --cache goes from a multi-hour, flaky crawl to a fast, verifiable seed.
- Much less aggregate load on NVD (everyone stops re-crawling 20+ years of history).
- Directly kills the most common mirror failure —
503 / HTTP-2 resets during the initial pull.
Summary
Publish the already-completed NVD years (2002 … last year) as an immutable,
project-native cache (e.g. attached to a GitHub Release), and let
vulnz cve --cachebootstrap from that base, then fetch only the recent delta from the NVD API.
This makes the first
--cacherun fast and reliable out of the box, without changing the"build your own mirror" philosophy — most of the machinery is already in the tool.
Problem
Since NVD retired the legacy JSON data feeds (2023-12-15), the only way to seed a cache is
to crawl the full history through the throttled 2.0 API. In practice the initial
cve --cache(and the docker mirror) routinely takes hours and dies on503/H2StreamResetExceptionstorms — the biggest onboarding pain. Every user, in every org,repeats this same full crawl.
Proposal
format (
nvdcve-<year>.json.gz+.meta), pluscache.properties. Completed years areimmutable, so each refresh only re-uploads the current-year file + properties — small.
cve --cacheseeds from that base, then queries the NVD API only for therecent window to close the gap — bounded to roughly the current year instead of the
entire history.
--base-url(default: the official release; overridable / disable-able) toauto-download and verify the base — the piece that makes it truly out-of-the-box.
How this differs from #56 and #257
It's your own canonical export, in this tool's native 2.0 format,
so users stay on your data and your code path. It's not a re-mirror; it's a published seed.
Most of this already works today
Reading
NvdMirrorService(v9.0.x), the hard parts are already implemented:.metaper file withsha256,size,gzSize,lastModifiedDate. A published base is self-verifying; nothing new needed.writeJsonMirroralready reads and merges existingnvdcve-<year>.json.gzfiles if present.
process()already readscache.properties:lastModifiedDateandissues a date-range API query for the gap.
So dropping a base into
--directoryand topping it up is essentially the existing codepath (we run exactly this against an internal mirror). The only genuinely new pieces are
publishing the base and (optionally) fetching it via
--base-url.One constraint worth calling out
The catch-up uses a single 120-day window and falls back to a full crawl if the cache is
>120 days old. So the published base needs refreshing within that window (monthly is
safe and cheap), not once a year. Optionally, looping consecutive 120-day windows would
let an arbitrarily-old base catch up without a full crawl — removing that cadence
requirement.
Payoff
cve --cachegoes from a multi-hour, flaky crawl to a fast, verifiable seed.503/ HTTP-2 resets during the initial pull.