Skip to content

Feature: publish an immutable historical NVD cache to bootstrap cve --cache (base from release + delta from API) #414

Description

@anatoliy-balakirev

Summary

Publish the already-completed NVD years (2002 … last year) as an immutable,
project-native cache (e.g. attached to a GitHub Release), and let vulnz cve --cache
bootstrap from that base, then fetch only the recent delta from the NVD API.
This makes the first --cache run fast and reliable out of the box, without changing the
"build your own mirror" philosophy — most of the machinery is already in the tool.

Problem

Since NVD retired the legacy JSON data feeds (2023-12-15), the only way to seed a cache is
to crawl the full history through the throttled 2.0 API. In practice the initial
cve --cache (and the docker mirror) routinely takes hours and dies on 503 /
H2StreamResetException storms — the biggest onboarding pain. Every user, in every org,
repeats this same full crawl.

Proposal

  1. As part of the release/mirror pipeline, publish the completed years in the native cache
    format (nvdcve-<year>.json.gz + .meta), plus cache.properties. Completed years are
    immutable, so each refresh only re-uploads the current-year file + properties — small.
  2. On a first run, cve --cache seeds from that base, then queries the NVD API only for the
    recent window to close the gap — bounded to roughly the current year instead of the
    entire history.
  3. (Optional) A --base-url (default: the official release; overridable / disable-able) to
    auto-download and verify the base — the piece that makes it truly out-of-the-box.

How this differs from #56 and #257

It's your own canonical export, in this tool's native 2.0 format,
so users stay on your data and your code path. It's not a re-mirror; it's a published seed.

Most of this already works today

Reading NvdMirrorService (v9.0.x), the hard parts are already implemented:

  • Integrity — the tool already writes a .meta per file with sha256, size,
    gzSize, lastModifiedDate. A published base is self-verifying; nothing new needed.
  • MergewriteJsonMirror already reads and merges existing nvdcve-<year>.json.gz
    files if present.
  • Delta catch-upprocess() already reads cache.properties:lastModifiedDate and
    issues a date-range API query for the gap.

So dropping a base into --directory and topping it up is essentially the existing code
path (we run exactly this against an internal mirror). The only genuinely new pieces are
publishing the base and (optionally) fetching it via --base-url.

One constraint worth calling out

The catch-up uses a single 120-day window and falls back to a full crawl if the cache is
>120 days old
. So the published base needs refreshing within that window (monthly is
safe and cheap)
, not once a year. Optionally, looping consecutive 120-day windows would
let an arbitrarily-old base catch up without a full crawl — removing that cadence
requirement.

Payoff

  • First cve --cache goes from a multi-hour, flaky crawl to a fast, verifiable seed.
  • Much less aggregate load on NVD (everyone stops re-crawling 20+ years of history).
  • Directly kills the most common mirror failure — 503 / HTTP-2 resets during the initial pull.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions