fix: resolve ClientPayloadError on CVE downloading (issue #5779)

[jeffrey-theog06]

Summary

This PR resolves issue #5779, where the CVE database downloading process (particularly when using -n json-mirror or other multiple-file sources) hangs at 60% and crashes with:
ClientPayloadError: Response payload is not completed: <ContentLengthError: 400, message='Not enough data to satisfy content length header.'>

Root Cause

In cve_bin_tool/fetch_json_db.py, the download tasks were created as raw session.get(url) context managers inside list comprehensions. All connections (typically ~250 for a signed mirror database) were initiated concurrently. However, their response bodies were read sequentially in a single-threaded loop (await resp.read()). While the first few files were being read and saved to disk, the server ended up severing the remaining idle/un-read connections, leading to the ClientPayloadError.

Changes

1. fetch_json_db.py:
   • Implemented an async helper method download_file(self, session, url) wrapping the request inside a proper async with block, forcing immediate read of the payload body and closing the connection.
   • Updated get_download_urls to queue self.download_file(...) coroutines instead of raw request context manager objects.
   • Simplified download_files to run and await the queued download tasks concurrently.
2. test_fetch_json_db.py:
   • Added a MockRequestContextManager class mimicking aiohttp._RequestContextManager to support both await and async with protocols, ensuring test mock compatibility.

Verification

• Configured a local virtual environment and ran automated tests successfully:

  python -m pytest test/test_fetch_json_db.py
  python -m pytest test/test_source_nvd.py test/test_source_osv.py

[ffontaine]

Thanks for this PR, @Thierry4135 can you confirm if it fixes your issue?

[jeffrey-theog06]

Please do check and let me know. Any clarifications? Am open to answer. If its acceptable.. Merging the PR would be appreciated. Thanks!

[Thierry4135]

HI, I tried the modification but it didn't fix my issue:
cve-bin-tool -u now : the download of CVE stops at 60% and cve-bin-tool crashes.
cve-bin-tool -u now -l debug -n json-mirror : the download of CVE stops at 60% and cve-bin-tool crashes
Error: Response payload is not completed: <ContentLengthError: 400, message='Not enough data to satisfy content length header (received 3740096 of
9553034 bytes).'>

[jeffrey-theog06]

Ok @Thierry4135 I have again fixed the issue and it has been pushed and you can try pulling the latest commits from your branch to verify. And let me know the results.

[Thierry4135]

Those modifications fix the crash of cve-bin-tool . Some data getting are in "timeout". Adn downloading achieved at 100%.

[jeffrey-theog06]

Okay.. Happy to hear that it fixed your issue. And hope it gets merged in the main code soon

[ffontaine]

@alex-ter or @QSchulz, whenever you have a moment, could you please review this PR?

[alex-ter]

I've taken a look and this particular rabbit hole is quite deep. Here are my notes:

1. First and foremost, this PR changes two files, not one as listed in the original description: fetch_json_db.py and nvd_source.py.
2. Despite the description, the former file has nothing to do with the error reported in the original bug or the perceived fix, because it only gets used when (and only when) the --use-mirror option is specified on the command line. Neither the original bug report, nor discussion in this thread indicate that.
   1. That option seems to be an early attempt to implement what -n json-mirror currently does, and it is actually broken now. The respective test passes (test/test_fetch_json_db.py), because it doesn't really use any network connection, so it happily parses the mock network data. However, this option requires an actual URL of the mirror and tries to open the metadata.json file there, which doesn't exist in the 2.0 version of the mirror. I guess it may have been the case for 1.0, but can't check as pre-2.0 no longer exist. @jeffrey-theog06, have you actually tried running it with this option enabled, or generally verify that the fix in that file actually gets used (beyond running that test file)?
3. The change in nvd_source.py is what most likely resolves the originally reported problem, if it does, as it adds a retry plus decreases the TCPConnector's per-host connection limit. However, I cannot reproduce the original problem using the command provided in the original bug report. It may have to do with the network peculiarities at @Thierry4135's, so needs to be clarified first.

All in all, my conclusion is: this PR is quite misguided and must not be merged as-is.

The change in fetch_json_db.py is unnecessary and is actually a mistake, because that code path is not used in the error in question. We should think about removing that part of the functionality, because it seems to have been overridden by the json-mirror logic anyway.

The changes in nvd_source.py may be relevant for making downloads more robust, but I haven't yet reviewed them in detail because we anyway first need to properly troubleshoot the issue at hand. I'll post relevant instructions for @Thierry4135 in the original bug in a moment.

[jeffrey-theog06]

Hey @alex-ter , I have made the changes of reverting back as you wanted.. please go through and let me know.. !