Merge branch 'main' into add-rockylinux-advisories

Author: ambuj-1211

CHANGELOG.rst

@@ -1,9 +1,118 @@
 Release notes
 =============
 
-Version (next)
+
+Version v35.1.0
+---------------------
+
+- Use AboutCode mirror for collecting CISA KEV #1685
+- Do not report ghost package as a fix for vulnerability #1679
+- Add pipeline to sort packages #1686
+- Fix urls for API #1678
+
+
+Version v35.0.0
+---------------------
+
+- Add scores in bulk search V1 API #1675
+- Add improver pipeline to flag ghost packages #644 #917 #1395 by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1533
+- Add base pipeline for importers and migrate PyPa importer to aboutcode pipeline by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1559
+- Remove dupe Package.get_non_vulnerable_versions by @pombredanne in https://github.com/aboutcode-org/vulnerablecode/pull/1570
+- Import data from GSD #706 by @ziadhany in https://github.com/aboutcode-org/vulnerablecode/pull/787
+- Add curl advisories importer by @ambuj-1211 in https://github.com/aboutcode-org/vulnerablecode/pull/1439
+- Update dependencies by @TG1999 in https://github.com/aboutcode-org/vulnerablecode/pull/1590
+- Bump django from 4.2.0 to 4.2.15 by @dependabot in https://github.com/aboutcode-org/vulnerablecode/pull/1591
+- Bump cryptography from 42.0.4 to 43.0.1 by @dependabot in https://github.com/aboutcode-org/vulnerablecode/pull/1582
+- Bump actions/download-artifact from 3 to 4.1.7 in /.github/workflows by @dependabot in https://github.com/aboutcode-org/vulnerablecode/pull/1581
+- Improve export command by @pombredanne in https://github.com/aboutcode-org/vulnerablecode/pull/1571
+- Fix typo in Kev requests import by @ziadhany in https://github.com/aboutcode-org/vulnerablecode/pull/1594
+- Prepare for release v34.0.1 by @TG1999 in https://github.com/aboutcode-org/vulnerablecode/pull/1595
+- Bump upload-artifact to v4 by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1596
+- Migrate Npm importer to aboutcode pipeline by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1574
+- Use correct regex for CVE by @pombredanne in https://github.com/aboutcode-org/vulnerablecode/pull/1599
+- Migrate Nginx importer to aboutcode pipeline by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1575
+- Migrate GitLab importer to aboutcode pipeline by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1580
+- Migrate GitHub importer to aboutcode pipeline by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1584
+- Migrate NVD importer to aboutcode pipeline by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1587
+- Match affected and fixed-by Packages by @johnmhoran in https://github.com/aboutcode-org/vulnerablecode/pull/1528
+- Add management command to commit exported data by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1600
+- Add support to Exploits model by @ziadhany in https://github.com/aboutcode-org/vulnerablecode/pull/1562
+- Fix 500 Server Error with DRF browsable API and resolve blank Swagger API documentation by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1603
+- Release v34.0.2 by @TG1999 in https://github.com/aboutcode-org/vulnerablecode/pull/1604
+- Bump VCIO version by @TG1999 in https://github.com/aboutcode-org/vulnerablecode/pull/1605
+- Bump django from 4.2.15 to 4.2.16 by @dependabot in https://github.com/aboutcode-org/vulnerablecode/pull/1608
+- Bump fetchcode from v0.3.0 to v0.6.0 by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1607
+- Use 4-tier system for storing package metadata by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1609
+- Fix vers range crash by @pombredanne in https://github.com/aboutcode-org/vulnerablecode/pull/1598
+- Add GitHub action to publish aboutcode.hashid PyPI by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1615
+- Segregate PackageRelatedVulnerability model to new models by @TG1999 in https://github.com/aboutcode-org/vulnerablecode/pull/1612
+- Add documentation for new pipeline design by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1621
+- Fix 500 error in /api/cpes endpoint by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1629
+- Migrate pysec importer to aboutcode pipeline by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1628
+- Avoid memory exhaustion during data migration by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1630
+- Add support for Calculating Risk in VulnerableCode by @ziadhany in https://github.com/aboutcode-org/vulnerablecode/pull/1593
+- Bulk create in migrations by @TG1999 in https://github.com/aboutcode-org/vulnerablecode/pull/1640
+- Update README.rst by @TG1999 in https://github.com/aboutcode-org/vulnerablecode/pull/1641
+- Prepare for release v34.1.0 by @TG1999 in https://github.com/aboutcode-org/vulnerablecode/pull/1642
+- Add V2 API endpoints by @TG1999 in https://github.com/aboutcode-org/vulnerablecode/pull/1631
+- Prepare for release v34.2.0 by @TG1999 in https://github.com/aboutcode-org/vulnerablecode/pull/1647
+- Refactor severity score model and fix incorrect suse scores by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1636
+- Add bulk search in v2 by @TG1999 in https://github.com/aboutcode-org/vulnerablecode/pull/1649
+- Prepare release v34.3.0 by @TG1999 in https://github.com/aboutcode-org/vulnerablecode/pull/1652
+- Add `on_failure` to handle cleanup during pipeline failure by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1651
+- Fix API bug by @TG1999 in https://github.com/aboutcode-org/vulnerablecode/pull/1654
+- Add reference score to package endpoint  by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1655
+- Prepare for release v34.3.2 by @TG1999 in https://github.com/aboutcode-org/vulnerablecode/pull/1656
+- Add support for storing  exploitability and weighted severity by @ziadhany in https://github.com/aboutcode-org/vulnerablecode/pull/1646
+- Avoid migrations on version bumps by @keshav-space in https://github.com/aboutcode-org/vulnerablecode/pull/1660
+- Prepare v35.0.0rc1 by @TG1999 in https://github.com/aboutcode-org/vulnerablecode/pull/1664
+
+
+
+Version v35.0.0rc1
+---------------------
+
+- Add support for storing exploitability and weighted severity #1646
+- Avoid migrations on version bumps #1660
+
+
+Version v34.3.2
+----------------
+
+- HOTFIX: Add reference score to package endpoint #1655
+
+
+Version v34.3.1
+----------------
+
+- HOTFIX: Fix API bug #1654
+
+
+Version v34.3.0
+-----------------
+
+- Add bulk search in v2 #1649 
+- Refactor severity score model and fix incorrect suse scores #1636
+
+
+Version v34.2.0
 -------------------
 
+- Add V2 API endpoints #1631
+
+
+Version v34.1.0
+-------------------
+
+- Add support for Calculating Package Vulnerability Risk #1593
+- Migrate pysec importer to aboutcode pipeline #1628
+- Fix 500 error in /api/cpes endpoint #1629
+- Add documentation for new pipeline design #1621
+- Segregate PackageRelatedVulnerability model to new models #1612
+- Add GitHub action to publish aboutcode.hashid PyPI #1615
+- Fix vers range crash #1598
+- Use 4-tier system for storing package metadata #1609
+
 
 Version v34.0.2
 -------------------

Makefile

@@ -42,6 +42,10 @@ else
 	SUDO_POSTGRES=
 endif
 
+ifeq ($(UNAME), Darwin)
+	GET_SECRET_KEY=`head /dev/urandom | base64 | head -c50`
+endif
+
 virtualenv:
 	@echo "-> Bootstrap the virtualenv with PYTHON_EXE=${PYTHON_EXE}"
 	@${PYTHON_EXE} ${VIRTUALENV_PYZ} --never-download --no-periodic-update ${VENV}

README.rst

@@ -80,7 +80,7 @@ Then run an importer for nginx advisories (which is small)
 
 .. code:: bash
 
-    docker compose exec vulnerablecode ./manage.py import vulnerabilities.importers.nginx.NginxImporter
+    docker compose exec vulnerablecode ./manage.py import nginx_importer
     docker compose exec vulnerablecode ./manage.py improve --all
 
 At this point, the VulnerableCode app and API should be up and running with
@@ -117,7 +117,7 @@ On a Debian system, use this
     make dev envfile postgres
     make test
     source venv/bin/activate
-    ./manage.py import vulnerabilities.importers.nginx.NginxImporter
+    ./manage.py import nginx_importer
     ./manage.py improve --all
     make run
 

aboutcode/hashid/CHANGELOG.rst

@@ -0,0 +1,13 @@
+Changelog
+=============
+
+
+v0.2.0 (December 05, 2024)
+---------------------------
+
+- Use 4-tier system for storing package metadata https://github.com/aboutcode-org/vulnerablecode/pull/1609
+
+v0.1.0 (September 12, 2024)
+---------------------------
+
+- Initial release of the ``aboutcode.hashid`` library.

aboutcode/hashid/__init__.py

@@ -19,6 +19,9 @@
 from packageurl import normalize_qualifiers
 from packageurl import normalize_subpath
 
+__version__ = "0.2.0"
+
+
 """
 General purpose utilities to create Vulnerability Ids aka. VCID and content-defined, hash-based
 paths to store Vulnerability and Package data using these paths in many balanced directories.

docker-compose.yml

@@ -3,10 +3,12 @@ version: "3"
 services:
   db:
     image: postgres:13
+    command: -c config_file=/etc/postgresql/postgresql.conf
     env_file:
       - docker.env
     volumes:
       - db_data:/var/lib/postgresql/data/
+      - ./etc/postgresql/postgresql.conf:/etc/postgresql/postgresql.conf
 
   vulnerablecode:
     build: .

docs/source/contributing.rst

@@ -89,7 +89,7 @@ Helpful Resources
 
 - Review our `comprehensive guide <https://scancode-toolkit.readthedocs.io/en/latest/contribute/index.html>`_
   for more details on how to add quality contributions to our codebase and documentation
-- Check this free resource on `how to contribute to an open source project on github <https://egghead.io/courses/how-to-contribute-to-an-open-source-project-on-github>`_
+- Check this free resource on `How to contribute to an open source project on github <https://egghead.io/lessons/javascript-identifying-how-to-contribute-to-an-open-source-project-on-github>`_
 - Follow `this wiki page <https://aboutcode.readthedocs.io/en/latest/contributing/writing_good_commit_messages.html>`_
   on how to write good commit messages
 - `Pro Git book <https://git-scm.com/book/en/v2>`_

docs/source/tutorial_add_importer_pipeline.rst

@@ -298,10 +298,14 @@ version management from `univers <https://github.com/aboutcode-org/univers>`_.
     **advisories_count** should never be directly added in steps.
 
 
+.. attention::
+
+   Implement ``on_failure`` to handle cleanup in case of pipeline failure.
+   Cleanup of downloaded archives or cloned repos is necessary to avoid potential resource leakage.
 
 .. note::
 
-   | Use ``make valid`` to format your code using black and isort automatically.
+   | Use ``make valid`` to format your new code using black and isort automatically.
    | Use ``make check`` to check for formatting errors.
 
 Register the Importer Pipeline

docs/source/tutorial_add_improver_pipeline.rst

@@ -187,6 +187,11 @@ methods.
             self.log(f"Successfully flagged {ghost_package_count:,d} ghost Packages")
 
 
+.. attention::
+
+   Implement ``on_failure`` to handle cleanup in case of pipeline failure.
+   Cleanup of downloaded archives or cloned repos is necessary to avoid potential resource leakage.
+
 .. note::
 
    | Use ``make valid`` to format your new code using black and isort automatically.

etc/postgresql/postgresql.conf

@@ -0,0 +1,12 @@
+# Default configuration for development build
+# DB Version: 13
+# OS Type: linux
+# DB Type: development
+# Data Storage: local
+
+listen_addresses = '*'
+max_connections = 100
+shared_buffers = 128MB
+dynamic_shared_memory_type = posix
+max_wal_size = 1GB
+min_wal_size = 80MB