Use the SCIO Pipeline Mechanism for v1 Importers/Improvers

[keshav-space]

We should adopt/reuse the pipeline mechanism in SCIO for Importers and Improvers.
With this, we would have a common programming model across tools, including ScanCode.io and PurlDB
See also:

• VCIO-next: Design new Improver #1395
• RFC: Adopt scancode.io "pipeline" approach for visit/map/collect  purldb#521

Migrate the improvers to aboutcode.pipeline:

• vulnerabilities/improvers/valid_versions.py
• vulnerabilities/improvers/vulnerability_kev.py
• vulnerabilities/improvers/vulnerability_status.py
• vulnerabilities/improvers/default.py merge this as a step in new importer pipeline
  • Completed in Add base pipeline for importers and migrate PyPa importer to aboutcode pipeline #1559
  • This is now available as a import_new_advisories step in VulnerableCodeBaseImporterPipeline.

Migrate the importers to aboutcode.pipeline:

• nvd Migrate NVD importer to aboutcode pipeline #1587
• github Migrate GitHub importer to aboutcode pipeline #1584
• gitlab Migrate GitLab importer to aboutcode pipeline #1580
• npm Migrate Npm importer to aboutcode pipeline #1574
• pypa Add base pipeline for importers and migrate PyPa importer to aboutcode pipeline #1559
• nginx Migrate Nginx importer to aboutcode pipeline #1575
• pysec Migrate pysec importer to aboutcode pipeline #1627
• alpine_linux VCIO-next: Migrate Alpine Linux importer to aboutcode pipeline #1633
• openssl
• redhat
• debian
• postgresql
• archlinux
• ubuntu
• debian_oval
• retiredotnet
• apache_httpd
• mozilla
• gentoo
• istio
• project_kb_msr2019
• suse_scores
• elixir_security
• apache_tomcat
• xen
• ubuntu_usn
• fireeye
• apache_kafka
• oss_fuzz
• ruby
• github_osv
• epss

[pombredanne]

@keshav-space do you have any commit for this yet?

[keshav-space]

@keshav-space do you have any commit for this yet?

@pombredanne here is WIP PR #1533

[tdruez]

aboutcode.pipeline 0.1 published to PyPI at https://pypi.org/project/aboutcode.pipeline/