Skip to content

Commit 7a89e5b

Browse files
TG1999TG1999
committed
Review all v2 pipelines
Signed-off-by: Tushar Goel <tushar.goel.dav@gmail.com>
1 parent 5126395 commit 7a89e5b

14 files changed

+114
-4
lines changed

‎CHANGELOG.rst‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,10 +13,16 @@ Version v37.0.0
1313
- We have added new models AdvisoryV2, AdvisoryAlias, AdvisoryReference, AdvisorySeverity, AdvisoryWeakness, PackageV2 and CodeFixV2.
1414
- We are using ``avid`` as an internal advisory ID for uniquely identifying advisories.
1515
- We have a new route ``/v2`` which only support package search which has information on packages that are reported to be affected or fixing by advisories.
16-
- This version introduces ``/api/v2/advisories-packages`` which has information on packages that are reported to be affected or fixing by advisories.
16+
- This version introduces ``/api/v3/packages`` which has information on packages that are reported to be affected or fixing by advisories.
1717
- Pipeline Dashboard improvements #1920.
1818
- Throttle API requests based on user permissions #1909.
1919
- Add pipeline to compute Advisory ToDos #1764
20+
- Use related advisory severity to calculate exploitibility, weighted severity and risk scores
21+
- Migrate all importers to use the new advisory models. All new advisories have a unique AVID and all importers will use this AVID as the unique identifier for advisories instead of CVE ID or other identifiers used by the data sources #1881.
22+
- Handle advisories with same and related data https://github.com/aboutcode-org/vulnerablecode/issues/2099.
23+
- Add a pipeline for exporting VulnerableCode data to FederatedCode #2110.
24+
- Plan storing of exploits and EPSS based advisories #2069.
25+
2026

2127
Version v36.1.3
2228
---------------------

‎PIPELINES-AVID.rst‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
+-------------------------------+--------------------------------------------------------------+
2+
| pipeline name | AVID |
3+
+===============================+==============================================================+
4+
| alpine_linux_importer_v2 | {package_name}/{distroversion}/{version}/{vulnerability_id} |
5+
+-------------------------------+--------------------------------------------------------------+
6+
| aosp_dataset_fix_commits | CVE ID of the record |
7+
+-------------------------------+--------------------------------------------------------------+
8+
| apache_httpd_importer_v2 | CVE ID of the record |
9+
+-------------------------------+--------------------------------------------------------------+
10+
| apache_kafka_importer_v2 | CVE ID of the record |
11+
+-------------------------------+--------------------------------------------------------------+
12+
| apache_tomcat_importer_v2 | {page_id}/{cve_id} |
13+
+-------------------------------+--------------------------------------------------------------+
14+
| archlinux_importer_v2 | AVG ID of the record |
15+
+-------------------------------+--------------------------------------------------------------+
16+
| curl_importer_v2 | CURL-CVE ID of the record |
17+
+-------------------------------+--------------------------------------------------------------+
18+
| debian_importer_v2 | {package_name}/{debian_record_id} |
19+
+-------------------------------+--------------------------------------------------------------+
20+
| elixir_security_importer_v2 | {package_name}/{file_id} |
21+
+-------------------------------+--------------------------------------------------------------+
22+
| epss_importer_v2 | CVE ID of the record |
23+
+-------------------------------+--------------------------------------------------------------+
24+
| fireeye_importer_v2 | {file_id} |
25+
+-------------------------------+--------------------------------------------------------------+
26+
| gentoo_importer_v2 | GLSA ID of the record |
27+
+-------------------------------+--------------------------------------------------------------+
28+
| github_osv_importer_v2 | ID of the OSV record |
29+
+-------------------------------+--------------------------------------------------------------+
30+
| gitlab_importer_v2 | Identifier of the GitLab community advisory record |
31+
+-------------------------------+--------------------------------------------------------------+
32+
| istio_importer_v2 | ISTIO-SECURITY-<ID> |
33+
+-------------------------------+--------------------------------------------------------------+
34+
| mattermost_importer_v2 | MMSA-<ID> |
35+
+-------------------------------+--------------------------------------------------------------+
36+
| mozilla_importer_v2 | MFSA-<ID> |
37+
+-------------------------------+--------------------------------------------------------------+
38+
| nginx_importer_v2 | First alias of the record |
39+
+-------------------------------+--------------------------------------------------------------+
40+
| nodejs_security_wg | NPM-<ID> |
41+
+-------------------------------+--------------------------------------------------------------+
42+
| nvd_importer_v2 | CVE ID of the record |
43+
+-------------------------------+--------------------------------------------------------------+
44+
| openssl_importer_v2 | CVE ID of the record |
45+
+-------------------------------+--------------------------------------------------------------+
46+
| oss_fuzz_importer_v2 | ID of the OSV record |
47+
+-------------------------------+--------------------------------------------------------------+
48+
| postgresql_importer_v2 | CVE ID of the record |
49+
+-------------------------------+--------------------------------------------------------------+
50+
| project-kb-msr-2019_v2 | Vulnerability ID of the record |
51+
+-------------------------------+--------------------------------------------------------------+
52+
| project-kb-statements_v2 | Vulnerability ID of the record |
53+
+-------------------------------+--------------------------------------------------------------+
54+
| pypa_importer_v2 | ID of the OSV record |
55+
+-------------------------------+--------------------------------------------------------------+
56+
| pysec_importer_v2 | ID of the OSV record |
57+
+-------------------------------+--------------------------------------------------------------+
58+
| redhat_importer_v2 | RHSA ID of the record |
59+
+-------------------------------+--------------------------------------------------------------+
60+
| retiredotnet_importer_v2 | retiredotnet-{file_id} |
61+
+-------------------------------+--------------------------------------------------------------+
62+
| ruby_importer_v2 | {file_id} |
63+
+-------------------------------+--------------------------------------------------------------+
64+
| suse_importer_v2 | CVE ID of the record |
65+
+-------------------------------+--------------------------------------------------------------+
66+
| ubuntu_osv_importer_v2 | ID of the OSV record |
67+
+-------------------------------+--------------------------------------------------------------+
68+
| vulnrichment_importer_v2 | CVE ID of the record |
69+
+-------------------------------+--------------------------------------------------------------+
70+
| xen_importer_v2 | XSA-<ID> |
71+
+-------------------------------+--------------------------------------------------------------+

‎vulnerabilities/pipelines/v2_importers/alpine_linux_importer.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
# See https://aboutcode.org for more information about nexB OSS projects.
88
#
99

10+
import json
1011
import logging
1112
from pathlib import Path
1213
from typing import Any
@@ -244,4 +245,5 @@ def load_advisories(
244245
references=references,
245246
affected_packages=affected_packages,
246247
url=url,
248+
original_advisory_text=json.dumps(pkg_infos, indent=2, ensure_ascii=False),
247249
)

‎vulnerabilities/pipelines/v2_importers/apache_tomcat_importer.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,7 @@ def collect_advisories(self) -> Iterable[AdvisoryDataV2]:
111111
summary=advisory_list[0].summary,
112112
affected_packages=affected_packages,
113113
url=page_url,
114+
original_advisory_text=str(content),
114115
)
115116

116117
except Exception as e:

‎vulnerabilities/pipelines/v2_importers/archlinux_importer.py‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,10 +14,13 @@
1414
from packageurl import PackageURL
1515
from univers.version_range import ArchLinuxVersionRange
1616

17+
from vulnerabilities import severity_systems
1718
from vulnerabilities.importer import AdvisoryDataV2
1819
from vulnerabilities.importer import AffectedPackageV2
1920
from vulnerabilities.importer import ReferenceV2
21+
from vulnerabilities.importer import VulnerabilitySeverity
2022
from vulnerabilities.pipelines import VulnerableCodeBaseImporterPipelineV2
23+
from vulnerabilities.severity_systems import SCORING_SYSTEMS
2124
from vulnerabilities.utils import fetch_response
2225

2326

@@ -54,6 +57,7 @@ def parse_advisory(self, record) -> AdvisoryDataV2:
5457
affected_packages = []
5558
references = []
5659
avg_name = record.get("name")
60+
severity = record.get("severity")
5761
aliases = record.get("issues", [])
5862
aliases.extend(record.get("advisories", []))
5963
summary = record.get("type", "")
@@ -92,13 +96,22 @@ def parse_advisory(self, record) -> AdvisoryDataV2:
9296
)
9397
)
9498

99+
severities = [
100+
VulnerabilitySeverity(
101+
system=severity_systems.GENERIC,
102+
value=severity,
103+
url="https://security.archlinux.org/{avg_name}.json"
104+
)
105+
]
106+
95107
return AdvisoryDataV2(
96108
advisory_id=avg_name,
97109
aliases=aliases,
98110
summary=summary,
99111
references=references,
100112
affected_packages=affected_packages,
113+
severities=severities,
101114
weaknesses=[],
102115
url=f"https://security.archlinux.org/{avg_name}.json",
103-
original_advisory_text=json.dumps(record),
116+
original_advisory_text=json.dumps(record, indent=2, ensure_ascii=False),
104117
)

‎vulnerabilities/pipelines/v2_importers/debian_importer.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
# See https://aboutcode.org for more information about nexB OSS projects.
88
#
99

10+
import json
1011
import re
1112
from typing import Any
1213
from typing import Iterable
@@ -171,6 +172,7 @@ def parse(self, pkg_name: str, records: Mapping[str, Any]) -> Iterable[AdvisoryD
171172
references=references,
172173
weaknesses=weaknesses,
173174
url=f"https://security-tracker.debian.org/tracker/{record_identifier}",
175+
original_advisory_text=json.dumps(record, indent=2, ensure_ascii=False),
174176
)
175177

176178

‎vulnerabilities/pipelines/v2_importers/epss_importer_v2.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,4 +83,5 @@ def collect_advisories(self) -> Iterable[AdvisoryDataV2]:
8383
severities=[severity],
8484
references=[references],
8585
url=self.advisory_url,
86+
original_advisory_text=",".join(epss_row),
8687
)

‎vulnerabilities/pipelines/v2_importers/mattermost_importer.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
# See https://aboutcode.org for more information about nexB OSS projects.
88
#
99

10+
import json
1011
from typing import Iterable
1112

1213
from packageurl import PackageURL
@@ -122,5 +123,7 @@ def collect_advisories(self) -> Iterable[AdvisoryDataV2]:
122123
summary=details,
123124
references=[reference],
124125
affected_packages=affected_packages,
126+
severities=severities,
125127
url=self.url,
128+
original_advisory_text=json.dumps(advisory, indent=2, ensure_ascii=False),
126129
)

‎vulnerabilities/pipelines/v2_importers/nginx_importer.py‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@ def collect_advisories(self):
6262
vulnerability_list = soup.select("li p")
6363
for vulnerability_info in vulnerability_list:
6464
ngnix_advisory = parse_advisory_data_from_paragraph(vulnerability_info)
65-
yield to_advisory_data(ngnix_advisory)
65+
yield to_advisory_data(ngnix_advisory, vulnerability_info)
6666

6767

6868
class NginxAdvisory(NamedTuple):
@@ -79,7 +79,7 @@ def to_dict(self):
7979
return self._asdict()
8080

8181

82-
def to_advisory_data(nginx_adv: NginxAdvisory) -> AdvisoryDataV2:
82+
def to_advisory_data(nginx_adv: NginxAdvisory, vulnerability_info) -> AdvisoryDataV2:
8383
"""
8484
Return AdvisoryDataV2 from an NginxAdvisory tuple.
8585
"""
@@ -150,6 +150,7 @@ def to_advisory_data(nginx_adv: NginxAdvisory) -> AdvisoryDataV2:
150150
references=nginx_adv.references,
151151
patches=nginx_adv.patches,
152152
url="https://nginx.org/en/security_advisories.html",
153+
original_advisory_text=str(vulnerability_info),
153154
)
154155

155156

‎vulnerabilities/pipelines/v2_importers/project_kb_msr2019_importer.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,7 @@ def collect_advisories(self) -> Iterable[AdvisoryDataV2]:
9090
patches=patches,
9191
references=references,
9292
url="https://github.com/SAP/project-kb/blob/main/MSR2019/dataset/vulas_db_msr2019_release.csv",
93+
original_advisory_text=",".join(row),
9394
)
9495

9596
def clean_downloads(self):

0 commit comments

Comments
 (0)