Skip to content

Commit 0dee7bc

Browse files
skywalke34skywalke34
authored
feat(parsers): add Qualys VMDR CSV parser (#14453)
* docs: add Qualys VMDR parser design document Design document for new Qualys VMDR parser supporting QID and CVE CSV export formats. Includes field mappings, architecture decisions, and test strategy. Authored by T. Walker - DefectDojo * docs: add qualys_vmdr implementation plan Detailed TDD implementation plan with 13 tasks covering: - Package structure and test files - helpers.py, qid_parser.py, cve_parser.py, parser.py - Comprehensive test coverage - Documentation following enhanced format structure Authored by T. Walker - DefectDojo * feat(parser): add qualys_vmdr package structure Authored by T. Walker - DefectDojo * test(parser): add QID format test files for qualys_vmdr Authored by T. Walker - DefectDojo * test(parser): add CVE format test files for qualys_vmdr Authored by T. Walker - DefectDojo * test(parser): add failing tests for qualys_vmdr basic structure TDD: Tests written before implementation. Authored by T. Walker - DefectDojo * feat(parser): add helpers module for qualys_vmdr Shared utilities for severity mapping, date parsing, description building, endpoint parsing, and tag handling. Authored by T. Walker - DefectDojo * feat(parser): add QID format parser for qualys_vmdr Parses QID-centric CSV exports from Qualys VMDR. Authored by T. Walker - DefectDojo * feat(parser): add CVE format parser for qualys_vmdr Parses CVE-centric CSV exports with CVSS scores from NVD. Authored by T. Walker - DefectDojo * feat(parser): add main qualys_vmdr parser with format detection Auto-detects QID vs CVE format and delegates to appropriate parser. Authored by T. Walker - DefectDojo * test(parser): add field validation tests for qualys_vmdr Comprehensive tests for severity mapping, endpoints, tags, CVE fields. Also fixed CSV test files to use standard format and updated parser format detection for proper CVE format recognition. Authored by T. Walker - DefectDojo * docs(parser): add qualys_vmdr parser documentation Includes field mapping tables, severity conversion, and processing notes. Authored by T. Walker - DefectDojo * fix: handle non-standard Qualys CSV format in VMDR parser The Qualys VMDR export uses a non-standard CSV format where fields are delimited by ,"" instead of the standard "," format. This caused the parser to fail when processing real Qualys exports. Changes: - Add format detection to distinguish standard vs non-standard CSV - Add custom parsing functions for non-standard Qualys format - Handle multi-line records with embedded newlines - Both parsers (QID and CVE) now use the unified parsing logic The parser now correctly handles both test files (standard CSV) and real Qualys exports (non-standard format). Authored by T. Walker - DefectDojo * fix: correctly parse escaped quotes in Qualys non-standard CSV format The previous parsing logic used simple string splitting on ,"" which failed when field values contained escaped quotes (""""). This caused field misalignment and empty/default values in parsed findings. The fix: 1. Remove outer quotes from the row 2. Unescape row-level quote doubling ("" -> ") 3. Parse the result as standard CSV using Python's csv module This correctly handles fields containing embedded quotes like: "Description with ""quoted text"" inside" Authored by T. Walker - DefectDojo * fix: correct multi-line record detection in Qualys CSV parser The previous end-of-record detection incorrectly treated any line ending with a single quote as a complete record. This caused multi-line records (where Results field contains embedded newlines) to be split incorrectly. In Qualys non-standard format, multi-field records always end with """ (the last field's closing "" plus the record's closing "). Single quote endings within a record are just field content, not record terminators. Authored by T. Walker - DefectDojo * feat: add CVE to vulnerability_ids in CVE parser Map the CVE field to unsaved_vulnerability_ids so it appears in the Vulnerability IDs column in DefectDojo, in addition to vuln_id_from_tool. Authored by T. Walker - DefectDojo * docs: update Qualys VMDR documentation for vulnerability_ids mapping Add documentation that CVE is mapped to both vuln_id_from_tool and unsaved_vulnerability_ids for proper CVE tracking in DefectDojo. Authored by T. Walker - DefectDojo * fix: use trailing-quote heuristic for multi-line record boundary detection Replace field-count-based record boundary detection in the Qualys VMDR nonstandard CSV parser with a trailing-quote heuristic. The old approach re-parsed accumulated rows each iteration and failed on malformed quote patterns (e.g. #table cols=""3"") that produce incorrect field counts. The new _is_record_end_line() helper counts trailing quotes: exactly 3 means record end, 4+ means record end only if preceded by a comma (empty field). This is O(1) per line and correctly handles all known Qualys export patterns. Also fixes pre-existing ruff lint issues in the state machine parser. Authored by T. Walker - DefectDojo * chore: remove internal planning docs from branch These design/plan files were used during development and should not be included in the upstream PR. Authored by T. Walker - DefectDojo * docs: add CSV format handling and data cleaning details to Qualys VMDR docs Document the non-standard CSV format, multi-line record support, metadata line detection, HTML stripping, and null marker filtering. Authored by T. Walker - DefectDojo * fix(parser): support V3_FEATURE_LOCATIONS in Qualys VMDR parser Replace direct Endpoint() instantiation with the new LocationData API when V3_FEATURE_LOCATIONS is enabled, falling back to Endpoint for the legacy code path. Tests pass under both flag states. Authored by T. Walker - DefectDojo
1 parent 29fb41e commit 0dee7bc

16 files changed

Lines changed: 1018 additions & 0 deletions

File tree

‎docs/content/supported_tools/parsers/file/qualys_vmdr.md‎

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
---
2+
title: "Qualys VMDR"
3+
toc_hide: true
4+
---
5+
6+
The [Qualys VMDR](https://www.qualys.com/apps/vulnerability-management-detection-response/) parser for DefectDojo supports imports from CSV format. This parser handles both QID-centric and CVE-centric export variants from Qualys VMDR (Vulnerability Management, Detection, and Response).
7+
8+
## Supported File Types
9+
10+
The Qualys VMDR parser accepts CSV file format in two variants:
11+
12+
**QID Format:** Primary vulnerability identifier is the Qualys QID
13+
**CVE Format:** Includes CVE identifiers and CVSS scores from NVD
14+
15+
To generate these files from Qualys VMDR:
16+
17+
1. Log into your Qualys VMDR console
18+
2. Navigate to Vulnerabilities > Vulnerability Management
19+
3. Select the assets or vulnerabilities to export
20+
4. Click "Download" and select CSV format
21+
5. Choose either QID-centric or CVE-centric export option
22+
6. Upload the downloaded CSV file to DefectDojo
23+
24+
## Default Deduplication
25+
26+
The parser uses `DEDUPE_ALGO_UNIQUE_ID_FROM_TOOL_OR_HASH_CODE`, which tries `unique_id_from_tool` (populated with the Qualys QID) first and falls back to hashcode deduplication.
27+
28+
**Hashcode fields:** `title`, `component_name`, `vuln_id_from_tool`
29+
30+
For more information, see [About Deduplication](https://docs.defectdojo.com/en/working_with_findings/finding_deduplication/about_deduplication/).
31+
32+
### Sample Scan Data
33+
34+
Sample Qualys VMDR scans can be found in the [sample scan data folder](https://github.com/DefectDojo/django-DefectDojo/tree/master/unittests/scans/qualys_vmdr).
35+
36+
## Link To Tool
37+
38+
- [Qualys VMDR](https://www.qualys.com/apps/vulnerability-management-detection-response/)
39+
- [Qualys Documentation](https://www.qualys.com/documentation/)
40+
41+
## QID Format (Primary Export)
42+
43+
### QID Format Field Mapping
44+
45+
<details>
46+
<summary>Click to expand Field Mapping Table</summary>
47+
48+
| Source Field | DefectDojo Field | Notes |
49+
| ------------ | ---------------- | ----- |
50+
| Title | title | Truncated to 500 characters |
51+
| Severity | severity | Mapped: 1=Info, 2=Low, 3=Medium, 4=High, 5=Critical |
52+
| Severity | severity_justification | Preserved as "Qualys Severity: X" |
53+
| QID | unique_id_from_tool | Native Qualys vulnerability identifier |
54+
| QID | vuln_id_from_tool | Also used as vulnerability ID |
55+
| First Detected | date | Parsed to date object |
56+
| Status | active | True if "ACTIVE", False otherwise |
57+
| Solution | mitigation | Remediation guidance |
58+
| Threat | impact | Threat description |
59+
| Asset Name | component_name | Asset/server name |
60+
| Category | service | Vulnerability category |
61+
| Asset IPV4 | unsaved_endpoints | Multiple endpoints if comma-separated |
62+
| Asset IPV6 | unsaved_endpoints | Fallback if no IPv4 |
63+
| Asset Tags | unsaved_tags | Split on comma |
64+
| Results | description | Included in structured description |
65+
66+
</details>
67+
68+
### Additional Finding Settings (QID Format)
69+
70+
| Finding Field | Default Value | Notes |
71+
|---------------|---------------|-------|
72+
| static_finding | True | Vulnerability scan data |
73+
| dynamic_finding | False | Not dynamic testing |
74+
75+
## CVE Format (Extended Export)
76+
77+
### CVE Format Field Mapping
78+
79+
<details>
80+
<summary>Click to expand Field Mapping Table</summary>
81+
82+
| Source Field | DefectDojo Field | Notes |
83+
| ------------ | ---------------- | ----- |
84+
| CVE | vuln_id_from_tool | CVE identifier (e.g., CVE-2021-44228) |
85+
| CVE | unsaved_vulnerability_ids | Also added for CVE tracking |
86+
| CVE-Description | description | Prepended to structured description |
87+
| CVSSv3.1 Base (nvd) | cvssv3_score | Numeric CVSS score |
88+
| Title | title | Truncated to 500 characters |
89+
| Severity | severity | Mapped: 1=Info, 2=Low, 3=Medium, 4=High, 5=Critical |
90+
| Severity | severity_justification | Preserved as "Qualys Severity: X" |
91+
| QID | unique_id_from_tool | Native Qualys vulnerability identifier |
92+
| First Detected | date | Parsed to date object |
93+
| Status | active | True if "ACTIVE", False otherwise |
94+
| Solution | mitigation | Remediation guidance |
95+
| Threat | impact | Threat description |
96+
| Asset Name | component_name | Asset/server name |
97+
| Category | service | Vulnerability category |
98+
| Asset IPV4 | unsaved_endpoints | Multiple endpoints if comma-separated |
99+
| Asset IPV6 | unsaved_endpoints | Fallback if no IPv4 |
100+
| Asset Tags | unsaved_tags | Split on comma |
101+
| Results | description | Included in structured description |
102+
103+
</details>
104+
105+
### Additional Finding Settings (CVE Format)
106+
107+
| Finding Field | Default Value | Notes |
108+
|---------------|---------------|-------|
109+
| static_finding | True | Vulnerability scan data |
110+
| dynamic_finding | False | Not dynamic testing |
111+
112+
## Special Processing Notes
113+
114+
### Severity Conversion
115+
116+
Qualys severity levels (1-5 numeric scale) are converted to DefectDojo severity levels:
117+
- `1` → Info
118+
- `2` → Low
119+
- `3` → Medium
120+
- `4` → High
121+
- `5` → Critical
122+
123+
The original Qualys severity is preserved in the severity_justification field as "Qualys Severity: X".
124+
125+
### Endpoint Handling
126+
127+
The parser creates Endpoint objects from IP addresses:
128+
- Multiple IPv4 addresses (comma-separated) create multiple endpoints
129+
- Falls back to IPv6 if no IPv4 address is present
130+
131+
### CSV Format Handling
132+
133+
Qualys VMDR exports use a non-standard CSV format where each row is wrapped in outer quotes and fields are delimited by `,""` instead of standard `","`. The parser automatically detects and handles both standard and non-standard CSV formats.
134+
135+
**Multi-line records:** Qualys fields such as Results and Threat may contain embedded newlines. The parser correctly assembles multi-line records that span multiple lines in the CSV file, including records containing malformed quote patterns in fields like Results.
136+
137+
**Metadata lines:** Some Qualys exports include 3 metadata lines (report title, date range, column count) before the header row. The parser auto-detects whether metadata is present and skips it accordingly.
138+
139+
### Data Cleaning
140+
141+
- **HTML tags** in fields like Threat (mapped to impact) are stripped automatically
142+
- **Qualys null markers** (`'-`) are filtered and treated as empty values
143+
- **Stray quotes** left by the non-standard CSV format are cleaned from field values
144+
145+
### Format Detection
146+
147+
The parser automatically detects whether the import file is QID format or CVE format by examining the first column of the header row:
148+
- If first column is "QID" → QID format parser is used
149+
- If first column is "CVE" → CVE format parser is used

‎dojo/settings/settings.dist.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1096,6 +1096,7 @@ def generate_url(scheme, double_slashes, user, password, host, port, path, param
10961096
"n0s1 Scanner": ["description"],
10971097
"IriusRisk Threats Scan": ["title", "component_name"],
10981098
"Orca Security Alerts": ["title", "component_name"],
1099+
"Qualys VMDR": ["title", "component_name", "vuln_id_from_tool"],
10991100
}
11001101

11011102
# Override the hardcoded settings here via the env var
@@ -1365,6 +1366,7 @@ def generate_url(scheme, double_slashes, user, password, host, port, path, param
13651366
"OpenReports": DEDUPE_ALGO_HASH_CODE,
13661367
"IriusRisk Threats Scan": DEDUPE_ALGO_HASH_CODE,
13671368
"Orca Security Alerts": DEDUPE_ALGO_HASH_CODE,
1369+
"Qualys VMDR": DEDUPE_ALGO_UNIQUE_ID_FROM_TOOL_OR_HASH_CODE,
13681370
}
13691371

13701372
# Override the hardcoded settings here via the env var

‎dojo/tools/qualys_vmdr/__init__.py‎

Whitespace-only changes.

‎dojo/tools/qualys_vmdr/cve_parser.py‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
from django.conf import settings
2+
3+
from dojo.models import Finding
4+
from dojo.tools.qualys_vmdr.helpers import (
5+
build_description_cve,
6+
build_severity_justification,
7+
is_qualys_null,
8+
map_qualys_severity,
9+
parse_cvss_score,
10+
parse_endpoints,
11+
parse_locations,
12+
parse_qualys_csv_content,
13+
parse_qualys_date,
14+
parse_tags,
15+
strip_html,
16+
truncate_title,
17+
)
18+
19+
20+
class QualysVMDRCVEParser:
21+
22+
def parse(self, content):
23+
findings = []
24+
rows = parse_qualys_csv_content(content)
25+
for row in rows:
26+
finding = self._create_finding(row)
27+
if finding:
28+
findings.append(finding)
29+
return findings
30+
31+
def _create_finding(self, row):
32+
title = truncate_title(row.get("Title", ""))
33+
severity = map_qualys_severity(row.get("Severity"))
34+
severity_justification = build_severity_justification(row.get("Severity"))
35+
36+
cve = row.get("CVE", "")
37+
qid = row.get("QID", "")
38+
39+
finding = Finding(
40+
title=title,
41+
severity=severity,
42+
severity_justification=severity_justification,
43+
description=build_description_cve(row),
44+
mitigation=row.get("Solution", ""),
45+
impact=strip_html(row.get("Threat", "")),
46+
unique_id_from_tool="" if is_qualys_null(qid) else qid,
47+
vuln_id_from_tool="" if is_qualys_null(cve) else cve,
48+
date=parse_qualys_date(row.get("First Detected")),
49+
active=(row.get("Status", "").upper() == "ACTIVE"),
50+
component_name=row.get("Asset Name", ""),
51+
service=row.get("Category", ""),
52+
static_finding=True,
53+
dynamic_finding=False,
54+
)
55+
56+
cvss_score = parse_cvss_score(row.get("CVSSv3.1 Base (nvd)"))
57+
if cvss_score is not None:
58+
finding.cvssv3_score = cvss_score
59+
60+
if settings.V3_FEATURE_LOCATIONS:
61+
finding.unsaved_locations = parse_locations(
62+
row.get("Asset IPV4", ""),
63+
row.get("Asset IPV6", ""),
64+
)
65+
else:
66+
# TODO: Delete this after the move to Locations
67+
finding.unsaved_endpoints = parse_endpoints(
68+
row.get("Asset IPV4", ""),
69+
row.get("Asset IPV6", ""),
70+
)
71+
finding.unsaved_tags = parse_tags(row.get("Asset Tags", ""))
72+
73+
if not is_qualys_null(cve):
74+
finding.unsaved_vulnerability_ids = [cve]
75+
76+
return finding

0 commit comments

Comments
 (0)