Skip to content

Commit 4028765

Browse files
Maffoochclaude
andauthored
feat(locations): LocationData.code factory + get_locations() hash symmetry + location docs (#15171)
* feat(locations): LocationData.code factory + get_locations() hash symmetry Adds a LocationData.code() factory so parsers can emit static-analysis code coordinates as first-class locations: identity is file_path (+ line), while volatile context (snippet, end_line, sink/source metadata) is carried separately and omitted when unset, so downstream location subtypes can route it onto the finding reference instead of the location identity. Also fixes an asymmetry in Finding.get_locations(): the saved path filters a finding's location references to URL type before feeding the "endpoints" hash ingredient, but the unsaved path hashed every unsaved location type (dependencies included). A finding whose hash was computed before save could therefore never match its own hash recomputed after save. The unsaved path now applies the same URL-type filter. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(locations): document source code locations and location drift matching Two new pages plus cross-links from the existing docs: - asset_modelling/locations/PRO__source_code_locations.md — the Code location subtype: file+line identity, scan-managed lifecycle, the All Source Code / asset-scoped views, the movement-history trail, and the toggle-independent reference status sync. - triage_findings/finding_deduplication/PRO__location_drift_matching.md — the full feature guide: the churn problem, enabling the per-tool toggle, the two-stage matching model (stable identity + evidence passes per location type), severity re-score tolerance, scan-owned field refresh with human-edit protection, scan-pure hash adoption, location history, upgrade/enablement guidance for existing data, hash-field guidance (all-volatile fallback, location-in-title), the dependency dedupe stable-key interaction, the historical churn consolidation command, and the safeguards/limits. - PRO__deduplication_tuning.md — Content Fingerprint hash field (with the backfill-before-select ordering) and the Track Findings as Locations Change toggle, plus best-practice entries. - PRO__locations_overview.md — Source Code Locations join the subtype list. - import_data/import_intro/reimport.md — cross-link from the "line number shift" troubleshooting note to the drift matching page. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(locations): parsers emit LocationData.code for source-code findings Follows the method PR #14395 established for dependencies: rather than downstream synthesis from finding scalars alone, static parsers now emit their code coordinates as LocationData.code() into unsaved_locations, gated on V3_FEATURE_LOCATIONS, alongside the unchanged scalar fields. Context the parser already extracts rides along (snippet, end_line, source/sink objects, taint source file/line) so location subtypes can route it onto the finding reference instead of the location identity. Coverage: 45 parser directories converted (~60 emission sites) — SAST, secret scanners, and IaC scanners. SCA parsers are deliberately skipped (their file_path is a dependency manifest; they already emit LocationData.dependency), as are dynamic findings and non-source paths (container-image filesystem hits, JIRA/Confluence URLs, cloud service names). Mixed parsers (trivy, checkmarx, aws_inspector2, wizcli, xygeni, mobsf, deepfence, rusty_hog, snyk_issue_api) emit only for their code-shaped findings. Multi-location: SARIF explodes results into one finding per location, each with its own emission; xygeni secrets emits one location per leaked line. Fabricated line numbers (trufflehog's line=0 dedup placeholder) are never propagated — real coordinates only. Secret scanners never pass matched secret content as snippets. unittests/test_code_location_emission.py exercises nine representative parsers against real fixtures: emission coherence with the finding scalars, snippet context, SARIF per-location behavior, and a gate-off class proving the sweep is inert without V3 (full unittests.tools suite: 1531 tests green with V3 off). Known follow-ups (out of scope): mend and meterian have no dependency emission from #14395 either; github_vulnerability's code-scanning branch parses no location fields at all today. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(locations): LocationData.code robust to unhashable context; progpilot scalars CI (the V3-on rest-framework matrix variant) caught two coupled defects the local V3-off run could not, since the emission only fires under V3: - LocationData.code filtered unset context with `value not in {"", None}` — a SET membership test that hashes `value`, so any unhashable value crashed with "cannot use 'list' as a set element". Switched to tuple membership (`in ("", None)`), which compares by equality and never hashes — robust for every call site, present and future. - progpilot reports taint-source fields as single-element arrays (source_name=["$sql"], source_line=[610]); the sweep passed those lists straight into the context kwargs. Now collapsed to scalars via a _first() helper, with str()/int guards. Audited every other converted parser's context kwargs: all pass .text/int/ string scalars (checkmarx pathnode elements, sarif get_snippet, etc.) — progpilot was the only list-valued source. Verified against the full unittests.tools suite run with V3_FEATURE_LOCATIONS=True. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * style(locations): lint-clean the unset-context filter Replace the crash-prone set-membership filter with a truthiness dict comprehension: never hashes the value (crash-robust for any parser input), and satisfies the repo's ruff rules (no manual loop, no != "" comparison). Drops None/""/0/empty — all "no data" for the code location's context fields. Same fix, lint-clean form. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(locations): pin the get_locations() unsaved-path URL filter with a raw AbstractLocation instance The existing symmetry tests feed LocationData, which the URL-only LocationManager already drops during cleaning — they pass even without the filter. Only a raw AbstractLocation instance (a plugin's Dependency/CodeLocation model object, passed through untyped by make_abstract_locations) reaches the hash comprehension, so this case is the one that actually fails without the fix: without the type filter its value leaks into the endpoints ingredient and drifts the pre-save hash. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(lint): hoist test imports to module top for CI ruff (PLC0415) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent 39ae28d commit 4028765

72 files changed

Lines changed: 1197 additions & 43 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

docs/content/asset_modelling/locations/PRO__locations_overview.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,12 +17,13 @@ The original Endpoints model was built around URLs and IP addresses — it carri
1717
2. **Performance ceiling.** Per-Finding Endpoint_Status rows and the URL-shaped schema did not scale well at large customer volumes.
1818
3. **Components were second-class.** Software libraries lived only as denormalised fields on a Finding, so a library could not exist independently of a vulnerability — making true SBOM management impossible.
1919

20-
Locations fix all three by introducing a **base `Location` object** with a typed payload, plus dedicated **subtypes** for each asset shape. The MVP ships two subtypes:
20+
Locations fix all three by introducing a **base `Location` object** with a typed payload, plus dedicated **subtypes** for each asset shape:
2121

2222
- **URL Locations** — functional equivalent of the old Endpoints, with the same protocol/host/port/path/query/fragment fields.
2323
- **Dependency Locations** — software libraries identified by [Package URL (pURL)](https://github.com/package-url/purl-spec), used to model SBOM contents.
24+
- **[Source Code Locations](/asset_modelling/locations/pro__source_code_locations/)** — where a static-analysis finding lives in source, identified by file path and line number. Scan-managed, and the substrate for [tracking findings as their code moves](/triage_findings/finding_deduplication/pro__location_drift_matching/).
2425

25-
Future Location types under consideration include cloud provider resource IDs (AWS ARN, Azure Resource ID, GCP Full Resource Name), container images (registry/repository:tag and SHA256 fingerprints), and code repositories.
26+
Future Location types under consideration include cloud provider resource IDs (AWS ARN, Azure Resource ID, GCP Full Resource Name) and container images (registry/repository:tag and SHA256 fingerprints).
2627

2728
## Key Concepts
2829

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
---
2+
title: "Source Code Locations"
3+
description: "Code locations model where a static-analysis finding lives in source, and record its movement history as code evolves"
4+
weight: 6
5+
audience: pro
6+
---
7+
8+
**Source Code Locations** extend the Locations model to static analysis: alongside URLs (DAST) and Dependencies (SCA), a **Code** location describes where a SAST finding lives in source — identified by its **file path and line number**.
9+
10+
> Source Code Locations require the Locations feature (Beta). To enable Locations on your instance, contact [support@defectdojo.com](mailto:support@defectdojo.com).
11+
12+
## What They Model
13+
14+
Every static finding that reports a file path gets a Code location. The location's canonical value is `path/to/file.py:42` (or just the file path when the tool reports no line). Like all Locations, code locations are shared objects: two findings at the same file and line reference the same location, and the location carries per-finding and per-asset reference statuses.
15+
16+
Code locations are **scan-managed**: they are created and updated by imports and reimports, not by hand. There is no "New Source Code Location" action — the scanner is the source of truth for where code findings live.
17+
18+
## Where to Find Them
19+
20+
- **All Source Code** in the sidebar lists every code location in the instance, with the same filtering and tagging as URLs and Dependencies.
21+
- **View Source Code** in an Asset's Locations menu scopes the list to one asset.
22+
- A finding's page shows its current code location and, when the finding has moved, its **location history**.
23+
24+
## Movement History
25+
26+
Source code moves constantly: commits shift line numbers, refactors rename files. When [Location Drift Matching](/triage_findings/finding_deduplication/pro__location_drift_matching/) is enabled for a tool, a finding that moves keeps its identity, and its code location references record the trail:
27+
28+
- The finding's reference to the **old** location is mitigated and stamped with *where the finding moved* and *why the match was made* (nearest line, dataflow, file rename ...).
29+
- A reference to the **new** location is created and stays active.
30+
31+
The result is a browsable supersession chain — "this finding lived at `auth.py:42`, then `auth.py:57`, then `session.py:31`" — rendered as a timeline on the finding page. The same history mechanism covers URL moves and dependency version bumps, so all three location types share one timeline UI.
32+
33+
History is recorded from the moment Locations is enabled on the instance. Findings that moved before that keep their current location; past hops were applied but not recorded. For instances with years of pre-feature history, the [churn consolidation command](/triage_findings/finding_deduplication/pro__location_drift_matching/#consolidating-historical-churn) can reconstruct trails while merging historical close-and-recreate chains.
34+
35+
## Status Correctness
36+
37+
Code location reference statuses are kept truthful by reimport on **every** matching algorithm, whether or not drift matching is enabled:
38+
39+
- A matched finding's current code reference is synced on each reimport, so a finding that moved does not leave its old reference active forever.
40+
- The same toggle-independent sync applies to dependency references: when an SCA finding's package version bumps, the old version's reference is mitigated rather than remaining active alongside the new one.
41+
42+
## Relationship to Finding Fields
43+
44+
The finding's own `file_path` / `line` fields remain the authoritative scalars (they are what filters, hashes, and the API expose); the Code location is the shared, reference-counted view of that same coordinate. Reimport refreshes the scalars from the latest scan and the location machinery derives locations from them — the two can not drift apart.

docs/content/import_data/import_intro/reimport.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,8 @@ Reimport decides whether an incoming item matches an existing Finding using **[R
106106

107107
If you are seeing Reimport close old Findings and create new Findings when only a minor attribute changes (for example, a line number shift), tune **Reimport Deduplication** for that tool to use stable identifiers that ignore those attributes (such as Unique ID From Tool).
108108

109+
**DefectDojo Pro** can solve this directly for tools without reliable unique IDs: enabling **[Location Drift Matching](/triage_findings/finding_deduplication/pro__location_drift_matching/)** makes Reimport recognize a Finding whose location moved — a line shift, file rename, URL move, or dependency version bump — as the *same* Finding, updating it in place and preserving its location history.
110+
109111
## Reimport via API - special note
110112

111113
Note that the /reimport API endpoint can both **extend an existing Test** (apply the method in this article) **or create a new Test** with new data \- an initial call to `/import`, or setting up a Test in advance is not required.

docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,16 @@ DefectDojo Pro offers the following deduplication methods for same-tool deduplic
3939
#### Hash Code
4040
Uses a combination of selected fields to generate a unique hash. When selected, a third dropdown will appear showing the fields being used to calculate the hash.
4141

42+
##### Content Fingerprint
43+
44+
**Content Fingerprint** is a selectable hash field (available in all three configuration areas) that provides a *location-invariant* identity for static-analysis findings. It is derived from the vulnerable code snippet a tool includes in the finding — normalized so that indentation, line-number annotations, and formatting differences do not change it. Two findings about the same vulnerable code hash identically even when the code moved to a different line or file.
45+
46+
Content Fingerprint is computed for tools that include a code snippet in the finding description — including **Bandit**, **Gosec**, **Brakeman**, **Checkmarx One**, and any tool whose description carries a fenced code block or SARIF snippet.
47+
48+
> **Before selecting Content Fingerprint as a hash field**, populate fingerprints for existing findings by running `./manage.py backfill_fingerprints`. Findings imported after the feature is present get fingerprints automatically, but pre-existing findings have none — selecting the field without backfilling makes existing and incoming findings hash differently, splitting every match until the backfill runs.
49+
50+
Content Fingerprint pairs well with **CWE** for tools that embed file paths or line numbers inside their titles, where other identity fields change every time the code moves. See [Location Drift Matching](/triage_findings/finding_deduplication/pro__location_drift_matching/#choosing-hash-fields-for-tracked-tools).
51+
4252
#### Unique ID From Tool
4353
Leverages the security tool's own internal identifier for findings, ensuring perfect deduplication when the scanner provides reliable unique IDs.
4454

@@ -118,6 +128,14 @@ The following algorithm options are available for Reimport Deduplication:
118128

119129
Reimport can completely discard Findings before they are recorded, so Reimport Deduplication settings should be adjusted with caution.
120130

131+
### Track Findings as Locations Change
132+
133+
When a tool's Reimport Deduplication algorithm is **Hash Code**, an additional toggle appears: **Track findings as locations change**. With it enabled, a finding whose location moved between reimports — a line shift or file rename, a URL move, or a dependency version bump — is treated as the *same* finding, even if the tool re-scored its severity. One finding is maintained in place and its location history is preserved, instead of the old finding closing and an identical new one being created.
134+
135+
The toggle is off by default and applies only to the Hash Code reimport algorithm (tools with a reliable Unique ID From Tool already track movement through their stable IDs). Enabling it automatically re-hashes the tool's existing findings in the background so historical data participates immediately.
136+
137+
See [Location Drift Matching](/triage_findings/finding_deduplication/pro__location_drift_matching/) for how the matching works, what is preserved, and guidance for enabling it on large instances.
138+
121139
## Running Deduplication Retroactively on Existing Data
122140

123141
A common situation when first turning on Deduplication Tuning is having a large backlog of Findings that were imported *before* the dedup configuration changed. In DefectDojo Pro, you do not need to run a separate command to dedupe this historical data — **changing the Deduplication Settings for a tool automatically triggers a background re-hash of all existing Findings associated with that test type**.
@@ -144,6 +162,8 @@ For optimal results with Deduplication Tuning:
144162
- **Use Hash Code for cross-tool deduplication**: When enabling cross-tool deduplication, select fields that reliably identify the same finding across different tools (such as vulnerability name, location, and severity). **IMPORTANT** Each tool enabled for cross-tool deduplication **MUST** have the same fields selected.
145163
- **Keep cross-tool sources in the same Asset**: Cross-Tool Deduplication is Asset-scoped. Findings split across separate Assets will not dedupe even with matching hash fields. See [Cross-Tool Deduplication is Scoped to a Single Asset](#cross-tool-deduplication-is-scoped-to-a-single-asset) above.
146164
- **Avoid overly broad deduplication**: Cross-tool deduplication with too few hash fields may result in false duplicates
165+
- **Backfill before selecting Content Fingerprint**: run `./manage.py backfill_fingerprints` first, then select the field — the triggered re-hash then has fingerprints to work with. See [Content Fingerprint](#content-fingerprint) above.
166+
- **Enable location tracking between scan runs**: the toggle's automatic re-hash covers the tool's whole backlog; on large instances let it finish before the next scheduled reimport. See [Location Drift Matching](/triage_findings/finding_deduplication/pro__location_drift_matching/#enabling-on-existing-data-upgrades).
147167

148168
By tuning deduplication settings to your specific tools, you can significantly reduce duplicate noise.
149169

0 commit comments

Comments
 (0)