Commit 4028765
feat(locations): LocationData.code factory + get_locations() hash symmetry + location docs (#15171)
* feat(locations): LocationData.code factory + get_locations() hash symmetry
Adds a LocationData.code() factory so parsers can emit static-analysis code
coordinates as first-class locations: identity is file_path (+ line), while
volatile context (snippet, end_line, sink/source metadata) is carried
separately and omitted when unset, so downstream location subtypes can route
it onto the finding reference instead of the location identity.
Also fixes an asymmetry in Finding.get_locations(): the saved path filters a
finding's location references to URL type before feeding the "endpoints"
hash ingredient, but the unsaved path hashed every unsaved location type
(dependencies included). A finding whose hash was computed before save could
therefore never match its own hash recomputed after save. The unsaved path
now applies the same URL-type filter.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(locations): document source code locations and location drift matching
Two new pages plus cross-links from the existing docs:
- asset_modelling/locations/PRO__source_code_locations.md — the Code
location subtype: file+line identity, scan-managed lifecycle, the
All Source Code / asset-scoped views, the movement-history trail,
and the toggle-independent reference status sync.
- triage_findings/finding_deduplication/PRO__location_drift_matching.md —
the full feature guide: the churn problem, enabling the per-tool
toggle, the two-stage matching model (stable identity + evidence
passes per location type), severity re-score tolerance, scan-owned
field refresh with human-edit protection, scan-pure hash adoption,
location history, upgrade/enablement guidance for existing data,
hash-field guidance (all-volatile fallback, location-in-title),
the dependency dedupe stable-key interaction, the historical churn
consolidation command, and the safeguards/limits.
- PRO__deduplication_tuning.md — Content Fingerprint hash field (with
the backfill-before-select ordering) and the Track Findings as
Locations Change toggle, plus best-practice entries.
- PRO__locations_overview.md — Source Code Locations join the subtype
list.
- import_data/import_intro/reimport.md — cross-link from the "line
number shift" troubleshooting note to the drift matching page.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(locations): parsers emit LocationData.code for source-code findings
Follows the method PR #14395 established for dependencies: rather than
downstream synthesis from finding scalars alone, static parsers now emit
their code coordinates as LocationData.code() into unsaved_locations,
gated on V3_FEATURE_LOCATIONS, alongside the unchanged scalar fields.
Context the parser already extracts rides along (snippet, end_line,
source/sink objects, taint source file/line) so location subtypes can
route it onto the finding reference instead of the location identity.
Coverage: 45 parser directories converted (~60 emission sites) — SAST,
secret scanners, and IaC scanners. SCA parsers are deliberately skipped
(their file_path is a dependency manifest; they already emit
LocationData.dependency), as are dynamic findings and non-source paths
(container-image filesystem hits, JIRA/Confluence URLs, cloud service
names). Mixed parsers (trivy, checkmarx, aws_inspector2, wizcli, xygeni,
mobsf, deepfence, rusty_hog, snyk_issue_api) emit only for their
code-shaped findings. Multi-location: SARIF explodes results into one
finding per location, each with its own emission; xygeni secrets emits
one location per leaked line. Fabricated line numbers (trufflehog's
line=0 dedup placeholder) are never propagated — real coordinates only.
Secret scanners never pass matched secret content as snippets.
unittests/test_code_location_emission.py exercises nine representative
parsers against real fixtures: emission coherence with the finding
scalars, snippet context, SARIF per-location behavior, and a gate-off
class proving the sweep is inert without V3 (full unittests.tools suite:
1531 tests green with V3 off).
Known follow-ups (out of scope): mend and meterian have no dependency
emission from #14395 either; github_vulnerability's code-scanning branch
parses no location fields at all today.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(locations): LocationData.code robust to unhashable context; progpilot scalars
CI (the V3-on rest-framework matrix variant) caught two coupled defects the
local V3-off run could not, since the emission only fires under V3:
- LocationData.code filtered unset context with `value not in {"", None}` — a
SET membership test that hashes `value`, so any unhashable value crashed with
"cannot use 'list' as a set element". Switched to tuple membership (`in
("", None)`), which compares by equality and never hashes — robust for every
call site, present and future.
- progpilot reports taint-source fields as single-element arrays
(source_name=["$sql"], source_line=[610]); the sweep passed those lists
straight into the context kwargs. Now collapsed to scalars via a _first()
helper, with str()/int guards.
Audited every other converted parser's context kwargs: all pass .text/int/
string scalars (checkmarx pathnode elements, sarif get_snippet, etc.) — progpilot
was the only list-valued source. Verified against the full unittests.tools suite
run with V3_FEATURE_LOCATIONS=True.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* style(locations): lint-clean the unset-context filter
Replace the crash-prone set-membership filter with a truthiness dict
comprehension: never hashes the value (crash-robust for any parser input),
and satisfies the repo's ruff rules (no manual loop, no != "" comparison).
Drops None/""/0/empty — all "no data" for the code location's context
fields. Same fix, lint-clean form.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(locations): pin the get_locations() unsaved-path URL filter with a raw AbstractLocation instance
The existing symmetry tests feed LocationData, which the URL-only
LocationManager already drops during cleaning — they pass even without
the filter. Only a raw AbstractLocation instance (a plugin's
Dependency/CodeLocation model object, passed through untyped by
make_abstract_locations) reaches the hash comprehension, so this case is
the one that actually fails without the fix: without the type filter its
value leaks into the endpoints ingredient and drifts the pre-save hash.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore(lint): hoist test imports to module top for CI ruff (PLC0415)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>1 parent 39ae28d commit 4028765
72 files changed
Lines changed: 1197 additions & 43 deletions
File tree
- docs/content
- asset_modelling/locations
- import_data/import_intro
- triage_findings/finding_deduplication
- dojo
- finding
- tools
- api_sonarqube
- aws_inspector2
- bandit
- bearer_cli
- brakeman
- checkmarx_cxflow_sast
- checkmarx_one
- checkmarx
- checkov
- codechecker
- coverity_api
- coverity_scan
- cred_scan
- deepfence_threatmapper
- detect_secrets
- eslint
- fortify
- generic
- ggshield
- github_sast
- github_secrets_detection_report
- gitlab_sast
- gitlab_secret_detection_report
- gitleaks
- gosec
- hadolint
- hcl_asoc_sast
- horusec
- huskyci
- kics
- kiuwan
- mobsf
- noseyparker
- php_security_audit_v2
- pmd
- progpilot
- pwn_sast
- rubocop
- rusty_hog
- sarif
- semgrep_pro
- semgrep
- snyk_issue_api
- solar_appscreener
- sonarqube
- spotbugs
- talisman
- terrascan
- tfsec
- trivy_operator
- trivy
- trufflehog3
- trufflehog
- veracode
- whispers
- wizcli_common_parsers
- xanitizer
- xygeni
- unittests
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 3 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
20 | | - | |
| 20 | + | |
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
| 24 | + | |
24 | 25 | | |
25 | | - | |
| 26 | + | |
26 | 27 | | |
27 | 28 | | |
28 | 29 | | |
| |||
Lines changed: 44 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
106 | 106 | | |
107 | 107 | | |
108 | 108 | | |
| 109 | + | |
| 110 | + | |
109 | 111 | | |
110 | 112 | | |
111 | 113 | | |
| |||
Lines changed: 20 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
39 | 39 | | |
40 | 40 | | |
41 | 41 | | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
42 | 52 | | |
43 | 53 | | |
44 | 54 | | |
| |||
118 | 128 | | |
119 | 129 | | |
120 | 130 | | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
121 | 139 | | |
122 | 140 | | |
123 | 141 | | |
| |||
144 | 162 | | |
145 | 163 | | |
146 | 164 | | |
| 165 | + | |
| 166 | + | |
147 | 167 | | |
148 | 168 | | |
149 | 169 | | |
| |||
0 commit comments