fix(dedupe): stable original across scan orders via content-ordered finding creation

[Maffooch]

[sc-13656]

Problem

Deduplication chose the "original" among findings that collide on the dedupe key by lowest DB id — i.e. whichever the scanner listed first in the file. Scanners don't guarantee a stable finding order between scans, so on reimport the original/duplicate assignment flips back and forth between scans even though the underlying vulnerabilities never change. Reported via a customer support ticket; observed with Fortify but not tool-specific (any parser without a stable export order can trigger it).

Fix

Make creation order content-stable on the reimport path instead of making dedupe comparisons content-aware:

• Add deduplication_ordering_key() — a stable key derived from finding content (hash_code, unique_id_from_tool, file_path, line, title, then id as final tiebreak).
• DefaultReImporter now fully prepares parsed findings up front (test/service/hash_code) and creates them in this key order, so among findings created by one reimport "lowest id" equals "canonical by content".
• Every dedupe comparison stays a pure lowest-id-wins comparison (semantics unchanged from before this PR).

Because dedupe picks the lowest id among collisions, the surviving original for intra-report collisions on reimport is now reproducible across re-scans regardless of the scanner's export order — in every execution mode (eager and async post-processing), for any report size — while pre-existing findings (always lower ids) keep winning, so an already-established original never flips.

The plain import path is untouched: an import's intra-report winner is still the first-in-file finding, but once established it is pre-existing for every later import/reimport and therefore permanently stable — the flip between scans was a reimport phenomenon. (Keeping the import path unchanged also keeps order-coupled fixtures such as the JIRA VCR cassettes valid.)

Why not compare by content inside dedupe?

An earlier revision of this PR stamped dedupe batches (_dedupe_batch_min_id) and content-compared "intra-batch" candidates inside _is_candidate_older. That made the "older" relation depend on which batch evaluated it, with two concrete defects:

• For reports larger than IMPORT_REIMPORT_DEDUPE_BATCH_SIZE (1000), a colliding pair split across two post-processing batches was evaluated with different batch minimums — each side could consider the other "older" (antisymmetry violation), so two concurrently running dedupe batches could mark the pair as duplicates of each other.
• The dedupe management command re-processes pre-existing findings through the same batch path, so re-running dedupe could flip long-established originals.

Sorting creation order keeps the comparison globally antisymmetric and makes both problems structurally impossible. A regression test for the antisymmetry property and a "re-running dedupe never flips an established original" test are included.

Backward compatibility

• No migration; no change to existing data at upgrade time. Dedupe only runs during import/reimport (or when explicitly re-run).
• Pre-existing findings always keep the original role — covered by a dedicated test and verified hands-on: data imported under the pre-PR code, then the reversed report reimported under this PR → 0 findings changed, no flip; manage.py dedupe re-run over the same data → no flips.
• Visible behavior change: findings created by one reimport get their ids in content-key order rather than report order, so id order no longer reflects the scanner's document order for reimported tests.

Tests

• End-to-end Fortify fixtures: two .fpr files whose findings share a unique_id_from_tool but differ in content, in opposite document order, each reimported through the real DefaultReImporter; the surviving original must be identical across both orders (fails pre-fix). Also asserts the reimporter creates findings in deduplication_ordering_key order.
• Batch dedupe keeps the lowest id as original, and re-running dedupe (management-command scenario) never flips an established original.
• _is_candidate_older antisymmetry regression test.
• Pre-existing-finding-stays-original guard and ordering-key order-independence test.
• Note for reviewers: the Fortify test forces unique_id_from_tool dedup via override_settings (open source defaults Fortify to legacy, where different-content findings would not collide at all). This mirrors the realistic customer configuration.

Related

Pairs with the corresponding Dojo Pro change (test-only after this rework — Pro inherits the reimporter creation ordering). They should merge together.

🤖 Generated with Claude Code