diff --git a/API_V3_DIVERGENCE_ANALYSIS.md b/API_V3_DIVERGENCE_ANALYSIS.md new file mode 100644 index 00000000000..f08e020756f --- /dev/null +++ b/API_V3_DIVERGENCE_ANALYSIS.md @@ -0,0 +1,504 @@ +# Finding write path — UI / v2 API / v3 API divergence analysis and fix proposal + +Status: analysis for review. No code, tests, or plan are changed by this document. + +Scope: the finding **write** path (create / update / delete). The v3 API's write path was extracted +into `dojo/finding/services.py` by reconciling two pre-existing implementations of the same behaviour: + +- the **v2 API serializer** — `FindingSerializer` / `FindingCreateSerializer` in + `dojo/finding/api/serializer.py` (`.create` / `.update` / `.validate`), plus the viewset overrides + in `dojo/finding/api/views.py`; and +- the **classic UI** — `EditFinding` in `dojo/finding/ui/views.py` (the `process_*` form flow), with + field-level validation in `FindingForm` (`dojo/finding/ui/forms.py`). + +The extraction chose the serializer as the reference for v3 and deferred UI-only behaviours. That was +a reasonable way to ship an additive alpha, but it means **the same user action can behave differently +depending on whether it arrives via the UI or the API** — and, in one case, differently between the v2 +and v3 APIs. Each such gap is a latent bug class. This document catalogues every divergence found in +the code (verified line-by-line, not taken on trust), proposes a canonical behaviour for each, and +sequences the fixes across the alpha PR and the post-alpha convergence track. + +This document supersedes the working divergence table that seeded it. It is self-contained; nothing +below depends on any other note. + +--- + +## 1. Summary + +Business logic for editing a finding currently lives in three places (UI view, v2 serializer, and — +now — the v3 service). Because they were each patched independently over time, they have drifted. The +observable symptoms fall into three buckets: + +1. **Side-effects the UI performs and the API does not** — when a finding is deactivated, closed, or + flipped out of false-positive, the UI runs mitigation stamping, endpoint/location mitigation, + retroactive false-positive history, and `last_reviewed` stamping. The API write path runs none of + these. So a finding closed through the UI and the "same" finding closed through the API end up in + materially different states (mitigation flags, endpoint statuses, review dates, and — for FP + history — other findings entirely). +2. **A v2-vs-v3 regression** — deleting a JIRA-linked finding through v3 does not close or reassign the + JIRA issue, whereas v2 (and the UI) do. +3. **Shape/consistency differences** — CWE relation resync on a scalar `cwe` update (v3 does it, v2 + does not), and a create-time JIRA "push all" asymmetry present in both APIs. + +**Fix strategy.** One service layer (`dojo/finding/services.py`) becomes the single home for finding +write orchestration and its side-effects. The v2 serializer and the UI view then migrate to call it +(the convergence track), so there is exactly one place a workflow bug can live. This document decides, +per divergence, *which* behaviour the single service should implement — defaulting to the behaviour +that is correct for a security workflow rather than merely the one the serializer happened to have — +and flags which of those decisions change what a v2 API consumer sees. + +**How to read the ratings.** Severity reflects the real-world consequence of the two channels +disagreeing, not the size of the diff. Most rows are low: the two implementations already agree, and +convergence is a pure refactor. The high/medium rows are where the API silently skips a workflow +control the UI enforces. + +--- + +## 2. Divergence catalogue + +Each entry states the behaviour today in all three implementations (with `file:line` anchors), the +concrete impact and its severity, the proposed canonical behaviour with rationale, and the fix plan +across the three consumers. "CONV1" = v2 serializer delegates to the service; "CONV2" = UI view +delegates to the service; "CONV3" = delete the now-dead duplicate bodies. + +Entries D1–D17 correspond to the seventeen reconciled rows. D18 (mandatory closing notes) and D19 +(note-creation side-effects) are divergences that were not in the original reconciliation table and +are surfaced here. + +--- + +### D1 — Status invariants (active / verified / duplicate / false_p / risk_accepted) + +- **v2 serializer:** enforced in `validate()` — update path `dojo/finding/api/serializer.py:538-554` + (with PATCH defaults pulled from the instance at `:523-536`); create path `:714-736`. +- **UI:** enforced field-level in `FindingForm.clean()` `dojo/finding/ui/forms.py:632-640`. Note the + "simple risk acceptance disabled" case is *not* raised in the UI — the form instead removes the + `risk_accepted` field entirely when simple risk acceptance is off and the finding is not already + accepted (`dojo/finding/ui/forms.py:590-595`), so the invalid state is unreachable rather than + rejected. +- **v3 service:** `_validate_status_invariants()` `dojo/finding/services.py:83-97`, called from both + `create_finding` (`:148-156`, defaults treat "absent" as falsy) and `update_finding` + (`:215-223`, PATCH defaults from the instance). +- **Impact:** low. All three enforce the same four core invariants. The only observable difference is + cosmetic: the API returns a 400 for a disabled-simple-risk-acceptance attempt, whereas the UI never + offers the control. Both outcomes are correct for their medium. +- **Proposed canonical:** the service's ported invariants. Keep the API's explicit 400 rejection — an + API must reject invalid input loudly rather than silently drop a field. +- **Fix plan:** service already correct. CONV1: serializer delegates — invisible to v2 (identical + logic). CONV2: the UI keeps its field-removal UX for the simple-risk-acceptance case (a template + concern) and delegates the four invariants; no UX change. No release note. + +### D2 — Risk-acceptance processing + +- **v2 serializer:** `process_risk_acceptance()` runs inside `validate()` (`:558`, defined at + `:425-435`): `simple_risk_accept` / `risk_unaccept`, gated on `enable_simple_risk_acceptance`, before + the field updates in `update()`. +- **UI:** `simple_risk_accept(..., perform_save=False)` / `risk_unaccept(..., perform_save=False)` + followed by an explicit `finding.save()` (`dojo/finding/ui/views.py:919-923`, saved at `:1045-1047`). +- **v3 service:** `_process_risk_acceptance()` (`dojo/finding/services.py:100-116`) runs before the + field updates (`:225`). +- **Impact:** low. Behaviours converge. The UI's `perform_save=False` variant is an ordering + optimisation tied to the single form save; the trigger conditions differ slightly in wording but + resolve to the same transitions. +- **Proposed canonical:** the service's port of the serializer path. It is the field-level authority + and does not depend on a form's save lifecycle. +- **Fix plan:** service correct. CONV1 invisible. CONV2: the UI drops its `perform_save=False` dance + and delegates; no behaviour change. No release note. + +### D3 — JIRA push on update + +- **v2 serializer:** `if push_to_jira or is_keep_in_sync(instance): push(force_sync=True)` and raise on + failure (`dojo/finding/api/serializer.py:499-503`). The viewset ORs `push_to_jira` with the JIRA + project's `push_all_issues` first (`dojo/finding/api/views.py:169-176`). +- **UI:** `push_to_jira = push_all_issues or checkbox or is_keep_in_sync` (`dojo/finding/ui/views.py:970`), + then `finding.save(push_to_jira=...)` (`:1045-1047`); failures surface as page messages, not + exceptions. +- **v3 service:** synchronous `push(force_sync=True)`, raising `ValidationError` on failure + (`dojo/finding/services.py:260-263`); the route ORs `push_all_issues` + (`dojo/finding/api_v3/routes.py:231-234`). +- **Impact:** low. v2 and v3 already behave identically; the UI's message-based error handling is + appropriate for a browser and its exception-free save is appropriate for a template render. +- **Proposed canonical:** the service's synchronous push that raises on failure. An API caller needs a + real error at request time, not a swallowed background failure. +- **Fix plan:** service correct. CONV1 invisible. CONV2: when the UI delegates, its wrapper must catch + the raised error and translate it into a page message (so the template still renders) — a small + adapter, not a behaviour change. No release note. + +### D4 — JIRA push on create + +- **v2 serializer:** `if push_to_jira: push(new_finding)` with **no** `force_sync` + (`dojo/finding/api/serializer.py:678-679`). The create path does **not** OR `push_all_issues`: the + viewset overrides `perform_update` but not `perform_create` (`dojo/finding/api/views.py:169`), so a + product configured to "push all issues" does not push findings created via the API until they are + next updated. +- **UI:** create is a separate view (add-finding), out of the reconciled edit flow. +- **v3 service:** mirrors the serializer — `if push_to_jira: push(new_finding)` + (`dojo/finding/services.py:187-188`); the create route does not OR `push_all_issues` + (`dojo/finding/api_v3/routes.py:198-217`). +- **Impact:** low-to-medium. The create-vs-update asymmetry is present identically in v2 and v3, so it + is not a channel divergence — but it is a latent bug: "push all issues" quietly fails to cover + API-created findings. +- **Proposed canonical:** honour `push_all_issues` on create as well as update, so "push all" means + what it says regardless of channel. This is a behaviour change for v2, so treat it as a deliberate, + release-noted convergence step rather than a silent fix. +- **Fix plan:** low priority. Optionally OR `push_all_issues` in `create_finding`'s callers (both the + v3 route and, at CONV1, the v2 create path). Behavioural change for v2 — release note. Not required + for alpha. + +### D5 — JIRA link / unlink / change issue key + +- **v2 serializer:** not supported. +- **UI:** `process_jira_form()` links, unlinks, or re-keys the JIRA issue + (`dojo/finding/ui/views.py:977-995`). +- **v3 service:** not implemented (deferred). +- **Impact:** low. This is a UI-only capability the API never had; its absence is a feature gap, not a + behavioural disagreement over a shared field. +- **Proposed canonical:** keep it out of the finding write contract. Linking/unlinking a JIRA issue is + a distinct action and belongs on a dedicated JIRA sub-resource, not folded into a field write where + it would be an API-only side-channel. +- **Fix plan:** none in the service now. CONV2: the UI keeps this logic (or it moves to a JIRA + sub-resource service later); it does not block the UI from delegating the rest of the edit flow. No + release note. + +### D6 — Finding-group handling and group push + +- **v2 serializer:** not supported. +- **UI:** `update_finding_group()` on save (`dojo/finding/ui/views.py:915-917`) and a post-save group + push (`:1050-1051`). +- **v3 service:** not implemented (deferred). +- **Impact:** low. Group membership is a separate write concern; the API never modelled it here. +- **Proposed canonical:** keep deferred; finding-group writes belong to their own endpoint. +- **Fix plan:** none now. CONV2: the UI retains group handling outside the shared service, or it moves + to a group sub-resource later. No release note. + +### D7 — `last_reviewed` / `last_reviewed_by` stamping + +- **v2 serializer:** does not stamp. +- **UI:** stamps `last_reviewed = now` and `last_reviewed_by = request.user` on **every** edit + (`dojo/finding/ui/views.py:911-912`). +- **v3 service:** does not stamp. +- **Impact:** **medium.** `last_reviewed` feeds "stale finding" / review-age reporting. A finding + edited through the UI is marked reviewed; the identical edit through the API is not. Review-age + metrics therefore depend on which channel a team happens to use. +- **Canonical — DECIDED (architect, 2026-07-21): NO — API field-writes do NOT stamp `last_reviewed`.** + The v3 service's current behaviour (no stamping on create/update field-writes) is canonical, not + provisional. Rationale: automated/bulk field writes (tag syncs, integrations) are not reviews, and + an API cannot distinguish a human edit from automation. `last_reviewed` stamping remains an + *explicit-action* concern: note creation already stamps it (v2-parity note side-effects), and future + workflow actions (`request_review`, `close`, verification transitions) are the natural stamping + points. +- **Fix plan (updated):** v3 service — no change needed. CONV2: when the UI edit flow migrates onto + the service, the UI's "every form edit stamps" behaviour must NOT be baked into the shared + field-write path; if the team wants to preserve it for interactive edits, the UI layer stamps + explicitly (or passes an explicit opt-in flag) — the shared service stays stamp-free on field + writes. Dropping the UI stamping entirely would be a UI behaviour change — release note at CONV2 + time. (See also D19: + note creation stamps `last_reviewed` from the note date, which is a separate, already-agreed rule.) + +### D8 — Auto-mitigation side-effects on deactivation (`process_mitigated_data`) + +- **v2 serializer:** does not run mitigation side-effects; `validate()` only gates *editing* the + `mitigated` / `mitigated_by` fields (`dojo/finding/api/serializer.py:508-521`). +- **UI:** when `active` is unchecked (or `false_p` / `out_of_scope` set) and the finding is not a + duplicate, and the user may edit mitigated data, sets `is_mitigated = True` and mitigates every + endpoint/location status (`dojo/finding/ui/views.py:833-863`, invoked at `:938`). +- **v3 service:** not implemented (deferred). +- **Impact:** **high.** A finding deactivated through the UI becomes `is_mitigated=True` with its + endpoint/location statuses mitigated; the same finding deactivated through the API (v2 or v3) is left + `is_mitigated=False` with live endpoint statuses. The two are then inconsistent for metrics, SLA, and + endpoint reporting, and the discrepancy is invisible to the caller. This is the clearest + "same action, different result" gap. +- **Proposed canonical:** **the UI behaviour is correct** — deactivating a finding should mitigate it + and its endpoints. Note that a shared `close_finding` helper already does exactly this consistently + and is reachable via the v2 `close` action (`dojo/finding/api/views.py:255-284`). The real defect is + that the *generic field-write path* lets a caller set `active=false` while bypassing the close + workflow. Preferred long-term shape: route status-closing through the close service so there is one + mitigation code path. Faithful interim: have `update_finding` apply the same mitigation side-effects + (guarded by `can_edit_mitigated_data`, `dojo/finding/helper.py:204-205`) when `active` flips off. +- **Fix plan:** service gains the side-effect (as a step toward, or a call into, the shared close + logic). This is a behavioural change for v2 when it delegates (CONV1) — a finding closed via the v2 + field-write would begin mitigating endpoints. Release note; strong candidate for a behaviour-pinning + test before the change so the "before" state is documented. Not a clear v3-alpha bug (v3 matches v2 + today), so this belongs in convergence, not alpha. + +### D9 — False-positive history (`process_false_positive_history`) + +- **v2 serializer:** not implemented. +- **UI:** when a finding is reactivated out of false-positive and both `false_positive_history` and + `retroactive_false_positive_history` are enabled, retroactively reactivates all matching findings + (`dojo/finding/ui/views.py:865-885`, invoked at `:939`). +- **v3 service:** not implemented (deferred). +- **Impact:** **medium** (gated on a non-default setting, but high blast radius when on — it edits + *other* findings). With retroactive FP history configured, clearing `false_p` through the UI + cascades to sibling findings; through the API it does not. +- **Proposed canonical:** the UI behaviour. This is a configured, system-wide policy; it should apply + regardless of the channel that triggers the transition. +- **Fix plan:** move the FP-history cascade into the service so both channels honour it. Behavioural + change for v2 (CONV1), visible only when the setting is on — arguably a bug fix, but still worth a + release note because it changes cross-finding state. Behaviour-pinning test first. Not alpha. + +### D10 — Burp request/response persistence + +- **v2 serializer:** exposes `request_response` as a read-only `SerializerMethodField` + (`dojo/finding/api/serializer.py:331`); it cannot be written. +- **UI:** `process_burp_request_response()` writes it (`dojo/finding/ui/views.py:887-900`, invoked at + `:940`). +- **v3 service:** not implemented (deferred). +- **Impact:** low. A UI-only write capability the API never had; a feature gap, not a disagreement. +- **Proposed canonical:** keep deferred; add as an explicit write field/sub-resource later if wanted. +- **Fix plan:** none now. CONV2: UI retains this outside the shared service. No release note. + +### D11 — GitHub issue sync + +- **v2 serializer:** not implemented. +- **UI:** `process_github_form()` (`dojo/finding/ui/views.py:1008-1021`, invoked at `:1035`). +- **v3 service:** not implemented (deferred). +- **Impact:** low. UI-only feature gap. +- **Proposed canonical:** keep deferred. +- **Fix plan:** none now. CONV2: UI retains it outside the shared service. No release note. + +### D12 — `numerical_severity` + +- **v2 serializer:** does not set it. +- **UI:** sets it explicitly on save (`dojo/finding/ui/views.py:910`). +- **v3 service:** does not set it. +- **Model:** `Finding.save()` always computes it (`dojo/finding/models.py:583`). +- **Impact:** none observable. The UI's explicit assignment is redundant with the model. +- **Proposed canonical:** rely on `Finding.save()`. Do not duplicate. +- **Fix plan:** service correct. CONV2: drop the redundant UI line when the view delegates. Invisible. + +### D13 — `finding_added` notification on create + +- **v2 serializer:** dispatches the `finding_added` notification + (`dojo/finding/api/serializer.py:682-690`). +- **UI:** create is a separate view, out of scope for the edit reconciliation. +- **v3 service:** dispatches it identically (`dojo/finding/services.py:190-198`). +- **Impact:** low. v2 and v3 agree. +- **Proposed canonical:** the service's dispatch. +- **Fix plan:** service correct. CONV1 invisible. No release note. + +### D14 — Notification on update + +- **v2 serializer:** none on the update path itself. +- **UI:** the edit flow dispatches no finding notification (notes and tags notify through their own + flows — see D19). +- **v3 service:** none (`update_finding` dispatches nothing). +- **Impact:** low. All agree. +- **Proposed canonical:** none on the field-write update path. +- **Fix plan:** service correct. Invisible. + +### D15 — Vulnerability-id and CWE persistence + +- **v2 serializer:** `save_vulnerability_ids` (also writing the first id into `Finding.cve`) and + `save_cwes`, both guarded on the nested field being present — update + (`dojo/finding/api/serializer.py:463-491`), create (`:644-676`). On the **scalar** `cwe` path, v2 + does **not** touch the `Finding_CWE` rows (only the nested `cwes` list does, `:489-491`). +- **UI:** the same helpers (`dojo/finding/ui/views.py:942-943`). +- **v3 service:** `save_vulnerability_ids` before save (create `:180`, update `:238-239`) with the + `cve` mirror written pre-save (create `:168-169`); `save_cwes` on create always (`:182`) and on + update whenever the scalar `cwe` is in the change set (`:253-254`). +- **Impact:** low-to-medium. v3 exposes a scalar `cwe` (not v2's nested `cwes` list) and **resyncs** + the `Finding_CWE` rows when the scalar changes; v2's scalar path leaves `Finding.cwe` and its + `Finding_CWE` rows able to drift apart. +- **Proposed canonical:** v3's resync — keep `Finding.cwe` and `Finding_CWE` consistent. It is the more + correct of the two. +- **Fix plan:** service correct. CONV1: when v2 delegates, v2's scalar-cwe path gains the resync — a + behavioural change (a consistency fix). Release note or silent-fix at the team's discretion; + behaviour-pinning test first. The contract difference (scalar `cwe` vs nested `cwes`) is an + intentional v3 simplification and stays. + +### D16 — `found_by` handling + +- **v2 serializer:** set when a non-empty list is provided; clear on an explicit empty list; leave + untouched when absent — update (`dojo/finding/api/serializer.py:467-475`), create set-only + (`:668-669`). +- **UI:** via the form. +- **v3 service:** the same set/clear/untouched semantics using an `_UNSET` sentinel to distinguish + "absent" from "empty" (update `dojo/finding/services.py:241-245`; create `:177-178`). +- **Impact:** none observable. Matches the reference exactly. +- **Proposed canonical:** the service's ported semantics. +- **Fix plan:** service correct. CONV1 invisible. + +### D17 — Delete-time dedup / grading / JIRA sync *(table corrected)* + +- **v2 viewset:** `destroy()` reads an optional `push_to_jira` query param via `get_request_boolean` + (which returns `None` when the param is absent, `dojo/api_v2/views.py:103-109`) and calls + `instance.delete(push_to_jira=...)` (`dojo/finding/api/views.py:178-185`). +- **UI:** the delete flow calls `finding.delete(push_to_jira=...)`. +- **v3 service:** `delete_finding()` calls `finding.delete()` with **no** `push_to_jira` + (`dojo/finding/services.py:267-275`). +- **What that actually means.** `Finding.delete()` defaults `push_to_jira` to a sentinel + (`DELETE_JIRA_SYNC_UNSET`, `dojo/finding/models.py:716`). Inside `finding_delete`, the JIRA + delete-sync is gated on `jira_sync_requested = push_to_jira is None or isinstance(push_to_jira, bool)` + (`dojo/finding/helper.py:578`). v2 passes `None` (or a real bool) ⇒ `jira_sync_requested = True`; the + sentinel from v3 is neither ⇒ `jira_sync_requested = False`. Consequently the two JIRA delete-sync + actions — reassigning the linked issue to the surviving duplicate original + (`dojo/finding/helper.py:585-591`) and closing the issue for the deleted finding (`:595-601`) — are + **both skipped on v3 deletes** and **run on v2 deletes**. The dedup-cluster reconfiguration and + product grading run in both (they are not JIRA-gated: `dojo/finding/models.py:719-733`). +- **Correction to the original table.** The seed table framed this as merely "the service omits the + `push_to_jira` query-param; delete-time JIRA closure is additive later." That understates it: because + of the sentinel default, v3 deletes perform **no** JIRA delete-sync at all, which is a behavioural + **regression against v2 and the UI**, not just a missing optional parameter. +- **Impact:** **medium-to-high.** Deleting a JIRA-linked finding through v3 leaves the JIRA issue + untouched — not closed, and (if the finding was the original of a duplicate cluster) not reassigned to + the surviving finding. DefectDojo and JIRA drift, silently. +- **Proposed canonical:** v3 delete should engage JIRA delete-sync exactly as v2 does. The minimal, + faithful fix is for `delete_finding` to pass `push_to_jira=None` by default (matching v2's + no-param behaviour), with an optional explicit override added later. +- **Fix plan:** **this is the one clear bug in v3's current choice — fix it in the alpha PR.** Change + `delete_finding` to call `finding.delete(push_to_jira=None)` (or thread an optional argument through + the route). No change needed for v2 (it already does this) or the UI. Invisible to v2. Add a v3 + test asserting a JIRA-linked finding's issue is closed/reassigned on delete. + +### D18 — Mandatory closing notes on status change *(not in the original table)* + +- **v2 serializer:** the field-write path does not enforce mandatory closing notes. (The dedicated + `close` action, `dojo/finding/api/views.py:255-284`, accepts a note but the generic update does not + require one.) +- **UI:** `validate_status_change()` blocks setting a finding inactive / false-positive / out-of-scope + unless all mandatory note types are present (`dojo/finding/ui/views.py:794-831`). +- **v3 service:** not enforced. +- **Impact:** **medium.** Where an installation configures mandatory note types, the UI refuses to + close a finding without the required justification notes; the API (v2 and v3) closes it without them. + An integration can therefore bypass a governance control the UI enforces. +- **Proposed canonical:** the UI behaviour is the correct governance stance, but the plain field-write + is the wrong place to demand a note (it has no note field). The right home is the close workflow + action, which already carries a note. Recommendation: enforce mandatory-note presence in the shared + close path, and either reject `active=false` on the generic field-write when mandatory notes are + configured, or document that the field-write bypasses the control and steer integrations to the close + action. +- **Fix plan:** not alpha. Decide alongside D8 (both concern closing a finding through the field-write + vs the close action). Behavioural change for v2 if the field-write starts rejecting note-less closes + — potentially **breaking** for an existing integration, so this one needs a deprecation window rather + than a silent flip. + +### D19 — Note-creation side-effects *(in progress for alpha — see §5)* + +Recorded here for completeness; **not proposed** in this document because it is being implemented in v3 +now by a separate work stream (the notes sub-resource). Adding a note through the UI +(`dojo/finding/ui/views.py:609-641`) stamps `last_reviewed` from the note date and +`last_reviewed_by` (`:624-625`), posts a JIRA comment when the finding or its group has an issue +(`:628-631`), and fires mention/tag notifications (`process_tag_notifications`, `:637`). See §5. + +--- + +## 3. v2 impact assessment + +The v3 alpha is additive: nothing here changes v2 today. The changes below are what a v2 API consumer +would observe **after CONV1**, when the v2 serializer delegates to the shared service and the canonical +behaviours above are adopted. Categorised by consumer-visible effect: + +| Divergence | v2-visible change | Category | Recommended communication | +|---|---|---|---| +| D8 auto-mitigation on deactivate | field-write closes now mitigate `is_mitigated` + endpoint/location statuses | **behavioural** | Release note. Behaviour-pinning test first. | +| D9 false-positive history | reactivating out of `false_p` cascades to matching findings (when the setting is on) | **behavioural** | Release note (cross-finding effect), even though it is a bug fix. | +| D15 CWE resync on scalar `cwe` | scalar `cwe` update now resyncs `Finding_CWE` rows | **behavioural** | Release note or silent fix (consistency correction). | +| D4 `push_all_issues` on create | findings created via API now push to JIRA under "push all issues" | **behavioural** | Release note. Optional; lowest priority. | +| D18 mandatory closing notes | field-write may begin rejecting note-less closes | **breaking** (if adopted as a rejection) | Deprecation window; do not silent-flip. | +| D7 `last_reviewed` on edit | *pending product decision* — not yet a committed change | (t.b.d.) | Decide first; whatever lands is behavioural. | +| D1 status invariants | none (service ports v2 logic verbatim) | invisible | — | +| D2 risk acceptance | none | invisible | — | +| D3 JIRA push on update | none (v2 already does this) | invisible | — | +| D12 `numerical_severity` | none (model computes it) | invisible | — | +| D13 `finding_added` notification | none | invisible | — | +| D14 update notification | none | invisible | — | +| D16 `found_by` | none | invisible | — | +| D17 delete JIRA sync | none for v2 (v2 already syncs; the fix is on v3) | invisible | — | +| D5/D6/D10/D11 UI-only features | none (v2 never had them) | invisible | — | + +**Tally of v2-visible changes:** breaking **1** (D18, only if the field-write starts rejecting +note-less closes); behavioural **4** (D8, D9, D15, D4), plus D7 pending a decision; invisible **the +remaining 12** rows (D1, D2, D3, D12, D13, D14, D16, D17, and the four deferred UI-only features D5/D6/ +D10/D11 which never touched v2). + +Guidance: the invisible rows are pure refactors and can ship in CONV1 with the existing `test_apiv2_*` +suite as the guard. The four behavioural rows should be layered *after* the pure refactor as separate, +individually release-noted commits, so a bisect can attribute any behaviour change to a single change. +The one potentially breaking row (D18) should go through a deprecation window. + +--- + +## 4. Sequencing proposal + +Guiding rule (from the convergence track): **behaviour-pinning tests land before each refactor**, never +after. CONV1 leans on the extensive `test_apiv2_*` suite; CONV2 needs new tests because UI-flow +coverage is thinner. + +**Step 0 — Alpha PR (only clear bugs in v3's *current* choice).** +- **D17** — make `delete_finding` engage JIRA delete-sync by default (pass `push_to_jira=None`), so v3 + deletes match v2/UI. Add a v3 test asserting the linked JIRA issue is closed/reassigned on delete. +- Everything else is either already-agreed (no change) or a convergence-track change to v2/UI (not a + v3-alpha concern). D15 (CWE resync) is v3's deliberate, already-implemented choice and stays. + +**Step 1 — CONV1 pure refactor (v2 serializer → service, zero contract change).** +- Pin current v2 behaviour first (extend `test_apiv2_*` to freeze exact output, *including* the + behaviours we intend to change later: no auto-mitigation, no FP cascade, no scalar-cwe resync). +- Refactor `FindingSerializer.update` / `FindingCreateSerializer.create` to call `create_finding` / + `update_finding`, preserving v2 semantics exactly (parameterise the service where v2 and the desired + canonical differ, so this step changes nothing observable). + +**Step 2 — CONV1 behavioural convergence (deliberate, release-noted, one change per commit).** +- D15 CWE resync → D9 FP history → D8 auto-mitigation → (optional) D4 push-all-on-create. Each with its + own behaviour-pinning test flipped from "old" to "new" and its own release note. D8 depends on the + shared close logic being the mitigation home, so land the close-path consolidation first. +- D18 (mandatory notes) and D7 (`last_reviewed`) are gated on decisions, not code — resolve those + before scheduling; D18 additionally needs a deprecation window. + +**Step 3 — CONV2 (UI view → service, view by view).** +- The service must first grow the UI-only side-effects it does not yet cover, as opt-in + parameters/hooks: D5 (JIRA link/unlink), D6 (finding-group), D7 (`last_reviewed`, per the decision), + D8 (already added in Step 2), D9 (already added), D10 (burp), D11 (github). Only then can `EditFinding` + delegate without losing behaviour. +- Write behaviour-pinning tests for each UI flow *before* migrating it (coverage is thin today). Start + with the flows the service already reconciled (edit / risk-acceptance). Zero template/UX change. + +**Step 4 — CONV3 (delete dead duplicates).** Remove the now-unused serializer `update`/`create` +internals and the duplicated UI `process_*` bodies, only after CONV1 and CONV2 have proven the service. + +**Dependency order:** Step 0 (alpha) → Step 1 (pin + delegate v2) → Step 2 (v2 behavioural, close-path +consolidation before D8) → Step 3 (service grows UI side-effects → pin UI → migrate views) → Step 4 +(delete duplicates). + +--- + +## 5. Addendum — note-creation side-effects (in progress for alpha) + +One divergence adjacent to the write path is **already being implemented in v3** by a parallel work +stream (the notes sub-resource) and is therefore recorded, not proposed, here. + +Adding a note to a finding through the UI (`dojo/finding/ui/views.py:609-641`) performs three +side-effects that a naive "create a note row" implementation would miss: + +1. **`last_reviewed` stamping from the note** — sets `finding.last_reviewed = new_note.date` and + `finding.last_reviewed_by` (`:624-625`). This is a distinct, already-agreed rule and should not be + conflated with D7 (edit-time stamping, which is still open). +2. **JIRA comment** — posts the note as a comment on the finding's JIRA issue, or the finding group's + issue (`:628-631`). +3. **Mentions / tag notifications** — `process_tag_notifications` for `@`-mentions and tag subscribers + (`:637`). + +The v3 notes work stream should reproduce all three so that a note added via the API has the same +downstream effect as one added via the UI. It is tracked there; this document makes no separate +proposal for it beyond flagging the three side-effects as the acceptance criteria. + +--- + +## Appendix — verification record + +Every row above was checked against the code rather than taken from the seed table. All 17 reconciled +rows were verified. Material corrections and additions: + +- **D17 corrected** — the delete-time JIRA sync is skipped entirely on v3 (sentinel-default gate, + `dojo/finding/helper.py:578`), a behavioural regression against v2/UI, not a merely-omitted optional + parameter as the seed table implied. Reclassified to a v3-alpha bug. +- **D8 sharpened** — raised to high severity and tied to the existing shared close helper; the seed + table listed it only as "deferred". +- **D1 refined** — the UI's status invariants live in `FindingForm.clean` (`dojo/finding/ui/forms.py:632-640`); + `validate_status_change` is a *separate*, stronger check (mandatory notes, now D18); the + simple-risk-acceptance case is handled by field removal (`:590-595`), not a raised error. +- **D4 refined** — noted the create-vs-update `push_all_issues` asymmetry, present in both v2 and v3. +- **D15 refined** — spelled out the scalar-`cwe` resync consistency difference between v2 and v3. +- **D18 added** — mandatory closing notes, a UI governance control absent from the API, not in the seed + table. +- **D19 added** — note-creation side-effects, in progress via the notes work stream (§5). diff --git a/API_V3_PLAN.md b/API_V3_PLAN.md new file mode 100644 index 00000000000..8bee84b81e8 --- /dev/null +++ b/API_V3_PLAN.md @@ -0,0 +1,578 @@ +# DefectDojo API v3 — Implementation Plan + +| | | +|---|---| +| **Status** | Approved for OS implementation (alpha). Downstream adoption phases are outline-only, not started. | +| **Audience** | Developers and architects implementing or reviewing v3. Written to be executable by an implementation-focused engineer or AI model without re-deriving decisions. | +| **Companion** | "API v3 Idea and Design Spec" (the original design proposal). Where this plan and the spec disagree, **this plan wins** — it incorporates codebase and downstream-consumer audits done after the spec was written. | +| **How to update** | Decisions live in §1 (numbered D1…). Don't silently change a decision — add a row to the Decision Log (§12) with rationale, then update the affected sections. Open questions also go to §12. | + +--- + +## 1. Architect Summary + +v3 is a new, parallel API mounted at `/api/v3/` next to the existing v2. v2 stays fully supported and **is not modified by this work**. The alpha covers seven objects (`product_type`, `product`, `engagement`, `test`, `finding`, `location`, `user`) plus a consolidated import endpoint. Everything else stays on v2 until v3 earns parity. + +The design decisions, with the reasoning that produced them: + +**D1 — Parallel, additive-only mount; alpha carries "alpha" in the URL.** v3 mounts in `dojo/urls.py` next to v2 — during alpha at **`/api/v3-alpha/`**, so the instability is impossible to miss (headers and docs banners are ignorable; the URL isn't). At beta it moves to `/api/v3/` and never changes again — one client-side URL migration total, timed to the additive-only contract freeze (§4.1). The entire OS deliverable is new code: no v2 serializer, viewset, or URL changes. This is what makes one big, easily-testable PR viable (§7) — nothing existing can regress. + +**D2 — Framework: Django Ninja pilot with a hard decision gate; DRF fallback.** Ninja (Pydantic v2 DTOs) is recommended because explicit schemas make the ref/expand model natural and schemas are subclassable (a downstream requirement, see D10). It is *not* justified by async — DefectDojo deploys WSGI (uwsgi), so async endpoints yield no concurrency win. Neither ninja nor pydantic is currently a dependency; adding them is a deliberate supply-chain decision surfaced in the PR. Phase OS1 ends with a go/no-go gate against seven exit criteria (§6, OS1); on failure we implement the **identical contract** on DRF. Everything in §4 is framework-agnostic. + +*Ecosystem trade-off (accepted):* Ninja has a much smaller third-party ecosystem than DRF — there is no drop-in equivalent for much of the DRF plugin landscape (filter integrations, throttling add-ons, spectacular extensions), so v3 writes more of its own kernel code (pagination, filter adapter, error handling — §6 OS1/OS2). We accept this deliberately: (a) the kernel pieces are small, contract-driven, and easier to own than to adapt — and with AI-assisted implementation the cost of writing well-specified glue code is far lower than it historically was; (b) the DRF plugin ecosystem is itself a liability in places — several widely-used packages are aging or effectively unmaintained, which is how v2 accumulated debt like the bespoke prefetch framework. Owning ~6 small kernel modules outright is preferable to depending on third-party code with uncertain maintenance. The OS1 gate still checks that this self-built surface stays small (exit criterion 4: django-filter reuse or costed equivalent). + +**D3 — Response model: slim default + `{id, name}` refs + `?expand=` + `?fields=`. No in-band memoization.** Every relation renders as a ref by default; `?expand=test.engagement` swaps refs for fuller objects and drives real `select_related`/`prefetch_related` (this is the actual N+1 fix — v2's `?prefetch=` is post-serialization and does per-row queries). We surveyed how major APIs handle nested relations: Stripe (id + `expand[]`), ServiceNow (`{value, link}` refs + display values), Jira (`fields=`/`expand=`), GitLab/GitHub (small objects inlined and repeated per row). **No mainstream API deduplicates repeated nested objects in-band** the way one downstream client implementation does; the only standardized dedup is JSON:API side-loading, which carries the client-rehydration complexity we're explicitly rejecting. gzip makes repeated `{id, name}` refs nearly free on the wire. Consequence: no memoization, no `prefetch` blob, no client rehydration contract. + +*Schemas are hand-declared DTOs, never auto-derived from models.* Ninja offers `ModelSchema` +(derive every field from the Django model); v3 deliberately does not use it — auto-derivation would +recreate v2's core disease: heavy-by-default responses where every model field leaks into the +contract and every model change is a silent breaking change. Each resource declares up to four +explicit classes — `Slim` (list/expansion shape: identity + status + refs + timestamps), +`Detail(Slim)` (adds the heavy fields; inherits so nothing is duplicated), `Write` and `Update` +(create/patch payloads — separate because writes take integer ids where reads render refs, forbid +unknown fields, and never expose server-managed fields). Pydantic auto-generates the *validation +and OpenAPI* from these declarations; the declarations themselves ARE the curated contract. Four +DTOs per resource is idiomatic ninja/FastAPI style, and the named classes are exactly what makes +schemas subclassable downstream (I4). + +**D4 — Pagination: one envelope, two modes; offset is the default; no two-phase pagination.** Envelope is always `{count, next, previous, results}` (+ optional `meta`). Default mode is limit/offset because the primary future consumer (a SPA with data tables) needs page-jump, offset URL state, and a count. Cursor/keyset mode is reserved as `?pagination=cursor` (GitLab precedent: same endpoints, opt-in keyset) for export/sync consumers: same envelope, `count: null`, opaque cursor in `next`, sort restricted to keyset-safe orderings. Alpha ships offset only; the reserved parameter makes cursor a later non-breaking addition. Clients must treat `next`/`previous` as opaque URLs — that's what makes both modes interchangeable. (Page-jump itself is a UI choice, not a law — if a future UI drops it, cursor becomes the recommended default for large collections with zero contract change; the dual-mode envelope is the hedge.) Counts follow industry practice (Jira's new endpoints drop totals; GitLab omits `x-total` >10k; Elasticsearch caps at 10k): **exact below a documented threshold, planner-estimated above it**. Below `COUNT_CAP` the count is exact (capped counting bounds the cost); above it, `count` is the Postgres planner's row estimate for the filtered query (`EXPLAIN`-based, the PostgREST `count=estimated` technique), clamped to ≥ CAP+1 and flagged `count_exact: false` in `meta`. Users see "≈1.2M" instead of "10000+"; cost stays bounded; never a full `COUNT(*)` and never a second rehydration query phase like the two-phase paginator observed in a downstream implementation, whose root cause (heavy list annotations) the slim default removes. + +**D5 — Locations, not Endpoints; M2M with status on the edge; URL-type only; flag-required.** v3 never exposes the legacy `Endpoint` model. The audit showed Finding↔Location is many-to-many via `LocationFindingReference`, which carries the status (`Active/Mitigated/FalsePositive/RiskAccepted/OutOfScope`), `auditor`, `audit_time` — so the original spec's singular `"location": {id, name}` field on Finding is wrong. Slim findings carry `locations_count`; the full list is the `/findings/{id}/locations` sub-resource whose rows are *(location ref + edge status)*. Only the `URL` location subtype has a persistable model today (`LocationData.code`/`.dependency` DTOs exist but are dropped by `LocationManager` on import), so v3 documents location support as **URL locations only** until Code/Dependency models land. v3 requires `V3_FEATURE_LOCATIONS=True` (already default-on in dev); with the flag off, `/api/v3/` is not mounted at all — v3 carries no legacy-endpoint code path. + +**D6 — One filter contract, many projections.** The most load-bearing pattern found in downstream API usage: the same filter parameters must drive the list, aggregate counts, future chart/aggregation endpoints, and future exports. Therefore (a) the per-object filter vocabulary is a documented, snapshot-tested artifact (django-filter-compatible grammar: `field`, `field__gte`, `field__in`, …, multi-sort `o=`, free-text `q=`); (b) `?include=counts` returns severity/status totals computed over the *filtered, authorized* queryset in the same response's `meta` — no second round-trip for "N active / M verified" headers; (c) future aggregation endpoints take identical filter params. Only `include=counts` ships in alpha; the grammar is what's locked. + +*Expressiveness ceiling (learned from downstream workarounds, not copied from them):* flat `field__lookup` params cannot express boolean OR/AND groups. When a downstream consumer hit this it accreted bespoke any/all-style suffix params and its dashboard layer abandoned GET params for POSTed filter dicts. v3 does not rebuild that ceiling: the flat grammar stays for the common case, and complex boolean filtering gets exactly one structured representation later — a reserved **`POST //query`** accepting a canonical JSON filter tree (also the future substrate for saved views and aggregation filters). Bespoke `_any`/`_all`-style suffix params are **banned**; if a filter need can't be expressed in the flat grammar, it waits for the structured representation rather than growing a one-off param. + +**D7 — Service layer is the kernel; routes are thin; v2 and UI views untouched (for now), converging later.** Business logic currently lives in **three** places: the classic Django UI views (e.g. the close/edit/risk-acceptance flows in `dojo/finding/views.py`), the v2 serializers (`FindingSerializer.update()` doing JIRA/risk/vuln-ids), and — without this decision — soon a third copy in v3. The UI-vs-API duplication is a known drift source (the paths have diverged on notification triggers and JIRA sync behavior because each is patched independently). Therefore: all side-effects and orchestration (JIRA push, risk acceptance, status transitions, import) live in `dojo//services.py` functions **written to serve all three consumers**, but in the alpha PR only v3 calls them — that preserves the additive property (D1). Schemas do field validation only; routes parse → call service → shape response. The import endpoint requires a new `dojo/importers/services.py` facade returning a structured `ImportResult` (the current importer returns a 7-tuple, and — correcting the spec — **no service wrapper exists today**; the v2 serializer calls `DefaultImporter` directly). **Extraction rule:** a service is extracted by reconciling *both* existing implementations (v2 serializer + UI view flow), not by copying the serializer — where they disagree, the canonical behavior is chosen consciously and recorded in §12; otherwise the later UI-view refactor silently becomes a behavior change instead of a pure refactor. v2 serializers and UI views migrate onto the services in the post-alpha **convergence track** (§7): end state is views, v2, and v3 as thin shells over one service layer — one place to fix a workflow bug. + +**D8 — Auth: token AND Django session+CSRF from day one; v2 tokens work unchanged.** Two authentication modes, both active on every endpoint: +- **Token — reuses the existing v2 tokens.** Same `Authorization: Token ` scheme, validated against the same store (`rest_framework.authtoken.models.Token`). No new token model, no migration, no re-issuing: a key that works on `/api/v2/` works on `/api/v3/` the same day. Implemented as a small header-auth class that parses the `Token ` prefix and resolves `request.user`. +- **Django session + CSRF** — session cookie plus `X-CSRFToken` on unsafe methods. Mandatory now because the eventual SPA consumer is session-authenticated; bolting it on later would ripple through every route's auth declaration. + +Auth classes are registered as an ordered list (`auth=[TokenAuth(), django_auth]`); each returns `None` on no-match so the next gets a chance; all failing → 401 problem+json. **The token scheme is intentionally replaceable:** because routes only see the pluggable auth list (invariant I7), the token backend can later be extended or swapped (scoped/expiring keys, JWT, OAuth) as a config-level change with no route changes — v2-token compatibility is the alpha baseline, not a permanent commitment. Authentication is separate from authorization: after any auth mode succeeds, all object access still flows through the existing `get_authorized_*` querysets and permission semantics (I8), so RBAC is identical to v2. Both modes are tested on the same endpoints from OS1. + +**D9 — Boring, standard conventions.** ISO-8601 UTC datetimes, explicit `null`s (no key-stripping), raw fields (no server-side display formatting of users etc.), errors as RFC 9457 `application/problem+json` with a `fields: {field: [messages]}` extension for validation errors. Where downstream consumers currently rely on non-standard conventions (Unix-ms timestamps, stripped nulls, preformatted user strings), those are client-side adaptations — they do not shape the v3 contract. + +**D10 — Forward-compatibility invariants instead of downstream features.** OS phases build **zero** downstream-specific features, but §5 defines invariants (composable schemas, router factories, pluggable auth, the filter contract, the `include`/`meta` mechanism, service-layer purity) that guarantee later downstream adoption phases are *additions on top of* v3, never *changes to* it. Every OS phase is reviewed against §5. + +**D11 — v3 speaks the new domain language: organizations and assets (wire surface only, unconditional).** The product has already renamed Product_Type→Organization and Product→Asset (`DD_ENABLE_V3_ORGANIZATION_ASSET_RELABEL`, default on); a brand-new API must not bake the legacy names into the surface consumers will codegen against for years, and the alpha is the only time the rename is free (post-beta it becomes aliases + deprecation forever). Scope: **wire surface only** — paths (`/organizations`, `/assets`), ref keys (`finding.asset`, `finding.organization`, `engagement.asset`, `asset.organization`), filter params (`asset=`, `organization=`, …), expand paths, schema class names (`AssetSlim`, `OrganizationSlim`), router factories, OpenAPI tags, import form fields (`asset_name`, `organization_name`), docs, examples, snapshots. **All-or-nothing** — no half-rename; a renamed path with a legacy ref key is the worst outcome. **Not renamed:** Django models, DB tables, and `dojo/product*/` module paths (the DTO layer is exactly what decouples wire names from models; internal relocation rides the module reorg later); `engagement`/`test`/`finding`/`user`/`location` are not part of the relabel. **Unconditional:** the API naming does NOT follow the UI relabel flag — a per-instance API contract is codegen poison; deployments that restore legacy UI labels still get `organizations`/`assets` on v3 (documented). The v2→v3 mapping table doubles as the rename documentation. + +--- + +## 2. Scope + +**OS alpha delivers:** `/api/v3/` mount; CRUD for the 7 objects; refs/slim/expand/fields/include; offset pagination with capped counts; documented+tested filter contract; generic notes/tags/files sub-resources; locations with edge status; consolidated `POST /api/v3/import` (mode auto/import/reimport); token+session auth; problem+json errors; OpenAPI with alpha marking; query-count regression tests. + +**Explicit non-goals for OS (deferred, see §8):** bulk operations, workflow actions (close/review/duplicate handling), aggregation/chart endpoints, saved views, CSV export, delete-impact preview, job-status resources, global search, permission-map/bootstrap endpoints, cursor pagination implementation (grammar reserved only), v2/UI-view refactors (deferred to the convergence track, §7). + +--- + +## 3. Evidence (audit summary) + +Three audits inform this plan (details in PR discussion / architect's notes): + +1. **OS v2 audit.** ~49 endpoint groups, ~106 serializers, now organized per-module (`dojo//api/`); `dojo/api_v2/serializers.py` is largely a re-export shim. `?prefetch=` is post-serialization with per-row-per-field queries (`dojo/api_v2/prefetch/mixins.py:7-48`) — its RBAC hole is already patched (deny-by-default registry, `authorized_querysets.py`; tests `unittests/test_apiv2_prefetch_rbac.py`), so the motivation to replace it is performance, not security. Business logic embedded in `FindingSerializer.update()/validate()` (`dojo/finding/api/serializer.py:438-560`): JIRA push, risk acceptance, vuln-ids/CWEs. Importer returns a 7-tuple (`dojo/importers/base_importer.py:183-188`); no service facade exists. Deps: DRF 3.17.1, drf-spectacular 0.30.0, django-filter 26.1; no ninja/pydantic. No existing v3 scaffolding. +2. **Downstream API-consumer audit.** A survey of how existing downstream consumers (a session-authenticated SPA among them) use the v2 API. Key findings: hard dependence on limit/offset + `count`, the OS `?prefetch=` mechanism, filtered-list totals alongside lists, bulk operations with per-item skip reasons, workflow actions, aggregation endpoints, saved views, poll-based async jobs (no websockets anywhere), delete-impact previews, and CSV export. Full capability list with dispositions: §9. +3. **Location audit.** Full parallel Location subsystem behind `V3_FEATURE_LOCATIONS` (default True): `Location` + `LocationFindingReference` (edge status incl. mitigation transitions in `dojo/importers/location_manager.py:230`) + `LocationProductReference`; `URL` is the only persistable subtype (`location_manager.py:402-405` drops others); when the flag is on, `Endpoint.__init__` raises — hard cutover, no dual-write. + +--- + +## 4. Contract specification + +This section is normative. Implementers follow it verbatim; deviations require a §12 decision-log entry. + +### 4.1 Mount & versioning +- **Alpha mounts at `/api/v3-alpha/`** in `dojo/urls.py` alongside the v2 mount (near the existing `re_path(r"^{}api/v2/"...)`, ~line 198). Entire mount is conditional on `settings.V3_FEATURE_LOCATIONS` (D5). Everywhere else in this document, `/api/v3/` is the *logical* name; during alpha the actual prefix is `/api/v3-alpha/` (a single settings/urls constant — do not hardcode the prefix anywhere else). +- **Stability-in-URL policy (Kubernetes/Google precedent, one migration total):** `/api/v3-alpha/` for the whole alpha period — the URL itself signals "contract may break at any time; do not build production dependencies on this." At beta the API moves to **`/api/v3/` and stays there through GA** — beta's additive-only contract freeze is exactly the stability a permanent URL implies. Beta/GA status is conveyed by header + docs, not further URL changes. Clients therefore migrate URLs exactly once (alpha→beta), at the moment they must re-verify against the settled contract anyway. +- OpenAPI `info.version = "3.0.0-alpha"`; every response carries header `X-API-Status: alpha` (later `beta`, then removed at GA); docs page states the alpha contract may change. +- OpenAPI JSON at `/api/v3-alpha/openapi.json`, interactive docs at `/api/v3-alpha/docs`. + +### 4.2 Auth (D8) +- **Token:** `Authorization: Token ` validated against the existing DRF token store (`rest_framework.authtoken.models.Token`) — same tokens work on v2 and v3. In Ninja: a small `APIKeyHeader` subclass parsing the `Token ` prefix; sets `request.user`. +- **Session:** Django session cookie + CSRF on unsafe methods. In Ninja: `ninja.security.django_auth` and `NinjaAPI(csrf=True)`. +- Both registered on the API instance: `auth=[TokenAuth(), django_auth]`. Anonymous → 401 problem+json. + +### 4.3 Envelope & pagination (D4) +Every list response: +```jsonc +{ + "count": 4321, // exact below COUNT_CAP, else planner estimate (see below) + "next": "https://.../api/v3/findings?limit=25&offset=50&severity=High", + "previous": null, + "results": [ ... ], + "meta": { } // present only when ?include= adds content, plus + // "count_exact": false when count was capped +} +``` +- `limit` default 25, max 250. `offset` ≥ 0. +- **Count strategy (hybrid exact → estimate), `COUNT_CAP = 10_000` (settings-overridable):** + 1. Capped exact count: `COUNT(*)` over a `LIMIT COUNT_CAP+1` subquery (`qs.order_by()[:COUNT_CAP+1].count()`-style) — bounded cost, and it detects whether the result is under the threshold. + 2. ≤ CAP → `count` is exact. + 3. = CAP+1 → `count` = the Postgres planner's row estimate for the same filtered queryset (`qs.explain(format="json")`, read `Plan Rows`), **clamped to `max(estimate, COUNT_CAP+1)`** so the reported count never contradicts step 1; response gains `"meta": {"count_exact": false}`. + - Never a full `COUNT(*)` on unbounded tables, never a second PK-prefetch phase. Estimates depend on planner statistics (autovacuum/ANALYZE freshness) and may drift — documented as approximate. Clients must tolerate an empty page when jumping near an estimated end (`results: []`, `next: null`). Tests assert the exact/estimate switchover, the clamp, and the `count_exact` flag — never specific estimate values (nondeterministic across environments). +- `?pagination=cursor` is **implemented** (forward-only keyset mode, GitLab-style; see the 2026-07-20 §12 entry). Same envelope with `count: null` and `previous: null` (forward-only — no backward cursor); `next` carries an opaque, tamper-proof signed `cursor=` token (`django.core.signing`, salt `dojo.api_v3.cursor`) encoding only the ordering + last-row key position (never the filters). Sorting is restricted to keyset-safe orderings — `id`, `created`, `updated` (each ± direction), whichever a resource's `FilterSpec` declares (derived per spec; a resource without `created`/`updated` allows `id` only) — always with a deterministic `id` tiebreaker appended; default is `id` ascending. Any other `o=` (or a multi-field `o=`) → 400 problem+json (pagination type); a tampered/undecodable/ordering-mismatched cursor → 400. Pages are read as `limit+1` rows to detect a next page with no `COUNT` query. Available on the seven top-level list endpoints; parent-scoped sub-resource/edge lists remain offset-only. Filters/`q=`/`fields=`/`expand=`/`include=counts` all compose unchanged. +- Sorting: `o=` comma-list, `-` prefix for descending (`o=-severity,date`). Allowed orderings are a curated per-object list (part of the filter contract, §4.9). + +### 4.4 Refs (D3) +```jsonc +{ "id": 42, "name": "Q3 Pentest" } +``` +- Produced by one shared `Ref` schema. The relation key conveys the type — no `type` field, **except** location refs, which add `"type": ""` because one key can hold heterogeneous location subtypes. +- Ref label registry (single source of truth, `dojo/api_v3/refs.py`): + +| Model | `name` source | +|---|---| +| Product_Type, Product, Engagement | `.name` | +| Test | `.title` if set, else `str(.test_type)` | +| Finding | `.title` | +| Dojo_User | `.username` (clients do their own display formatting — D9) | +| Location | `.location_value` | +| Test_Type, Development_Environment | `.name` | + +### 4.5 Slim shapes (starting field lists) +Slim = identity + primary status fields + parent refs + timestamps. **No per-row computed fields** (no `age`, no SLA math, no counts requiring extra queries — `locations_count` is a queryset annotation, which is allowed). Implementer: verify each field exists on the model before wiring; adjust only by removing, not adding, without a §12 entry. + +- **FindingSlim:** `id, title, severity, active, verified, false_p, duplicate, risk_accepted, out_of_scope, is_mitigated, date, cwe, cwes[], vulnerability_ids[], test{ref}, engagement{ref}, product{ref}, product_type{ref}, reporter{ref}, locations_count, tags[], created, updated`. + - `cwes` (flat `list[int]`) and `vulnerability_ids` (flat `list[str]`) are identity-critical and always present on slim (no per-row detail fetch); populated by two fixed `prefetch_related` paths (`finding_cwe_set`, `vulnerability_id_set`). See §12 (2026-07-21). + - `engagement/product/product_type` are denormalized parent refs populated by one `select_related("test__engagement__product__prod_type")` — no extra queries. +- **TestSlim:** `id, name(=title), test_type{ref}, engagement{ref}, product{ref}, product_type{ref}, environment{ref|null}, lead{ref|null}, target_start, target_end, percent_complete, tags[], created, updated`. +- **EngagementSlim:** `id, name, product{ref}, product_type{ref}, lead{ref|null}, status, engagement_type, target_start, target_end, active, tags[], created, updated`. +- **ProductSlim:** `id, name, description, product_type{ref}, lifecycle, tags[], created, updated`. +- **ProductTypeSlim:** `id, name, description, critical_product, key_product, created, updated`. +- **UserSlim:** `id, username, first_name, last_name, email, is_active, is_superuser, last_login`. +- **LocationSlim:** `id, name(=location_value), type(=location_type), tags[]`. URL-subtype detail fields (`protocol, host, port, path, query, fragment`) appear on retrieve/expand only. + +Detail (`GET /{id}`) returns Slim + a documented per-object set of heavier fields (e.g. Finding adds `description, mitigation, impact, steps_to_reproduce, severity_justification, references, file_path, line, mitigated, mitigated_by{ref}`). List returns Slim only. + +### 4.6 `?expand=` (D3) +- Comma-separated dotted paths: `?expand=test.engagement,reporter`. Each path segment must be a registered expandable relation of the current schema (per-schema registry `EXPANDABLE: dict[str, tuple[django_relation_path, TargetSchema]]`). +- Expanding swaps the ref for the target's Slim schema **inline** (no side blob). +- No fixed depth cap. Two guards, both returning 400 problem+json: + - **Cycle guard:** while walking a path, track the model classes from the root; a segment whose target model already appears in its own ancestry chain is rejected (`finding → test → ... → findings` style loops). + - **Budget:** total expanded relation *nodes* across all paths ≤ `EXPAND_BUDGET = 10` (settings-overridable). `expand=test.engagement,reporter` = 3 nodes. +- **The expand set drives the queryset:** FK/O2O segments append `select_related` paths; M2M segments append `prefetch_related`. Implemented centrally in `dojo/api_v3/expand.py` (`plan_queryset(qs, paths)`), never per-route. +- Special case: `expand=locations` on a finding swaps `locations_count` for `locations: [{location: {ref+type}, status, audit_time}]` (bounded by the same budget; uses `prefetch_related`). + +### 4.7 `?fields=` (D3) +- Comma-separated allowlist per object (documented enum = the **Detail** field names + the schema's `EXPANDABLE` keys). Unknown field → 400 (dedicated `fields` problem type). `id` always included. This is the "picker/dropdown" facility (`?fields=id,name`), replacing the need for separate restricted-list endpoints. +- **On a list, `?fields=` may opt UP into the Detail field set (Jira-style).** The default (no `?fields=`) list is the Slim shape exactly. When a requested field is beyond Slim, the row serializes with the Detail shape and is then projected to the requested fields; when all requested fields are within Slim, behaviour is identical to Slim. Because `?fields=` names are plain **row-columns**, opting up is a wider `SELECT` on the same single query — never a per-row cost. List querysets therefore `defer()` the heavy detail columns that were not requested, so the default list never fetches them and `?fields=impact` un-defers exactly `impact` (Detail-only *relation* refs, e.g. `finding.mitigated_by`, add a fixed `select_related` join, applied only when requested). **Guard:** only row-columns are eligible for `?fields=`; anything computed per row (SLA/age math, per-row counts) is permanently excluded — it would reintroduce the N+1 the Slim default removes. Detail `GET /{id}` routes are never deferred. Implemented resource-agnostically by the kernel (`plan_list_fields` / `serialize_list_row` in `dojo/api_v3/expand.py`); the location-edge sub-resource lists serialize fixed edge shapes and are out of scope (§12). + +### 4.8 `?include=` (D6) +- Namespaced response add-ons rendered into `meta`. Alpha: `include=counts` on findings lists only: +```jsonc +"meta": { "counts": { "total": 812, "active": 400, "verified": 123, "duplicate": 12, + "severity": {"Critical": 3, "High": 40, "Medium": 200, "Low": 500, "Info": 69} } } +``` +- Computed on the **filtered, authorized** queryset via one aggregate query. The mechanism (`dojo/api_v3/include.py`) is generic: a registry of `include_name -> callable(filtered_qs) -> dict`, so later includes (and downstream-defined ones) plug in without contract changes. + +### 4.9 Filter contract (D6) +- Grammar (fixed): exact `field=`, lookups `field__gte/__lte/__gt/__lt/__in/__icontains/__isnull`, multi-sort `o=`, free-text `q=`. `__in` takes comma-separated values. +- Per-object vocabulary is **curated** and documented; alpha findings vocabulary: `id__in, title__icontains, severity, severity__in, active, verified, duplicate, false_p, risk_accepted, out_of_scope, is_mitigated, date__gte/__lte, cwe, cwe__in, product, product__in, product_type, engagement, test, reporter, tags__in (any-match), created__gte/__lte, updated__gte/__lte`; orderings: `o= id, date, severity, title, created, updated` (± each). Other objects: analogous minimal sets (id/name/parent-FK/dates). +- Implementation reuses django-filter `FilterSet`s where practical (framework-light: `FilterSet(request_params, queryset).qs`); severity ordering must sort by severity *rank*, not alphabetically (reuse v2's approach). +- **Snapshot-tested:** a test renders each object's vocabulary (params + orderings) to a checked-in JSON snapshot; contract drift fails CI (§6 OS2). + +### 4.10 Errors (D9) +RFC 9457 problem+json, `Content-Type: application/problem+json`: +```jsonc +{ "type": "https://docs.defectdojo.com/api/v3/errors/validation", + "title": "Validation failed", "status": 400, + "detail": "2 fields failed validation", + "fields": { "severity": ["Not a valid choice."], "date": ["Required."] } } +``` +- 401 unauthenticated, 403 forbidden (RBAC), 404 unknown-or-unauthorized object (do not leak existence), 400 validation/expand/filter errors. One exception handler in `dojo/api_v3/errors.py`; routes never hand-build error bodies. + +### 4.11 Conventions (D9) +ISO-8601 UTC with `Z` suffix for datetimes; dates as `YYYY-MM-DD`; explicit `null`s; tags as `list[str]`; enums as their stored string values; write payloads reference relations by integer id (`"test": 9`), reads render refs — asymmetry is intentional and documented. + +**Update verbs (`PATCH` + `PUT`).** Every writable resource (findings, assets, organizations, engagements, tests, users; locations stay read-only) offers both. `PATCH` is a partial update (`model_dump(exclude_unset=True)` — only supplied fields change). `PUT` is a **full replace**: it validates against the resource's create-shaped Write schema (required fields enforced, `extra="forbid"`) and applies `payload.dict()` **without** `exclude_unset`, so an omitted optional resets to its schema default — mirroring v2's DRF `update(partial=False)`. The `PUT` schema default of a **non-null / non-blank-with-default** column is that column's *model* default (not `None`), so a full replace never violates a `NOT NULL`/blank constraint. Both verbs share the identical permission ladder (authorized-queryset resolve → 404, object edit permission → 403), reject unknown fields with a 400 problem+json, and treat an `editable=False` parent (finding `test`, test `engagement`) as non-reassignable. Finding writes (both verbs) flow through `dojo/finding/services.py` (I6). See §12 (2026-07-20 PUT reversal). + +### 4.12 Sub-resources: notes / tags / files +One generic factory each (`dojo/api_v3/subresources.py`), attached to all seven resources: +``` +GET/POST /api/v3//{id}/notes { "entry": "...", "private": false } +GET/PUT/POST /api/v3//{id}/tags { "tags": ["pci"] } # PUT replaces, POST appends +DELETE /api/v3//{id}/tags/{tag} +GET/POST /api/v3//{id}/files (multipart: file, title) +GET /api/v3//{id}/files/{file_id}/download +``` +Authorization inherited from the parent object (mirror v2's related-object permission classes and note privacy rules). `NoteSchema`: `id, entry, author{ref}, private, edited, created, updated`. `FileSchema`: `id, title, size, created`. + +### 4.13 Consolidated import +``` +POST /api/v3/import (multipart/form-data) + mode: auto | import | reimport # default auto + file, scan_type, engagement | test | (product_name+engagement_name+auto_create_context), + close_old_findings?, do_not_reactivate?, ...CommonImportScan fields +``` +- `auto` resolves via existing `AutoCreateContextManager.get_target_test_if_exists(...)`. +- **Destructive flags are never implied by mode** — if omitted, importer defaults apply and the response echoes effective values. +- **v3 imports are synchronous** — the response returns once the import completes; there is no job resource and no deferred-execution path for imports (removed from v3 scope by architect directive; §12, 2026-07-20). +- Endpoint delegates entirely to the new `dojo/importers/services.py` facade (§6 OS1). Response: +```jsonc +{ "mode_resolved": "reimport", "test": { "id": 9, "name": "ZAP Scan" }, + "statistics": { "new": 4, "reactivated": 1, "closed": 2, "untouched": 37 }, + "close_old_findings": true } +``` + +### 4.14 Locations (D5) +- `GET /api/v3/locations` + `GET /api/v3/locations/{id}` (read-only in alpha; lifecycle is import-driven). Filter: `type`, `name__icontains`, `product`. +- `GET /api/v3/findings/{id}/locations` and `GET /api/v3/products/{id}/locations` return edge rows: `{ location: {id, name, type}, status, audit_time }` (+ `auditor{ref}` on findings). +- Documented: URL locations only in alpha; other types 501/absent until models exist. + +### 4.15 CSV export (D6 — the second projection) +- `GET //export.csv` on the seven top-level list resources (`findings`, `assets`, `organizations`, `engagements`, `tests`, `users`, `locations` — `locations` keeps its superuser gate). Registered as a **literal** path so the `{int:id}` detail route's integer converter cannot match `export.csv` (no collision, verified). +- **Identical filter contract as the list:** filters, `o=` (the full offset-mode ordering vocabulary), `q=`, and `?fields=` (including the Detail opt-up). **No pagination** — the export is the whole filtered, authorized set. `expand`/`include`/`limit`/`offset`/`pagination`/`cursor` are **not applicable → 400** problem+json (`type=.../errors/export`), keeping the reserved-param handling coherent with the list. +- **Row cap:** `API_V3_EXPORT_MAX_ROWS` (env `DD_API_V3_EXPORT_MAX_ROWS`, default 100 000). Measured with the shared capped-count helper; if the filtered count exceeds the cap the request is a **400** telling the client to narrow the filter — **never a silent truncation**. +- **Response:** `StreamingHttpResponse`, `Content-Type: text/csv; charset=utf-8`, `Content-Disposition: attachment; filename="-export.csv"`, `X-API-Status: alpha`. Streamed by a `csv.writer` over `queryset.iterator(chunk_size=…)` (chunked prefetch), so memory is bounded and the query count is independent of the row count (pinned by `assertNumQueries`). +- **Column flattening (one generic, resource-agnostic kernel — `dojo/api_v3/csv_export.py`; no per-resource column lists):** columns are the serialized row's keys in schema order, restricted to the `?fields=` projection. A `{id, name}` ref → two columns `_id`,`_name`; a location `{id, name, type}` ref → three columns `_id`,`_name`,`_type`; `tags` (any list field) → one semicolon-joined column; datetimes ISO-8601 `Z`, dates `YYYY-MM-DD`, booleans lowercase; `null` → empty cell. A zero-row export still emits the full header row. Row values are reused from `serialize_list_row` (the same shaping as the list) then flattened, so the CSV never diverges from the JSON. +- **CSV-injection hardening (security product):** any cell whose text starts with `=`, `+`, `-`, `@` or a TAB is prefixed with a single quote (the standard spreadsheet formula-injection defense). + +--- + +## 5. Downstream forward-compatibility invariants (MUST hold in every OS phase) + +These guarantee downstream adoption phases are additions, never rework. **PR review checklist — every item verified before merge:** + +- **I1 Envelope is closed.** `{count, next, previous, results, meta}` and nothing else; `pagination=`, `include=`, `expand=`, `fields=`, `o=`, `q=` are reserved query params; new capabilities extend `meta`/`include`, never the envelope. +- **I2 Filter contract is a tested artifact.** Vocabulary snapshot in CI; grammar (§4.9) never varies per endpoint. +- **I3 Ref shape is closed.** `{id, name}` (+`type` for locations only). Extensions would break every consumer's ref handling. +- **I4 Schemas are composable.** Every response schema is a named, importable class; no anonymous/inline schemas; downstream code must be able to `class ProFindingSlim(FindingSlim): epss_score: float | None` and get correct OpenAPI. +- **I5 Routers are factories.** Each resource exposes `build__router(*, schema=None, filters=None, queryset_hook=None, auth=None) -> Router` with OS defaults. The OS mount calls the factory; a downstream distribution can call it with overrides and mount the result under its own prefix. No route logic at module import time. +- **I6 Services own all writes/side-effects.** Routes contain zero business logic (parse → authorize → service → shape). Service functions: keyword-only args, explicit `user`, return domain objects or result dataclasses, importable and callable without any HTTP context. +- **I7 Auth is a pluggable list** (§4.2); adding an auth class must require no route changes. +- **I8 RBAC only via `get_authorized_*` querysets + permission checks** — never ad-hoc `user.is_staff` logic in routes; expanded/included data is always drawn from authorized querysets. +- **I9 Error contract is closed** (§4.10); new error kinds get new `type` URIs, not new shapes. +- **I10 No UI-flavored formatting** in responses (D9) — formatting belongs to clients. + +--- + +## 6. OS phases (OS1–OS6) + +All phases land on one feature branch as **one PR** (§7), one merge-worthy commit (or small commit series) per phase. Each phase lists: goal, files, key details, tests, acceptance criteria. + +**Test suite layout — v3 is a separate package.** Unlike the flat `unittests/test_apiv2_*.py` files, all v3 tests live in a dedicated package so the suite is separately runnable, shardable in CI, and never interleaved with v2: +``` +unittests/api_v3/ + __init__.py + base.py # ApiV3TestCase(DojoAPITestCase): v3 URL-prefix helper, + # token + session/CSRF client helpers + test_apiv3_.py # all files referenced in the phases below live here + snapshots/filters.json # filter-contract snapshot (OS2) +``` +Django test discovery handles packages natively: `unittests.api_v3` runs the whole v3 suite, `unittests.api_v3.test_apiv3_findings` one module. Implementer: verify the CI unit-test workflow discovers the subpackage (standard Django labels do; a hardcoded file glob would need a tweak). Run tests exactly as repo convention requires: +```bash +./run-unittest.sh --test-case unittests.api_v3.test_apiv3_findings 2>&1 | tee /tmp/test_output.log +``` + +**Test client policy.** Ninja endpoints are plain Django views, so the standard in-process Django test client (via `DojoAPITestCase`) works against v3 unchanged — no socket, same thread: server-side exceptions propagate as real tracebacks (`raise_request_exception` default), logging is visible, `mock.patch` affects what routes see, `assertNumQueries` works, and the test transaction is shared (DB-state assertions for import equivalence). **All contract/integration tests use it** (full URLconf/middleware/auth/CSRF stack). `ninja.testing.TestClient` (invokes router operations directly, skipping URL resolution and middleware — and therefore real auth) is allowed **only** for kernel-internal unit tests (expand parser, pagination math). + +### OS1 — Foundation, findings read path, import, and the framework gate +**Goal:** working `/api/v3/` skeleton proving the whole contract on its hardest consumer (findings), plus the import endpoint; produce the evidence for the framework decision. + +Files: +- `requirements.txt`: add `django-ninja` (pin latest stable 1.x at implementation time; it vendors/pulls pydantic v2 — note both in the PR description). +- `dojo/api_v3/__init__.py` — `NinjaAPI` instance: `csrf=True`, `auth=[TokenAuth(), django_auth]`, exception handlers registered, `X-API-Status: alpha` via a response hook/middleware, version string. +- `dojo/api_v3/auth.py` — `TokenAuth` (parse `Authorization: Token ` → `Token.objects.select_related("user")`, set `request.user`; return None on mismatch so session auth can try). +- `dojo/api_v3/refs.py` — `Ref` schema + label registry (§4.4) + `to_ref(obj)` helper. +- `dojo/api_v3/pagination.py` — envelope paginator (§4.3) incl. the hybrid exact→planner-estimate count (offset mode) and forward-only keyset cursor mode (`?pagination=cursor`, opaque signed cursors). +- `dojo/api_v3/errors.py` — problem+json handlers (validation, 401/403/404, expand/filter errors). +- `dojo/api_v3/expand.py` — parse/validate/plan (§4.6). OS1 may hard-scope the registry to finding relations; generic walker completed in OS2. +- `dojo/api_v3/filtering.py` — django-filter adapter: build FilterSet from a declarative per-object spec; apply `o=`/`q=`. +- `dojo/api_v3/include.py` — include registry + `counts` implementation (§4.8). +- `dojo/finding/api_v3/schemas.py` — `FindingSlim`, `FindingDetail` (§4.5); `dojo/finding/api_v3/routes.py` — `build_findings_router()` (I5) with `GET /findings`, `GET /findings/{id}`. Queryset: `get_authorized_findings(Permissions.Finding_View)` + base `select_related("test__test_type", "test__engagement__product__prod_type", "reporter")` + `annotate(locations_count=Count("locations"))`. +- `dojo/importers/services.py` — the facade (framework-neutral; useful even if the gate fails): + ```python + @dataclass(frozen=True) + class ImportResult: + test: Test + mode_resolved: str # "import" | "reimport" + new: int; closed: int; reactivated: int; untouched: int + test_import: Test_Import | None + close_old_findings: bool; do_not_reactivate: bool # effective values + + def import_scan(*, user, scan_file, scan_type, ...) -> ImportResult + def reimport_scan(*, user, scan_file, test, ...) -> ImportResult + def auto_import_scan(*, user, ...) -> ImportResult # resolves via AutoCreateContextManager + ``` + Internally constructs the importer context exactly as `ImportScanSerializer.save()` does today (`dojo/api_v2/serializers.py:526-551` is the reference implementation; do not modify it) and unpacks the 7-tuple into `ImportResult`. +- `dojo/api_v3/import_routes.py` — `POST /import` (§4.13): multipart parse, permission check mirroring `UserHasImportPermission`/`UserHasReimportPermission` semantics, delegate to facade. +- `dojo/urls.py` — conditional mount (D5/§4.1). + +Tests (OS1): `test_apiv3_auth.py` (token OK, session+CSRF OK, anonymous 401, both on same endpoint); `test_apiv3_findings.py` (slim shape incl. denormalized parent refs + `locations_count`; `expand=test.engagement`; `fields=`; `include=counts` under RBAC; filter basics; pagination envelope; count switchover exact→estimate with a lowered `COUNT_CAP` override, incl. clamp + `count_exact` flag); `test_apiv3_findings_queries.py` (**`assertNumQueries` constant** for 100-row list, with and without expand — the headline guarantee); `test_apiv3_import.py` (import/reimport/auto DB-state equivalence vs the v2 endpoints on identical payloads — same counts, same finding states, incl. `close_old_findings`). + +**Gate (architect-reviewed, not self-certified):** the implementer produces a gate report (criteria results + benchmark numbers, posted on the draft PR) and **stops**; the architect reviews, records the outcome in §12, and releases OS2. Criteria: (1) query-count tests green; (2) import equivalence green; (3) RBAC via authorized querysets works as an auth dependency; (4) django-filter reuse demonstrated or equivalent costed; (5) both auth modes green; (6) OpenAPI renders + client codegen sane for Ref/Slim/expand; (7) I4/I5 seam demo (subclass `FindingSlim` + remount router, no fork). Failure → re-implement OS1 surface on DRF with identical contract; OS2+ unchanged. + +### OS2 — Kernel hardening +**Goal:** finish the generic machinery so OS3 resources are declaration-only. +- `expand.py`: generic registry-driven walker, cycle guard, budget, M2M prefetch planning, `expand=locations` edge-row special case. +- `filtering.py`: severity rank ordering; `q=` per-object field sets; vocabulary **snapshot test** (`test_apiv3_filter_contract.py` writes/compares `unittests/api_v3/snapshots/filters.json`). +- `include.py`: registry finalized; counts on any findings-shaped queryset. +- `errors.py`: full coverage sweep — every failure path returns problem+json (add tests for expand-budget 400, cycle 400, unknown field/filter 400, unauthorized 404-vs-403 policy). +- OpenAPI polish: tags per resource, alpha banner text, `openapi.json` served; CI check that schema generation succeeds (mirror how v2's spectacular check runs in CI if present; otherwise add a unit test that renders the schema). + +Acceptance: kernel modules have no per-resource imports (dependency direction: resources import kernel, never reverse — keeps I5 honest). + +### OS3 — Core resources + finding write path +**Goal:** CRUD for `product_type`, `product`, `engagement`, `test`, `user`; write endpoints for `finding`. +- Per resource: `dojo//api_v3/{schemas,routes}.py`, router factory, filter spec, authorized queryset (`dojo//queries.py` — all exist), permission checks mirroring the v2 permission classes for create/update/delete. +- **Write schemas (create/update):** the editable subset of the detail fields; required-vs-optional mirrors the v2 serializer for that resource; relations by integer id (§4.11); server-managed fields (`id`, `created`, `updated`, computed refs) are never writable. Ambiguity → conservative choice + §12 entry (per §10.2). +- **Finding writes go through a new service** (`dojo/finding/services.py`), extracted as *new code* from the logic in `FindingSerializer.update()/validate()` (`dojo/finding/api/serializer.py:438-560` is the reference; leave it untouched): + ```python + def update_finding(finding, *, changes: dict, vulnerability_ids: list[str] | None = None, + push_to_jira: bool = False, user) -> Finding + def create_finding(*, test, data: dict, user, push_to_jira: bool = False) -> Finding + def delete_finding(finding, *, user) -> None + ``` + Must reproduce: JIRA push + keep-in-sync (`jira_services.push`, `is_keep_in_sync`), risk-acceptance processing, vuln-id/CWE persistence, mitigated-field edit rules, active/verified/duplicate/false_p invariants. Port the *validation semantics* into schema+service validation errors. + **Extraction rule (D7):** reconcile the v2 serializer path with the corresponding UI view flows (`dojo/finding/views.py` — edit, close, risk-acceptance) before finalizing each service function. List every behavioral divergence found (notifications, JIRA sync, status side-effects) in §12 with the chosen canonical behavior — these services are the future home for the UI views too (CONV2), so API-only semantics must not get baked in unnoticed. +- Users: read + self; admin-only writes (mirror v2 `UserViewSet` permission behavior). +- Tests: per-resource `test_apiv3_.py` (CRUD happy path, RBAC 404/403, filters, expand where applicable); `test_apiv3_finding_writes.py` asserting service side-effects (JIRA mocked, risk acceptance, vuln ids) match v2 behavior for the same payloads. + +### OS4 — Locations +Per §4.14: `dojo/location/api_v3/`, finding/product location sub-resource routes, `locations_count` already wired in OS1. Read-only. Tests: edge-status shape, URL-only registry behavior, flag-off ⇒ whole `/api/v3/` absent (404), RBAC inheritance from parent. + +### OS5 — Notes / tags / files sub-resources +`dojo/api_v3/subresources.py` factories (§4.12) attached to all seven resources. Reference semantics: the v2 `@action` implementations (`dojo/finding/api/views.py:412-586`) and note-privacy/permission classes. Tests: `test_apiv3_subresources.py` — matrix over (resource × notes/tags/files × CRUD), privacy rules, parent-authorization inheritance, multipart upload + download. + +### OS6 — Verification sweep, docs, examples, release readiness +- Port the intent of `unittests/test_apiv2_prefetch_rbac.py` to expand: `test_apiv3_expand_rbac.py` — a user must never see an expanded/included object outside their authorized querysets. +- Repeat query-count + latency benchmark v2 (`?prefetch=`) vs v3 (`?expand=`), 100-row findings list on the largest fixture; record numbers in the PR description. +- Docs-site page (`docs/content/...`): v3 alpha overview, contract summary, v2→v3 mapping table for the seven objects, "one way to do notes/tags/files", import consolidation, alpha disclaimer. +- **`api_v3_examples.md` (repo root):** real captured request/response pairs (not hand-written) for developer/architect review — findings (detail, list with filtering+pagination+expand+include=counts, notes sub-resource, locations sub-resource, import) and products (the simple-entity contrast: detail, list, CRUD). Captured by executing real requests through the in-process test client against fixture data; bodies verbatim. +- **Docs UI:** built-in Swagger UI already serves `/api/v3-alpha/docs` (try-it works with session auth). Evaluate swapping to **Scalar** (modern OpenAPI reference UI; self-hostable single JS asset pointing at `/openapi.json` — must be vendored locally like the v2 spectacular sidecar, no CDN). Keep Swagger if vendoring is awkward — this is polish, not contract. +- Final §5 invariant checklist pass; final ruff/lint pass; ensure v2 test suite untouched and green. + +--- + +### Post-alpha OS backlog (explicitly deferred TODOs — architect-confirmed) + +Deliberately out of the alpha, to be scheduled in later phases. Each is enabled by the service +layer and the closed contract, so all are additive: + +- [ ] **Bulk operations** (`bulk_update`/`bulk_delete`/… with per-item skip reasons — checklist #5) +- [ ] **Workflow actions** (`close`, `request_review`, `mark_duplicate`, `unaccept_risk`, … — checklist #6) +- [ ] Delete-impact preview (`GET //{id}/delete-preview` — checklist #14) +- [ ] **XLSX export** from the filter contract (checklist #15 — CSV shipped in the alpha, §4.15; XLSX is the remaining format) +- [ ] delete-time `push_to_jira` param, `configuration_permissions` on user + writes, v3 self-profile endpoint, filter vocabularies for the location-edge sub-resources + (PUT full replace was pulled forward out of this bundle and shipped — §12, 2026-07-20) +- [x] **Accept a `cwes` list on finding writes** — DONE: finding create/update/replace now accept a `cwes` list (flat `list[int]`, primary-first) symmetric with the read shape and parallel to `vulnerability_ids` (§12, 2026-07-21). +- [ ] **Ref-registry completeness guard test** — a serialization audit (2026-07-21) confirmed every `Ref`/collection field on every read schema has a resolver and every fall-through field is a plain scalar (no manager/model/tagulous object reaches Pydantic). The one latent soft-spot: `ref_label()` (`dojo/api_v3/refs.py`) falls back to `str(obj)` for a model **not** in `_LABELERS`, so a future `Ref` to an unregistered model would silently emit a repr-ish label instead of failing. Add a test that walks every `Ref`-typed field across all schemas and asserts its target model is registered in `_LABELERS` — converting the silent mislabel into a caught failure (same self-extending spirit as the query/authz sweeps). +- **Port the v2 endpoint-level test corpora to v3** (architect-requested). Priority order: + - [x] **(1) the import/reimport corpus** (`test_import_reimport.py`'s mixin, `test_apiv2_scan_import_options.py`, + `test_importers_closeold.py`) — DONE via the dual-endpoint adapter `unittests/api_v3/import_corpus_shim.py` + (§12, 2026-07-21). Zero v2 test files modified. + - [x] **(2) JIRA push flows** (`test_jira_import_and_pushing_api.py` intent) against v3 finding + writes + import — DONE via `unittests/api_v3/test_apiv3_jira_push.py` (targeted ports, mocked JIRA layer) + + the `push_to_jira` ImportForm field / route `push_all_issues` OR (§12, 2026-07-21). + - **(3) evaluate the rest of `test_apiv2_*` per scope** — DONE: the full disposition is the + **§9.1 table**. The remaining PORT-LATER items are the only open checkboxes: + - [x] `TestDetail` matching-policy read fields (`deduplication_algorithm` + `hash_code_fields`, **rejected**-on-write) — ported via `unittests/api_v3/test_apiv3_test_dedupe_policy.py` (§12, 2026-07-21) + - [ ] `authorized_users` member-management on asset writes gated by `Product_Manage_Members` — ports `test_product_authorized_users_api_authz.py` + - (`configuration_permissions` on user writes + v3 self-profile endpoint are already tracked in the bundle above; they also close `ConfigurationPermissionTest`/`UserProfileTest`.) + +## 7. PR strategy + +- **One feature branch, one PR** for OS1–OS6. Rationale: the deliverable is additive-only (D1), so blast radius is contained; a single PR deploys/tests as one unit; reviewers get one coherent contract instead of six partial views. Keep one merge-worthy commit per phase so the history doubles as the phase log. +- The OS1 framework gate happens **on the branch** (draft PR): benchmark + exit-criteria results posted as a PR comment before OS2 work proceeds. +- **Post-alpha convergence track (separate PRs, not part of alpha; ordered by risk):** + - **CONV1 — v2 delegates to services.** Refactor `FindingSerializer.update()` and `ImportScanSerializer` (then other resources) to call the services. Small diffs; the extensive `test_apiv2_*` suite pins behavior. Zero contract change. + - **CONV2 — UI views delegate to services.** View-by-view (never big-bang), starting with the flows the services already reconciled (finding edit/close, risk acceptance, import). UI view test coverage is thinner than the API's, so each flow gets behavior-pinning tests *before* its refactor. Zero template/UX change. + - **CONV3 — delete dead duplicates.** Remove the now-unused logic bodies (serializer `update()` internals, per-resource notes/files action implementations the views duplicate). Only after CONV1+2 prove the services in production. + +## 8. Downstream adoption phases (outline only — do not start; the §5 invariants are what make these pure additions) + +- **DS1 — Consume:** mount OS v3 in the downstream distribution; verify session auth end-to-end; adapt the SPA's list controller to the v3 envelope/filter grammar (both offset-based — mechanical); migrate 1–2 read-heavy tables (findings first) behind a feature flag. +- **DS2 — Extend:** subclass schemas (EPSS, priority, …), extend filter specs, remount via router factories (I4/I5); replace restricted picker endpoints with `?fields=`. +- **DS3 — Write parity:** bulk operations (partial-success result shape: `{updated, skipped: [{id, reason}]}`) and workflow actions (`close`, `request_review`, `mark_duplicate`, …) as routers over the OS service layer; delete-impact preview; CSV export from the filter contract. +- **DS4 — Aggregation:** dashboard and metrics needs served by a **small set of generic aggregation primitives** (group-by, time-bucket, top-N) taking the identical filter params (D6) and returning chart-neutral data — never per-chart-type endpoints whose envelopes couple the API to one charting library; chart shaping moves to the client. New `include=` members where cheap. +- **DS5 — App services:** saved table preferences (backed by the structured filter representation, D6), permission map, global search. Any bootstrap-style mega-endpoint is **split into small cacheable resources** when it moves — do not port a monolith. +- **DS6 — Retirement:** legacy private API surfaces retired domain-by-domain; v2 deprecation window last. + +## 9. Appendix — downstream capability checklist (from the downstream-consumer audit) + +| # | Capability | Disposition | +|---|---|---| +| 1 | `{count,next,previous,results}` + limit/offset (max 250) | **OS1** (D4) | +| 2 | Filtered-list totals in `meta` | **OS1** `include=counts` (D6) | +| 3 | Relation inlining (v2 `?prefetch=`, 30+ SPA call sites) | **OS1** refs+`expand` — strictly better | +| 4 | In-band memoized nested objects | **Rejected** (D3) — no mainstream precedent; gzip; client machinery gets deleted, not ported | +| 5 | Bulk ops w/ per-item skip reasons | DS3 (services designed for it: I6) | +| 6 | Workflow actions (close/review/duplicate/…) | DS3 (same service extractions as OS3) | +| 7 | Dashboard aggregation engine | DS4 (enabled by D6) — as generic group-by/time-bucket/top-N primitives, not per-chart-type endpoints | +| 8 | Metrics/insights rollups | DS4 | +| 9 | Per-object count endpoints | Covered by `include=counts` pattern | +| 10 | Saved views / table preferences | DS5 | +| 11 | django-filter vocabulary + multi-sort + match modes | **OS1/OS2** filter contract (D6) | +| 12 | Restricted/picker list variants | Covered by `?fields=` + RBAC querysets | +| 13 | Long-running job status polling (import / report / celery) | Intentionally removed from v3 scope by architect directive (§12, 2026-07-20) — v3 imports are synchronous; no job resource is reserved | +| 14 | Delete-impact preview | DS3 (`GET //{id}/delete-preview` fits contract) | +| 15 | CSV/XLSX export | **CSV shipped in the alpha** (§4.15, `GET //export.csv`, same filter contract — D6); XLSX deferred | +| 16 | Global search + `meta.by_type` | DS5 | +| 17 | Auth bootstrap + permission map | DS5 (session auth ready since OS1: D8) | +| 18 | Unix-ms datetimes / null-stripping / preformatted users | **Rejected** (D9) — client-side adaptations | +| 19 | Error contract for field errors + retry markers | **OS1** problem+json with `fields` extension (D9) | + +### 9.1 v2 API test-corpus disposition (backlog priority 3 — the evaluation) + +Inventory + classification of every remaining v2 API-shaped test suite (the ones the §6 backlog +item names, plus every `test_apiv2_*`, plus files discovered by an `APIClient`/`rest_framework`/`-list` +scan). **Legend:** **PORTED** = covered by the v3 suite today · **PORT-LATER** = maps to a v3 +surface not yet built (a remaining checkbox) · **SKIP** = covers endpoints/features v3 deliberately +lacks in the alpha (§2) · **SUPERSEDED** = v3 ships a stronger equivalent. `test_rest_framework.py` +(the 4241-line master suite) is split by area. + +| v2 suite / area | Disposition | Covered-by / needs / reason | +|---|---|---| +| `test_apiv2_scan_import_options.py` (ScanImportOptions) | **PORTED** | `test_apiv3_import_options_corpus.py` (2 scenarios ported via `import_zap_scan` override) + `test_apiv3_import_corpus.py` (priority 1). | +| `test_rest_framework.py` → ImportScanTest / ReimportScanTest | **PORTED** | `test_apiv3_import_corpus.py` (71 cases) + `test_apiv3_import.py` (DB-state equivalence). | +| `test_jira_import_and_pushing_api.py` (import/reimport push, push_all, keep-in-sync, PATCH/PUT push, verified-enforcement — non-group) | **PORTED** | `test_apiv3_jira_push.py` (priority 2, mocked JIRA layer) + `test_apiv3_finding_writes.py` (PATCH keep-in-sync + push-failure-400). Coverage map in §12 (2026-07-21). | +| `test_rest_framework.py` → Findings/Products(Assets)/ProductType(Organization)/Engagements/Tests/Users CRUD + schema + prefetch + RBAC | **PORTED** | per-resource `test_apiv3_{findings,assets,organizations,engagements,tests,users}.py` (+ finding writes). | +| `test_rest_framework.py` → Location / URL / LocationFindingReference / LocationProductReference (reads + edge shape) | **PORTED** | `test_apiv3_locations.py` (read-only + edge-status shape, URL-only registry). Edge/reference **writes** → SKIP (v3 locations read-only in alpha, D5). | +| `test_rest_framework.py` → Notes / Files | **PORTED** | `test_apiv3_subresources.py` (notes/tags/files matrix incl. note-create side-effects). | +| `test_rest_framework.py` → FindingCreateUpdateMitigatedFields | **PORTED** | `test_apiv3_finding_writes.py` (EDITABLE_MITIGATED_DATA both states). | +| `test_apiv2_prefetch_rbac.py` (PrefetchRBAC gate) | **SUPERSEDED** | `test_apiv3_expand_rbac.py` — prefetch→expand RBAC leakage sweep (OS6). | +| `test_apiv2_methods_and_endpoints.py` (method/endpoint smoke) | **SUPERSEDED** | `test_apiv3_openapi.py` (path/tag guard) + `test_apiv3_authz_sweep.py` + `test_apiv3_authz_static.py` + per-resource query/RBAC sweeps. | +| `test_rest_framework.py` → SchemaChecker (per-endpoint OpenAPI validation) | **SUPERSEDED** | `test_apiv3_openapi.py` (schema renders + path/tag presence); pydantic DTOs generate the schema (D3). | +| `test_fk_reassignment_authorization.py` (FK-reassign authz) | **SUPERSEDED** | v3 write RBAC re-checks `add` on a *reassigned* parent (engagement→product, test→api_scan_config) — `test_apiv3_engagements.py`/`test_apiv3_tests.py`/`test_apiv3_assets.py` (§12 OS3b). | +| `test_api_finding_filter_no_multiply.py` (to-many filters don't multiply rows) | **SUPERSEDED** (for v3's vocabulary) | v3 `tags__in` uses `distinct=True` + the `assertNumQueries` list guards row multiplication. The specific `finding_group`/`found_by`/`reviewers`/`risk_acceptance` filters → SKIP (not in the v3 alpha finding vocabulary, §4.9). | +| `test_apiv2_user.py` (User CRUD + perms) | **PORTED** (core) | `test_apiv3_users.py` (CRUD, RBAC, password guards). `configuration_permissions` on user writes → **PORT-LATER** (already a §6 backlog item). | +| `test_apiv2_test_dedupe_policy.py` (Test API exposes matching policy read-only) | **PORTED** | `test_apiv3_test_dedupe_policy.py` — `TestDetail` exposes `deduplication_algorithm` + `hash_code_fields` as read-only **computed** fields (reusing the v2 `Test.deduplication_algorithm`/`Test.hash_code_fields` model properties verbatim). v3 **rejects** writes to them (400 unknown-field, `extra="forbid"`) where v2 silently **ignores** them — deliberate hardening (§12, 2026-07-21). | +| `test_product_authorized_users_api_authz.py` (`authorized_users` gated by `Product_Manage_Members`) | **PORT-LATER** | v3 `AssetWrite` does not yet expose `authorized_users` (member-management is a distinct op). When added, needs the `Product_Manage_Members` gate + these authz cases. | +| `test_rest_framework.py` → UserProfileTest | **PORT-LATER** | v3 self-profile endpoint (already a §6 backlog item). | +| `test_rest_framework.py` → ConfigurationPermissionTest | **PORT-LATER** | `configuration_permissions` (same §6 backlog item as the user sliver). | +| `test_rest_framework.py` → FindingClose / FindingVerify / EngagementCloseReopen / FindingActionAuthz | **SKIP** | workflow actions — §2 non-goal (DS3). Status *fields* are writable and tested; the action endpoints are not. | +| `test_bulk_risk_acceptance_api.py` | **SKIP** | bulk operations — §2 non-goal (DS3). | +| `test_risk_acceptance_api.py` + `test_rest_framework.py` → RiskAcceptanceTest | **SKIP** (endpoint) | the `Risk_Acceptance` CRUD/workflow resource is not a v3 alpha surface. The finding-level `risk_accepted` invariant (simple_risk_accept / unaccept) **is** PORTED in `test_apiv3_finding_writes.py`. | +| `test_apiv2_endpoint.py` + `test_rest_framework.py` → Endpoint / EndpointStatus / V3EndpointStatus + `test_endpoint_status_cross_product_authz.py` | **SKIP** | legacy `Endpoint` model — v3 exposes Locations instead (D5); `/api/v3/` carries no Endpoint path. | +| `test_endpoint_meta_import.py` / `test_generic_meta_import.py` / `test_apiv2_metadata.py` (DojoMeta) + `test_rest_framework.py` → FindingMetadata | **SKIP** | metadata + endpoint-meta import endpoints — not a v3 alpha surface (§2). | +| `test_apiv2_notifications.py` + `test_rest_framework.py` → Notifications / NotificationWebhooks / Announcement | **SKIP** | notifications / webhooks / announcements endpoints — §2 non-goal. | +| `test_rest_framework.py` → Jira{Instances,Issues,Project} | **SKIP** (config endpoints) | JIRA *configuration* endpoints not in v3 alpha; JIRA *push behavior* is covered (priority 2). | +| `test_rest_framework.py` → Sonarqube* / {Product,Asset}_API_Scan_Configuration / ToolConfigurations / ToolProductSettings / ToolTypes / AppAnalysis + `test_api_sonarqube_updater.py` | **SKIP** | tool-config / SCA / scan-configuration endpoints — not a v3 alpha surface. | +| `test_rest_framework.py` → Language / LanguageType / ImportLanguages / NoteTypes / FindingTemplates / BurpRawRequestResponse / ReportGenerateFormat + `test_apiv2_limit_reqresp.py` | **SKIP** | languages / note-types / templates / burp req-resp pairs / report generation — none are v3 alpha surfaces (§2). | +| `test_rest_framework.py` → DevelopmentEnvironment / TestType (standalone CRUD) | **SKIP** (as endpoints) | v3 exposes these only as `{id,name}` refs / expand targets (§4.4), not as CRUD endpoints in the alpha; the ref rendering is PORTED. | +| `test_importers_importer.py` | **SKIP** (unit, not endpoint) | importer-internal unit tests; v3 endpoint equivalence is covered by the import corpus. | +| `test_location_reference_write_authz.py` | **SKIP** | location-reference *write* authz — v3 locations are read-only in the alpha (D5). | + +**Trivial high-value ports done from this table:** none at evaluation time. Every PORT-LATER item +required new v3 schema/route surface (matching-policy read fields, `configuration_permissions`, +self-profile, `authorized_users` member-management) that exceeded the <30-min trivial-port bar and +was already tracked as a §6 backlog checkbox — so nothing was implemented during the evaluation +itself (conservative, §10.2). *Subsequently closed:* the matching-policy read fields were ported in +`test_apiv3_test_dedupe_policy.py` (§12, 2026-07-21). + +**⚠ v2-sunset prerequisites (do NOT delete these v2 test files without doing the step first):** + +1. **`test_jira_import_and_pushing_api.py` is the ONLY consumer of the JIRA VCR cassettes.** + Cassettes are not self-executing — deleting this suite silently orphans the sole coverage of + the shared JIRA *engine* behaviors (issue transitions on mitigation/reactivation, `keep_sync` + reimport flows, `enforce_verified_status` gating). The v3 JIRA tests deliberately cover only + dispatch/plumbing at the service boundary and inherit engine coverage from this suite. + *Before removal:* record v3-keyed cassettes (or an equivalent live-JIRA integration harness) + for the engine scenarios — naturally paired with the post-alpha workflow-actions work, which + makes those transitions v3-endpoint-observable. +2. **`test_import_reimport.py` is imported by the v3 corpus shim.** `unittests/api_v3/ + import_corpus_shim.py` subclasses `ImportReimportMixin` from it (the anti-fork design). + *Before removal:* relocate the mixin to a shared module (e.g. `unittests/import_corpus_mixin.py`) + so the v3 corpus keeps running verbatim. + +## 10. Implementer instructions (read before writing code) + +1. **Never modify v2** (serializers, viewsets, urls, prefetch) or any existing test. v3 is additive. The only shared-file edits allowed: `dojo/urls.py` (mount), `requirements.txt`, settings (new constants `COUNT_CAP`, `EXPAND_BUDGET`). +2. **Contract is §4; invariants are §5.** If the contract is ambiguous or a model field doesn't match §4.5, do not improvise silently — pick the conservative option, implement it, and record the question + choice in §12. +3. **Tests:** all v3 tests live in the `unittests/api_v3/` package (base class `ApiV3TestCase`); run only via `./run-unittest.sh --test-case unittests.api_v3[.module] 2>&1 | tee /tmp/test_output.log`; analyze with `grep -E "PASSED|FAILED|ERROR" /tmp/test_output.log`; narrate each iteration (what failed, the fix, then re-run). Never `pytest`/`manage.py test` directly. Contract tests go through the Django test client (in-process, full stack); `ninja.testing.TestClient` only for kernel-internal units (§6 preamble). +4. **Reuse, don't rewrite:** RBAC only through `dojo//queries.py` `get_authorized_*` + `dojo.authorization` permission semantics; import only through `dojo/importers/*` wrapped by the new facade; JIRA through `dojo/jira/services.py`. Reference implementations to *read but not touch*: `dojo/finding/api/serializer.py:438-560`, `dojo/api_v2/serializers.py:526-551`, `dojo/finding/api/views.py:412-586`. +5. **Every list endpoint ships with an `assertNumQueries` test** proving query count is independent of row count, with and without `expand`. This is non-negotiable — it is the project's headline claim for v3. +6. **Lint:** ruff config is in-repo; run it before finishing a phase. Match surrounding code style; comments only for non-obvious constraints. +7. **Never commit** `CLAUDE.md`, anything under `.claude/`, or scratch/log files. +8. Phase order is OS1→OS6; do not start OS2 before the OS1 gate outcome is recorded in §12 **by the architect** — the gate is reviewed, never self-certified. Produce the gate report, then stop and wait. + +## 11. Verification summary (release bar for the OS PR) + +- Query-count regression suite green (constant queries, ±expand) — the headline claim, tested. +- Import equivalence: v3 `POST /import` (all three modes) reproduces v2 import/reimport DB state for identical payloads, incl. `close_old_findings`. +- RBAC parity: expand/include never leak unauthorized objects (port of prefetch-RBAC test intent). +- Contract tests: slim shapes (incl. denormalized finding parent refs, `locations_count`), expand permutations + cycle guard + budget, `fields`, `include=counts`, filter vocabulary snapshot, problem+json on every failure path, both auth modes. +- Locations: edge-status shape, URL-only, flag-off ⇒ no `/api/v3/`. +- OpenAPI generation + client codegen succeed (alpha-tagged). +- Benchmark artifact in PR: v2 `?prefetch=` vs v3 `?expand=` (query count + latency), 100-row findings list. +- Full existing v2 test suite untouched and green. + +## 12. Decision log & open questions + +| Date | Entry | +|---|---| +| 2026-07-19 | Plan created from the design spec + three audits (OS v2 API, downstream consumer usage, Location subsystem). Decisions D1–D10 recorded in §1; supersedes spec where they conflict (notably: finding↔location cardinality, pagination default, memoization rejection, importer-facade-as-new-work). | +| 2026-07-19 | Downstream-anchoring review ("current downstream usage is not written in stone"): four inherited habits replaced with better designs — (1) structured `POST //query` filter representation reserved; bespoke `_any`/`_all` suffix params banned (D6); (2) generic `/jobs/{id}` resource reserved for all async work instead of the three per-feature status mechanisms observed downstream (§4.13); (3) DS4 aggregation as chart-neutral primitives, not chart-library-shaped per-chart endpoints; (4) DS5 splits any bootstrap-style mega-endpoint into cacheable resources. Checked and kept deliberately: offset-default pagination (dual-mode envelope is the hedge if page-jump UX ever goes) and session+CSRF auth (independently correct for first-party SPAs). | +| 2026-07-19 | Duplication across UI views / v2 / v3 addressed via convergence strategy (D7, §7): v3 services are written for all three consumers but only v3 calls them in the alpha PR (stays additive); extraction reconciles serializer + UI-view semantics with divergences logged here; post-alpha CONV1 (v2→services) then CONV2 (UI views→services, behavior-pinning tests first) then CONV3 (delete dead duplicates). | +| 2026-07-19 | Alpha URL carries the stability marker: mount at `/api/v3-alpha/`, moving to `/api/v3/` at beta and staying there through GA (D1, §4.1). Rationale: URL is the only unmissable instability signal (K8s/Google precedent); structuring beta+GA on one URL limits clients to a single migration, timed to the additive-only contract freeze. Prefix is one constant; router factories make the remount a one-liner. | +| 2026-07-19 | Count strategy upgraded from "capped at COUNT_CAP" to hybrid exact→planner-estimate (D4, §4.3): above the cap, `count` becomes the Postgres `EXPLAIN` row estimate (PostgREST `count=estimated` technique), clamped to ≥ CAP+1, flagged `count_exact: false`. Rationale: real approximate numbers beat a "10000+" floor for pagination UX at ~zero extra cost; `pg_class.reltuples` rejected (whole-table only, useless under filters/RBAC). | +| 2026-07-19 | **OS1 GATE: GO on django-ninja — architect released all OS phases.** All 7 exit criteria met; benchmark shows v3 query count constant (5 no-expand / 7 with `expand=test.engagement`) at both 10 and 100 rows vs v2 scaling 91→271 (no prefetch) / 222→762 (`?prefetch=test`). Evidence in `.claude/os1-gate-report.md` (gate report reviewed by coordinator). Coordinator review found+fixed one pre-commit bug: `locations_count` needed `Count(..., distinct=True)` against `tags__in` joins. Latency benchmark deferred to OS6. | +| 2026-07-19 | **Trailing slash decided: NO trailing slash** on route paths (`/findings`, `/findings/{id}`, `/import`) — ninja-idiomatic, matches GitHub/Stripe convention. §4.3 example updated. `next`/`previous` are opaque (D4) so this binds only documentation and client codegen. | +| 2026-07-20 | **RESOLVED (2026-07-20, architect sign-off):** exact `django-ninja`/`pydantic` pins + license/supply-chain sign-off — pins accepted as-is. **OS1 implementer note:** pinned `django-ninja==1.6.2` (latest stable 1.x at implementation; MIT). It requires `pydantic>=2,<3`; `pydantic==2.13.4` (+`pydantic-core==2.46.4`, MIT) is already present in the image as a transitive dependency, so no *new* top-level pin was added for pydantic. Both are widely used, actively maintained. | +| 2026-07-20 | **RESOLVED (2026-07-20, architect sign-off):** `COUNT_CAP`/`EXPAND_BUDGET` defaults (10 000 / 10 proposed) — defaults accepted as-is. **OS1 implementer note:** implemented as `DD_API_V3_COUNT_CAP=10000` and `DD_API_V3_EXPAND_BUDGET=10` env settings (`API_V3_COUNT_CAP`/`API_V3_EXPAND_BUDGET`), settings-overridable per §4.3/§4.6; tests exercise a lowered cap. | +| 2026-07-19 | **OS1 impl — CSRF wiring (§4.2 conservative choice):** §4.2 specifies `NinjaAPI(csrf=True)`, but `django-ninja` 1.6.x removed the `csrf` parameter from `NinjaAPI.__init__`. Conservative choice: rely on ninja's `SessionAuth` (`ninja.security.django_auth`), which defaults `csrf=True` and enforces CSRF on unsafe methods for cookie/session auth, while `TokenAuth` (an `APIKeyHeader`) requires no CSRF. This achieves the D8 contract exactly (token bypasses CSRF; session enforces it) — verified by `test_apiv3_auth`. No `csrf=` kwarg is passed to `NinjaAPI`. | +| 2026-07-19 | **OS1 impl — query-count fidelity (beyond §6 literal base-queryset):** §6 OS1 lists the findings base queryset as `select_related(...) + annotate(locations_count)`. To keep the headline query count constant, the route also `prefetch_related("tags")` (FindingSlim exposes `tags[]`, §4.5) and the `expand` planner additionally pre-loads each expand-target schema's own `SELECT_RELATED`/`PREFETCH_RELATED` (e.g. `test__tags`) so serializing expanded objects issues no per-row queries. Purely additive to the spec; required by criterion 1. | +| 2026-07-19 | **OS1 impl — expand-target slim schemas placement:** §6 OS1 lists only `FindingSlim`/`FindingDetail` in `dojo/finding/api_v3/schemas.py`, but `?expand=test.engagement` needs slim schemas for the targets. `TestSlim`, `EngagementSlim`, `ProductSlim`, `ProductTypeSlim`, `UserSlim`, `TestTypeSlim`, `EnvironmentSlim` are defined there for OS1 (findings-only). OS3 relocates the canonical copies to their resource modules (`dojo//api_v3/schemas.py`); these will re-export or move. | +| 2026-07-19 | **OS1 impl — login-redirect exemption (settings):** DefectDojo's `LoginRequiredMiddleware` redirects unauthenticated requests to `/login` unless the path matches `LOGIN_EXEMPT_URLS`. v3 was appended to that list (mirroring `^api/v2/`) so anonymous v3 requests get a 401 problem+json instead of a UI redirect. Uses the single-source `API_V3_URL_PREFIX` constant. | +| 2026-07-19 | **OS1 impl — route path trailing slash (§4.3 example vs implementation):** routes are registered without a trailing slash (`/findings`, `/findings/{id}`, `/import`) — django-ninja's idiomatic convention and it resolves cleanly under `APPEND_SLASH`. The §4.3 example shows `.../findings/?...` with a trailing slash; this does not bind clients because `next`/`previous` are opaque URLs (D4) built from the actual request path. Flagged for architect confirmation; switching to trailing slashes is a one-line change per route if preferred (would then need the v2-style POST-trailing-slash middleware handling). | +| 2026-07-19 | **OS1 impl — response rendering:** routes return a pre-built `JsonResponse` (custom encoder gives ISO-8601 `Z` datetimes + `X-API-Status` header) which ninja passes through unchanged, so `?expand=`/`?fields=` can reshape `results` dynamically while the declared `response=` schema still documents the base shape in OpenAPI. Base slim serialization is schema-driven (`Schema.model_validate(obj)` runs ninja resolvers), so a subclassed schema serializes its new fields automatically (I4). | +| 2026-07-19 | **OS2 impl — severity ordering direction (§4.9 ambiguity, conservative choice):** `o=severity` sorts by *rank* (Critical first) and `o=-severity` reverses it (Info first). §4.9 names the rank order "(Critical>High>Medium>Low>Info)" but does not say which sign is "ascending"; resolved so plain `o=severity` yields exactly that listed order (also DefectDojo's canonical display order and `Finding.Meta.ordering`). Implemented in `filtering.py` as a query-time `Case/When` (`SEVERITY_RANK` Critical=0…Info=4) exposed via `FilterSpec.order_expressions`, mirroring v2's `numerical_severity` ordering but *computed at query time* so it never depends on the denormalized `numerical_severity` column being populated. | +| 2026-07-19 | **OS2 impl — unknown filter param → 400 (§6 OS2 item 2, conservative choice):** django-filter silently ignores query params it doesn't declare; §6 OS2 says "keep unknown-filter 400 behavior". Resolved by making `apply_filters` reject any query param that is neither reserved (`RESERVED_PARAMS`) nor a declared filter with a 400 `filter` problem+json, strengthening the single filter contract (D6). Caveat: cache-buster/tracking params (e.g. jQuery's `_=`) would 400 — acceptable for an alpha API and consistent with a strict typed contract. | +| 2026-07-19 | **OS2 impl — `?fields=` gets its own error type (I9):** unknown `?fields=` values previously reused the `expand` problem type; OS2 adds a dedicated `fields` type URI (`.../errors/fields`) so the closed error contract has one type per error kind (I9). Pure add; no shape change. | +| 2026-07-19 | **OS2 impl — `expand=locations` edge-row shape (§4.6/§4.14 reconciliation):** `expand=locations` on a finding renders `[{location: {id,name,type}, status, audit_time}]` and *replaces* `locations_count` in the output (§4.6 "swaps"), driven by `prefetch_related("locations__location")` (query count stays constant, verified by test). §4.14's additional `auditor{ref}` on finding-location rows is deferred to the OS4 `/findings/{id}/locations` sub-resource (OS2 delivers only the `expand=` projection per the OS2 task's explicit shape). Implemented generically via `ExpandRel.special`/`prefetch_paths`/`replaces` so any resource can declare a computed, prefetch-backed, count-replacing expansion. | +| 2026-07-19 | **OS2 impl — filter-contract snapshot storage (read-only test FS):** `unittests/api_v3/snapshots/filters.json` is checked in and compared on every run; the auto-create-on-missing branch of `test_apiv3_filter_contract` only works on a writable FS (the docker test bind-mount is read-only), so the baseline is generated on the host. Deliberate regeneration: `DD_API_V3_UPDATE_SNAPSHOTS=1` (host run). | +| 2026-07-19 | **OS2 impl — I5 dependency direction, verified:** the seven reusable kernel machinery modules (`auth`, `errors`, `expand`, `filtering`, `include`, `pagination`, `refs`) import **zero** resource/service modules. The only resource import under `dojo/api_v3/*` is a **deferred** import inside `api.build_api()` (the composition root/mount that wires the OS router factories) and `import_routes.py`'s use of `dojo.importers.*` (services, explicitly allowed). Reusable machinery stays resource-agnostic, keeping I5 honest; whether `api.py` (the mount) belongs in the kernel package or a separate mount package is left as an OS-cleanup question for the architect. | +| 2026-07-19 | **OS3a scope + no PUT (§6 OS3, decision recorded per task):** CRUD for the three simpler resources `product_type`, `product`, `user` shipped (`dojo//api_v3/{schemas,routes}.py` + `build__router()` factories, mounted in `build_api()`). Each exposes `GET` list, `GET /{id}`, `POST`, `PATCH /{id}`, `DELETE /{id}`, no trailing slash. **No `PUT` in alpha** — PATCH-only partial update (pydantic `model_dump(exclude_unset=True)`); a full-replace `PUT` is an additive, non-breaking addition available later if a consumer needs it. Engagement/test/finding **writes** are explicitly OS3b, not delivered here. | +| 2026-07-19 | **OS3a impl — canonical slim relocation (fulfils the OS1 §12 placement note):** `ProductTypeSlim`/`ProductSlim`/`UserSlim` moved from `dojo/finding/api_v3/schemas.py` to their resource modules (`dojo//api_v3/schemas.py`); the finding module now imports (re-exports) them so there is exactly **one** canonical class per model — verified at runtime that `finding.api_v3.schemas.ProductSlim is product.api_v3.schemas.ProductSlim` (and likewise product_type/user), so finding `expand=` targets and the resource endpoints serialize through the same schema (I4). No duplicate class. `TestSlim`/`EngagementSlim`/`TestTypeSlim`/`EnvironmentSlim` stay in the finding module (their resources are OS3b). Import direction `finding → product → product_type`, `finding → user` (no cycle). All 65 OS1/OS2 tests stay green. | +| 2026-07-19 | **OS3a v2-semantics finding — deletion (mirrored exactly):** `ProductTypeViewSet.destroy` / `ProductViewSet.destroy` both do: `async_delete().delete(instance)` when `get_setting("ASYNC_OBJECT_DELETE")` is truthy, else a **synchronous** `instance.delete()` wrapped in `with Endpoint.allow_endpoint_init():` (the `Endpoint` context is required while `V3_FEATURE_LOCATIONS` is on, because `Endpoint.__init__` otherwise raises during the delete cascade). v3 `DELETE` reproduces this verbatim (`_destroy()` in each routes module). `UsersViewSet.destroy` is different: **no async, no Endpoint context** — a plain `instance.delete()`, except a user may **not delete themselves** (v2 returns 400 "Users may not delete themselves"); v3 mirrors this (400 problem+json, self left intact). | +| 2026-07-19 | **OS3a v2-semantics finding — user visibility (v2-parity `view_user` gate + guaranteed self-read; coordinator-corrected):** §6 OS3 says "Users: read + self; admin-only writes". v2 `UsersViewSet` gates all access behind `UserHasConfigurationPermissionSuperuser` (DjangoModelPermissions: `auth.view_user` for GET) and returns `User.objects.all()`. v3 `_base_queryset` mirrors this (conservative per §10.2, never widening PII/email exposure beyond v2): holders of `auth.view_user` (superusers/staff via the OS bypass; a non-staff holder via explicit grant) get the RBAC-scoped `get_authorized_users("view", user=request.user)` queryset — for a non-staff holder that is the collaborator subset (co-members of their authorized products/types, **plus superusers**), i.e. **≤** v2's "all users" exposure; **everyone else sees only themselves** (`Dojo_User.objects.filter(pk=request.user.pk)`), so a plain user lists exactly their own record and `GET /users/{own id}` always resolves (the previous collaborator-for-all design was rejected: it exposed email to every authenticated user and 404'd a plain user's own record). **Writes** stay admin/superuser-only via the `auth.add_user`/`change_user`/`delete_user` configuration permissions (`user_has_configuration_permission`; superusers pass automatically). `UserSerializer.validate()` rules ported verbatim into the route: only superusers may add/edit superusers or staff; `password` is write-only and **cannot** be changed via PATCH (400); a password is required on create when `REQUIRE_PASSWORD_ON_USER` (400 otherwise); password run through Django `validate_password`. `configuration_permissions` (a v2 write field) is intentionally **out of the alpha write surface** (not in the read shape; M2M member-management concern) — additive later. | +| 2026-07-19 | **OS3a v2-semantics finding — write permission checks (mirror the v2 permission classes exactly):** `product_type` (`UserHasProductTypePermission`): create = `user_has_global_permission(user, "add")` (the literal string `"add"` is kept, not the `Permissions` enum, because the `dojo.add_product_type` config-permission carve-out keys on `permission == "add"`); update = object `edit`; delete = object `delete` (⇒ **staff-only** for a non-staff member under the legacy model, so a member can view/edit but a member `DELETE` is 403 — used as the "authorized-to-see-but-not-modify" 403 test). `product` (`UserHasProductPermission`): create = `add` on the target `prod_type` in the payload (404 if that product type doesn't exist, 403 if unauthorized — mirrors `check_post_permission`'s `get_object_or_404`); update = object `edit` **plus** `add` on a *reassigned* `prod_type` (re-checked only when the FK actually changes, mirroring `check_update_permission`); delete = object `delete`. Reads resolve the object through the authorized `view` queryset first → **404** for unknown-or-unauthorized, **403** only when visible-but-not-modifiable. `permission_to_action` was verified to map the `Permissions` enum and the action string identically, so either form is safe for the object checks. | +| 2026-07-19 | **OS3a impl — write-schema strictness + shapes:** create/update schemas are ninja `Schema`s with `model_config = {"extra": "forbid"}`, so an unknown body field is a **400** problem+json — consistent with the kernel's strict query contract (unknown `filter`/`fields`/`expand` params also 400; I9). Required-vs-optional mirrors the v2 serializers: `ProductType` requires `name`; `Product` requires `name`+`description`+`prod_type` (`description` is a non-null model field; `sla_configuration` falls back to the model default when omitted); `User` requires `username`+`email` (+password per `REQUIRE_PASSWORD_ON_USER`). Relations by integer id (§4.11). §4.5 does not enumerate detail fields for these resources, so the conservative additions are: `ProductDetail` += `business_criticality, platform, origin, external_audience, internet_accessible` + `product_manager/technical_contact/team_manager` refs; `UserDetail` += `is_staff, date_joined`; `ProductTypeDetail` == slim (no heavier fields). Filter vocabularies follow §4.9's minimal template (`id__in`, `name__icontains`, parent FK where present, date ranges; orderings id/name/created/updated) and are captured in the regenerated `snapshots/filters.json` (diff = exactly the three new resources, 73 insertions / 0 deletions, finding section untouched). Per-list-endpoint `assertNumQueries`-style constant-query tests pass (product_types 2, users 2, products 3 queries, unchanged at 10 vs 100 rows; products constant with `expand=product_type` too). | +| 2026-07-19 | **OS3a impl — no service layer for the three simple resources (I6 note):** unlike finding writes (which own JIRA/risk/notification side-effects and get a `dojo/finding/services.py` in OS3b), product_type/product/user writes are plain ORM CRUD with no cross-cutting orchestration, so the (thin) create/update/delete logic lives directly in the route factories. The one real side-effect — the delete strategy (async vs sync-inside-`Endpoint.allow_endpoint_init()`) — is a per-route `_destroy()` helper mirroring v2 exactly. `204 No Content` responses return an empty body via `HttpResponse(status=204)` (not JSON `null`) while still carrying `X-API-Status` (§4.1); the kernel `errors.py` was **not** extended (only `api.py` mount was touched, per §10). | +| 2026-07-19 | **OS3b scope + canonical-slim relocation:** engagement + test CRUD (`dojo/engagement/api_v3/`, `dojo/test/api_v3/`) and the finding write path shipped. `EngagementSlim` relocated to `dojo/engagement/api_v3/schemas.py`; `TestSlim`/`TestTypeSlim`/`EnvironmentSlim` relocated to `dojo/test/api_v3/schemas.py`; the finding module now **re-exports** all four so there is exactly one canonical class per model (is-identity asserted in `test_apiv3_engagements.TestApiV3EngagementsRelocation` and `test_apiv3_tests.TestApiV3TestsRelocation`, mirroring the OS3a pattern). Import direction stays acyclic: `finding → {engagement, test}`, `test → engagement`, `engagement → {product, product_type, user}`. No `PUT` (PATCH-only, per the OS3a decision). Snapshot regenerated additively: `filters.json` = +engagement +test (66 insertions / 1 formatting deletion; finding/product/product_type/user sections unchanged, verified by the order-independent contract test). All 197 `unittests.api_v3` tests green. | +| 2026-07-19 | **OS3b v2-semantics finding — engagement/test write permissions (mirror the v2 permission classes exactly):** `engagement` (`UserHasEngagementPermission`): create = `add` on the target `product` in the payload (404 if the product doesn't exist, 403 if unauthorized — mirrors `check_post_permission(request, Product, "product", "add")`); update = object `edit` **plus** `add` on a *reassigned* `product` (re-checked only when the FK actually changes, mirroring `check_update_permission`); delete = object `delete` (⇒ **staff-only** for a non-staff member under the legacy model). `test` (`UserHasTestPermission`): create = `add` on the target `engagement` (404/403) **plus** `view` on `api_scan_configuration` when present (mirrors the `required=False` sibling `check_post_permission`); update = object `edit` **plus** `view` on a *reassigned* `api_scan_configuration` (mirrors `check_update_permission(request, obj, "view", "api_scan_configuration")`) — note the update path does **not** re-authorize any `engagement` reassignment because `engagement` is `editable=False` and therefore not writable on update (mirrors v2, where the update serializer treats it read-only); delete = object `delete` (staff-only). Reads resolve through the authorized `view` queryset first → **404** for unknown-or-unauthorized, **403** only when visible-but-not-modifiable. `Permissions.Engagement_Add`/`Test_Add`/`Finding_Add` map to `Action.Add` identically to the literal string `"add"` (verified via `permission_to_action`), so either form is safe. | +| 2026-07-19 | **OS3b v2-semantics finding — engagement/test deletion (mirrored exactly; differs from product/product_type):** `EngagementViewSet.destroy` and `TestsViewSet.destroy` both do `async_delete().delete(instance)` when `ASYNC_OBJECT_DELETE` is truthy, else a **plain** synchronous `instance.delete()` — crucially with **no** `Endpoint.allow_endpoint_init()` wrapper (unlike `ProductViewSet`/`ProductTypeViewSet` destroy, which do wrap it). v3 `_destroy()` in each routes module reproduces this verbatim: no Endpoint context for engagement/test. (Cascade to findings under `V3_FEATURE_LOCATIONS` goes through `LocationFindingReference`, not the legacy `Endpoint`, so the wrapper is unnecessary here — and mirroring v2 exactly is the mandate.) | +| 2026-07-19 | **OS3b impl — engagement/test write-schema shapes (editable subset, conservative FK set):** required-vs-optional mirrors the v2 `ModelSerializer` requiredness rule (`required = not (blank or has_default or null)`): `Engagement` requires `product`+`target_start`+`target_end`; `Test` requires `engagement`+`test_type`+`target_start`+`target_end` (`environment` is `null=True` ⇒ optional). `EngagementSerializer.validate` (POST: `target_start > target_end` → 400) ported to `_validate_dates`, applied on create and on any update that changes either date. Conservative FK subset in the write surface (additive later, per §10.2): engagement exposes `lead` only (dropped `preset`/`report_type`/`requester`/`build_server`/`scm_server`/`orchestration_engine` tool-config FKs — niche, and each needs its own resolve+validate); test exposes `test_type`/`environment`/`lead` (resolved by id → 400 on bad pk, mirroring DRF `PrimaryKeyRelatedField`) and `api_scan_configuration` (permission-gated: 404 if absent, 403 without `view`). `EngagementDetail`/`TestDetail` add conservative heavier read fields (§4.5 doesn't enumerate them): engagement += description/version/first_contacted/reason/tracker/test_strategy/threat_model/api_test/pen_test/check_list/build_id/commit_hash/branch_tag/source_code_management_uri/deduplication_on_engagement; test += description/scan_type/version/build_id/commit_hash/branch_tag (`api_scan_configuration` left out of the read shape — its model has no `Ref` label registry entry; additive later). Per-list `assertNumQueries`-style constant-query tests pass, including with `expand`. | +| 2026-07-19 | **OS3b — finding write service (D7 flagship extraction) reconciliation + divergences:** `dojo/finding/services.py` provides `create_finding`/`update_finding`/`delete_finding` (I6: keyword-only, explicit `user`, no HTTP context; validation raises `rest_framework.exceptions.ValidationError` — the same exception the reference serializer raises — which the kernel's DRF boundary adapter maps to 400 problem+json, §12 OS1). It reconciles the v2 serializer path (`FindingSerializer`/`FindingCreateSerializer` `.update`/`.create`/`.validate`, `dojo/finding/api/serializer.py:328-744`) with the UI flow (`EditFinding` in `dojo/finding/ui/views.py:718-1053`). **The serializer path is the chosen canonical for the API** (the plan names it as the reference). UI-only side-effects are consciously **deferred to the convergence track** (CONV2), not baked into the API. Full divergence table in `.claude/os3b-report.md`. The material divergences and chosen canonical behavior: **(1) risk-acceptance** — serializer's `process_risk_acceptance` (simple_risk_accept/risk_unaccept, gated on `enable_simple_risk_acceptance`, run in validate) is canonical; the UI's `perform_save=False` + explicit `finding.save()` variant is deferred. **(2) JIRA push** — canonical = serializer semantics: update pushes synchronously with `force_sync=True` and raises on failure (`push_to_jira or is_keep_in_sync`); create pushes without force_sync; the route OR-s `push_to_jira` with `jira_project.push_all_issues` (mirrors `perform_update`). UI-only jira link/unlink/change-key and finding-group push are deferred. **(3) status side-effects** — the UI's `process_mitigated_data` (auto-`is_mitigated`+location-edge mitigation when active flips off), `process_false_positive_history` (retroactive FP reactivation), `last_reviewed`/`last_reviewed_by` stamping, burp req/resp, finding-group handling, and github are **all UI-only and deferred**; the API mirrors only the serializer's field-level updates. **(4) notifications** — create dispatches the `finding_added` notification exactly as the serializer does; update dispatches none (matches the serializer). **(5) `numerical_severity`** — not set in the service (the model's `save()` computes it; the UI sets it explicitly, redundantly). | +| 2026-07-19 | **OS3b impl — finding write specifics (conscious choices):** **(a) `cve` mirror** — the v3 write schema exposes a flat `vulnerability_ids: list[str]` (§4.11), not the v2 nested `vulnerability_id_set`; `create_finding` writes `vulnerability_ids[0]` into `Finding.cve` **before** the initial `save()` (mirroring the serializer, which sets `validated_data["cve"]` pre-save) because `save_vulnerability_ids` only sets `finding.cve` in memory; `update_finding` calls `save_vulnerability_ids` before `finding.save()` so the mirror persists. Empty/absent `vulnerability_ids` is a no-op (mirrors the serializer's truthy guard). **(b) CWE** — v3 exposes a scalar `cwe` (not the v2 nested `cwes` list). Create always persists the primary `Finding.cwe` as a `Finding_CWE` row (`save_cwes`, mirrors serializer). On **update**, `save_cwes` is called whenever the scalar `cwe` is in the change set (the closest analog to the v2 `finding_cwe_set`-provided guard) so `Finding.cwe` and its `Finding_CWE` rows stay consistent — this resyncs to just the primary CWE, wiping any extra imported `Finding_CWE` rows (documented divergence: v2's scalar-cwe update alone does not touch `Finding_CWE`). **(c) `delete_finding`** — mirrors `FindingViewSet.destroy`'s dedup/grading hooks via the model's `Finding.delete()` (`finding_helper.finding_delete` + `perform_product_grading`); the plan-mandated signature `delete_finding(finding, *, user)` omits v2 destroy's `push_to_jira` query-param (niche delete-time JIRA closure push) — additive later. **(d) mitigated-edit rules** — `can_edit_mitigated_data` (requires `EDITABLE_MITIGATED_DATA` **and** superuser) gates any attempt to set non-null `mitigated`/`mitigated_by`, on both create and update (ported verbatim from both serializers). | +| 2026-07-19 | **OS3b — RBAC 403 test shape under OS legacy auth (view-but-not-edit not expressible):** the task asks for a "403 view-but-not-edit" finding test, but the OS legacy authorization model (`user_has_permission`) grants View/Edit/Add **equally** to any `authorized_users` member (only Delete is staff-gated). So a member who can *see* a finding/engagement/test can also *edit* it — a view-but-not-**edit** 403 is not expressible without a granular RBAC role model. Conservative resolution (mirrors the OS3a product RBAC test): the "authorized-to-see-but-not-modify" 403 is demonstrated on **delete** (member GET 200, member DELETE 403 staff-only), and the 404-for-unauthorized case is tested with a non-member (`limited`) user. Recorded so the API contract test is not mistaken for asserting an edit/view permission split that OS legacy does not implement. | +| 2026-07-19 | **OS4 scope + sub-resource placement:** read-only Locations resource (`GET /locations`, `GET /locations/{id}`) + the `GET /findings/{id}/locations` and `GET /products/{id}/locations` edge sub-resources shipped in a new `dojo/location/api_v3/{schemas,routes}.py`. **All three routers are factories (I5)** mounted in `build_api()`. **Placement choice:** the finding/product *location* sub-resources live in the **location** module (not the finding/product route factories) so all location code stays cohesive and the finding/product factories are untouched (lower regression risk); each is an independent `build__router()` and the `/findings/{id}/locations` path resolves cleanly alongside `/findings/{id}` (distinct URL patterns). Read-only: locations are import-driven (§4.14); no write routes. All 229 `unittests.api_v3` tests green (198 prior + 31 new). | +| 2026-07-19 | **OS4 v2-semantics finding — Location RBAC (verified superuser-only; mirrored):** the v2 `LocationViewSet` (`dojo/location/api/views.py`) is a `ReadOnlyModelViewSet` with `permission_classes = (IsSuperUser, DjangoModelPermissions)` — AND-ed, and `IsSuperUser.has_permission` = `request.user.is_superuser`, so the resource is **superuser-only** (confirmed). v3 `/locations` mirrors this exactly: the router gates the whole resource behind `request.user.is_superuser` → **403** problem+json for any non-superuser (both list and detail), and draws rows from `get_authorized_locations("view", user=...)` (I8, forward-compatible — a downstream distribution can scope the queryset without a route change; in OS that helper returns all locations, matching v2's `Location.objects.order_by_id()`). The **sub-resources** are not superuser-gated: they use **parent-inherited authorization** — the parent finding/product is resolved through `get_authorized_findings`/`get_authorized_products` and an unknown *or unauthorized* parent is a **404** (never leak existence, §4.10); edges are then read from the parent's own reverse manager (`finding.locations` / `product.locations`), matching the v2 `LocationFindingReference`/`LocationProductReference` viewsets' RBAC-scoped querysets. | +| 2026-07-19 | **OS4 — product edge shape reconciliation (§4.14 shorthand vs model):** §4.14 lists both finding- and product-location edge rows as `{location, status, audit_time}`, but `LocationProductReference` has **no** `audit_time`/`auditor` columns (only `status`, `relationship`, `relationship_data`). Resolved per the OS4 task's explicit item-2 shapes and the model: **finding** edge = `{location: {id,name,type}, status, audit_time, auditor: {id,name}|null}`; **product** edge = `{location: {id,name,type}, status}`. `LocationRef` (the one ref subtype carrying `type`) is emitted for the `location` field. | +| 2026-07-19 | **OS4 — `auditor` added to `expand=locations` edge rows (supersedes the OS2 deferral):** OS2 (§12) deferred the `auditor{ref}` on finding-location rows to OS4; the OS4 task adds it to **both** the `/findings/{id}/locations` sub-resource **and** the existing `expand=locations` projection. `_finding_location_edges` now emits `{location, status, audit_time, auditor}` with `prefetch_paths=("locations__location", "locations__auditor")` so the query count stays constant (verified). The pre-existing `test_apiv3_findings.test_expand_locations_swaps_count_for_edge_rows` edge-key assertion was updated to include `auditor` (a v3 test reflecting the new intended contract, not a v2 test). | +| 2026-07-19 | **OS4 — `?fields=` / `?expand=` interplay (OS2 open question, coordinator decision):** the `?fields=` allowlist is now **schema fields ∪ the schema's registered `EXPANDABLE` keys**, implemented as one kernel helper `dojo.api_v3.expand.allowed_field_names(schema)` (the single permitted kernel edit besides the mount) and applied at **all** resource list/detail `parse_fields` call sites (findings/products/product_types/users/engagements/tests/locations) so the contract is uniform. Effect: `?expand=locations&fields=id,title,locations` works; a genuinely unknown name still 400s (dedicated `fields` type URI, I9); naming an expand key in `fields=` **without** `expand=` is accepted (no 400) but renders nothing (`apply_fields` keeps only keys present in the serialized dict). Purely widens the allowlist, so no existing 200 becomes a 400. | +| 2026-07-19 | **OS4 — flag-off behaviour tested in-process (D5):** `V3_FEATURE_LOCATIONS=False` ⇒ the whole `/api/v3-alpha/` tree is absent (the mount in `dojo/urls.py` is import-time conditional). Made cleanly testable via a **URLconf reload**: `test_apiv3_locations.TestApiV3LocationsFlagOff` reloads `dojo.urls` under `override_settings(V3_FEATURE_LOCATIONS=False)` (+`clear_url_caches()`), asserts every v3 path (`locations`/`findings`/`products`/`import`) raises `Resolver404` against the reloaded urlconf, then restores the real (flag-on) urlconf in a `finally` (reload + `clear_url_caches()`) so the rest of the suite is unaffected. Verified: the test passes and the other 228 tests stay green in the same run, so the reload does not pollute the shared test process. | +| 2026-07-19 | **OS4 — URL-subtype detail fields + query-count fidelity:** `LocationDetail` adds `protocol/host/port/path/query/fragment` read from the `URL` subtype via the reverse one-to-one (`location.url`, `related_name="%(class)s"`), loaded with `select_related("url")` on the detail fetch so the six resolvers issue **no** extra query; a non-URL location (none exist in alpha, D5) renders those fields `null` (the reverse-O2O accessor's `RelatedObjectDoesNotExist` is caught). Per-list `assertNumQueries`-style constant-query tests pass for `/locations`, `/findings/{id}/locations` and `/products/{id}/locations` (sub-resources use `select_related("location"[, "auditor"])` — constant regardless of edge count). Filter vocabulary (`type`, `name__icontains`, `product` via `products__product`; orderings `id`/`name`) captured in `snapshots/filters.json` as an **additive-only** diff (+`location` block, all other sections byte-identical; verified by the passing snapshot contract test). Whole-surface query sweep extended with the four new GET endpoints (with location-edge fan-out so per-row queries can't hide) and stays green. | +| 2026-07-19 | **OS5 scope + storage support matrix (coordinator scope correction: attach only where the MODEL stores it, verified against `dojo/*/models.py`; §4.12's "all seven resources" is wrong):** three generic factories in `dojo/api_v3/subresources.py` (`build_notes_router`/`build_files_router`/`build_tags_router`). Model storage: **notes** (`Notes` M2M) — engagement, test, finding; **files** (`FileUpload` M2M) — engagement, test, finding; **tags** (`TagField`) — product, engagement, test, finding, location. product_type/user have **none**. Attachment: notes+files on **finding/engagement/test**; tags on **finding/engagement/test/product**. **Location tags NOT attached** despite the `TagField`: location is a read-only, superuser-only resource (OS4) with **no v2 tag-mutation endpoint** to mirror, and its tags are already surfaced via `LocationSlim.tags[]` (§4.5) — a tag sub-resource would be a redundant read + an invented write (conservative, §10.2). Unsupported combos return **404** (path not registered); tested. Total surface: 13 new GET paths + POST/PUT/DELETE. All 252 `unittests.api_v3` tests green (229 prior + 23 new). | +| 2026-07-19 | **OS5 — note privacy = v2 parity (return all; `private` is not a per-user read filter):** verified v2's notes `@action` GET returns `parent.notes.all()` (no privacy filter) and the UI `comments.html` shows every note to any viewer; `private` only excludes a note from **generated reports** (`dojo/reports/ui/views.py` `finding.notes.filter(private=False)`). So v3 mirrors v2 exactly: the notes list returns **all** notes (private included) to anyone who can view the parent, with `private` exposed on `NoteSchema` so clients can label/exclude. Inventing per-user filtering would be a behavior divergence, not a mirror. Tested (`test_private_note_visible_to_other_authorized_user_v2_parity`). | +| 2026-07-19 | **OS5 — NoteSchema/FileSchema field mappings (model has no matching columns; conservative):** `NoteSchema.created`←`Notes.date`, `updated`←`Notes.edit_time` (the model has no `created`/`updated`; `edit_time` defaults to now at creation, and `edited` distinguishes a real edit). `note_type` is **out of the alpha write/read surface** (§4.12 POST body is `{entry, private?}`; the v2 single-note-type check is moot without it — additive later). `FileSchema.created` is always **`null`**: `FileUpload` has no creation timestamp column; `size`←`file.size` (a storage stat, not a DB query — keeps list query counts flat). | +| 2026-07-19 | **OS5 — tags sub-resource semantics (new convenience API; v2 has only `remove_tags`):** body is a JSON `{"tags": [...]}` list (§4.11), `extra="forbid"` (unknown field → 400). GET → `{"tags":[...]}`; **PUT** replaces, **POST** appends (order-preserving, case-insensitive dedup), both return `200 {"tags":[...]}`; **DELETE /tags/{tag}** removes one and returns `204`, **404 if absent** (the OS5 task's explicit contract — differs from v2 `remove_tags`'s 400-on-absent). Write path mirrors v2 `remove_tags` exactly: `parent.tags = ; parent.save()` — so tagulous `force_lowercase` normalization and the tag-inheritance signals fire identically (assigning a *list*, not a rendered string, is safe for tags with spaces/commas). The `{tag}` path param is lowercased before matching (stored tags are force-lowercase). No `FilterSpec` (parent-scoped, simple) ⇒ **no snapshot regeneration** (§6 OS5). | +| 2026-07-19 | **OS5 — file upload/download (mirror v2 validation):** extension validated via `FileUpload.clean()` against `settings.FILE_UPLOAD_TYPES` (mirrors v2 `FileSerializer.validate`); v2 enforces **no size cap**, so v3 doesn't either. `FileUpload.title` is globally `unique=True`, so a pre-save `exists()` check returns **400** on duplicate title (mirrors DRF `UniqueValidator`, avoids an IntegrityError→500). Download streams via `dojo.utils.generate_file_response` (correct content-type + `Content-Disposition: attachment`, mirrors v2 `download_file`); the `FileResponse` is returned through ninja unchanged with `X-API-Status` set manually. Multipart parsed with ninja `Form(...)`/`File(...)`. Roundtrip tested (bytes + disposition). | +| 2026-07-19 | **OS5 — parent-inherited authorization + permission values (mirror the v2 related-object permission classes exactly):** the parent is resolved through its `get_authorized_*` view queryset → **404** unknown-or-unauthorized (never leak existence, §4.10); the applicable permission is then checked on the parent via `user_has_permission` → **403**. Permission *values* mirror v2: notes GET/POST = `view` (`UserHas*NotePermission` post_permission is `view`); tags GET = `view`, PUT/POST/DELETE = `edit` (`UserHas*RelatedObjectPermission`); files GET/download = `Product_Tracking_Files_View`, POST = `Product_Tracking_Files_Add` (`UserHas*FilePermission`). **403-write not naturally expressible for members under OS legacy** (view==edit==add for `authorized_users` members; only Delete is staff-gated, and no sub-resource write maps to Delete — same finding as OS3b §12): the 403 code path is therefore tested by making the permission check fail (`mock` `user_has_permission`→False while the parent stays visible); the 404 path is tested with a real non-member. Parent-view resolvers: finding/product take `user=request.user`, engagement/test use crum's current user (matching their `get_authorized_*` signatures and the OS3 routes). | +| 2026-07-19 | **OS5 — kernel purity + wiring + unique operation ids:** `subresources.py` imports **no parent-resource model** (finding/product/…); the parent model/queryset/label/permissions all arrive as factory arguments. It imports only its **own** storage models (`Notes`/`NoteHistory`/`FileUpload`), the `generate_file_response` helper, and `user_has_permission` (authorization infra, not a resource/service module — same class as the kernel's existing rest_framework/django imports; I8). Wiring lives in `dojo/api_v3/api.py::_mount_subresources` (the composition root, alongside the existing deferred router-factory imports — consistent with the §12 OS2 "only `build_api()`/`import_routes.py` import resources under `dojo/api_v3/*`" rule). Inner view functions are registered **manually** (`router.get(path,...)(fn)`) after setting a unique `fn.__name__` per resource, so the once-per-resource factory calls don't collide into duplicate OpenAPI `operationId`s. Query sweep extended with all 13 new GET endpoints (notes/files/tags fan-out + a real file per parent for `download`); `capture_request` made streaming-aware (the download `FileResponse` has no `.content`). | +| 2026-07-19 | **OS6 — expand/include/sub-resource RBAC sweep (ports `test_apiv2_prefetch_rbac.py` intent):** `unittests/api_v3/test_apiv3_expand_rbac.py` (15 tests, all green). Builds a two-product world, authorizes a non-superuser on product A only (legacy `authorized_users` M2M), and proves **no** v3 projection discloses product B: the list/`?expand=test.engagement,product,product_type` rows are all product A; a product-B finding detail (even with `?expand=`) is **404**; `?product=` intersects the authorized queryset → empty; `?expand=reporter` never surfaces product B's distinctive reporter (the v2 4a user-enumeration analogue — note expand swaps the ref for `UserSlim`, keyed by `username`); `?include=counts` totals reflect only product A (B's severities stay 0, admin's total strictly exceeds the member's); and the `/findings/{id}/{notes,locations}` sub-resources + `expand=locations` return 404 for an unauthorized parent (parent-inherited authorization). Structural reason it holds: every projection is computed over `get_authorized_findings(...)`, so nothing reachable from an authorized finding can belong to another product (I8). | +| 2026-07-19 | **OS6 — latency + query-count benchmark (honest, in-process; deferred from OS1 gate).** Harness `unittests/api_v3/test_apiv3_benchmark.py`, CI-excluded via `@skipUnless(DD_API_V3_BENCH=1)`; run `docker compose exec -e DD_API_V3_BENCH=1 uwsgi python manage.py test unittests.api_v3.test_apiv3_benchmark`. Seeded 1000 findings on one test, `limit=100`, N=30 timed in-process GETs. **Results (real capture, 1021 findings):** v2 `?prefetch=test` = **636 queries, median 521 ms / p95 721 ms**; v2 no-prefetch = 229 q, 193/424 ms; **v3 slim = 5 q, 38/45 ms**; **v3 `?expand=test.engagement` = 7 q, 60/231 ms**. **CAVEATS (binding on the latency numbers):** in-process only — no HTTP/uwsgi/network/gzip layer, single-threaded, no pool warmup, shared test transaction; treat **latency as directional** and the **query counts (5/7 vs 636, constant vs row count) as the load-bearing, environment-independent evidence**. The wall-clock-against-uwsgi procedure in `.claude/os1-gate-report.md` remains the way to quote production latency. Full report: `.claude/os6-benchmark.md`. | +| 2026-07-19 | **OS6 — captured examples + docs page.** `api_v3_examples.md` (repo root) is generated by `unittests/api_v3/test_apiv3_examples.py` (CI-excluded, `DD_API_V3_EXAMPLES=1`) making **real** in-process requests; bodies verbatim, tokens redacted, lists truncated to ~3 rows. Covers findings (detail slim; `expand=test.engagement,locations`; filtered list + page-2 envelope showing non-null `next`/`previous`; `include=counts` meta; notes POST+GET; locations edge rows; PATCH; `POST /import`) and products (detail/list/POST/PATCH). Docs-site page: `docs/content/automation/api/api-v3-alpha-docs.md` (Doks/thulite theme, `weight: 3`, next to the v2 `api-v2-docs.md` — **not** the `asset_modelling` pages). Written in **plain markdown only** (blockquote instead of an `{{% alert %}}` shortcode) because the theme is a not-locally-installed Hugo module and an unknown shortcode would break the build; covers overview, alpha disclaimer (URL→`/api/v3/` at beta), token+session auth, envelope/refs/expand/fields/include/filter summary, the v2→v3 mapping table for the seven objects + import, one-way notes/tags/files, **Known alpha gaps**, and the `/api/v3-alpha/docs` link. | +| 2026-07-19 | **OS6 — Docs UI (Scalar) evaluation — coordinator decision: DEFER; alpha keeps ninja's built-in Swagger at `/api/v3-alpha/docs`.** Rationale: swapping to Scalar means vendoring an unreviewed third-party JS bundle into a **security-product** repo, which needs its own supply-chain review — the same class of decision as adding django-ninja (D2), not "polish". What the swap would take when approved: one template view (Hugo shortcode or a small Django `TemplateView`) rendering the Scalar ` + + +""" + + +def scalar_reference(request: HttpRequest) -> HttpResponse: + """Render the Scalar reference shell (no data of its own — the schema URL does the work).""" + return HttpResponse(_PAGE.format( + openapi_url=reverse("api_v3:openapi-json"), + docs_url=reverse("api_v3:openapi-view"), + script_url=static(SCALAR_STATIC_PATH), + )) diff --git a/dojo/api_v3/refs.py b/dojo/api_v3/refs.py new file mode 100644 index 00000000000..d2628b119f9 --- /dev/null +++ b/dojo/api_v3/refs.py @@ -0,0 +1,77 @@ +""" +Relation references for API v3 (D3 / §4.4). + +Every relation renders by default as a slim ``{id, name}`` ref produced by a single shared +schema. The relation key conveys the type, so refs carry no ``type`` field -- the sole +exception being location refs, whose one key can hold heterogeneous location subtypes. + +This module is the single source of truth for the ref *label registry* (which model attribute +supplies ``name`` for each model). Invariant I3: the ref shape is closed -- do not extend it. +""" +from __future__ import annotations + +from typing import TYPE_CHECKING + +from ninja import Schema + +if TYPE_CHECKING: + from collections.abc import Callable + + from django.db.models import Model + + +class Ref(Schema): + + """Closed ref shape (I3): identity + human label.""" + + id: int + name: str + + +class LocationRef(Ref): + + """Location refs additionally carry ``type`` because one key holds heterogeneous subtypes.""" + + type: str + + +def _test_label(obj: Model) -> str: + # Test: .title if set, else str(.test_type) (§4.4). + return obj.title or str(obj.test_type) + + +# Label registry (§4.4): model class -> callable(obj) -> name. Registered lazily by import path +# name to avoid importing every model at kernel-import time (keeps the kernel resource-agnostic). +_LABELERS: dict[str, Callable[[Model], str]] = { + "Product_Type": lambda o: o.name, + "Product": lambda o: o.name, + "Engagement": lambda o: o.name, + "Test": _test_label, + "Finding": lambda o: o.title, + "Dojo_User": lambda o: o.username, + "Location": lambda o: o.location_value, + "Test_Type": lambda o: o.name, + "Development_Environment": lambda o: o.name, +} + + +def ref_label(obj: Model) -> str: + """Return the human label for ``obj`` per the registry, falling back to ``str(obj)``.""" + labeler = _LABELERS.get(type(obj).__name__) + if labeler is None: + return str(obj) + return labeler(obj) + + +def to_ref(obj: Model | None) -> dict | None: + """Render ``obj`` as a closed ``{id, name}`` ref, or ``None`` when ``obj`` is ``None``.""" + if obj is None: + return None + return {"id": obj.pk, "name": ref_label(obj)} + + +def to_location_ref(location: Model | None) -> dict | None: + """Render a Location as ``{id, name, type}`` (the one ref subtype carrying ``type``).""" + if location is None: + return None + return {"id": location.pk, "name": location.location_value, "type": location.location_type} diff --git a/dojo/api_v3/subresources.py b/dojo/api_v3/subresources.py new file mode 100644 index 00000000000..1978b852781 --- /dev/null +++ b/dojo/api_v3/subresources.py @@ -0,0 +1,362 @@ +""" +Generic notes / tags / files sub-resource router factories for API v3 (§4.12, OS5). + +Three router factories (I5), each **parameterized by the parent resource** so this kernel module +imports **no parent-resource model** (finding/product/engagement/test/...): the parent's URL path +segment, a human label, its authorized-view queryset resolver, and the RBAC permission values all +arrive as factory arguments. The only models imported here are the sub-resources' *own* storage +models (``Notes``/``NoteHistory``/``FileUpload``) plus the shared authorization/file helpers -- the +factory legitimately owns those. + +**Authorization is inherited from the parent, mirroring the v2 related-object permission classes** +(``UserHas*NotePermission`` / ``*FilePermission`` / ``*RelatedObjectPermission``): + +1. the parent is resolved through its ``get_authorized_*`` view queryset -- an unknown *or + unauthorized* parent is a **404** (never leak existence, §4.10); +2. the applicable per-method permission is then checked on the parent via ``user_has_permission`` + -- failure is a **403**. The permission *values* mirror v2 exactly: + + ========= ============================ ===================================== + sub-res read (GET) write + ========= ============================ ===================================== + notes ``view`` POST create -> ``view`` (v2 note post_permission) + tags ``view`` PUT/POST/DELETE -> ``edit`` + files ``Product_Tracking_Files_View`` POST -> ``Product_Tracking_Files_Add`` + ========= ============================ ===================================== + +Routes are thin (I6). Sub-resource lists are parent-scoped and simple -- no expand/fields/filter +machinery, only the shared pagination envelope (§12); therefore no FilterSpecs and no snapshot +regeneration. + +**Note side-effects stay out of the kernel (I5/I6).** v2's per-resource notes ``@action`` fires +resource-specific side-effects on create (finding: JIRA comment sync + ``last_reviewed`` stamping + +@mention notifications; engagement/test: @mention notifications only). To reach v2 parity without +importing any JIRA/notification machinery here, ``build_notes_router`` takes an optional +``on_note_created(parent, note, *, user)`` callback, invoked *after* the note is persisted and linked +to the parent. The callback is a resource **service** function (``dojo//services.py``) that +owns all such side-effects; the kernel merely calls it. + +Inner view functions are registered manually (not via ``@router.get``) so each can be given a +unique ``__name__`` per resource -- the factory is called once per parent resource, and identical +closure names would otherwise collide into duplicate OpenAPI ``operationId``s. +""" +from __future__ import annotations + +from datetime import datetime # noqa: TC003 -- runtime import: ninja/pydantic resolves the schema field types +from typing import TYPE_CHECKING + +from django.conf import settings +from django.core.exceptions import PermissionDenied +from django.core.exceptions import ValidationError as DjangoValidationError +from django.http import HttpResponse +from ninja import File, Form, Router, Schema +from ninja.constants import NOT_SET +from ninja.files import UploadedFile # noqa: TC002 -- runtime import: ninja resolves the File() param type + +from dojo.api_v3.errors import json_response, not_found_problem, validation_problem +from dojo.api_v3.pagination import paginate +from dojo.api_v3.refs import Ref, to_ref +from dojo.authorization.authorization import user_has_permission +from dojo.file_uploads.models import FileUpload +from dojo.notes.models import NoteHistory, Notes +from dojo.utils import generate_file_response + +if TYPE_CHECKING: + from collections.abc import Callable + + from django.db.models import Model, QuerySet + from django.http import HttpRequest + + +# --- Schemas (documentation only; runtime serialization is manual) ---------------------------- + +class NoteSchema(Schema): + + """A note (§4.12). ``created`` maps to ``Notes.date``; ``updated`` to ``Notes.edit_time`` (§12).""" + + id: int + entry: str + author: Ref | None + private: bool + edited: bool + created: datetime | None + updated: datetime | None + + +class NoteCreate(Schema): + + """POST body for a note (§4.12). ``note_type`` is out of the alpha write surface (§12).""" + + model_config = {"extra": "forbid"} + + entry: str + private: bool = False + + +class NoteListResponse(Schema): + count: int + next: str | None + previous: str | None + results: list[NoteSchema] + meta: dict | None = None + + +class FileSchema(Schema): + + """An uploaded file (§4.12). ``FileUpload`` has no creation column, so ``created`` is null (§12).""" + + id: int + title: str + size: int + created: datetime | None + + +class FileListResponse(Schema): + count: int + next: str | None + previous: str | None + results: list[FileSchema] + meta: dict | None = None + + +class TagsResponse(Schema): + tags: list[str] + + +class TagsBody(Schema): + model_config = {"extra": "forbid"} + + tags: list[str] + + +# --- Shared helpers --------------------------------------------------------------------------- + +def _resolve_parent( + request: HttpRequest, + get_parent_queryset: Callable[[HttpRequest], QuerySet], + parent_label: str, + parent_id: int, +) -> Model: + """Resolve the parent through its authorized-view queryset; 404 unknown-or-unauthorized (§4.10).""" + parent = get_parent_queryset(request).filter(pk=parent_id).first() + if parent is None: + msg = f"{parent_label} {parent_id} not found" + raise not_found_problem(msg) + return parent + + +def _require(request: HttpRequest, parent: Model, permission) -> None: + """403 when the caller lacks ``permission`` on the (already-viewable) parent (I8).""" + if not user_has_permission(request.user, parent, permission): + raise PermissionDenied + + +def _serialize_note(note: Notes) -> dict: + return { + "id": note.pk, + "entry": note.entry, + "author": to_ref(note.author), + "private": note.private, + "edited": note.edited, + "created": note.date, + "updated": note.edit_time, + } + + +def _serialize_file(file_obj: FileUpload) -> dict: + # ``file.size`` is a storage stat (filesystem), not a DB query -- keeps list query counts flat. + return {"id": file_obj.pk, "title": file_obj.title, "size": file_obj.file.size, "created": None} + + +def _read_tags(parent: Model) -> list[str]: + return [tag.name for tag in parent.tags.all()] + + +def _write_tags(parent: Model, names: list[str]) -> None: + """ + Replace the parent's tags with ``names`` via the v2 write path (assignment + ``save()``): + tagulous normalises (``force_lowercase``) on save and the tag-inheritance signals fire exactly + as they do for v2's ``remove_tags`` (``dojo/finding/api/views.py``). Assigning a *list* (not a + rendered string) is safe for tags containing spaces/commas. + """ + parent.tags = names + parent.save() + + +# --- Factories -------------------------------------------------------------------------------- + +def build_notes_router( + *, + resource: str, + parent_label: str, + get_parent_queryset: Callable[[HttpRequest], QuerySet], + view_permission="view", + create_permission="view", + on_note_created: Callable[..., None] | None = None, + auth=NOT_SET, +) -> Router: + """ + ``GET`` (paginated) + ``POST`` ``/{resource}/{id}/notes`` (§4.12). + + ``on_note_created(parent, note, *, user)`` -- optional resource-service callback fired after the + new note is persisted and linked; it owns the resource's v2 note side-effects (JIRA comment sync, + ``last_reviewed`` stamping, @mention notifications). The kernel imports none of that machinery. + """ + router = Router(tags=[resource], auth=auth) + notes_path = f"/{resource}/{{int:parent_id}}/notes" + + def list_notes(request: HttpRequest, parent_id: int): + parent = _resolve_parent(request, get_parent_queryset, parent_label, parent_id) + _require(request, parent, view_permission) + # Mirror v2 exactly: return every note incl. private ones. In v2 the notes @action returns + # `parent.notes.all()`; `private` only excludes a note from generated reports, it is not a + # per-user read filter (§12). `select_related("author")` keeps the list query count flat. + notes = parent.notes + page_qs = notes.select_related("author").order_by("-date", "-id") + envelope = paginate(request, count_qs=notes.all(), page_qs=page_qs, serialize=_serialize_note) + return json_response(envelope) + + def create_note(request: HttpRequest, parent_id: int, payload: NoteCreate): + parent = _resolve_parent(request, get_parent_queryset, parent_label, parent_id) + _require(request, parent, create_permission) + note = Notes(entry=payload.entry, author=request.user, private=payload.private) + note.save() + history = NoteHistory.objects.create(data=note.entry, time=note.date, current_editor=note.author) + note.history.add(history) + parent.notes.add(note) + # Resource-specific side-effects (JIRA comment sync, last_reviewed stamping, @mention + # notifications) live in the resource's service layer (I5/I6); the kernel imports none of + # that machinery -- it only invokes the callback the factory was configured with, matching + # v2's per-resource notes @action (dojo//api/views.py). + if on_note_created is not None: + on_note_created(parent, note, user=request.user) + return json_response(_serialize_note(note), status=201) + + list_notes.__name__ = f"list_{resource}_notes" + create_note.__name__ = f"create_{resource}_note" + router.get(notes_path, response=NoteListResponse, url_name=f"{resource}_notes_list")(list_notes) + router.post(notes_path, response=NoteSchema, url_name=f"{resource}_notes_create")(create_note) + return router + + +def build_files_router( + *, + resource: str, + parent_label: str, + get_parent_queryset: Callable[[HttpRequest], QuerySet], + view_permission, + add_permission, + auth=NOT_SET, +) -> Router: + """``GET`` list + ``POST`` (multipart) + ``GET .../{file_id}/download`` (§4.12).""" + router = Router(tags=[resource], auth=auth) + files_path = f"/{resource}/{{int:parent_id}}/files" + download_path = f"/{resource}/{{int:parent_id}}/files/{{int:file_id}}/download" + + def list_files(request: HttpRequest, parent_id: int): + parent = _resolve_parent(request, get_parent_queryset, parent_label, parent_id) + _require(request, parent, view_permission) + files = parent.files + page_qs = files.all().order_by("id") + envelope = paginate(request, count_qs=files.all(), page_qs=page_qs, serialize=_serialize_file) + return json_response(envelope) + + def create_file( + request: HttpRequest, + parent_id: int, + title: str = Form(...), + file: UploadedFile = File(...), # noqa: B008 -- ninja's declarative param default + ): + parent = _resolve_parent(request, get_parent_queryset, parent_label, parent_id) + _require(request, parent, add_permission) + # Mirror v2 FileSerializer: extension validation via FileUpload.clean() (settings + # .FILE_UPLOAD_TYPES); v2 does NOT enforce a size cap. Title is globally unique on the + # model, so pre-check it here (mirrors DRF's UniqueValidator -> 400, avoids a 500). + upload = FileUpload(title=title, file=file) + try: + upload.clean() + except DjangoValidationError as exc: + raise validation_problem({"file": list(exc.messages)}) from exc + if FileUpload.objects.filter(title=upload.title).exists(): + raise validation_problem({"title": ["A file with this title already exists."]}) + upload.save() + parent.files.add(upload) + return json_response(_serialize_file(upload), status=201) + + def download_file(request: HttpRequest, parent_id: int, file_id: int): + parent = _resolve_parent(request, get_parent_queryset, parent_label, parent_id) + _require(request, parent, view_permission) + file_object = parent.files.filter(id=file_id).first() + if file_object is None: + msg = f"File {file_id} not associated with {parent_label} {parent_id}" + raise not_found_problem(msg) + # Streamed FileResponse with the correct content-type + Content-Disposition (mirrors v2's + # download_file via generate_file_response). Ninja passes HttpResponse subclasses through. + response = generate_file_response(file_object) + response["X-API-Status"] = settings.API_V3_STATUS + return response + + list_files.__name__ = f"list_{resource}_files" + create_file.__name__ = f"create_{resource}_file" + download_file.__name__ = f"download_{resource}_file" + router.get(files_path, response=FileListResponse, url_name=f"{resource}_files_list")(list_files) + router.post(files_path, response=FileSchema, url_name=f"{resource}_files_create")(create_file) + router.get(download_path, url_name=f"{resource}_files_download")(download_file) + return router + + +def build_tags_router( + *, + resource: str, + parent_label: str, + get_parent_queryset: Callable[[HttpRequest], QuerySet], + view_permission="view", + write_permission="edit", + auth=NOT_SET, +) -> Router: + """``GET`` + ``PUT`` (replace) + ``POST`` (append) ``/tags`` and ``DELETE /tags/{tag}`` (§4.12).""" + router = Router(tags=[resource], auth=auth) + tags_path = f"/{resource}/{{int:parent_id}}/tags" + tag_item_path = f"/{resource}/{{int:parent_id}}/tags/{{tag}}" + + def get_tags(request: HttpRequest, parent_id: int): + parent = _resolve_parent(request, get_parent_queryset, parent_label, parent_id) + _require(request, parent, view_permission) + return json_response({"tags": _read_tags(parent)}) + + def replace_tags(request: HttpRequest, parent_id: int, payload: TagsBody): + parent = _resolve_parent(request, get_parent_queryset, parent_label, parent_id) + _require(request, parent, write_permission) + _write_tags(parent, list(payload.tags)) + return json_response({"tags": _read_tags(parent)}) + + def append_tags(request: HttpRequest, parent_id: int, payload: TagsBody): + parent = _resolve_parent(request, get_parent_queryset, parent_label, parent_id) + _require(request, parent, write_permission) + # Case-insensitive dedup (stored tags are lowercased); order-preserving. + merged = list(dict.fromkeys([*_read_tags(parent), *(t.lower() for t in payload.tags)])) + _write_tags(parent, merged) + return json_response({"tags": _read_tags(parent)}) + + def delete_tag(request: HttpRequest, parent_id: int, tag: str): + parent = _resolve_parent(request, get_parent_queryset, parent_label, parent_id) + _require(request, parent, write_permission) + current = _read_tags(parent) + target = tag.lower() # stored tags are force_lowercase; match case-insensitively (§12) + if target not in current: + msg = f"Tag '{tag}' not found on {parent_label} {parent_id}" + raise not_found_problem(msg) + _write_tags(parent, [name for name in current if name != target]) + response = HttpResponse(status=204) + response["X-API-Status"] = settings.API_V3_STATUS + return response + + get_tags.__name__ = f"get_{resource}_tags" + replace_tags.__name__ = f"replace_{resource}_tags" + append_tags.__name__ = f"append_{resource}_tags" + delete_tag.__name__ = f"delete_{resource}_tag" + router.get(tags_path, response=TagsResponse, url_name=f"{resource}_tags_list")(get_tags) + router.put(tags_path, response=TagsResponse, url_name=f"{resource}_tags_replace")(replace_tags) + router.post(tags_path, response=TagsResponse, url_name=f"{resource}_tags_append")(append_tags) + router.delete(tag_item_path, url_name=f"{resource}_tags_delete")(delete_tag) + return router diff --git a/dojo/engagement/api_v3/__init__.py b/dojo/engagement/api_v3/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/dojo/engagement/api_v3/routes.py b/dojo/engagement/api_v3/routes.py new file mode 100644 index 00000000000..fd4e95040e2 --- /dev/null +++ b/dojo/engagement/api_v3/routes.py @@ -0,0 +1,336 @@ +""" +Engagement CRUD routes for API v3 (§4.5, §4.9, §4.11, OS3b). + +``build_engagements_router()`` is a router factory (I5), same shape as ``build_assets_router``. +Routes are thin (I6): authorize -> filter -> plan queryset -> serialize -> shape. RBAC flows only +through ``get_authorized_engagements`` for reads (I8) and the v2 ``user_has_permission`` semantics +for writes, mirroring the v2 ``UserHasEngagementPermission`` permission class exactly. Per D11 the +target ``Product`` is exposed on the wire as ``asset`` (write FK -> model ``product``); the model / +ORM paths / permission enums are not renamed (§12): + +- create (POST): ``add`` permission on the target asset referenced in the payload + (404 if the asset doesn't exist, 403 if unauthorized -- mirrors + ``check_post_permission(request, Product, "product", "add")``) +- update (PATCH): object ``edit`` permission, plus ``add`` on a *reassigned* asset + (mirrors ``check_update_permission(request, obj, "add", "product")``) +- delete (DELETE): object ``delete`` permission (staff-only for non-staff members, legacy model) +- read: object ``view`` via the authorized queryset (404 for unknown-or-unauthorized) + +Deletion mirrors the v2 ``EngagementViewSet.destroy`` exactly: async delete when +``ASYNC_OBJECT_DELETE`` is set, else a plain synchronous ``instance.delete()`` -- note there is **no** +``Endpoint.allow_endpoint_init()`` wrapper here (unlike asset/organization destroy), mirroring v2 +(§12). Relations are referenced by integer id (§4.11). +""" +from __future__ import annotations + +from typing import TYPE_CHECKING + +from django.conf import settings +from django.core.exceptions import PermissionDenied +from django.core.exceptions import ValidationError as DjangoValidationError +from django.http import HttpResponse +from ninja import Router, Schema +from ninja.constants import NOT_SET + +from dojo.api_v3.csv_export import stream_csv_export +from dojo.api_v3.errors import json_response, not_found_problem, validation_problem +from dojo.api_v3.expand import ( + allowed_field_names, + apply_fields, + parse_fields, + plan, + plan_list_fields, + plan_queryset, + serialize, + serialize_list_row, +) +from dojo.api_v3.filtering import ( + FilterSpec, + apply_filters, + filter_field, + register_filter_spec, +) +from dojo.api_v3.include import apply_includes +from dojo.api_v3.pagination import paginate +from dojo.authorization.authorization import user_has_permission +from dojo.authorization.roles_permissions import Permissions +from dojo.engagement.api_v3.schemas import ( + EngagementDetail, + EngagementReplace, + EngagementSlim, + EngagementUpdate, + EngagementWrite, +) +from dojo.engagement.queries import get_authorized_engagements +from dojo.models import Dojo_User, Engagement, Product +from dojo.utils import async_delete, get_object_or_none, get_setting + +if TYPE_CHECKING: + from collections.abc import Callable + + from django.db.models import QuerySet + from django.http import HttpRequest + +# --- Engagement filter vocabulary (§4.9, minimal set) ----------------------------------------- + +ENGAGEMENT_FILTER_SPEC = register_filter_spec("engagement", FilterSpec( + model=Engagement, + filters={ + "id__in": filter_field("id", "in", "number"), + "name__icontains": filter_field("name", "icontains", "char"), + "asset": filter_field("product", "exact", "number"), + "asset__in": filter_field("product", "in", "number"), + "organization": filter_field("product__prod_type", "exact", "number"), + "lead": filter_field("lead", "exact", "number"), + "status": filter_field("status", "exact", "char"), + "engagement_type": filter_field("engagement_type", "exact", "char"), + "target_start__gte": filter_field("target_start", "gte", "date"), + "target_start__lte": filter_field("target_start", "lte", "date"), + "target_end__gte": filter_field("target_end", "gte", "date"), + "target_end__lte": filter_field("target_end", "lte", "date"), + "created__gte": filter_field("created", "gte", "datetime"), + "created__lte": filter_field("created", "lte", "datetime"), + "updated__gte": filter_field("updated", "gte", "datetime"), + "updated__lte": filter_field("updated", "lte", "datetime"), + }, + orderings={ + "id": "id", + "name": "name", + "target_start": "target_start", + "created": "created", + "updated": "updated", + }, + search_fields=["name", "description"], +)) + +# Sentinel distinguishing "tags omitted" from "tags set to null/empty" on PATCH. +_UNSET = object() + + +class EngagementListResponse(Schema): + + """OpenAPI documentation of the list envelope (I1); runtime serialization is manual.""" + + count: int + next: str | None + previous: str | None + results: list[EngagementSlim] + meta: dict | None = None + + +def _base_queryset(request: HttpRequest, queryset_hook: Callable | None) -> QuerySet: + # Reads flow only through the authorized queryset (I8). The OS helper resolves the current user + # from crum (its signature takes no user kwarg), exactly as the v2 viewset relies on. + qs = get_authorized_engagements(Permissions.Engagement_View) + if queryset_hook is not None: + qs = queryset_hook(qs, request) + return qs + + +def _validation_from_django(exc: DjangoValidationError) -> Exception: + if hasattr(exc, "message_dict"): + return validation_problem({k: list(v) for k, v in exc.message_dict.items()}) + return validation_problem({"non_field_errors": list(exc.messages)}) + + +def _resolve_lead(pk: int | None) -> Dojo_User | None: + if pk is None: + return None + lead = get_object_or_none(Dojo_User, pk=pk) + if lead is None: + raise validation_problem({"lead": [f"user {pk} does not exist"]}) + return lead + + +def _validate_dates(target_start, target_end) -> None: + # Mirror EngagementSerializer.validate (POST): target start must not exceed target end. + if target_start is not None and target_end is not None and target_start > target_end: + raise validation_problem( + {"target_start": ["Your target start date exceeds your target end date"]}, + ) + + +def _destroy(instance: Engagement) -> None: + """Mirror v2 ``EngagementViewSet.destroy`` exactly (no Endpoint context wrapper -- §12).""" + if get_setting("ASYNC_OBJECT_DELETE"): + async_delete().delete(instance) + else: + instance.delete() + + +def _apply_scalars(instance: Engagement, data: dict) -> None: + if "lead" in data: + instance.lead = _resolve_lead(data.pop("lead")) + for key, value in data.items(): + setattr(instance, key, value) + + +def build_engagements_router( + *, + schema: type = EngagementSlim, + detail_schema: type = EngagementDetail, + filter_spec: FilterSpec = ENGAGEMENT_FILTER_SPEC, + queryset_hook: Callable | None = None, + auth=NOT_SET, +) -> Router: + """Build the engagements router (I5).""" + router = Router(tags=["engagements"], auth=auth) + + @router.get("/engagements", response=EngagementListResponse, url_name="engagements_list") + def list_engagements(request: HttpRequest): + filtered = apply_filters(request, _base_queryset(request, queryset_hook), filter_spec) + + expand_tree, select_related, prefetch = plan(schema, request.GET.get("expand")) + # ?fields= may opt up into the detail field set (§4.7 Part A); defer the heavy detail + # columns not requested (Part B). + fields = parse_fields(request.GET.get("fields"), allowed_field_names(detail_schema)) + fplan = plan_list_fields(schema, detail_schema, fields) + page_qs = filtered.select_related(*schema.SELECT_RELATED, *fplan.select_related).prefetch_related(*schema.PREFETCH_RELATED) + page_qs = plan_queryset(page_qs, select_related, prefetch) + if fplan.defer: + page_qs = page_qs.defer(*fplan.defer) + + def serialize_row(obj: object) -> dict: + return serialize_list_row(obj, fplan, expand_tree) + + envelope = paginate(request, count_qs=filtered, page_qs=page_qs, serialize=serialize_row, filter_spec=filter_spec) + include_meta = apply_includes(request, filtered, allowed=set()) + if include_meta: + envelope.setdefault("meta", {}).update(include_meta) + return json_response(envelope) + + # Literal ``export.csv`` cannot match the ``{int:engagement_id}`` detail route (int converter), + # so no collision. Same filter contract as the list; whole filtered set streamed as CSV (§4.15). + @router.get("/engagements/export.csv", url_name="engagements_export_csv") + def export_engagements(request: HttpRequest): + filtered = apply_filters(request, _base_queryset(request, queryset_hook), filter_spec) + fields = parse_fields(request.GET.get("fields"), allowed_field_names(detail_schema)) + fplan = plan_list_fields(schema, detail_schema, fields) + page_qs = filtered.select_related(*schema.SELECT_RELATED, *fplan.select_related).prefetch_related(*schema.PREFETCH_RELATED) + if fplan.defer: + page_qs = page_qs.defer(*fplan.defer) + return stream_csv_export(request, resource="engagements", count_qs=filtered, page_qs=page_qs, plan=fplan) + + @router.get("/engagements/{int:engagement_id}", response=detail_schema, url_name="engagements_detail") + def get_engagement(request: HttpRequest, engagement_id: int): + expand_tree, select_related, prefetch = plan(detail_schema, request.GET.get("expand")) + qs = _base_queryset(request, queryset_hook).select_related(*detail_schema.SELECT_RELATED).prefetch_related(*detail_schema.PREFETCH_RELATED) + qs = plan_queryset(qs, select_related, prefetch) + obj = qs.filter(pk=engagement_id).first() + if obj is None: + msg = f"Engagement {engagement_id} not found" + raise not_found_problem(msg) + fields = parse_fields(request.GET.get("fields"), allowed_field_names(detail_schema)) + return json_response(apply_fields(serialize(obj, detail_schema, expand_tree), fields)) + + @router.post("/engagements", response=detail_schema, url_name="engagements_create") + def create_engagement(request: HttpRequest, payload: EngagementWrite): + data = payload.dict() + tags = data.pop("tags") + product_id = data.pop("asset") + # Mirror check_post_permission(request, Product, "product", "add"): 404 if the target + # asset doesn't exist, 403 if the user can't add engagements to it. + product = get_object_or_none(Product, pk=product_id) + if product is None: + msg = f"Asset {product_id} not found" + raise not_found_problem(msg) + if not user_has_permission(request.user, product, Permissions.Engagement_Add): + raise PermissionDenied + _validate_dates(data.get("target_start"), data.get("target_end")) + + instance = Engagement(product=product) + _apply_scalars(instance, {k: v for k, v in data.items() if v is not None}) + try: + instance.save() + except DjangoValidationError as exc: + raise _validation_from_django(exc) from exc + if tags is not None: + instance.tags = tags + instance.save() + return json_response(serialize(instance, detail_schema, {}), status=201) + + @router.patch("/engagements/{int:engagement_id}", response=detail_schema, url_name="engagements_update") + def update_engagement(request: HttpRequest, engagement_id: int, payload: EngagementUpdate): + instance = _base_queryset(request, queryset_hook).filter(pk=engagement_id).first() + if instance is None: + msg = f"Engagement {engagement_id} not found" + raise not_found_problem(msg) # 404: unknown or unauthorized-to-view + if not user_has_permission(request.user, instance, Permissions.Engagement_Edit): + raise PermissionDenied # 403: visible but not editable + + data = payload.dict(exclude_unset=True) + tags = data.pop("tags", _UNSET) + if "asset" in data: + new_product_id = data.pop("asset") + # Mirror check_update_permission: only re-check `add` when the FK actually changes. + if new_product_id is not None and new_product_id != instance.product_id: + new_product = get_object_or_none(Product, pk=new_product_id) + if new_product is None: + msg = f"Asset {new_product_id} not found" + raise not_found_problem(msg) + if not user_has_permission(request.user, new_product, Permissions.Engagement_Add): + raise PermissionDenied + instance.product = new_product + + target_start = data.get("target_start", instance.target_start) + target_end = data.get("target_end", instance.target_end) + _validate_dates(target_start, target_end) + + _apply_scalars(instance, data) + if tags is not _UNSET: + instance.tags = tags if tags is not None else [] + try: + instance.save() + except DjangoValidationError as exc: + raise _validation_from_django(exc) from exc + return json_response(serialize(instance, detail_schema, {})) + + @router.put("/engagements/{int:engagement_id}", response=detail_schema, url_name="engagements_replace") + def replace_engagement(request: HttpRequest, engagement_id: int, payload: EngagementReplace): + # Full replace (PUT). Validates against the create-shaped EngagementReplace (required + # asset/target_start/target_end, extra="forbid") and applies payload.dict() WITHOUT + # exclude_unset, so omitted optionals reset to their schema defaults (§4.11). Permission + # ladder identical to PATCH: authorized-view resolve (404) then object Edit (403); the + # required asset is re-authorized only when it actually changes (mirrors PATCH). + instance = _base_queryset(request, queryset_hook).filter(pk=engagement_id).first() + if instance is None: + msg = f"Engagement {engagement_id} not found" + raise not_found_problem(msg) # 404: unknown or unauthorized-to-view + if not user_has_permission(request.user, instance, Permissions.Engagement_Edit): + raise PermissionDenied # 403: visible but not editable + + data = payload.dict() + tags = data.pop("tags") + new_product_id = data.pop("asset") + if new_product_id != instance.product_id: + new_product = get_object_or_none(Product, pk=new_product_id) + if new_product is None: + msg = f"Asset {new_product_id} not found" + raise not_found_problem(msg) + if not user_has_permission(request.user, new_product, Permissions.Engagement_Add): + raise PermissionDenied + instance.product = new_product + + _validate_dates(data.get("target_start"), data.get("target_end")) + _apply_scalars(instance, data) + instance.tags = tags if tags is not None else [] + try: + instance.save() + except DjangoValidationError as exc: + raise _validation_from_django(exc) from exc + return json_response(serialize(instance, detail_schema, {})) + + @router.delete("/engagements/{int:engagement_id}", url_name="engagements_delete") + def delete_engagement(request: HttpRequest, engagement_id: int): + instance = _base_queryset(request, queryset_hook).filter(pk=engagement_id).first() + if instance is None: + msg = f"Engagement {engagement_id} not found" + raise not_found_problem(msg) + if not user_has_permission(request.user, instance, Permissions.Engagement_Delete): + raise PermissionDenied + _destroy(instance) + response = HttpResponse(status=204) + response["X-API-Status"] = settings.API_V3_STATUS + return response + + return router diff --git a/dojo/engagement/api_v3/schemas.py b/dojo/engagement/api_v3/schemas.py new file mode 100644 index 00000000000..dcdfa7b6645 --- /dev/null +++ b/dojo/engagement/api_v3/schemas.py @@ -0,0 +1,205 @@ +""" +Engagement response + write schemas for API v3 (§4.5, §4.11, OS3b). + +``EngagementSlim`` is the canonical Engagement slim -- relocated here from ``dojo/finding/api_v3`` +(where OS1 first defined it so finding ``?expand=`` had a target); the finding module now re-exports +this copy so there is exactly one class per model (verified is-identity in the tests, mirroring the +OS3a relocation pattern -- see §12). ``EngagementDetail`` adds the documented heavier read fields +(§4.5). ``asset``/``organization``/``lead`` are expandable relations (§4.6). Per D11 the +Product/Product_Type models are exposed on the wire as ``asset``/``organization`` (the ref keys and +the ``asset`` write FK -> model ``product``); the models/DB/module paths are not renamed (§12). + +Write schemas mirror the v2 ``EngagementSerializer`` (a ``ModelSerializer`` excluding +``inherited_tags``): the model requires ``target_start``, ``target_end`` and ``product`` (exposed on +the wire as ``asset``); everything else is optional. Relations are referenced by integer id (§4.11); +``editable=False`` / +server-managed fields (``active``, ``notes``, ``files``, ``progress``, ``risk_acceptance``, +``done_testing``, ``id``, ``created``, ``updated``) are never writable; unknown fields are rejected +(``extra="forbid"``). +""" +from __future__ import annotations + +from datetime import date, datetime # noqa: TC003 -- runtime import: pydantic resolves the schema field types +from typing import ClassVar + +from ninja import Schema + +from dojo.api_v3.expand import ExpandRel +from dojo.api_v3.refs import Ref, to_ref +from dojo.models import Engagement +from dojo.product.api_v3.schemas import AssetSlim +from dojo.product_type.api_v3.schemas import OrganizationSlim +from dojo.user.api_v3.schemas import UserSlim + + +class EngagementSlim(Schema): + django_model: ClassVar = Engagement + SELECT_RELATED: ClassVar[tuple] = ("product__prod_type", "lead") + PREFETCH_RELATED: ClassVar[tuple] = ("tags",) + EXPANDABLE: ClassVar[dict[str, ExpandRel]] = {} + + id: int + name: str | None + asset: Ref + organization: Ref + lead: Ref | None + status: str | None + engagement_type: str | None + target_start: date | None + target_end: date | None + active: bool | None + tags: list[str] + created: datetime | None + updated: datetime | None + + @staticmethod + def resolve_asset(obj) -> dict | None: + return to_ref(obj.product) + + @staticmethod + def resolve_organization(obj) -> dict | None: + return to_ref(obj.product.prod_type) + + @staticmethod + def resolve_lead(obj) -> dict | None: + return to_ref(obj.lead) + + @staticmethod + def resolve_tags(obj) -> list[str]: + return [t.name for t in obj.tags.all()] + + +EngagementSlim.EXPANDABLE = { + "asset": ExpandRel(attr="product", path="product", schema=AssetSlim), + "organization": ExpandRel(attr="product.prod_type", path="product__prod_type", schema=OrganizationSlim), + "lead": ExpandRel(attr="lead", path="lead", schema=UserSlim), +} + + +class EngagementDetail(EngagementSlim): + + """Slim + the documented heavier read fields (§4.5). Retrieve returns detail; list returns slim.""" + + description: str | None + version: str | None + first_contacted: date | None + reason: str | None + tracker: str | None + test_strategy: str | None + threat_model: bool | None + api_test: bool | None + pen_test: bool | None + check_list: bool | None + build_id: str | None + commit_hash: str | None + branch_tag: str | None + source_code_management_uri: str | None + deduplication_on_engagement: bool | None + + +class EngagementWrite(Schema): + + """Create payload (POST). ``asset``/``target_start``/``target_end`` required (mirrors v2).""" + + model_config = {"extra": "forbid"} + + asset: int + target_start: date + target_end: date + name: str | None = None + description: str | None = None + version: str | None = None + first_contacted: date | None = None + lead: int | None = None + reason: str | None = None + tracker: str | None = None + test_strategy: str | None = None + threat_model: bool | None = None + api_test: bool | None = None + pen_test: bool | None = None + check_list: bool | None = None + status: str | None = None + engagement_type: str | None = None + build_id: str | None = None + commit_hash: str | None = None + branch_tag: str | None = None + source_code_management_uri: str | None = None + deduplication_on_engagement: bool | None = None + tags: list[str] | None = None + + +class EngagementUpdate(Schema): + + """Partial update payload (PATCH). Every field optional; only provided keys are applied.""" + + model_config = {"extra": "forbid"} + + asset: int | None = None + target_start: date | None = None + target_end: date | None = None + name: str | None = None + description: str | None = None + version: str | None = None + first_contacted: date | None = None + lead: int | None = None + reason: str | None = None + tracker: str | None = None + test_strategy: str | None = None + threat_model: bool | None = None + api_test: bool | None = None + pen_test: bool | None = None + check_list: bool | None = None + status: str | None = None + engagement_type: str | None = None + build_id: str | None = None + commit_hash: str | None = None + branch_tag: str | None = None + source_code_management_uri: str | None = None + deduplication_on_engagement: bool | None = None + tags: list[str] | None = None + + +class EngagementReplace(Schema): + + """ + Full-replace payload (PUT). ``asset``/``target_start``/``target_end`` required (create-shaped); + ``asset`` is required and reassignment is re-authorized when it changes (mirrors PATCH). + + A dedicated Replace schema is required because ``EngagementWrite`` cannot serve full-replace + semantics (§12): a full replace applies ``payload.dict()`` without ``exclude_unset``, so an + omitted optional resets to the schema default -- and ``threat_model``/``api_test``/``pen_test``/ + ``check_list``/``deduplication_on_engagement`` are ``NOT NULL`` boolean columns, so + ``EngagementWrite``'s create-appropriate ``None`` default (dropped by the create path) would + violate the constraint on an existing row. They default to their model defaults here + (``True`` for the four coverage flags, ``False`` for ``deduplication_on_engagement``). + ``status``/``engagement_type`` are ``null=True`` but ``blank=False`` with model defaults, and + ``Engagement.save()`` runs ``full_clean``, so they likewise default to their model defaults + (``"Not Started"``/``"Interactive"``) rather than ``None``. The remaining nullable scalar fields + reset to ``None`` when omitted. + """ + + model_config = {"extra": "forbid"} + + asset: int + target_start: date + target_end: date + name: str | None = None + description: str | None = None + version: str | None = None + first_contacted: date | None = None + lead: int | None = None + reason: str | None = None + tracker: str | None = None + test_strategy: str | None = None + threat_model: bool = True + api_test: bool = True + pen_test: bool = True + check_list: bool = True + status: str = "Not Started" + engagement_type: str = "Interactive" + build_id: str | None = None + commit_hash: str | None = None + branch_tag: str | None = None + source_code_management_uri: str | None = None + deduplication_on_engagement: bool = False + tags: list[str] | None = None diff --git a/dojo/engagement/services.py b/dojo/engagement/services.py index 6a5b7570103..5d947c534ca 100644 --- a/dojo/engagement/services.py +++ b/dojo/engagement/services.py @@ -1,6 +1,7 @@ # # engagements import logging +from crum import get_current_request from django.db.models.signals import pre_save from django.dispatch import receiver from django.urls import reverse @@ -9,12 +10,31 @@ from dojo.celery_dispatch import dojo_dispatch_task from dojo.jira import services as jira_services from dojo.models import Engagement -from dojo.notifications.helper import create_notification +from dojo.notifications.helper import create_notification, process_tag_notifications from dojo.utils import calculate_grade logger = logging.getLogger(__name__) +def process_note_added(engagement, note, *, user): + """ + Fire the same side-effects as the v2 engagement notes ``@action`` create branch + (``dojo/engagement/api/views.py``) after a note is persisted and linked: @mention notifications + only -- the engagement @action has **no** JIRA comment sync and **no** ``last_reviewed`` stamping. + Reuses the exact v2 parsing/notification helper (``process_tag_notifications``); the request is read + from crum (see ``dojo/finding/services.py``), and mentions are skipped with no request. ``user`` is + part of the callback contract (I6) but unused here (no side-effect needs it). + """ + request = get_current_request() + if request is not None: + process_tag_notifications( + request=request, + note=note, + parent_url=request.build_absolute_uri(reverse("view_engagement", args=(engagement.id,))), + parent_title=f"Engagement: {engagement.name}", + ) + + def close_engagement(eng): eng.active = False eng.status = "Completed" diff --git a/dojo/finding/api_v3/__init__.py b/dojo/finding/api_v3/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/dojo/finding/api_v3/routes.py b/dojo/finding/api_v3/routes.py new file mode 100644 index 00000000000..021b1d5adf4 --- /dev/null +++ b/dojo/finding/api_v3/routes.py @@ -0,0 +1,327 @@ +""" +Findings read + write routes for API v3 (§4.5, §4.6, §4.8, OS1 read / OS3b write). + +``build_findings_router()`` is a router *factory* (I5): the OS mount calls it with defaults; a +downstream distribution can call it with a subclassed schema / extra filters / a queryset hook and +mount the result under its own prefix -- no fork. Routes are thin (I6): authorize -> parse -> +service -> serialize; all RBAC flows through ``get_authorized_findings`` (reads) and the v2 +``UserHasFindingPermission`` semantics (writes, I8), and **all** write side-effects/orchestration +live in ``dojo/finding/services.py`` (D7), never in the route: + +- create (POST): ``add`` permission on the target ``test`` referenced in the payload (404 if it + doesn't exist, 403 if unauthorized -- mirrors + ``check_post_permission(request, Test, "test", "add")``). +- update (PATCH): object ``edit`` permission (404 for unknown-or-unauthorized, 403 if visible but + not editable). Mirrors ``perform_update``: ``push_to_jira`` is OR-ed with the + JIRA project's ``push_all_issues`` when JIRA is enabled. +- delete (DELETE): object ``delete`` permission; delegates to the service, whose ``Finding.delete()`` + runs the same dedup/grading hooks as the v2 ``FindingViewSet.destroy`` (§12). +""" +from __future__ import annotations + +from typing import TYPE_CHECKING + +from django.conf import settings +from django.core.exceptions import PermissionDenied +from django.db.models import Count +from django.http import HttpResponse +from ninja import Router, Schema +from ninja.constants import NOT_SET + +from dojo.api_v3.csv_export import stream_csv_export +from dojo.api_v3.errors import json_response, not_found_problem +from dojo.api_v3.expand import ( + allowed_field_names, + apply_fields, + parse_fields, + plan, + plan_list_fields, + plan_queryset, + serialize, + serialize_list_row, +) +from dojo.api_v3.filtering import ( + FilterSpec, + apply_filters, + filter_field, + register_filter_spec, + severity_rank_order, +) +from dojo.api_v3.include import apply_includes +from dojo.api_v3.pagination import paginate +from dojo.authorization.authorization import user_has_permission +from dojo.authorization.roles_permissions import Permissions +from dojo.finding.api_v3.schemas import ( + FindingDetail, + FindingReplace, + FindingSlim, + FindingUpdate, + FindingWrite, +) +from dojo.finding.queries import get_authorized_findings +from dojo.finding.services import create_finding, delete_finding, update_finding +from dojo.jira import services as jira_services +from dojo.models import Finding, Test +from dojo.utils import get_object_or_none, get_system_setting + +if TYPE_CHECKING: + from collections.abc import Callable + + from django.db.models import QuerySet + from django.http import HttpRequest + +# --- Findings filter vocabulary (§4.9) -------------------------------------------------------- + +FINDING_FILTER_SPEC = register_filter_spec("finding", FilterSpec( + model=Finding, + filters={ + "id__in": filter_field("id", "in", "number"), + "title__icontains": filter_field("title", "icontains", "char"), + "severity": filter_field("severity", "exact", "char"), + "severity__in": filter_field("severity", "in", "char"), + "active": filter_field("active", "exact", "bool"), + "verified": filter_field("verified", "exact", "bool"), + "duplicate": filter_field("duplicate", "exact", "bool"), + "false_p": filter_field("false_p", "exact", "bool"), + "risk_accepted": filter_field("risk_accepted", "exact", "bool"), + "out_of_scope": filter_field("out_of_scope", "exact", "bool"), + "is_mitigated": filter_field("is_mitigated", "exact", "bool"), + "date__gte": filter_field("date", "gte", "date"), + "date__lte": filter_field("date", "lte", "date"), + "cwe": filter_field("cwe", "exact", "number"), + "cwe__in": filter_field("cwe", "in", "number"), + "asset": filter_field("test__engagement__product", "exact", "number"), + "asset__in": filter_field("test__engagement__product", "in", "number"), + "organization": filter_field("test__engagement__product__prod_type", "exact", "number"), + "engagement": filter_field("test__engagement", "exact", "number"), + "test": filter_field("test", "exact", "number"), + "reporter": filter_field("reporter", "exact", "number"), + "tags__in": filter_field("tags__name", "in", "char", distinct=True), + "created__gte": filter_field("created", "gte", "datetime"), + "created__lte": filter_field("created", "lte", "datetime"), + "updated__gte": filter_field("updated", "gte", "datetime"), + "updated__lte": filter_field("updated", "lte", "datetime"), + }, + orderings={ + "id": "id", + "date": "date", + "title": "title", + "created": "created", + "updated": "updated", + }, + # `o=severity` sorts by rank (Critical first), not alphabetically (§4.9). + order_expressions={"severity": severity_rank_order}, + search_fields=["title", "description"], +)) + +_ALLOWED_INCLUDES = {"counts"} + + +class FindingListResponse(Schema): + + """ + OpenAPI documentation of the list envelope (I1). Runtime serialization is manual so + ``?expand=``/``?fields=`` can reshape ``results`` dynamically; this schema documents the base + slim shape for client codegen. + """ + + count: int + next: str | None + previous: str | None + results: list[FindingSlim] + meta: dict | None = None + + +def _base_queryset(request: HttpRequest, queryset_hook: Callable | None) -> QuerySet: + qs = get_authorized_findings(Permissions.Finding_View, user=request.user) + if queryset_hook is not None: + qs = queryset_hook(qs, request) + return qs + + +def _detail_object(request: HttpRequest, queryset_hook: Callable | None, detail_schema: type, finding_id: int): + """ + Fetch a finding through the authorized queryset with the detail schema's relations + + ``locations_count`` annotation loaded, so the write responses serialize identically to GET. + """ + qs = ( + _base_queryset(request, queryset_hook) + .select_related(*detail_schema.SELECT_RELATED) + .prefetch_related(*detail_schema.PREFETCH_RELATED) + .annotate(locations_count=Count("locations", distinct=True)) + ) + return qs.filter(pk=finding_id).first() + + +def build_findings_router( + *, + schema: type = FindingSlim, + detail_schema: type = FindingDetail, + filter_spec: FilterSpec = FINDING_FILTER_SPEC, + queryset_hook: Callable | None = None, + auth=NOT_SET, +) -> Router: + """Build the findings router (I5).""" + router = Router(tags=["findings"], auth=auth) + + @router.get("/findings", response=FindingListResponse, url_name="findings_list") + def list_findings(request: HttpRequest): + filtered = apply_filters(request, _base_queryset(request, queryset_hook), filter_spec) + + expand_tree, select_related, prefetch = plan(schema, request.GET.get("expand")) + # ?fields= may opt up into the detail field set (§4.7 Part A); the LIST queryset defers the + # heavy detail columns that were not requested (Part B). + fields = parse_fields(request.GET.get("fields"), allowed_field_names(detail_schema)) + fplan = plan_list_fields(schema, detail_schema, fields) + page_qs = ( + filtered.select_related(*schema.SELECT_RELATED, *fplan.select_related) + .prefetch_related(*schema.PREFETCH_RELATED) + .annotate(locations_count=Count("locations", distinct=True)) + ) + page_qs = plan_queryset(page_qs, select_related, prefetch) + if fplan.defer: + page_qs = page_qs.defer(*fplan.defer) + + def serialize_row(obj: object) -> dict: + return serialize_list_row(obj, fplan, expand_tree) + + envelope = paginate(request, count_qs=filtered, page_qs=page_qs, serialize=serialize_row, filter_spec=filter_spec) + + include_meta = apply_includes(request, filtered, allowed=_ALLOWED_INCLUDES) + if include_meta: + envelope.setdefault("meta", {}).update(include_meta) + + return json_response(envelope) + + # ``export.csv`` is registered as a literal path; the ``{int:finding_id}`` detail route's int + # converter cannot match "export.csv", so the two never collide regardless of registration order. + @router.get("/findings/export.csv", url_name="findings_export_csv") + def export_findings(request: HttpRequest): + # Same filter contract as the list (filters, o=, q=, fields= incl. detail opt-up); no + # pagination/expand/include (rejected in the kernel). Streams the whole filtered set (§4.15). + filtered = apply_filters(request, _base_queryset(request, queryset_hook), filter_spec) + fields = parse_fields(request.GET.get("fields"), allowed_field_names(detail_schema)) + fplan = plan_list_fields(schema, detail_schema, fields) + page_qs = ( + filtered.select_related(*schema.SELECT_RELATED, *fplan.select_related) + .prefetch_related(*schema.PREFETCH_RELATED) + .annotate(locations_count=Count("locations", distinct=True)) + ) + if fplan.defer: + page_qs = page_qs.defer(*fplan.defer) + return stream_csv_export(request, resource="findings", count_qs=filtered, page_qs=page_qs, plan=fplan) + + @router.get("/findings/{int:finding_id}", response=detail_schema, url_name="findings_detail") + def get_finding(request: HttpRequest, finding_id: int): + expand_tree, select_related, prefetch = plan(detail_schema, request.GET.get("expand")) + qs = ( + _base_queryset(request, queryset_hook) + .select_related(*detail_schema.SELECT_RELATED) + .prefetch_related(*detail_schema.PREFETCH_RELATED) + .annotate(locations_count=Count("locations", distinct=True)) + ) + qs = plan_queryset(qs, select_related, prefetch) + obj = qs.filter(pk=finding_id).first() + if obj is None: + # 404 for unknown *or unauthorized* -- never leak existence (§4.10). + msg = f"Finding {finding_id} not found" + raise not_found_problem(msg) + + allowed_fields = allowed_field_names(detail_schema) + fields = parse_fields(request.GET.get("fields"), allowed_fields) + data = apply_fields(serialize(obj, detail_schema, expand_tree), fields) + return json_response(data) + + @router.post("/findings", response=detail_schema, url_name="findings_create") + def create_finding_route(request: HttpRequest, payload: FindingWrite): + data = payload.dict() + test_id = data.pop("test") + push_to_jira = data.pop("push_to_jira") + vulnerability_ids = data.pop("vulnerability_ids") + cwes = data.pop("cwes") + # Mirror UserHasFindingPermission -> check_post_permission(request, Test, "test", "add"): + # 404 if the test doesn't exist, 403 if the user can't add findings to it. + test = get_object_or_none(Test, pk=test_id) + if test is None: + msg = f"Test {test_id} not found" + raise not_found_problem(msg) + if not user_has_permission(request.user, test, Permissions.Finding_Add): + raise PermissionDenied + finding = create_finding( + test=test, data=data, user=request.user, + push_to_jira=push_to_jira, vulnerability_ids=vulnerability_ids, cwes=cwes, + ) + obj = _detail_object(request, queryset_hook, detail_schema, finding.pk) or finding + return json_response(serialize(obj, detail_schema, {}), status=201) + + @router.patch("/findings/{int:finding_id}", response=detail_schema, url_name="findings_update") + def update_finding_route(request: HttpRequest, finding_id: int, payload: FindingUpdate): + finding = _base_queryset(request, queryset_hook).filter(pk=finding_id).first() + if finding is None: + msg = f"Finding {finding_id} not found" + raise not_found_problem(msg) # 404: unknown or unauthorized-to-view + if not user_has_permission(request.user, finding, Permissions.Finding_Edit): + raise PermissionDenied # 403: visible but not editable + + changes = payload.dict(exclude_unset=True) + push_to_jira = changes.pop("push_to_jira", False) + vulnerability_ids = changes.pop("vulnerability_ids", None) + cwes = changes.pop("cwes", None) + # Mirror FindingViewSet.perform_update: OR push_to_jira with the project's push_all_issues. + jira_project = jira_services.get_project(finding) + if get_system_setting("enable_jira") and jira_project: + push_to_jira = push_to_jira or jira_project.push_all_issues + + update_finding( + finding, changes=changes, user=request.user, + push_to_jira=push_to_jira, vulnerability_ids=vulnerability_ids, cwes=cwes, + ) + obj = _detail_object(request, queryset_hook, detail_schema, finding_id) or finding + return json_response(serialize(obj, detail_schema, {})) + + @router.put("/findings/{int:finding_id}", response=detail_schema, url_name="findings_replace") + def replace_finding_route(request: HttpRequest, finding_id: int, payload: FindingReplace): + # Full replace (PUT). Validates against the create-shaped FindingReplace (required fields, + # extra="forbid") and applies payload.dict() WITHOUT exclude_unset, so omitted optionals + # reset to their schema defaults -- a true full replace, mirroring v2's update(partial=False) + # (§4.11). ``test`` is not in the replace schema (editable=False; never reassigned on update, + # like PATCH). Goes through the service exactly like PATCH (full changes dict), so JIRA / + # risk-acceptance / vuln-id side-effects flow through dojo/finding/services.py (I6). Permission + # ladder identical to PATCH: authorized-view resolve (404) then object Edit (403). + finding = _base_queryset(request, queryset_hook).filter(pk=finding_id).first() + if finding is None: + msg = f"Finding {finding_id} not found" + raise not_found_problem(msg) # 404: unknown or unauthorized-to-view + if not user_has_permission(request.user, finding, Permissions.Finding_Edit): + raise PermissionDenied # 403: visible but not editable + + changes = payload.dict() + push_to_jira = changes.pop("push_to_jira", False) + vulnerability_ids = changes.pop("vulnerability_ids", None) + cwes = changes.pop("cwes", None) + # Mirror FindingViewSet.perform_update: OR push_to_jira with the project's push_all_issues. + jira_project = jira_services.get_project(finding) + if get_system_setting("enable_jira") and jira_project: + push_to_jira = push_to_jira or jira_project.push_all_issues + + update_finding( + finding, changes=changes, user=request.user, + push_to_jira=push_to_jira, vulnerability_ids=vulnerability_ids, cwes=cwes, + ) + obj = _detail_object(request, queryset_hook, detail_schema, finding_id) or finding + return json_response(serialize(obj, detail_schema, {})) + + @router.delete("/findings/{int:finding_id}", url_name="findings_delete") + def delete_finding_route(request: HttpRequest, finding_id: int): + finding = _base_queryset(request, queryset_hook).filter(pk=finding_id).first() + if finding is None: + msg = f"Finding {finding_id} not found" + raise not_found_problem(msg) + if not user_has_permission(request.user, finding, Permissions.Finding_Delete): + raise PermissionDenied + delete_finding(finding, user=request.user) + response = HttpResponse(status=204) + response["X-API-Status"] = settings.API_V3_STATUS + return response + + return router diff --git a/dojo/finding/api_v3/schemas.py b/dojo/finding/api_v3/schemas.py new file mode 100644 index 00000000000..a53a2a91726 --- /dev/null +++ b/dojo/finding/api_v3/schemas.py @@ -0,0 +1,340 @@ +""" +Finding response + write schemas for API v3 (§4.5, §4.11). + +``FindingSlim`` (list) and ``FindingDetail`` (retrieve). Every schema is a named, importable, +subclassable ninja Schema (I4) and declares (as ``ClassVar`` so pydantic does not treat them as +fields): + +- ``django_model`` -- the mapped model (used by the expand cycle guard), +- ``SELECT_RELATED`` / ``PREFETCH_RELATED`` -- the relation paths its resolvers read, so the + expand planner can keep the query count constant, +- ``EXPANDABLE`` -- its expandable relations (§4.6). + +The parent slims (``EngagementSlim``/``TestSlim``/``TestTypeSlim``/``EnvironmentSlim``/``AssetSlim`` +/``OrganizationSlim``/``UserSlim``) live in their own resource modules; this module **re-exports** +them (OS3a relocated product/product_type/user; OS3b relocated engagement/test) so there is exactly +one canonical class per model shared by the resource endpoints and the finding ``?expand=`` targets +(I4). Per D11 the Product/Product_Type models are exposed on the wire as ``asset``/``organization`` +(the ref keys ``finding.asset``/``finding.organization`` and expand keys below). See §12. + +Write schemas (``FindingWrite`` create, ``FindingUpdate`` PATCH) are the editable subset of the +detail fields; required-vs-optional mirrors the v2 ``FindingCreateSerializer`` / ``FindingSerializer``. +Relations are referenced by integer id (§4.11); ``test`` is writable only on create +(``editable=False`` on the model, mirrors v2). Server-managed fields are never writable and unknown +fields are rejected (``extra="forbid"``). All side-effect / status-invariant validation lives in the +service (``dojo/finding/services.py``, D7); the schemas do field typing + strictness only. +""" +from __future__ import annotations + +import datetime # noqa: TC003 -- runtime import: pydantic resolves the schema field types +from typing import ClassVar + +from ninja import Schema + +from dojo.api_v3.expand import ExpandRel +from dojo.api_v3.refs import Ref, to_location_ref, to_ref + +# Canonical parent slims live in their own resource modules; re-exported here so the finding +# expand targets (below) and the resource endpoints serialize through the same schema (I4, §12). +from dojo.engagement.api_v3.schemas import EngagementSlim +from dojo.finding.cwe import cwe_number +from dojo.models import Finding +from dojo.product.api_v3.schemas import AssetSlim +from dojo.product_type.api_v3.schemas import OrganizationSlim +from dojo.test.api_v3.schemas import EnvironmentSlim, TestSlim, TestTypeSlim +from dojo.user.api_v3.schemas import UserSlim + +__all__ = [ + "AssetSlim", + "EngagementSlim", + "EnvironmentSlim", + "FindingDetail", + "FindingReplace", + "FindingSlim", + "FindingUpdate", + "FindingWrite", + "OrganizationSlim", + "TestSlim", + "TestTypeSlim", + "UserSlim", +] + + +class FindingSlim(Schema): + django_model: ClassVar = Finding + # Base relation paths the finding resolvers read; applied by the route regardless of expand. + SELECT_RELATED: ClassVar[tuple] = ("test__test_type", "test__engagement__product__prod_type", "reporter") + PREFETCH_RELATED: ClassVar[tuple] = ("tags", "vulnerability_id_set", "finding_cwe_set") + EXPANDABLE: ClassVar[dict[str, ExpandRel]] = {} + + id: int + title: str + severity: str + active: bool + verified: bool + false_p: bool + duplicate: bool + risk_accepted: bool + out_of_scope: bool + is_mitigated: bool + date: datetime.date | None + cwe: int | None + cwes: list[int] + vulnerability_ids: list[str] + test: Ref + engagement: Ref + asset: Ref + organization: Ref + reporter: Ref | None + locations_count: int + tags: list[str] + created: datetime.datetime | None + updated: datetime.datetime | None + + @staticmethod + def resolve_test(obj) -> dict | None: + return to_ref(obj.test) + + @staticmethod + def resolve_engagement(obj) -> dict | None: + return to_ref(obj.test.engagement) + + @staticmethod + def resolve_asset(obj) -> dict | None: + return to_ref(obj.test.engagement.product) + + @staticmethod + def resolve_organization(obj) -> dict | None: + return to_ref(obj.test.engagement.product.prod_type) + + @staticmethod + def resolve_reporter(obj) -> dict | None: + return to_ref(obj.reporter) + + @staticmethod + def resolve_locations_count(obj) -> int: + return getattr(obj, "locations_count", 0) or 0 + + @staticmethod + def resolve_tags(obj) -> list[str]: + return [t.name for t in obj.tags.all()] + + @staticmethod + def resolve_vulnerability_ids(obj) -> list[str]: + # Flat strings in storage order (first = the id mirrored into ``cve``); symmetric with what + # ``FindingWrite`` accepts, NOT v2's object list. Reads the prefetched ``vulnerability_id_set`` + # reverse relation (default related_name; v2 uses the same source) -- no per-row query. + return [v.vulnerability_id for v in obj.vulnerability_id_set.all()] + + @staticmethod + def resolve_cwes(obj) -> list[int]: + # ``Finding_CWE`` rows store canonical ``CWE-`` strings and ``save_cwes`` persists the + # primary ``Finding.cwe`` as a row too, so the primary is already in the set. Expose the + # numeric part (``cwe_number`` parses ``CWE-`` -> n) for consistency with the scalar + # ``cwe: int``. Reads the prefetched ``finding_cwe_set`` reverse relation -- no per-row query. + return [n for n in (cwe_number(row.cwe) for row in obj.finding_cwe_set.all()) if n is not None] + + +def _finding_location_edges(finding) -> list[dict]: + """ + ``expand=locations`` special renderer (§4.6): swap the cheap ``locations_count`` for the edge + rows ``[{location: {id, name, type}, status, audit_time, auditor: {id, name}|null}]``. + ``finding.locations`` is the ``LocationFindingReference`` reverse manager (edge carries + ``status``/``audit_time``/``auditor``); the ``locations__location`` and ``locations__auditor`` + prefetches declared on the ExpandRel keep the query count constant. The ``auditor`` ref was + deferred from OS2 and added in OS4 to match the ``/findings/{id}/locations`` sub-resource (§12). + """ + return [ + { + "location": to_location_ref(ref.location), + "status": ref.status, + "audit_time": ref.audit_time, + "auditor": to_ref(ref.auditor), + } + for ref in finding.locations.all() + ] + + +FindingSlim.EXPANDABLE = { + "test": ExpandRel(attr="test", path="test", schema=TestSlim), + "reporter": ExpandRel(attr="reporter", path="reporter", schema=UserSlim), + "engagement": ExpandRel(attr="test.engagement", path="test__engagement", schema=EngagementSlim), + "asset": ExpandRel(attr="test.engagement.product", path="test__engagement__product", schema=AssetSlim), + "organization": ExpandRel( + attr="test.engagement.product.prod_type", + path="test__engagement__product__prod_type", + schema=OrganizationSlim, + ), + "locations": ExpandRel( + attr="locations", + path="locations", + to_many=True, + special=_finding_location_edges, + prefetch_paths=("locations__location", "locations__auditor"), + replaces="locations_count", + ), +} + + +class FindingDetail(FindingSlim): + + """Slim + the documented heavier fields (§4.5). List returns slim; retrieve returns detail.""" + + # Fixed join for the ``mitigated_by`` ref when a LIST ``?fields=`` opts up into it (§4.7 Part A): + # the kernel adds this select_related only when ``mitigated_by`` is requested, so the ref renders + # from the join with no per-row query. (GET /{id} loads it lazily; a single object query is fine.) + DETAIL_SELECT_RELATED: ClassVar[dict[str, tuple[str, ...]]] = {"mitigated_by": ("mitigated_by",)} + + description: str | None + mitigation: str | None + impact: str | None + steps_to_reproduce: str | None + severity_justification: str | None + references: str | None + file_path: str | None + line: int | None + mitigated: datetime.datetime | None + mitigated_by: Ref | None + + @staticmethod + def resolve_mitigated_by(obj) -> dict | None: + return to_ref(obj.mitigated_by) + + +# --- Write schemas (§4.11, §6 OS3 write-schema rule) ------------------------------------------ + +class FindingWrite(Schema): + + """ + Create payload (POST /findings). ``test``/``title``/``severity``/``description``/``active``/ + ``verified`` required (mirrors ``FindingCreateSerializer``: ``active``/``verified`` carry + ``extra_kwargs required=True``). ``vulnerability_ids`` is a flat ``list[str]`` (§4.11); the + service persists them and mirrors the first into the ``cve`` field. ``cwes`` is a flat + ``list[int]`` (plain CWE numbers, e.g. ``79``), symmetric with ``FindingSlim.cwes`` and parallel + to ``vulnerability_ids``: an explicit scalar ``cwe`` wins as the primary when both are supplied; + when only ``cwes`` is supplied its first entry is mirrored into ``cwe`` (§12). ``found_by`` + references ``Test_Type`` ids; ``reporter``/``mitigated_by`` reference user ids. + """ + + model_config = {"extra": "forbid"} + + test: int + title: str + severity: str + description: str + active: bool + verified: bool + date: datetime.date | None = None + cwe: int | None = None + false_p: bool | None = None + duplicate: bool | None = None + out_of_scope: bool | None = None + risk_accepted: bool | None = None + is_mitigated: bool | None = None + mitigation: str | None = None + impact: str | None = None + steps_to_reproduce: str | None = None + severity_justification: str | None = None + references: str | None = None + file_path: str | None = None + line: int | None = None + mitigated: datetime.datetime | None = None + mitigated_by: int | None = None + reporter: int | None = None + found_by: list[int] | None = None + vulnerability_ids: list[str] | None = None + cwes: list[int] | None = None + tags: list[str] | None = None + push_to_jira: bool = False + + +class FindingUpdate(Schema): + + """Partial update payload (PATCH). ``test`` is not writable (editable=False, mirrors v2).""" + + model_config = {"extra": "forbid"} + + title: str | None = None + severity: str | None = None + description: str | None = None + active: bool | None = None + verified: bool | None = None + date: datetime.date | None = None + cwe: int | None = None + false_p: bool | None = None + duplicate: bool | None = None + out_of_scope: bool | None = None + risk_accepted: bool | None = None + is_mitigated: bool | None = None + mitigation: str | None = None + impact: str | None = None + steps_to_reproduce: str | None = None + severity_justification: str | None = None + references: str | None = None + file_path: str | None = None + line: int | None = None + mitigated: datetime.datetime | None = None + mitigated_by: int | None = None + reporter: int | None = None + found_by: list[int] | None = None + vulnerability_ids: list[str] | None = None + cwes: list[int] | None = None + tags: list[str] | None = None + push_to_jira: bool = False + + +class FindingReplace(Schema): + + """ + Full-replace payload (PUT). A dedicated Replace schema is required because ``FindingWrite`` + cannot serve full-replace semantics (§12): + + - ``test`` is dropped -- it is ``editable=False`` on the model and not writable on update + (mirrors PATCH / v2's ``FindingSerializer`` which treats it read-only); so PUT, like PATCH, + never reassigns the parent test. + - the non-null status booleans default to their **model defaults** (not ``None``): a full + replace applies ``payload.dict()`` without ``exclude_unset``, so an omitted optional is reset + to the schema default -- and ``false_p``/``duplicate``/``out_of_scope``/``risk_accepted``/ + ``is_mitigated`` are ``NOT NULL`` columns, so resetting them to ``None`` (``FindingWrite``'s + create-appropriate default, which the create path drops) would violate the constraint. + ``active``/``verified`` stay required (create-shaped). + + Required, strict (``extra="forbid"``) and applied without ``exclude_unset`` by the route so + omitted optionals reset to the defaults below. All side-effect/status-invariant validation still + lives in the service (``dojo/finding/services.py`` ``update_finding``, D7/I6). + + ``vulnerability_ids``/``cwes`` default to ``None`` and the service treats ``None`` as "not + supplied" (rows left untouched) -- so on a PUT that omits them the reset-to-``None`` is a no-op, + NOT a reset to empty. This is the documented no-reset-on-omit deviation for the identity-row + fields, symmetric with ``reporter`` (§12); an explicit ``cwes: []`` clears the extra rows. + """ + + model_config = {"extra": "forbid"} + + title: str + severity: str + description: str + active: bool + verified: bool + date: datetime.date | None = None + cwe: int | None = None + false_p: bool = False + duplicate: bool = False + out_of_scope: bool = False + risk_accepted: bool = False + is_mitigated: bool = False + mitigation: str | None = None + impact: str | None = None + steps_to_reproduce: str | None = None + severity_justification: str | None = None + references: str | None = None + file_path: str | None = None + line: int | None = None + mitigated: datetime.datetime | None = None + mitigated_by: int | None = None + reporter: int | None = None + found_by: list[int] | None = None + vulnerability_ids: list[str] | None = None + cwes: list[int] | None = None + tags: list[str] | None = None + push_to_jira: bool = False diff --git a/dojo/finding/services.py b/dojo/finding/services.py new file mode 100644 index 00000000000..95d195b2429 --- /dev/null +++ b/dojo/finding/services.py @@ -0,0 +1,357 @@ +""" +Finding write service for API v3 (D7 -- the flagship service extraction). + +This module is the single home for finding create/update/delete orchestration and side-effects +(JIRA push + keep-in-sync, risk-acceptance processing, vulnerability-id / CWE persistence, found_by +handling, mitigated-field edit rules, status invariants, deletion dedup/grading hooks). It is +written to serve *all three* consumers (v2 serializer, UI views, v3 routes), but in the alpha PR only +v3 calls it -- so the additive property (D1) is preserved. The v2 ``FindingSerializer`` / +``FindingCreateSerializer`` and the UI ``EditFinding`` view remain the live implementations and are +**not modified** here. + +Extraction rule (D7): the functions below reconcile *both* reference implementations -- +``dojo/finding/api/serializer.py`` (``FindingSerializer.update``/``validate`` and +``FindingCreateSerializer.create``/``validate``) and the UI flow in ``dojo/finding/ui/views.py`` +(``EditFinding.process_finding_form`` / ``process_jira_form`` / ``process_forms``). Where they +diverge, the canonical behavior chosen here is recorded in API_V3_PLAN.md §12 (see the OS3b +divergence entry). The serializer path is the primary reference (the plan names it), so where the UI +adds API-absent side-effects (last_reviewed stamping, mitigated-status propagation to location edges, +false-positive history, burp req/resp, finding-group handling, github, jira link/unlink) those are +consciously **deferred** to the convergence track rather than baked into the API here. + +Invariant I6: keyword-only args, an explicit ``user``, no HTTP context. Validation failures raise +``rest_framework.exceptions.ValidationError`` -- exactly the exception the reference serializer +raises -- which the v3 kernel error boundary maps to a 400 ``application/problem+json`` (§12 OS1). +""" +from __future__ import annotations + +import logging +from typing import TYPE_CHECKING + +from crum import get_current_request +from django.urls import reverse +from django.utils.translation import gettext_lazy as _ +from rest_framework.exceptions import ValidationError + +from dojo.celery_dispatch import dojo_dispatch_task +from dojo.finding.helper import ( + can_edit_mitigated_data, + save_cwes, + save_vulnerability_ids, +) +from dojo.jira import services as jira_services +from dojo.models import SEVERITIES, Dojo_User, Finding, Test_Type +from dojo.notifications.helper import async_create_notification, process_tag_notifications +from dojo.utils import get_object_or_none + +if TYPE_CHECKING: + from dojo.models import Test + from dojo.notes.models import Notes + +logger = logging.getLogger(__name__) + +# Distinguishes "key absent" from "key present with value None" in a PATCH change set. +_UNSET = object() + +# Scalar Finding fields consumed via **kwargs on create; relations/side-effect keys are handled +# explicitly (popped) before this set is applied. +_SPECIAL_KEYS = frozenset({"reporter", "mitigated_by", "found_by", "tags", "vulnerability_ids", "cwes", "push_to_jira", "test"}) + + +# --- shared validation (ported from FindingSerializer.validate / FindingCreateSerializer.validate) -- + +def _validate_severity(severity: str) -> None: + if severity is not None and severity not in SEVERITIES: + msg = f"Severity must be one of the following: {SEVERITIES}" + raise ValidationError({"severity": [msg]}) + + +def _validate_mitigated_editable(data: dict, user) -> None: + """Mirror the serializer's mitigated-field editability gate (settings.EDITABLE_MITIGATED_DATA).""" + attempting = any( + (field in data) and (data.get(field) is not None) + for field in ("mitigated", "mitigated_by") + ) + if attempting and not can_edit_mitigated_data(user): + errors = {} + if ("mitigated" in data) and (data.get("mitigated") is not None): + errors["mitigated"] = ["Editing mitigated timestamp is disabled (EDITABLE_MITIGATED_DATA=false)"] + if ("mitigated_by" in data) and (data.get("mitigated_by") is not None): + errors["mitigated_by"] = ["Editing mitigated_by is disabled (EDITABLE_MITIGATED_DATA=false)"] + if errors: + raise ValidationError(errors) + + +def _validate_status_invariants(*, is_active, is_verified, is_duplicate, is_false_p, + is_risk_accepted, product, was_risk_accepted) -> None: + """Port the active/verified/duplicate/false_p/risk_accepted invariants (both serializers).""" + if (is_active or is_verified) and is_duplicate: + msg = "Duplicate findings cannot be verified or active" + raise ValidationError(msg) + if is_false_p and is_verified: + msg = "False positive findings cannot be verified." + raise ValidationError(msg) + if is_risk_accepted and not was_risk_accepted and not product.enable_simple_risk_acceptance: + msg = "Simple risk acceptance is disabled for this product, use the UI to accept this finding." + raise ValidationError(msg) + if is_active and is_risk_accepted: + msg = "Active findings cannot be risk accepted." + raise ValidationError(msg) + + +def _process_risk_acceptance(finding: Finding, changes: dict, user) -> None: + """ + Port ``FindingSerializer.process_risk_acceptance`` (runs in validate(), i.e. before field + updates). No-op unless ``risk_accepted`` is an explicit bool in the change set. + """ + import dojo.risk_acceptance.helper as ra_helper # noqa: PLC0415 -- lazy import, avoids circular dependency + + is_risk_accepted = changes.get("risk_accepted") + if not isinstance(is_risk_accepted, bool): + return + if (is_risk_accepted and not finding.risk_accepted + and finding.test.engagement.product.enable_simple_risk_acceptance + and not changes.get("active")): + ra_helper.simple_risk_accept(user, finding) + elif not is_risk_accepted and finding.risk_accepted: + ra_helper.risk_unaccept(user, finding) + + +def _resolve_user(pk: int) -> Dojo_User: + user = get_object_or_none(Dojo_User, pk=pk) + if user is None: + raise ValidationError({"reporter": [f"user {pk} does not exist"]}) + return user + + +def _resolve_test_types(ids: list[int]) -> list[Test_Type]: + resolved = [] + for pk in ids: + tt = get_object_or_none(Test_Type, pk=pk) + if tt is None: + raise ValidationError({"found_by": [f"Test_Type {pk} does not exist"]}) + resolved.append(tt) + return resolved + + +# --- public service API (I6) ------------------------------------------------------------------ + +def create_finding(*, test: Test, data: dict, user, push_to_jira: bool = False, + vulnerability_ids: list[str] | None = None, + cwes: list[int] | None = None) -> Finding: + """ + Create a finding under ``test``. Ports ``FindingCreateSerializer.create``/``validate``: + reporter defaulting, status invariants, vulnerability-id + CWE persistence, found_by, JIRA push, + and the ``finding_added`` notification. + + ``cwes`` is a flat ``list[int]`` symmetric with the read shape and parallel to + ``vulnerability_ids`` (§12): an explicit scalar ``cwe`` wins as the primary; when only ``cwes`` + is supplied its first entry is mirrored into ``cwe`` before save (so the primary persists, like + ``cve``). The persisted ``Finding_CWE`` rows are the primary first then the rest (``save_cwes``). + """ + data = {k: v for k, v in data.items() if k not in _SPECIAL_KEYS or k in {"reporter", "mitigated_by", "found_by", "tags"}} + + _validate_severity(data.get("severity")) + _validate_mitigated_editable(data, user) + _validate_status_invariants( + is_active=data.get("active"), + is_verified=data.get("verified"), + is_duplicate=data.get("duplicate"), + is_false_p=data.get("false_p"), + is_risk_accepted=data.get("risk_accepted"), + product=test.engagement.product, + was_risk_accepted=False, + ) + + reporter_id = data.pop("reporter", None) + reporter = _resolve_user(reporter_id) if reporter_id is not None else user + mitigated_by_id = data.pop("mitigated_by", None) + mitigated_by = _resolve_user(mitigated_by_id) if mitigated_by_id is not None else None + found_by_ids = data.pop("found_by", None) + tags = data.pop("tags", None) + + scalars = {k: v for k, v in data.items() if v is not None} + # Mirror the serializer: the first vuln id is written into `cve` *before* the initial save so it + # is persisted (save_vulnerability_ids below only sets it in memory). + if vulnerability_ids: + scalars["cve"] = vulnerability_ids[0] + # CWE list precedence (mirror vulnerability_ids -> cve): an explicit scalar `cwe` is the primary; + # when only `cwes` is supplied, mirror its first entry into `cwe` before the initial save so the + # primary persists. `"cwe" not in scalars` means the scalar was omitted/None on this create. + if cwes and "cwe" not in scalars: + scalars["cwe"] = cwes[0] + new_finding = Finding(test=test, reporter=reporter, **scalars) + if mitigated_by is not None: + new_finding.mitigated_by = mitigated_by + # Persist vuln ids so model save computes the hash including them (mirror the serializer). + new_finding.unsaved_vulnerability_ids = vulnerability_ids or [] + new_finding.save() + + if found_by_ids: + new_finding.found_by.set(_resolve_test_types(found_by_ids)) + if vulnerability_ids: + save_vulnerability_ids(new_finding, vulnerability_ids) + # Create always persists the primary Finding.cwe as a Finding_CWE row (mirror serializer); when a + # `cwes` list is supplied, save_cwes persists the full set (primary first, then the rest). `cwes` + # values are plain ints -- finding_cwe_labels/cwe_label accept ints and canonicalize to CWE-. + if cwes is not None: + new_finding.unsaved_cwes = cwes + save_cwes(new_finding) + if tags is not None: + new_finding.tags = tags + new_finding.save() + + if push_to_jira: + jira_services.push(new_finding) + + dojo_dispatch_task( + async_create_notification, + event="finding_added", + title=_("Addition of %s") % new_finding.title, + finding_id=new_finding.id, + description=_('Finding "%s" was added by %s') % (new_finding.title, new_finding.reporter), + url=reverse("view_finding", args=(new_finding.id,)), + icon="exclamation-triangle", + ) + return new_finding + + +def update_finding(finding: Finding, *, changes: dict, user, push_to_jira: bool = False, + vulnerability_ids: list[str] | None = None, + cwes: list[int] | None = None) -> Finding: + """ + Update ``finding`` from a partial ``changes`` dict. Ports ``FindingSerializer.update``/ + ``validate``: mitigated-edit rules, status invariants (PATCH defaults from the instance), + risk-acceptance processing, vulnerability-id + CWE persistence, found_by set/clear, reporter, + and the synchronous JIRA push (force_sync, raising on failure) + keep-in-sync. + + ``cwes`` (flat ``list[int]``, §12): a supplied list drives the ``Finding_CWE`` rows (primary + first, then the rest); an explicit scalar ``cwe`` in the same request stays the primary, else the + first list entry is mirrored into ``cwe`` before save. Omitting ``cwes`` (``None``) leaves the + rows untouched except the existing scalar-``cwe``-change resync (no-reset-on-omit, symmetric with + ``vulnerability_ids``/``reporter``); an explicit ``cwes: []`` clears the extra rows, resyncing to + just the scalar-derived primary. + """ + changes = dict(changes) + + if "severity" in changes: + _validate_severity(changes["severity"]) + _validate_mitigated_editable(changes, user) + _validate_status_invariants( + is_active=changes.get("active", finding.active), + is_verified=changes.get("verified", finding.verified), + is_duplicate=changes.get("duplicate", finding.duplicate), + is_false_p=changes.get("false_p", finding.false_p), + is_risk_accepted=changes.get("risk_accepted", finding.risk_accepted), + product=finding.test.engagement.product, + was_risk_accepted=finding.risk_accepted, + ) + # Runs before the field updates (mirrors validate() -> process_risk_acceptance ordering). + _process_risk_acceptance(finding, changes, user) + + reporter_id = changes.pop("reporter", None) + if reporter_id is not None: + finding.reporter = _resolve_user(reporter_id) + mitigated_by = changes.pop("mitigated_by", _UNSET) + if mitigated_by is not _UNSET: + finding.mitigated_by = _resolve_user(mitigated_by) if mitigated_by is not None else None + found_by_ids = changes.pop("found_by", _UNSET) + tags = changes.pop("tags", _UNSET) + cwe_provided = "cwe" in changes + # Only a non-None scalar `cwe` counts as "explicitly supplied" for the list-vs-scalar precedence. + # A `None` (including PUT's reset-to-default, where `cwe` is always present in the full-replace + # change set) is not an explicit primary, so a `cwes` list still mirrors its first entry. + cwe_explicit = cwe_provided and changes.get("cwe") is not None + + # Persist vuln ids first so model save computes the hash including them (mirror serializer). + if vulnerability_ids: + save_vulnerability_ids(finding, vulnerability_ids) + + if found_by_ids is not _UNSET: + if found_by_ids: + finding.found_by.set(_resolve_test_types(found_by_ids)) + else: + finding.found_by.clear() + + for key, value in changes.items(): + setattr(finding, key, value) + # CWE list precedence (mirror vulnerability_ids -> cve): mirror the first list entry into the + # scalar `cwe` before save so the primary persists, unless an explicit non-None scalar was + # supplied in this request (in which case that scalar stays the primary). + if cwes and not cwe_explicit: + finding.cwe = cwes[0] + finding.save() + + # Resync the Finding_CWE rows (§12). An explicit `cwes` list wins: it drives the rows (primary + # first, then the rest); an explicit empty list clears the extras and resyncs to the primary. + # Otherwise a scalar-only `cwe` change resyncs the rows to just the scalar-derived primary + # (existing behavior); omitting both leaves the rows untouched. + if cwes is not None: + finding.unsaved_cwes = cwes + save_cwes(finding) + elif cwe_provided: + save_cwes(finding) + + if tags is not _UNSET: + finding.tags = tags if tags is not None else [] + finding.save() + + if push_to_jira or jira_services.is_keep_in_sync(finding): + success, message = jira_services.push(finding, force_sync=True) + if not success: + raise ValidationError(message) + return finding + + +def delete_finding(finding: Finding, *, user, push_to_jira: bool | None = None) -> None: + """ + Delete ``finding``. Mirrors the v2 ``FindingViewSet.destroy`` calculation/dedup hooks: the + model's ``Finding.delete()`` runs ``finding_helper.finding_delete`` (dedup reassignment) and + ``perform_product_grading`` (§12). ``user`` is part of the service contract (I6); the model + resolves the acting user via crum. + + ``push_to_jira`` is the v2 tri-state: ``None`` (default) lets the JIRA delete-sync run with its + default semantics — exactly what v2's ``destroy`` passes when the query param is absent. Passing + no value at all would hit ``Finding.delete()``'s suppress-sentinel and silently skip the JIRA + close/reassign (divergence D17, a confirmed v3 regression — see API_V3_DIVERGENCE_ANALYSIS.md). + """ + logger.debug("api_v3 delete_finding id=%s by user=%s", finding.pk, getattr(user, "username", user)) + finding.delete(push_to_jira=push_to_jira) + + +def process_note_added(finding: Finding, note: Notes, *, user) -> None: + """ + Fire the same side-effects as the v2 finding notes ``@action`` create branch + (``dojo/finding/api/views.py`` -- ``finding.last_reviewed`` stamping, ``process_tag_notifications``, + and ``jira_services.add_comment``) after a note has been persisted and linked to ``finding``: + + 1. ``last_reviewed`` / ``last_reviewed_by`` stamping (finding notes only -- engagement/test don't); + 2. @mention notifications -- the **exact** v2 parsing/notification helper is reused, not reimplemented; + 3. JIRA comment sync on the finding's linked issue, else its finding-group issue -- same conditions as v2. + + I6: keyword-only ``user``, no HTTP object in the signature. The v2 @mention helper + (``process_tag_notifications``) needs the request to build the absolute mention URL, so it is read + from crum -- set for the whole request by ``CurrentRequestUserMiddleware``, the same context bridge + ``delete_finding`` relies on. With no request available (a pure non-HTTP call) mentions are skipped + while the other side-effects still fire (§12). + """ + # (1) last_reviewed / last_reviewed_by stamping -- mirrors the v2 finding notes @action exactly. + finding.last_reviewed = note.date + finding.last_reviewed_by = user + finding.save(update_fields=["last_reviewed", "last_reviewed_by", "updated"]) + + # (2) @mention notifications -- reuse the exact v2 parsing/notification code path. + request = get_current_request() + if request is not None: + process_tag_notifications( + request=request, + note=note, + parent_url=request.build_absolute_uri(reverse("view_finding", args=(finding.id,))), + parent_title=f"Finding: {finding.title}", + ) + + # (3) JIRA comment sync -- same conditions as v2 (linked issue, else finding-group issue). + if finding.has_jira_issue: + jira_services.add_comment(finding, note) + elif finding.has_jira_group_issue: + jira_services.add_comment(finding.finding_group, note) diff --git a/dojo/importers/services.py b/dojo/importers/services.py new file mode 100644 index 00000000000..932fefc412d --- /dev/null +++ b/dojo/importers/services.py @@ -0,0 +1,311 @@ +""" +Importer service facade (D7 / §6 OS1). + +The current importer returns a 7-tuple and **no service wrapper exists today** -- the v2 +``ImportScanSerializer`` calls ``DefaultImporter`` directly. This facade is framework-neutral (no +HTTP/DRF context): it constructs the importer options exactly as the v2 serializers do today +(``dojo/api_v2/serializers.py:526-903`` are the reference implementations; they are not modified) +and unpacks the 7-tuple into a structured ``ImportResult``. + +The importer 7-tuple is ``(test, updated_count, new, closed, reactivated, untouched, +test_import)`` for both the importer and the reimporter (the importer always reports +reactivated/untouched as 0). See ``default_importer.py:165`` / ``default_reimporter.py:165``. + +I6: keyword-only args, explicit ``user``, returns a result dataclass, callable without any HTTP +context. In the alpha PR only v3 calls this; v2 migrates onto it in the CONV1 convergence track. +""" +from __future__ import annotations + +from dataclasses import dataclass +from datetime import datetime +from typing import TYPE_CHECKING, Any + +from django.utils import timezone + +from dojo.importers.auto_create_context import AutoCreateContextManager +from dojo.importers.default_importer import DefaultImporter +from dojo.importers.default_reimporter import DefaultReImporter +from dojo.models import Development_Environment, Dojo_User + +if TYPE_CHECKING: + from datetime import date + + from django.core.files.uploadedfile import UploadedFile + + from dojo.models import Engagement, Test, Test_Import + + +@dataclass(frozen=True) +class ImportResult: + + """Structured import/reimport outcome (§4.13).""" + + test: Test + mode_resolved: str # "import" | "reimport" + new: int + closed: int + reactivated: int + untouched: int + test_import: Test_Import | None + close_old_findings: bool # effective value + do_not_reactivate: bool # effective value + + +def _resolve_environment(name: str | None, *, auto_create: bool) -> Development_Environment: + """Mirror ``setup_common_context`` environment resolution (§ v2 serializer).""" + name = name or "Development" + if auto_create: + return Development_Environment.objects.get_or_create(name=name)[0] + try: + return Development_Environment.objects.get(name=name) + except Development_Environment.DoesNotExist: + msg = f"Environment named {name} does not exist." + raise ValueError(msg) + + +def _aware_scan_date(scan_date: date | datetime | None) -> datetime | None: + """Make the scan date timezone-aware exactly as the v2 serializer does.""" + if scan_date is None: + return None + if isinstance(scan_date, datetime): + return timezone.make_aware(scan_date) if timezone.is_naive(scan_date) else scan_date + return timezone.make_aware(datetime.combine(scan_date, datetime.min.time())) + + +def _build_options( + *, + user: Dojo_User, + scan_type: str, + environment: Development_Environment, + scan_date: datetime | None, + minimum_severity: str, + active: bool | None, + verified: bool | None, + tags: list[str] | None, + service: str | None, + version: str | None, + group_by: str | None, + test_title: str | None, + deduplication_execution_mode: str | None, + push_to_jira: bool, + close_old_findings: bool, + close_old_findings_product_scope: bool, + do_not_reactivate: bool, + extra: dict[str, Any], +) -> dict[str, Any]: + options: dict[str, Any] = { + "user": user, + "scan_type": scan_type, + "environment": environment, + "scan_date": scan_date, + "minimum_severity": minimum_severity, + "active": active, + "verified": verified, + "tags": tags, + "service": service, + "version": version, + "group_by": group_by, + "test_title": test_title, + "deduplication_execution_mode": Dojo_User.resolve_deduplication_execution_mode( + user, deduplication_execution_mode, + ), + "push_to_jira": push_to_jira, + "close_old_findings": close_old_findings, + "close_old_findings_product_scope": close_old_findings_product_scope, + "do_not_reactivate": do_not_reactivate, + } + # Drop None values so the importer's own defaults apply (matches serializer behaviour where a + # field absent from initial_data is not forced). + options = {k: v for k, v in options.items() if v is not None} + options.update(extra) + return options + + +def _unpack(result_tuple: tuple, *, mode: str, close_old_findings: bool, do_not_reactivate: bool) -> ImportResult: + test, _updated_count, new, closed, reactivated, untouched, test_import = result_tuple + return ImportResult( + test=test, + mode_resolved=mode, + new=new, + closed=closed, + reactivated=reactivated, + untouched=untouched, + test_import=test_import, + close_old_findings=close_old_findings, + do_not_reactivate=do_not_reactivate, + ) + + +def import_scan( + *, + user: Dojo_User, + scan_file: UploadedFile | None, + scan_type: str, + engagement: Engagement, + environment: str | None = "Development", + auto_create_context: bool = False, + minimum_severity: str = "Info", + active: bool | None = None, + verified: bool | None = None, + close_old_findings: bool = False, + close_old_findings_product_scope: bool = False, + do_not_reactivate: bool = False, + tags: list[str] | None = None, + scan_date: date | datetime | None = None, + service: str | None = None, + version: str | None = None, + test_title: str | None = None, + group_by: str | None = None, + deduplication_execution_mode: str | None = None, + push_to_jira: bool = False, + **extra: Any, +) -> ImportResult: + """Import a scan into a new test on ``engagement`` (mirrors ``ImportScanSerializer.save()``).""" + options = _build_options( + user=user, + scan_type=scan_type, + environment=_resolve_environment(environment, auto_create=auto_create_context), + scan_date=_aware_scan_date(scan_date), + minimum_severity=minimum_severity, + active=active, + verified=verified, + tags=tags, + service=service, + version=version, + group_by=group_by, + test_title=test_title, + deduplication_execution_mode=deduplication_execution_mode, + push_to_jira=push_to_jira, + close_old_findings=close_old_findings, + close_old_findings_product_scope=close_old_findings_product_scope, + do_not_reactivate=do_not_reactivate, + extra=extra, + ) + options["engagement"] = engagement + importer = DefaultImporter(**options) + result_tuple = importer.process_scan(scan_file) + return _unpack(result_tuple, mode="import", close_old_findings=close_old_findings, do_not_reactivate=do_not_reactivate) + + +def reimport_scan( + *, + user: Dojo_User, + scan_file: UploadedFile | None, + test: Test, + scan_type: str | None = None, + environment: str | None = "Development", + auto_create_context: bool = False, + minimum_severity: str = "Info", + active: bool | None = None, + verified: bool | None = None, + close_old_findings: bool = True, + close_old_findings_product_scope: bool = False, + do_not_reactivate: bool = False, + tags: list[str] | None = None, + scan_date: date | datetime | None = None, + service: str | None = None, + version: str | None = None, + test_title: str | None = None, + group_by: str | None = None, + deduplication_execution_mode: str | None = None, + push_to_jira: bool = False, + **extra: Any, +) -> ImportResult: + """Reimport a scan into an existing ``test`` (mirrors ``ReImportScanSerializer.save()``).""" + options = _build_options( + user=user, + scan_type=scan_type or test.test_type.name, + environment=_resolve_environment(environment, auto_create=auto_create_context), + scan_date=_aware_scan_date(scan_date), + minimum_severity=minimum_severity, + active=active, + verified=verified, + tags=tags, + service=service, + version=version, + group_by=group_by, + test_title=test_title, + deduplication_execution_mode=deduplication_execution_mode, + push_to_jira=push_to_jira, + close_old_findings=close_old_findings, + close_old_findings_product_scope=close_old_findings_product_scope, + do_not_reactivate=do_not_reactivate, + extra=extra, + ) + options["test"] = test + options["engagement"] = test.engagement + reimporter = DefaultReImporter(**options) + result_tuple = reimporter.process_scan(scan_file) + return _unpack(result_tuple, mode="reimport", close_old_findings=close_old_findings, do_not_reactivate=do_not_reactivate) + + +def auto_import_scan( + *, + user: Dojo_User, + scan_file: UploadedFile | None, + scan_type: str, + engagement: Engagement | None = None, + test: Test | None = None, + product_name: str | None = None, + engagement_name: str | None = None, + product_type_name: str | None = None, + test_title: str | None = None, + auto_create_context: bool = False, + close_old_findings: bool | None = None, + do_not_reactivate: bool = False, + **kwargs: Any, +) -> ImportResult: + """ + Resolve the target test via ``AutoCreateContextManager`` and dispatch to reimport (existing + test) or import (new). Mirrors ``ReImportScanSerializer`` auto-create semantics: when a brand + new test is created, ``close_old_findings`` is forced False (nothing to compare against). + """ + auto = AutoCreateContextManager() + context: dict[str, Any] = { + "scan_type": scan_type, + "engagement": engagement, + "test": test, + "product_name": product_name, + "engagement_name": engagement_name, + "product_type_name": product_type_name, + "test_title": test_title, + "auto_create_context": auto_create_context, + } + try: + auto.process_import_meta_data_from_dict(context) + context["product"] = auto.get_target_product_if_exists(**context) + context["engagement"] = auto.get_target_engagement_if_exists(**context) + target_test = auto.get_target_test_if_exists(**context) + except (ValueError, TypeError) as exc: + raise ValueError(str(exc)) + + if target_test is not None: + return reimport_scan( + user=user, + scan_file=scan_file, + test=target_test, + scan_type=scan_type, + close_old_findings=True if close_old_findings is None else close_old_findings, + do_not_reactivate=do_not_reactivate, + auto_create_context=auto_create_context, + test_title=test_title, + **kwargs, + ) + + if auto_create_context: + resolved_engagement = auto.get_or_create_engagement(**context) + return import_scan( + user=user, + scan_file=scan_file, + scan_type=scan_type, + engagement=resolved_engagement, + # Do not close old findings when creating a brand new test. + close_old_findings=False, + do_not_reactivate=do_not_reactivate, + auto_create_context=auto_create_context, + test_title=test_title, + **kwargs, + ) + + msg = "A test could not be found, and auto_create_context was not enabled to create one." + raise ValueError(msg) diff --git a/dojo/location/api_v3/__init__.py b/dojo/location/api_v3/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/dojo/location/api_v3/routes.py b/dojo/location/api_v3/routes.py new file mode 100644 index 00000000000..779129c34f8 --- /dev/null +++ b/dojo/location/api_v3/routes.py @@ -0,0 +1,216 @@ +""" +Location read routes + finding/asset location sub-resources for API v3 (§4.14, OS4). + +Three router factories (I5), all **read-only** (location lifecycle is import-driven, §4.14). Per D11 +the product location sub-resource is exposed on the wire as ``/assets/{id}/locations`` (the model / +``get_authorized_products`` / ORM paths are not renamed -- §12): + +- ``build_locations_router()`` -> ``GET /locations`` + ``GET /locations/{id}`` +- ``build_finding_locations_router()``-> ``GET /findings/{id}/locations`` (edge rows) +- ``build_asset_locations_router()`` -> ``GET /assets/{id}/locations`` (edge rows) + +Routes are thin (I6): authorize -> filter/plan -> serialize -> shape. RBAC: + +- ``/locations`` mirrors the v2 ``LocationViewSet`` access model **exactly** -- that viewset stacks + ``permission_classes = (IsSuperUser, DjangoModelPermissions)``, i.e. **superuser-only** (verified, + see §12). v3 gates the whole resource behind ``request.user.is_superuser`` (403 otherwise) and + draws rows from ``get_authorized_locations`` (I8, forward-compatible: a downstream distribution can + scope the queryset without a route change). +- the sub-resources use **parent-inherited authorization**: the parent finding/asset is resolved + through ``get_authorized_findings``/``get_authorized_products`` (I8); an unknown *or unauthorized* + parent is a 404 (never leak existence, §4.10). The edge rows are then drawn from the parent's own + reverse manager, so a caller who can see the parent sees its edges. + +These live in the location module (not the finding/asset route factories) so all location code +stays together and the finding/asset factories are untouched (§12). +""" +from __future__ import annotations + +from typing import TYPE_CHECKING + +from django.core.exceptions import PermissionDenied +from ninja import Router +from ninja.constants import NOT_SET + +from dojo.api_v3.csv_export import stream_csv_export +from dojo.api_v3.errors import json_response, not_found_problem +from dojo.api_v3.expand import ( + allowed_field_names, + apply_fields, + parse_fields, + plan, + plan_list_fields, + plan_queryset, + serialize, + serialize_list_row, +) +from dojo.api_v3.filtering import FilterSpec, apply_filters, filter_field, register_filter_spec +from dojo.api_v3.pagination import paginate +from dojo.authorization.roles_permissions import Permissions +from dojo.finding.queries import get_authorized_findings +from dojo.location.api_v3.schemas import ( + AssetLocationListResponse, + FindingLocationListResponse, + LocationDetail, + LocationListResponse, + LocationSlim, + asset_location_edge, + finding_location_edge, +) +from dojo.location.models import Location, LocationFindingReference, LocationProductReference +from dojo.location.queries import get_authorized_locations +from dojo.product.queries import get_authorized_products + +if TYPE_CHECKING: + from collections.abc import Callable + + from django.db.models import QuerySet + from django.http import HttpRequest + +# --- Location filter vocabulary (§4.14) ------------------------------------------------------- + +LOCATION_FILTER_SPEC = register_filter_spec("location", FilterSpec( + model=Location, + filters={ + "type": filter_field("location_type", "exact", "char"), + "name__icontains": filter_field("location_value", "icontains", "char"), + # A location relates to an asset via LocationProductReference (mirrors the v2 LocationFilter + # `product` filter, field_name="products__product"). Distinct: the join can duplicate rows. + "asset": filter_field("products__product", "exact", "number", distinct=True), + }, + orderings={ + "id": "id", + "name": "location_value", + }, + search_fields=["location_value"], +)) + + +def build_locations_router( + *, + schema: type = LocationSlim, + detail_schema: type = LocationDetail, + filter_spec: FilterSpec = LOCATION_FILTER_SPEC, + queryset_hook: Callable | None = None, + auth=NOT_SET, +) -> Router: + """Build the read-only locations router (I5).""" + router = Router(tags=["locations"], auth=auth) + + def _base_queryset(request: HttpRequest) -> QuerySet: + # Mirror v2 LocationViewSet: superuser-only (IsSuperUser). Everyone else -> 403 (§12). + if not request.user.is_superuser: + raise PermissionDenied + qs = get_authorized_locations("view", user=request.user) + if queryset_hook is not None: + qs = queryset_hook(qs, request) + return qs + + @router.get("/locations", response=LocationListResponse, url_name="locations_list") + def list_locations(request: HttpRequest): + filtered = apply_filters(request, _base_queryset(request), filter_spec) + + expand_tree, select_related, prefetch = plan(schema, request.GET.get("expand")) + # ?fields= may opt up into the detail field set (§4.7 Part A); the URL-subtype detail fields + # are read through the `url` reverse-O2O (not own-model columns) so nothing is deferrable, but + # the join is added only when such a field is requested (Part B / DETAIL_SELECT_RELATED). + fields = parse_fields(request.GET.get("fields"), allowed_field_names(detail_schema)) + fplan = plan_list_fields(schema, detail_schema, fields) + page_qs = filtered.select_related(*schema.SELECT_RELATED, *fplan.select_related).prefetch_related(*schema.PREFETCH_RELATED) + page_qs = plan_queryset(page_qs, select_related, prefetch) + if fplan.defer: + page_qs = page_qs.defer(*fplan.defer) + + def serialize_row(obj: object) -> dict: + return serialize_list_row(obj, fplan, expand_tree) + + envelope = paginate(request, count_qs=filtered, page_qs=page_qs, serialize=serialize_row, filter_spec=filter_spec) + return json_response(envelope) + + # Literal ``export.csv`` cannot match the ``{int:location_id}`` detail route (int converter), so + # no collision. ``_base_queryset`` enforces the superuser gate (403 for non-superusers) exactly + # like the list/detail routes before any streaming begins (§12 OS4); whole filtered set as CSV. + @router.get("/locations/export.csv", url_name="locations_export_csv") + def export_locations(request: HttpRequest): + filtered = apply_filters(request, _base_queryset(request), filter_spec) + fields = parse_fields(request.GET.get("fields"), allowed_field_names(detail_schema)) + fplan = plan_list_fields(schema, detail_schema, fields) + page_qs = filtered.select_related(*schema.SELECT_RELATED, *fplan.select_related).prefetch_related(*schema.PREFETCH_RELATED) + if fplan.defer: + page_qs = page_qs.defer(*fplan.defer) + return stream_csv_export(request, resource="locations", count_qs=filtered, page_qs=page_qs, plan=fplan) + + @router.get("/locations/{int:location_id}", response=detail_schema, url_name="locations_detail") + def get_location(request: HttpRequest, location_id: int): + expand_tree, select_related, prefetch = plan(detail_schema, request.GET.get("expand")) + qs = _base_queryset(request).select_related(*detail_schema.SELECT_RELATED).prefetch_related(*detail_schema.PREFETCH_RELATED) + qs = plan_queryset(qs, select_related, prefetch) + obj = qs.filter(pk=location_id).first() + if obj is None: + msg = f"Location {location_id} not found" + raise not_found_problem(msg) + fields = parse_fields(request.GET.get("fields"), allowed_field_names(detail_schema)) + return json_response(apply_fields(serialize(obj, detail_schema, expand_tree), fields)) + + return router + + +def build_finding_locations_router(*, queryset_hook: Callable | None = None, auth=NOT_SET) -> Router: + """ + Build the ``GET /findings/{id}/locations`` sub-resource router (I5). Edge rows carry the location + ref + edge ``status``/``audit_time``/``auditor`` (§4.14). ``select_related("location", "auditor")`` + keeps the query count constant regardless of the number of edges. + """ + router = Router(tags=["findings"], auth=auth) + + @router.get( + "/findings/{int:finding_id}/locations", + response=FindingLocationListResponse, + url_name="finding_locations_list", + ) + def list_finding_locations(request: HttpRequest, finding_id: int): + findings = get_authorized_findings(Permissions.Finding_View, user=request.user) + if queryset_hook is not None: + findings = queryset_hook(findings, request) + finding = findings.filter(pk=finding_id).first() + if finding is None: + # 404 for unknown *or unauthorized* parent -- parent-inherited authorization (§4.10). + msg = f"Finding {finding_id} not found" + raise not_found_problem(msg) + + edges = LocationFindingReference.objects.filter(finding=finding).order_by("id") + page_qs = edges.select_related("location", "auditor") + envelope = paginate(request, count_qs=edges, page_qs=page_qs, serialize=finding_location_edge) + return json_response(envelope) + + return router + + +def build_asset_locations_router(*, queryset_hook: Callable | None = None, auth=NOT_SET) -> Router: + """ + Build the ``GET /assets/{id}/locations`` sub-resource router (I5). Edge rows carry the location + ref + edge ``status`` only -- ``LocationProductReference`` has no audit columns (§12). + ``select_related("location")`` keeps the query count constant. + """ + router = Router(tags=["assets"], auth=auth) + + @router.get( + "/assets/{int:asset_id}/locations", + response=AssetLocationListResponse, + url_name="asset_locations_list", + ) + def list_asset_locations(request: HttpRequest, asset_id: int): + assets = get_authorized_products(Permissions.Product_View, user=request.user) + if queryset_hook is not None: + assets = queryset_hook(assets, request) + asset = assets.filter(pk=asset_id).first() + if asset is None: + msg = f"Asset {asset_id} not found" + raise not_found_problem(msg) + + edges = LocationProductReference.objects.filter(product=asset).order_by("id") + page_qs = edges.select_related("location") + envelope = paginate(request, count_qs=edges, page_qs=page_qs, serialize=asset_location_edge) + return json_response(envelope) + + return router diff --git a/dojo/location/api_v3/schemas.py b/dojo/location/api_v3/schemas.py new file mode 100644 index 00000000000..9e9196620a7 --- /dev/null +++ b/dojo/location/api_v3/schemas.py @@ -0,0 +1,214 @@ +""" +Location response schemas + the finding/asset location edge schemas for API v3 (§4.5, §4.14, OS4). + +``LocationSlim`` (list) and ``LocationDetail`` (retrieve). Every schema is a named, importable, +subclassable ninja Schema (I4) and declares (as ``ClassVar`` so pydantic does not treat them as +fields) ``django_model`` / ``SELECT_RELATED`` / ``PREFETCH_RELATED`` / ``EXPANDABLE`` -- the same +contract the kernel expand planner reads from every resource schema. + +Locations are **read-only** in alpha (lifecycle is import-driven, §4.14). Only the ``URL`` location +subtype is persistable today (D5), so ``LocationDetail`` adds the URL-subtype fields +(``protocol/host/port/path/query/fragment``) pulled from the ``url`` reverse one-to-one; for a +non-URL location (none exist in alpha) those fields render ``null``. + +``FindingLocationEdge`` / ``AssetLocationEdge`` document the edge-row shape of the +``/findings/{id}/locations`` and ``/assets/{id}/locations`` sub-resources for OpenAPI (I1/I4); +their runtime serialization is manual dicts (like the list envelopes) so ``LocationRef`` is emitted +with its ``type`` field. (Per D11 the product location sub-resource is exposed on the wire as +``/assets/{id}/locations``; the model/module paths are not renamed -- §12.) +""" +from __future__ import annotations + +import datetime # noqa: TC003 -- runtime import: pydantic resolves the schema field types +from typing import TYPE_CHECKING, ClassVar + +from django.core.exceptions import ObjectDoesNotExist +from ninja import Schema + +# ``LocationRef``/``Ref`` are pydantic field types (runtime-resolved); ``to_location_ref``/``to_ref`` +# back the edge-row serializers below (runtime) -- co-located with the edge schemas, mirroring the +# finding module's ``_finding_location_edges``. +from dojo.api_v3.refs import LocationRef, Ref, to_location_ref, to_ref +from dojo.location.models import Location + +if TYPE_CHECKING: + # Only referenced in a ClassVar annotation (never a pydantic field), so a typing-only import. + from dojo.api_v3.expand import ExpandRel + +__all__ = [ + "AssetLocationEdge", + "AssetLocationListResponse", + "FindingLocationEdge", + "FindingLocationListResponse", + "LocationDetail", + "LocationListResponse", + "LocationSlim", + "asset_location_edge", + "finding_location_edge", +] + + +def _url_of(location: Location): + """ + The location's ``URL`` subtype (reverse one-to-one), or ``None`` for a non-URL location. The + reverse one-to-one accessor raises ``RelatedObjectDoesNotExist`` (an ``ObjectDoesNotExist`` + subtype) when absent; catch it so non-URL locations render null URL fields. Loaded via + ``select_related("url")`` on the detail fetch so this issues no extra query. + """ + try: + return location.url + except ObjectDoesNotExist: + return None + + +class LocationSlim(Schema): + django_model: ClassVar = Location + SELECT_RELATED: ClassVar[tuple] = () + PREFETCH_RELATED: ClassVar[tuple] = ("tags",) + EXPANDABLE: ClassVar[dict[str, ExpandRel]] = {} + + id: int + name: str + type: str + tags: list[str] + + @staticmethod + def resolve_name(obj) -> str: + return obj.location_value + + @staticmethod + def resolve_type(obj) -> str: + return obj.location_type + + @staticmethod + def resolve_tags(obj) -> list[str]: + return [t.name for t in obj.tags.all()] + + +class LocationDetail(LocationSlim): + + """Slim + the URL-subtype detail fields (§4.5, §4.14). Retrieve/expand only; list is slim.""" + + # The URL subtype is a reverse one-to-one; load it in the detail fetch so the resolvers below + # issue no per-object query. + SELECT_RELATED: ClassVar[tuple] = ("url",) + # The URL-subtype fields are read through the ``url`` reverse-O2O (not own-model columns, so never + # deferrable); when a LIST ``?fields=`` opts up into any of them the kernel adds this fixed join + # so the resolvers issue no per-row query (§4.7 Part A). + DETAIL_SELECT_RELATED: ClassVar[dict[str, tuple[str, ...]]] = dict.fromkeys( + ("protocol", "host", "port", "path", "query", "fragment"), ("url",), + ) + + protocol: str | None + host: str | None + port: int | None + path: str | None + query: str | None + fragment: str | None + + @staticmethod + def resolve_protocol(obj) -> str | None: + url = _url_of(obj) + return url.protocol if url is not None else None + + @staticmethod + def resolve_host(obj) -> str | None: + url = _url_of(obj) + return url.host if url is not None else None + + @staticmethod + def resolve_port(obj) -> int | None: + url = _url_of(obj) + return url.port if url is not None else None + + @staticmethod + def resolve_path(obj) -> str | None: + url = _url_of(obj) + return url.path if url is not None else None + + @staticmethod + def resolve_query(obj) -> str | None: + url = _url_of(obj) + return url.query if url is not None else None + + @staticmethod + def resolve_fragment(obj) -> str | None: + url = _url_of(obj) + return url.fragment if url is not None else None + + +# --- Sub-resource edge schemas (OpenAPI documentation of the manual dict shapes) -------------- + +class FindingLocationEdge(Schema): + + """One row of ``GET /findings/{id}/locations`` (§4.14): location ref + edge status/audit.""" + + location: LocationRef + status: str + audit_time: datetime.datetime | None + auditor: Ref | None + + +class AssetLocationEdge(Schema): + + """ + One row of ``GET /assets/{id}/locations`` (§4.14): location ref + edge status. + ``LocationProductReference`` has no ``audit_time``/``auditor`` columns, so the asset edge + carries only ``status`` (§12). + """ + + location: LocationRef + status: str + + +class LocationListResponse(Schema): + + """OpenAPI documentation of the ``/locations`` list envelope (I1); serialization is manual.""" + + count: int + next: str | None + previous: str | None + results: list[LocationSlim] + meta: dict | None = None + + +class FindingLocationListResponse(Schema): + + """OpenAPI documentation of the ``/findings/{id}/locations`` envelope (I1).""" + + count: int + next: str | None + previous: str | None + results: list[FindingLocationEdge] + meta: dict | None = None + + +class AssetLocationListResponse(Schema): + + """OpenAPI documentation of the ``/assets/{id}/locations`` envelope (I1).""" + + count: int + next: str | None + previous: str | None + results: list[AssetLocationEdge] + meta: dict | None = None + + +# --- Edge-row serializers (manual dict shape; co-located with the edge schemas) --------------- + +def finding_location_edge(ref) -> dict: + """Serialize a ``LocationFindingReference`` edge row (§4.14): location ref + status/audit/auditor.""" + return { + "location": to_location_ref(ref.location), + "status": ref.status, + "audit_time": ref.audit_time, + "auditor": to_ref(ref.auditor), + } + + +def asset_location_edge(ref) -> dict: + """Serialize a ``LocationProductReference`` edge row (§4.14): location ref + status (no audit).""" + return { + "location": to_location_ref(ref.location), + "status": ref.status, + } diff --git a/dojo/product/api_v3/__init__.py b/dojo/product/api_v3/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/dojo/product/api_v3/routes.py b/dojo/product/api_v3/routes.py new file mode 100644 index 00000000000..4709afb35d5 --- /dev/null +++ b/dojo/product/api_v3/routes.py @@ -0,0 +1,331 @@ +""" +Asset CRUD routes for API v3 (§4.5, §4.9, §4.11, OS3a; renamed per D11). + +**D11 wire rename:** the ``Product`` model is exposed on the wire as ``asset`` and its parent +``Product_Type`` FK as ``organization`` -- paths (``/assets``), the OpenAPI tag, the filter registry +name, the schema classes and the write FK field (``organization`` -> model ``prod_type``) all use the +new domain language. The Django model / DB table / ``dojo/product/`` module path are **not** renamed +(see §12); ORM field paths, ``get_authorized_products`` and the ``Product_*`` permission enums keep +their names (they point at the real model). The wire field ``asset_manager`` maps to the model's +``product_manager`` FK (relabel term "Asset Manager"); ``technical_contact``/``team_manager`` are +unchanged (no product token). See §12. + +``build_assets_router()`` is a router factory (I5), same signature style as ``build_findings_router``. +Routes are thin (I6): authorize -> filter -> plan queryset -> serialize -> shape. RBAC flows only +through ``get_authorized_products`` for reads (I8) and the v2 ``user_has_permission`` semantics for +writes, mirroring the v2 ``UserHasProductPermission`` class: + +- create (POST): ``add`` permission on the target organization (``prod_type``) in the payload +- update (PATCH): object ``edit`` permission, plus ``add`` on a *reassigned* organization + (mirrors ``check_update_permission(request, obj, "add", "prod_type")``) +- delete (DELETE): object ``delete`` permission (staff-only for non-staff members, legacy model) +- read: object ``view`` via the authorized queryset (404 for unknown-or-unauthorized) + +Deletion mirrors the v2 ``ProductViewSet.destroy`` exactly: async delete when +``ASYNC_OBJECT_DELETE`` is set, else a synchronous delete inside ``Endpoint.allow_endpoint_init()`` +(§12). Relations are referenced by integer id (§4.11). +""" +from __future__ import annotations + +from typing import TYPE_CHECKING + +from django.conf import settings +from django.core.exceptions import PermissionDenied +from django.core.exceptions import ValidationError as DjangoValidationError +from django.http import HttpResponse +from ninja import Router, Schema +from ninja.constants import NOT_SET + +from dojo.api_v3.csv_export import stream_csv_export +from dojo.api_v3.errors import json_response, not_found_problem, validation_problem +from dojo.api_v3.expand import ( + allowed_field_names, + apply_fields, + parse_fields, + plan, + plan_list_fields, + plan_queryset, + serialize, + serialize_list_row, +) +from dojo.api_v3.filtering import ( + FilterSpec, + apply_filters, + filter_field, + register_filter_spec, +) +from dojo.api_v3.include import apply_includes +from dojo.api_v3.pagination import paginate +from dojo.authorization.authorization import user_has_permission +from dojo.authorization.roles_permissions import Permissions +from dojo.models import Dojo_User, Endpoint, Product, Product_Type, SLA_Configuration +from dojo.product.api_v3.schemas import ( + AssetDetail, + AssetReplace, + AssetSlim, + AssetUpdate, + AssetWrite, +) +from dojo.product.queries import get_authorized_products +from dojo.utils import async_delete, get_object_or_none, get_setting + +if TYPE_CHECKING: + from collections.abc import Callable + + from django.db.models import QuerySet + from django.http import HttpRequest + +# --- Asset filter vocabulary (§4.9, minimal set) ---------------------------------------------- + +ASSET_FILTER_SPEC = register_filter_spec("asset", FilterSpec( + model=Product, + filters={ + "id__in": filter_field("id", "in", "number"), + "name__icontains": filter_field("name", "icontains", "char"), + "organization": filter_field("prod_type", "exact", "number"), + "organization__in": filter_field("prod_type", "in", "number"), + "created__gte": filter_field("created", "gte", "datetime"), + "created__lte": filter_field("created", "lte", "datetime"), + "updated__gte": filter_field("updated", "gte", "datetime"), + "updated__lte": filter_field("updated", "lte", "datetime"), + }, + orderings={ + "id": "id", + "name": "name", + "created": "created", + "updated": "updated", + }, + search_fields=["name", "description"], +)) + +# Wire write-field -> model attribute for the user-role FKs. ``asset_manager`` is the D11 wire name +# for the model's ``product_manager`` FK; the other two are unchanged (no product token). +_USER_FK_FIELDS = { + "asset_manager": "product_manager", + "technical_contact": "technical_contact", + "team_manager": "team_manager", +} + +# Sentinel distinguishing "tags omitted" from "tags set to null/empty" on PATCH. +_UNSET = object() + + +class AssetListResponse(Schema): + + """OpenAPI documentation of the list envelope (I1); runtime serialization is manual.""" + + count: int + next: str | None + previous: str | None + results: list[AssetSlim] + meta: dict | None = None + + +def _base_queryset(request: HttpRequest, queryset_hook: Callable | None) -> QuerySet: + qs = get_authorized_products(Permissions.Product_View, user=request.user) + if queryset_hook is not None: + qs = queryset_hook(qs, request) + return qs + + +def _validation_from_django(exc: DjangoValidationError) -> Exception: + if hasattr(exc, "message_dict"): + return validation_problem({k: list(v) for k, v in exc.message_dict.items()}) + return validation_problem({"non_field_errors": list(exc.messages)}) + + +def _resolve_user_fk(field: str, pk: int) -> Dojo_User: + user = get_object_or_none(Dojo_User, pk=pk) + if user is None: + raise validation_problem({field: [f"user {pk} does not exist"]}) + return user + + +def _destroy(instance: Product) -> None: + """Mirror v2 ``ProductViewSet.destroy`` exactly (§12).""" + if get_setting("ASYNC_OBJECT_DELETE"): + async_delete().delete(instance) + else: + with Endpoint.allow_endpoint_init(): # TODO: remove after full move to Locations (mirrors v2) + instance.delete() + + +def _apply_optional_relations_and_scalars(instance: Product, data: dict) -> None: + """Apply the user-FK, SLA and scalar fields present in ``data`` (create + update share this).""" + for wire_field, model_attr in _USER_FK_FIELDS.items(): + if wire_field in data: + pk = data.pop(wire_field) + setattr(instance, model_attr, _resolve_user_fk(wire_field, pk) if pk is not None else None) + if "sla_configuration" in data: + pk = data.pop("sla_configuration") + if pk is not None: + sla = get_object_or_none(SLA_Configuration, pk=pk) + if sla is None: + raise validation_problem({"sla_configuration": [f"SLA configuration {pk} does not exist"]}) + instance.sla_configuration = sla + for key, value in data.items(): + setattr(instance, key, value) + + +def build_assets_router( + *, + schema: type = AssetSlim, + detail_schema: type = AssetDetail, + filter_spec: FilterSpec = ASSET_FILTER_SPEC, + queryset_hook: Callable | None = None, + auth=NOT_SET, +) -> Router: + """Build the assets router (I5).""" + router = Router(tags=["assets"], auth=auth) + + @router.get("/assets", response=AssetListResponse, url_name="assets_list") + def list_assets(request: HttpRequest): + filtered = apply_filters(request, _base_queryset(request, queryset_hook), filter_spec) + + expand_tree, select_related, prefetch = plan(schema, request.GET.get("expand")) + # ?fields= may opt up into the detail field set (§4.7 Part A); defer the heavy detail + # columns not requested (Part B). + fields = parse_fields(request.GET.get("fields"), allowed_field_names(detail_schema)) + fplan = plan_list_fields(schema, detail_schema, fields) + page_qs = filtered.select_related(*schema.SELECT_RELATED, *fplan.select_related).prefetch_related(*schema.PREFETCH_RELATED) + page_qs = plan_queryset(page_qs, select_related, prefetch) + if fplan.defer: + page_qs = page_qs.defer(*fplan.defer) + + def serialize_row(obj: object) -> dict: + return serialize_list_row(obj, fplan, expand_tree) + + envelope = paginate(request, count_qs=filtered, page_qs=page_qs, serialize=serialize_row, filter_spec=filter_spec) + include_meta = apply_includes(request, filtered, allowed=set()) + if include_meta: + envelope.setdefault("meta", {}).update(include_meta) + return json_response(envelope) + + # Literal ``export.csv`` cannot match the ``{int:asset_id}`` detail route (int converter), so no + # collision. Same filter contract as the list; whole filtered set streamed as CSV (§4.15). + @router.get("/assets/export.csv", url_name="assets_export_csv") + def export_assets(request: HttpRequest): + filtered = apply_filters(request, _base_queryset(request, queryset_hook), filter_spec) + fields = parse_fields(request.GET.get("fields"), allowed_field_names(detail_schema)) + fplan = plan_list_fields(schema, detail_schema, fields) + page_qs = filtered.select_related(*schema.SELECT_RELATED, *fplan.select_related).prefetch_related(*schema.PREFETCH_RELATED) + if fplan.defer: + page_qs = page_qs.defer(*fplan.defer) + return stream_csv_export(request, resource="assets", count_qs=filtered, page_qs=page_qs, plan=fplan) + + @router.get("/assets/{int:asset_id}", response=detail_schema, url_name="assets_detail") + def get_asset(request: HttpRequest, asset_id: int): + expand_tree, select_related, prefetch = plan(detail_schema, request.GET.get("expand")) + qs = _base_queryset(request, queryset_hook).select_related(*detail_schema.SELECT_RELATED).prefetch_related(*detail_schema.PREFETCH_RELATED) + qs = plan_queryset(qs, select_related, prefetch) + obj = qs.filter(pk=asset_id).first() + if obj is None: + msg = f"Asset {asset_id} not found" + raise not_found_problem(msg) + fields = parse_fields(request.GET.get("fields"), allowed_field_names(detail_schema)) + return json_response(apply_fields(serialize(obj, detail_schema, expand_tree), fields)) + + @router.post("/assets", response=detail_schema, url_name="assets_create") + def create_asset(request: HttpRequest, payload: AssetWrite): + data = payload.dict() + tags = data.pop("tags") + organization_id = data.pop("organization") + # Mirror check_post_permission(request, Product_Type, "prod_type", "add"): 404 if the target + # organization doesn't exist, 403 if the user can't add assets to it. + prod_type = get_object_or_none(Product_Type, pk=organization_id) + if prod_type is None: + msg = f"Organization {organization_id} not found" + raise not_found_problem(msg) + if not user_has_permission(request.user, prod_type, Permissions.Product_Type_Add_Product): + raise PermissionDenied + instance = Product(prod_type=prod_type) + # Drop unset (None) scalars so the model field defaults apply on create. + _apply_optional_relations_and_scalars(instance, {k: v for k, v in data.items() if v is not None}) + if tags is not None: + instance.tags = tags + try: + instance.save() + except DjangoValidationError as exc: + raise _validation_from_django(exc) from exc + return json_response(serialize(instance, detail_schema, {}), status=201) + + @router.patch("/assets/{int:asset_id}", response=detail_schema, url_name="assets_update") + def update_asset(request: HttpRequest, asset_id: int, payload: AssetUpdate): + instance = _base_queryset(request, queryset_hook).filter(pk=asset_id).first() + if instance is None: + msg = f"Asset {asset_id} not found" + raise not_found_problem(msg) # 404: unknown or unauthorized-to-view + if not user_has_permission(request.user, instance, Permissions.Product_Edit): + raise PermissionDenied # 403: visible but not editable + + data = payload.dict(exclude_unset=True) + tags = data.pop("tags", _UNSET) + if "organization" in data: + new_org_id = data.pop("organization") + # Mirror check_update_permission: only re-check when the FK actually changes. + if new_org_id is not None and new_org_id != instance.prod_type_id: + new_pt = get_object_or_none(Product_Type, pk=new_org_id) + if new_pt is None: + msg = f"Organization {new_org_id} not found" + raise not_found_problem(msg) + if not user_has_permission(request.user, new_pt, Permissions.Product_Type_Add_Product): + raise PermissionDenied + instance.prod_type = new_pt + + _apply_optional_relations_and_scalars(instance, data) + if tags is not _UNSET: + instance.tags = tags if tags is not None else [] + try: + instance.save() + except DjangoValidationError as exc: + raise _validation_from_django(exc) from exc + return json_response(serialize(instance, detail_schema, {})) + + @router.put("/assets/{int:asset_id}", response=detail_schema, url_name="assets_replace") + def replace_asset(request: HttpRequest, asset_id: int, payload: AssetReplace): + # Full replace (PUT). Validates against the create-shaped AssetReplace (required + # name/description/organization, extra="forbid") and applies payload.dict() WITHOUT + # exclude_unset, so omitted optionals reset to their schema defaults (§4.11). Permission + # ladder identical to PATCH: authorized-view resolve (404) then object Edit (403); the + # required organization is re-authorized only when it actually changes (mirrors PATCH). + instance = _base_queryset(request, queryset_hook).filter(pk=asset_id).first() + if instance is None: + msg = f"Asset {asset_id} not found" + raise not_found_problem(msg) # 404: unknown or unauthorized-to-view + if not user_has_permission(request.user, instance, Permissions.Product_Edit): + raise PermissionDenied # 403: visible but not editable + + data = payload.dict() + tags = data.pop("tags") + new_org_id = data.pop("organization") + if new_org_id != instance.prod_type_id: + new_pt = get_object_or_none(Product_Type, pk=new_org_id) + if new_pt is None: + msg = f"Organization {new_org_id} not found" + raise not_found_problem(msg) + if not user_has_permission(request.user, new_pt, Permissions.Product_Type_Add_Product): + raise PermissionDenied + instance.prod_type = new_pt + + _apply_optional_relations_and_scalars(instance, data) + instance.tags = tags if tags is not None else [] + try: + instance.save() + except DjangoValidationError as exc: + raise _validation_from_django(exc) from exc + return json_response(serialize(instance, detail_schema, {})) + + @router.delete("/assets/{int:asset_id}", url_name="assets_delete") + def delete_asset(request: HttpRequest, asset_id: int): + instance = _base_queryset(request, queryset_hook).filter(pk=asset_id).first() + if instance is None: + msg = f"Asset {asset_id} not found" + raise not_found_problem(msg) + if not user_has_permission(request.user, instance, Permissions.Product_Delete): + raise PermissionDenied + _destroy(instance) + response = HttpResponse(status=204) + response["X-API-Status"] = settings.API_V3_STATUS + return response + + return router diff --git a/dojo/product/api_v3/schemas.py b/dojo/product/api_v3/schemas.py new file mode 100644 index 00000000000..aa2e71c0518 --- /dev/null +++ b/dojo/product/api_v3/schemas.py @@ -0,0 +1,176 @@ +""" +Asset response + write schemas for API v3 (§4.5, §4.11, OS3a; renamed per D11). + +**D11 wire rename:** v3 speaks the new domain language -- the ``Product`` model is exposed on the +wire as ``asset`` and its parent ``Product_Type`` FK as ``organization``. The schema classes are +``Asset*``; the Django model (``Product``) / DB table / module path are deliberately **not** renamed +(the DTO layer is what decouples wire names from models). See §12. + +``AssetSlim`` is the canonical parent slim (relocated here from ``dojo/finding/api_v3`` where OS1 +first defined it -- see §12; the finding module now re-exports this copy). ``AssetDetail`` adds the +documented heavier read fields (§4.5). ``organization`` is an expandable relation (§4.6). + +Write schemas mirror the v2 ``ProductSerializer`` required/optional split: the model requires +``name``, ``description`` and ``prod_type`` (exposed on the wire as ``organization``); everything +else is optional (``sla_configuration`` defaults to the model default when omitted). Relations are +referenced by integer id (§4.11); server-managed fields are never writable; unknown fields are +rejected (``extra="forbid"``). The user-role ref ``asset_manager`` is the wire name for the model's +``product_manager`` FK (the UI relabel's canonical term is "Asset Manager"); ``critical_product`` / +``key_product`` (on ``organization``) keep their model-column names because the relabel itself +retains them (``org.critical_product_label``). See §12. +""" +from __future__ import annotations + +from datetime import datetime # noqa: TC003 -- runtime import: pydantic resolves the schema field types +from typing import ClassVar + +from ninja import Schema + +from dojo.api_v3.expand import ExpandRel +from dojo.api_v3.refs import Ref, to_ref +from dojo.models import Product +from dojo.product_type.api_v3.schemas import OrganizationSlim + + +class AssetSlim(Schema): + django_model: ClassVar = Product + SELECT_RELATED: ClassVar[tuple] = ("prod_type",) + PREFETCH_RELATED: ClassVar[tuple] = ("tags",) + EXPANDABLE: ClassVar[dict[str, ExpandRel]] = {} + + id: int + name: str + description: str | None + organization: Ref + lifecycle: str | None + tags: list[str] + created: datetime | None + updated: datetime | None + + @staticmethod + def resolve_organization(obj) -> dict | None: + return to_ref(obj.prod_type) + + @staticmethod + def resolve_tags(obj) -> list[str]: + return [t.name for t in obj.tags.all()] + + +AssetSlim.EXPANDABLE = { + "organization": ExpandRel(attr="prod_type", path="prod_type", schema=OrganizationSlim), +} + + +class AssetDetail(AssetSlim): + + """Slim + the documented heavier read fields (§4.5). Retrieve returns detail; list returns slim.""" + + # Detail fetch pulls the extra parent FKs so the ref resolvers below issue no extra queries. + SELECT_RELATED: ClassVar[tuple] = ("prod_type", "product_manager", "technical_contact", "team_manager") + # Fixed joins for the user-role refs when a LIST ``?fields=`` opts up into them (§4.7 Part A). The + # wire field ``asset_manager`` maps to the model FK ``product_manager`` (D11), so the join path is + # declared here rather than derived from the wire name; added by the kernel only when requested. + DETAIL_SELECT_RELATED: ClassVar[dict[str, tuple[str, ...]]] = { + "asset_manager": ("product_manager",), + "technical_contact": ("technical_contact",), + "team_manager": ("team_manager",), + } + + business_criticality: str | None + platform: str | None + origin: str | None + external_audience: bool | None + internet_accessible: bool | None + asset_manager: Ref | None + technical_contact: Ref | None + team_manager: Ref | None + + @staticmethod + def resolve_asset_manager(obj) -> dict | None: + return to_ref(obj.product_manager) + + @staticmethod + def resolve_technical_contact(obj) -> dict | None: + return to_ref(obj.technical_contact) + + @staticmethod + def resolve_team_manager(obj) -> dict | None: + return to_ref(obj.team_manager) + + +class AssetWrite(Schema): + + """Create payload (POST). ``name``/``description``/``organization`` required (§6 OS3, mirrors v2).""" + + model_config = {"extra": "forbid"} + + name: str + description: str + organization: int + business_criticality: str | None = None + platform: str | None = None + lifecycle: str | None = None + origin: str | None = None + asset_manager: int | None = None + technical_contact: int | None = None + team_manager: int | None = None + sla_configuration: int | None = None + external_audience: bool | None = None + internet_accessible: bool | None = None + tags: list[str] | None = None + + +class AssetUpdate(Schema): + + """Partial update payload (PATCH). Every field optional; only provided keys are applied.""" + + model_config = {"extra": "forbid"} + + name: str | None = None + description: str | None = None + organization: int | None = None + business_criticality: str | None = None + platform: str | None = None + lifecycle: str | None = None + origin: str | None = None + asset_manager: int | None = None + technical_contact: int | None = None + team_manager: int | None = None + sla_configuration: int | None = None + external_audience: bool | None = None + internet_accessible: bool | None = None + tags: list[str] | None = None + + +class AssetReplace(Schema): + + """ + Full-replace payload (PUT). ``name``/``description``/``organization`` required (create-shaped); + ``organization`` is required and reassignment is re-authorized when it changes (mirrors PATCH). + + A dedicated Replace schema is required because ``AssetWrite`` cannot serve full-replace semantics + (§12): a full replace applies ``payload.dict()`` without ``exclude_unset``, so an omitted + optional resets to the schema default -- and ``external_audience``/``internet_accessible`` are + ``NOT NULL`` boolean columns, so ``AssetWrite``'s create-appropriate ``None`` default (which the + create path drops) would violate the constraint on an existing row. They default to their model + default ``False`` here. The nullable scalar fields reset to ``None``; ``sla_configuration`` (a + non-null FK with a model default) is left unchanged when omitted (the route helper only reassigns + it when a non-null id is supplied). + """ + + model_config = {"extra": "forbid"} + + name: str + description: str + organization: int + business_criticality: str | None = None + platform: str | None = None + lifecycle: str | None = None + origin: str | None = None + asset_manager: int | None = None + technical_contact: int | None = None + team_manager: int | None = None + sla_configuration: int | None = None + external_audience: bool = False + internet_accessible: bool = False + tags: list[str] | None = None diff --git a/dojo/product_type/api_v3/__init__.py b/dojo/product_type/api_v3/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/dojo/product_type/api_v3/routes.py b/dojo/product_type/api_v3/routes.py new file mode 100644 index 00000000000..c9a4919b5d8 --- /dev/null +++ b/dojo/product_type/api_v3/routes.py @@ -0,0 +1,253 @@ +""" +Organization CRUD routes for API v3 (§4.5, §4.9, §4.11, OS3a; renamed per D11). + +**D11 wire rename:** the ``Product_Type`` model is exposed on the wire as ``organization`` -- paths +(``/organizations``), the OpenAPI tag, the filter registry name and the schema classes all use the +new domain language. The Django model / DB table / ``dojo/product_type/`` module path are **not** +renamed (see §12); ORM field paths, ``get_authorized_product_types`` and the ``Product_Type_*`` +permission enums keep their names (they point at the real model). + +``build_organizations_router()`` is a router factory (I5), same signature style as +``build_findings_router``. Routes are thin (I6): authorize -> filter -> plan queryset -> serialize +-> shape. RBAC flows only through ``get_authorized_product_types`` for reads (I8) and the v2 +``user_has_permission``/``user_has_global_permission`` semantics for writes, mirroring the v2 +``UserHasProductTypePermission`` permission class exactly: + +- create (POST): global ``add`` permission (``user_has_global_permission(user, "add")``) +- update (PATCH): object ``edit`` permission +- delete (DELETE): object ``delete`` permission (staff-only for non-staff members, per the legacy model) +- read: object ``view`` via the authorized queryset (404 for unknown-or-unauthorized) + +Deletion mirrors the v2 ``ProductTypeViewSet.destroy`` exactly: async delete when +``ASYNC_OBJECT_DELETE`` is set, else a synchronous delete inside ``Endpoint.allow_endpoint_init()`` +(required while ``V3_FEATURE_LOCATIONS`` is on -- see §12). +""" +from __future__ import annotations + +from typing import TYPE_CHECKING + +from django.conf import settings +from django.core.exceptions import PermissionDenied +from django.core.exceptions import ValidationError as DjangoValidationError +from django.http import HttpResponse +from ninja import Router, Schema +from ninja.constants import NOT_SET + +from dojo.api_v3.csv_export import stream_csv_export +from dojo.api_v3.errors import json_response, not_found_problem, validation_problem +from dojo.api_v3.expand import ( + allowed_field_names, + apply_fields, + parse_fields, + plan, + plan_list_fields, + plan_queryset, + serialize, + serialize_list_row, +) +from dojo.api_v3.filtering import ( + FilterSpec, + apply_filters, + filter_field, + register_filter_spec, +) +from dojo.api_v3.include import apply_includes +from dojo.api_v3.pagination import paginate +from dojo.authorization.authorization import user_has_global_permission, user_has_permission +from dojo.authorization.roles_permissions import Permissions +from dojo.models import Endpoint, Product_Type +from dojo.product_type.api_v3.schemas import ( + OrganizationDetail, + OrganizationSlim, + OrganizationUpdate, + OrganizationWrite, +) +from dojo.product_type.queries import get_authorized_product_types +from dojo.utils import async_delete, get_setting + +if TYPE_CHECKING: + from collections.abc import Callable + + from django.db.models import QuerySet + from django.http import HttpRequest + +# --- Organization filter vocabulary (§4.9, minimal set) --------------------------------------- + +ORGANIZATION_FILTER_SPEC = register_filter_spec("organization", FilterSpec( + model=Product_Type, + filters={ + "id__in": filter_field("id", "in", "number"), + "name__icontains": filter_field("name", "icontains", "char"), + "created__gte": filter_field("created", "gte", "datetime"), + "created__lte": filter_field("created", "lte", "datetime"), + "updated__gte": filter_field("updated", "gte", "datetime"), + "updated__lte": filter_field("updated", "lte", "datetime"), + }, + orderings={ + "id": "id", + "name": "name", + "created": "created", + "updated": "updated", + }, + search_fields=["name", "description"], +)) + + +class OrganizationListResponse(Schema): + + """OpenAPI documentation of the list envelope (I1); runtime serialization is manual.""" + + count: int + next: str | None + previous: str | None + results: list[OrganizationSlim] + meta: dict | None = None + + +def _base_queryset(request: HttpRequest, queryset_hook: Callable | None) -> QuerySet: + # Reads flow only through the authorized queryset (I8). The OS helper resolves the current user + # from crum (its signature takes no user kwarg), exactly as the v2 viewset relies on. + qs = get_authorized_product_types(Permissions.Product_Type_View) + if queryset_hook is not None: + qs = queryset_hook(qs, request) + return qs + + +def _validation_from_django(exc: DjangoValidationError) -> Exception: + """Map a model ``full_clean`` failure onto the field-keyed problem+json contract (§4.10).""" + if hasattr(exc, "message_dict"): + return validation_problem({k: list(v) for k, v in exc.message_dict.items()}) + return validation_problem({"non_field_errors": list(exc.messages)}) + + +def _destroy(instance: Product_Type) -> None: + """Mirror v2 ``ProductTypeViewSet.destroy`` exactly (§12).""" + if get_setting("ASYNC_OBJECT_DELETE"): + async_delete().delete(instance) + else: + with Endpoint.allow_endpoint_init(): # TODO: remove after full move to Locations (mirrors v2) + instance.delete() + + +def build_organizations_router( + *, + schema: type = OrganizationSlim, + detail_schema: type = OrganizationDetail, + filter_spec: FilterSpec = ORGANIZATION_FILTER_SPEC, + queryset_hook: Callable | None = None, + auth=NOT_SET, +) -> Router: + """Build the organizations router (I5).""" + router = Router(tags=["organizations"], auth=auth) + + @router.get("/organizations", response=OrganizationListResponse, url_name="organizations_list") + def list_organizations(request: HttpRequest): + filtered = apply_filters(request, _base_queryset(request, queryset_hook), filter_spec) + + expand_tree, select_related, prefetch = plan(schema, request.GET.get("expand")) + # ?fields= may opt up into the detail field set (§4.7 Part A); defer the heavy detail + # columns not requested (Part B). OrganizationDetail == slim, so both are no-ops here, but + # the wiring is uniform across resources. + fields = parse_fields(request.GET.get("fields"), allowed_field_names(detail_schema)) + fplan = plan_list_fields(schema, detail_schema, fields) + page_qs = filtered.select_related(*schema.SELECT_RELATED, *fplan.select_related).prefetch_related(*schema.PREFETCH_RELATED) + page_qs = plan_queryset(page_qs, select_related, prefetch) + if fplan.defer: + page_qs = page_qs.defer(*fplan.defer) + + def serialize_row(obj: object) -> dict: + return serialize_list_row(obj, fplan, expand_tree) + + envelope = paginate(request, count_qs=filtered, page_qs=page_qs, serialize=serialize_row, filter_spec=filter_spec) + include_meta = apply_includes(request, filtered, allowed=set()) + if include_meta: + envelope.setdefault("meta", {}).update(include_meta) + return json_response(envelope) + + # Literal ``export.csv`` cannot match the ``{int:organization_id}`` detail route (int converter), + # so no collision. Same filter contract as the list; whole filtered set streamed as CSV (§4.15). + @router.get("/organizations/export.csv", url_name="organizations_export_csv") + def export_organizations(request: HttpRequest): + filtered = apply_filters(request, _base_queryset(request, queryset_hook), filter_spec) + fields = parse_fields(request.GET.get("fields"), allowed_field_names(detail_schema)) + fplan = plan_list_fields(schema, detail_schema, fields) + page_qs = filtered.select_related(*schema.SELECT_RELATED, *fplan.select_related).prefetch_related(*schema.PREFETCH_RELATED) + if fplan.defer: + page_qs = page_qs.defer(*fplan.defer) + return stream_csv_export(request, resource="organizations", count_qs=filtered, page_qs=page_qs, plan=fplan) + + @router.get("/organizations/{int:organization_id}", response=detail_schema, url_name="organizations_detail") + def get_organization(request: HttpRequest, organization_id: int): + expand_tree, select_related, prefetch = plan(detail_schema, request.GET.get("expand")) + qs = _base_queryset(request, queryset_hook).select_related(*detail_schema.SELECT_RELATED).prefetch_related(*detail_schema.PREFETCH_RELATED) + qs = plan_queryset(qs, select_related, prefetch) + obj = qs.filter(pk=organization_id).first() + if obj is None: + msg = f"Organization {organization_id} not found" + raise not_found_problem(msg) + fields = parse_fields(request.GET.get("fields"), allowed_field_names(detail_schema)) + return json_response(apply_fields(serialize(obj, detail_schema, expand_tree), fields)) + + @router.post("/organizations", response=detail_schema, url_name="organizations_create") + def create_organization(request: HttpRequest, payload: OrganizationWrite): + # Mirror UserHasProductTypePermission: POST requires global "add". + if not user_has_global_permission(request.user, "add"): + raise PermissionDenied + instance = Product_Type(**payload.dict()) + try: + instance.save() + except DjangoValidationError as exc: + raise _validation_from_django(exc) from exc + return json_response(serialize(instance, detail_schema, {}), status=201) + + @router.patch("/organizations/{int:organization_id}", response=detail_schema, url_name="organizations_update") + def update_organization(request: HttpRequest, organization_id: int, payload: OrganizationUpdate): + instance = _base_queryset(request, queryset_hook).filter(pk=organization_id).first() + if instance is None: + msg = f"Organization {organization_id} not found" + raise not_found_problem(msg) # 404: unknown or unauthorized-to-view + if not user_has_permission(request.user, instance, Permissions.Product_Type_Edit): + raise PermissionDenied # 403: visible but not editable + for key, value in payload.dict(exclude_unset=True).items(): + setattr(instance, key, value) + try: + instance.save() + except DjangoValidationError as exc: + raise _validation_from_django(exc) from exc + return json_response(serialize(instance, detail_schema, {})) + + @router.put("/organizations/{int:organization_id}", response=detail_schema, url_name="organizations_replace") + def replace_organization(request: HttpRequest, organization_id: int, payload: OrganizationWrite): + # Full replace (PUT). Validates against the create-shaped OrganizationWrite (required + # fields, extra="forbid") and applies payload.dict() WITHOUT exclude_unset, so omitted + # optionals reset to their schema defaults (§4.11). Permission ladder is identical to PATCH: + # authorized-view resolve (404) then object Edit permission (403). + instance = _base_queryset(request, queryset_hook).filter(pk=organization_id).first() + if instance is None: + msg = f"Organization {organization_id} not found" + raise not_found_problem(msg) # 404: unknown or unauthorized-to-view + if not user_has_permission(request.user, instance, Permissions.Product_Type_Edit): + raise PermissionDenied # 403: visible but not editable + for key, value in payload.dict().items(): + setattr(instance, key, value) + try: + instance.save() + except DjangoValidationError as exc: + raise _validation_from_django(exc) from exc + return json_response(serialize(instance, detail_schema, {})) + + @router.delete("/organizations/{int:organization_id}", url_name="organizations_delete") + def delete_organization(request: HttpRequest, organization_id: int): + instance = _base_queryset(request, queryset_hook).filter(pk=organization_id).first() + if instance is None: + msg = f"Organization {organization_id} not found" + raise not_found_problem(msg) + if not user_has_permission(request.user, instance, Permissions.Product_Type_Delete): + raise PermissionDenied + _destroy(instance) + # 204 No Content: an empty body (not "null"); still carry the alpha status header (§4.1). + response = HttpResponse(status=204) + response["X-API-Status"] = settings.API_V3_STATUS + return response + + return router diff --git a/dojo/product_type/api_v3/schemas.py b/dojo/product_type/api_v3/schemas.py new file mode 100644 index 00000000000..cac52b3e0a8 --- /dev/null +++ b/dojo/product_type/api_v3/schemas.py @@ -0,0 +1,77 @@ +""" +Organization response + write schemas for API v3 (§4.5, §4.11, OS3a; renamed per D11). + +**D11 wire rename:** v3 speaks the new domain language -- the ``Product_Type`` model is exposed on +the wire as ``organization``. The schema classes are ``Organization*`` and the Django model +(``Product_Type``) / DB table / module path are deliberately **not** renamed (the DTO layer is what +decouples wire names from models). See §12. + +``OrganizationSlim`` is the canonical parent slim (relocated here from ``dojo/finding/api_v3`` where +OS1 first defined it -- see §12; the finding module now re-exports this copy so there is one class, +not two, keeping expand targets and this resource in lock-step). Every schema is a named, +importable, subclassable ninja ``Schema`` (I4) and declares the ``ClassVar`` machinery the expand +planner reads (``django_model``/``SELECT_RELATED``/``PREFETCH_RELATED``/``EXPANDABLE``). + +Write schemas (``OrganizationWrite`` create, ``OrganizationUpdate`` PATCH) are the editable subset of +the detail fields; required-vs-optional mirrors the v2 ``ProductTypeSerializer`` (a ``ModelSerializer`` +over the model -- ``name`` required, the rest defaulted). Server-managed fields (``id``/``created``/ +``updated``) are never writable, and unknown fields are rejected (``extra="forbid"``) so write +validation is consistent with the kernel's strict query contract (§12). ``critical_product``/ +``key_product`` keep their model-column names (D11 excludes DB columns; the UI relabel itself retains +them, e.g. ``org.critical_product_label`` -- see §12). +""" +from __future__ import annotations + +from datetime import datetime # noqa: TC003 -- runtime import: pydantic resolves the schema field types +from typing import TYPE_CHECKING, ClassVar + +from ninja import Schema + +from dojo.models import Product_Type + +if TYPE_CHECKING: + from dojo.api_v3.expand import ExpandRel + + +class OrganizationSlim(Schema): + django_model: ClassVar = Product_Type + SELECT_RELATED: ClassVar[tuple] = () + PREFETCH_RELATED: ClassVar[tuple] = () + EXPANDABLE: ClassVar[dict[str, ExpandRel]] = {} + + id: int + name: str + description: str | None + critical_product: bool | None + key_product: bool | None + created: datetime | None + updated: datetime | None + + +class OrganizationDetail(OrganizationSlim): + + """Detail (retrieve) shape. Organization has no heavier read fields today, so it equals slim.""" + + +class OrganizationWrite(Schema): + + """Create payload (POST). ``name`` is required; the rest mirror the model defaults (§6 OS3).""" + + model_config = {"extra": "forbid"} + + name: str + description: str | None = None + critical_product: bool = False + key_product: bool = False + + +class OrganizationUpdate(Schema): + + """Partial update payload (PATCH). Every field optional; only provided keys are applied.""" + + model_config = {"extra": "forbid"} + + name: str | None = None + description: str | None = None + critical_product: bool | None = None + key_product: bool | None = None diff --git a/dojo/settings/settings.dist.py b/dojo/settings/settings.dist.py index b590576a62e..ddf47b4e74a 100644 --- a/dojo/settings/settings.dist.py +++ b/dojo/settings/settings.dist.py @@ -288,6 +288,15 @@ DD_REQUESTS_TIMEOUT=(int, 30), # Dictates if v3 functionality will be enabled (on by default as of 3.0.0; set to False to revert to the legacy Endpoint model) DD_V3_FEATURE_LOCATIONS=(bool, True), + # API v3 (alpha) list pagination: threshold below which `count` is exact; above it the + # response reports the Postgres planner's row estimate (flagged count_exact=false). See §4.3. + DD_API_V3_COUNT_CAP=(int, 10000), + # API v3 (alpha) ?expand= guard: maximum number of expanded relation nodes across all paths. See §4.6. + DD_API_V3_EXPAND_BUDGET=(int, 10), + # API v3 (alpha) CSV export row cap: the whole filtered set is streamed as CSV; if the filtered + # count exceeds this cap the request is a 400 telling the client to narrow the filter (never a + # silent truncation). See §4.15. + DD_API_V3_EXPORT_MAX_ROWS=(int, 100000), # Dictates if v3 org/asset relabeling (+url routing) will be enabled (on by default as of 3.0.0; set to False to restore Product/Product Type labels and URLs) DD_ENABLE_V3_ORGANIZATION_ASSET_RELABEL=(bool, True), # Shared cache backend (django.core.cache). When set, Django uses RedisCache @@ -680,6 +689,27 @@ def generate_url(scheme, double_slashes, user, password, host, port, path, param # V3 Feature Flags V3_FEATURE_LOCATIONS = env("DD_V3_FEATURE_LOCATIONS") +# ------------------------------------------------------------------------------ +# API v3 (alpha) +# ------------------------------------------------------------------------------ +# Logical name of the API is /api/v3/; during alpha the actual mount prefix carries the +# instability marker (D1/§4.1). This is the single source of truth for the prefix and version +# string -- do not hardcode them anywhere else. +API_V3_URL_PREFIX = "api/v3-alpha" +API_V3_VERSION = "3.0.0-alpha" +API_V3_STATUS = "alpha" +# Count/expand tuning (§4.3, §4.6); settings-overridable per the plan. +API_V3_COUNT_CAP = env("DD_API_V3_COUNT_CAP") +API_V3_EXPAND_BUDGET = env("DD_API_V3_EXPAND_BUDGET") +# CSV export row cap (§4.15); settings-overridable per the plan. +API_V3_EXPORT_MAX_ROWS = env("DD_API_V3_EXPORT_MAX_ROWS") +# List pagination bounds (§4.3). +API_V3_PAGE_LIMIT_DEFAULT = 25 +API_V3_PAGE_LIMIT_MAX = 250 +# v3 handles its own auth (token + session); exempt it from the UI login-redirect middleware +# exactly as /api/v2/ is (so anonymous requests get a 401 problem+json, not a /login redirect). +LOGIN_EXEMPT_URLS += (rf"^{URL_PREFIX}{API_V3_URL_PREFIX}/",) + # ------------------------------------------------------------------------------ # ADMIN diff --git a/dojo/templates/base.html b/dojo/templates/base.html index 92a6433a169..4da2504b29b 100644 --- a/dojo/templates/base.html +++ b/dojo/templates/base.html @@ -465,8 +465,17 @@ {% endif %} {% endif %} - {% trans "API v2 OpenAPI3 Docs" %} + {% trans "API v2 Docs" %} + {% if V3_FEATURE_LOCATIONS %} + {# v3 mounts conditionally on this flag (D5) — an unguarded url tag would break every page when it is off #} + + {% trans "Open API v3 alpha Docs (Swagger)" %} + + + {% trans "Open API v3 alpha Docs (Scalar)" %} + + {% endif %} {% trans "Documentation" %} diff --git a/dojo/templates_classic/base.html b/dojo/templates_classic/base.html index a8574ef6c01..cc227df9efa 100644 --- a/dojo/templates_classic/base.html +++ b/dojo/templates_classic/base.html @@ -179,10 +179,25 @@ {% endif %}
  • - - {% trans "API v2 OpenAPI3 Docs" %} + + {% trans "API v2 Docs" %}
    • + {% if V3_FEATURE_LOCATIONS %} + {# v3 mounts conditionally on this flag (D5) — an unguarded url tag would break every page when it is off #} +
  • + + + {% trans "Open API v3 alpha Docs (Swagger)" %} + +
    • +
  • + + + {% trans "Open API v3 alpha Docs (Scalar)" %} + +
    • + {% endif %}
  • diff --git a/dojo/test/api_v3/__init__.py b/dojo/test/api_v3/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/dojo/test/api_v3/routes.py b/dojo/test/api_v3/routes.py new file mode 100644 index 00000000000..9c78f4833eb --- /dev/null +++ b/dojo/test/api_v3/routes.py @@ -0,0 +1,331 @@ +""" +Test CRUD routes for API v3 (§4.5, §4.9, §4.11, OS3b). + +``build_tests_router()`` is a router factory (I5), same shape as ``build_assets_router``. Routes +are thin (I6): authorize -> filter -> plan queryset -> serialize -> shape. RBAC flows only through +``get_authorized_tests`` for reads (I8) and the v2 ``user_has_permission`` semantics for writes, +mirroring the v2 ``UserHasTestPermission`` permission class exactly: + +- create (POST): ``add`` permission on the target ``engagement`` referenced in the payload + (404 if it doesn't exist, 403 if unauthorized -- mirrors + ``check_post_permission(request, Engagement, "engagement", "add")``), plus + ``view`` on ``api_scan_configuration`` when present (mirrors the ``required=False`` + sibling check). +- update (PATCH): object ``edit`` permission, plus ``view`` on a *reassigned* + ``api_scan_configuration`` (mirrors + ``check_update_permission(request, obj, "view", "api_scan_configuration")``). + ``engagement`` is not writable on update (``editable=False``, mirrors v2). +- delete (DELETE): object ``delete`` permission (staff-only for non-staff members, legacy model) +- read: object ``view`` via the authorized queryset (404 for unknown-or-unauthorized) + +Deletion mirrors the v2 ``TestsViewSet.destroy`` exactly: async delete when ``ASYNC_OBJECT_DELETE`` +is set, else a plain synchronous ``instance.delete()`` (no ``Endpoint`` context wrapper -- §12). +Relations are referenced by integer id (§4.11). +""" +from __future__ import annotations + +from typing import TYPE_CHECKING + +from django.conf import settings +from django.core.exceptions import PermissionDenied +from django.core.exceptions import ValidationError as DjangoValidationError +from django.http import HttpResponse +from ninja import Router, Schema +from ninja.constants import NOT_SET + +from dojo.api_v3.csv_export import stream_csv_export +from dojo.api_v3.errors import json_response, not_found_problem, validation_problem +from dojo.api_v3.expand import ( + allowed_field_names, + apply_fields, + parse_fields, + plan, + plan_list_fields, + plan_queryset, + serialize, + serialize_list_row, +) +from dojo.api_v3.filtering import ( + FilterSpec, + apply_filters, + filter_field, + register_filter_spec, +) +from dojo.api_v3.include import apply_includes +from dojo.api_v3.pagination import paginate +from dojo.authorization.authorization import user_has_permission +from dojo.authorization.roles_permissions import Permissions +from dojo.models import ( + Development_Environment, + Dojo_User, + Engagement, + Product_API_Scan_Configuration, + Test, + Test_Type, +) +from dojo.test.api_v3.schemas import ( + TestDetail, + TestReplace, + TestSlim, + TestUpdate, + TestWrite, +) +from dojo.test.queries import get_authorized_tests +from dojo.utils import async_delete, get_object_or_none, get_setting + +if TYPE_CHECKING: + from collections.abc import Callable + + from django.db.models import QuerySet + from django.http import HttpRequest + +# --- Test filter vocabulary (§4.9, minimal set) ----------------------------------------------- + +TEST_FILTER_SPEC = register_filter_spec("test", FilterSpec( + model=Test, + filters={ + "id__in": filter_field("id", "in", "number"), + "title__icontains": filter_field("title", "icontains", "char"), + "engagement": filter_field("engagement", "exact", "number"), + "engagement__in": filter_field("engagement", "in", "number"), + "asset": filter_field("engagement__product", "exact", "number"), + "asset__in": filter_field("engagement__product", "in", "number"), + "organization": filter_field("engagement__product__prod_type", "exact", "number"), + "test_type": filter_field("test_type", "exact", "number"), + "environment": filter_field("environment", "exact", "number"), + "lead": filter_field("lead", "exact", "number"), + "target_start__gte": filter_field("target_start", "gte", "datetime"), + "target_start__lte": filter_field("target_start", "lte", "datetime"), + "target_end__gte": filter_field("target_end", "gte", "datetime"), + "target_end__lte": filter_field("target_end", "lte", "datetime"), + "created__gte": filter_field("created", "gte", "datetime"), + "created__lte": filter_field("created", "lte", "datetime"), + "updated__gte": filter_field("updated", "gte", "datetime"), + "updated__lte": filter_field("updated", "lte", "datetime"), + }, + orderings={ + "id": "id", + "title": "title", + "target_start": "target_start", + "created": "created", + "updated": "updated", + }, + search_fields=["title", "description"], +)) + +# Sentinel distinguishing "tags omitted" from "tags set to null/empty" on PATCH. +_UNSET = object() + +# Optional relation fields resolved by integer id (with existence validation -> 400). +_SIMPLE_FK = { + "test_type": Test_Type, + "environment": Development_Environment, + "lead": Dojo_User, +} + + +class TestListResponse(Schema): + + """OpenAPI documentation of the list envelope (I1); runtime serialization is manual.""" + + count: int + next: str | None + previous: str | None + results: list[TestSlim] + meta: dict | None = None + + +def _base_queryset(request: HttpRequest, queryset_hook: Callable | None) -> QuerySet: + qs = get_authorized_tests(Permissions.Test_View) + if queryset_hook is not None: + qs = queryset_hook(qs, request) + return qs + + +def _validation_from_django(exc: DjangoValidationError) -> Exception: + if hasattr(exc, "message_dict"): + return validation_problem({k: list(v) for k, v in exc.message_dict.items()}) + return validation_problem({"non_field_errors": list(exc.messages)}) + + +def _resolve_simple_fk(field: str, pk: int): + """Resolve a body-referenced FK by id (400 if it doesn't exist -- mirrors DRF PK validation).""" + obj = get_object_or_none(_SIMPLE_FK[field], pk=pk) + if obj is None: + raise validation_problem({field: [f"{_SIMPLE_FK[field].__name__} {pk} does not exist"]}) + return obj + + +def _authorize_api_scan_configuration(request: HttpRequest, pk: int) -> Product_API_Scan_Configuration: + """Mirror check_post_permission(..., 'view'): 404 if it doesn't exist, 403 if no view perm.""" + config = get_object_or_none(Product_API_Scan_Configuration, pk=pk) + if config is None: + msg = f"Product_API_Scan_Configuration {pk} not found" + raise not_found_problem(msg) + if not user_has_permission(request.user, config, Permissions.Product_API_Scan_Configuration_View): + raise PermissionDenied + return config + + +def _destroy(instance: Test) -> None: + """Mirror v2 ``TestsViewSet.destroy`` exactly (no Endpoint context wrapper -- §12).""" + if get_setting("ASYNC_OBJECT_DELETE"): + async_delete().delete(instance) + else: + instance.delete() + + +def _apply_relations_and_scalars(request: HttpRequest, instance: Test, data: dict) -> None: + for field in _SIMPLE_FK: + if field in data: + pk = data.pop(field) + setattr(instance, field, _resolve_simple_fk(field, pk) if pk is not None else None) + if "api_scan_configuration" in data: + pk = data.pop("api_scan_configuration") + instance.api_scan_configuration = _authorize_api_scan_configuration(request, pk) if pk is not None else None + for key, value in data.items(): + setattr(instance, key, value) + + +def build_tests_router( + *, + schema: type = TestSlim, + detail_schema: type = TestDetail, + filter_spec: FilterSpec = TEST_FILTER_SPEC, + queryset_hook: Callable | None = None, + auth=NOT_SET, +) -> Router: + """Build the tests router (I5).""" + router = Router(tags=["tests"], auth=auth) + + @router.get("/tests", response=TestListResponse, url_name="tests_list") + def list_tests(request: HttpRequest): + filtered = apply_filters(request, _base_queryset(request, queryset_hook), filter_spec) + + expand_tree, select_related, prefetch = plan(schema, request.GET.get("expand")) + # ?fields= may opt up into the detail field set (§4.7 Part A); defer the heavy detail + # columns not requested (Part B). + fields = parse_fields(request.GET.get("fields"), allowed_field_names(detail_schema)) + fplan = plan_list_fields(schema, detail_schema, fields) + page_qs = filtered.select_related(*schema.SELECT_RELATED, *fplan.select_related).prefetch_related(*schema.PREFETCH_RELATED) + page_qs = plan_queryset(page_qs, select_related, prefetch) + if fplan.defer: + page_qs = page_qs.defer(*fplan.defer) + + def serialize_row(obj: object) -> dict: + return serialize_list_row(obj, fplan, expand_tree) + + envelope = paginate(request, count_qs=filtered, page_qs=page_qs, serialize=serialize_row, filter_spec=filter_spec) + include_meta = apply_includes(request, filtered, allowed=set()) + if include_meta: + envelope.setdefault("meta", {}).update(include_meta) + return json_response(envelope) + + # Literal ``export.csv`` cannot match the ``{int:test_id}`` detail route (int converter), so no + # collision. Same filter contract as the list; whole filtered set streamed as CSV (§4.15). + @router.get("/tests/export.csv", url_name="tests_export_csv") + def export_tests(request: HttpRequest): + filtered = apply_filters(request, _base_queryset(request, queryset_hook), filter_spec) + fields = parse_fields(request.GET.get("fields"), allowed_field_names(detail_schema)) + fplan = plan_list_fields(schema, detail_schema, fields) + page_qs = filtered.select_related(*schema.SELECT_RELATED, *fplan.select_related).prefetch_related(*schema.PREFETCH_RELATED) + if fplan.defer: + page_qs = page_qs.defer(*fplan.defer) + return stream_csv_export(request, resource="tests", count_qs=filtered, page_qs=page_qs, plan=fplan) + + @router.get("/tests/{int:test_id}", response=detail_schema, url_name="tests_detail") + def get_test(request: HttpRequest, test_id: int): + expand_tree, select_related, prefetch = plan(detail_schema, request.GET.get("expand")) + qs = _base_queryset(request, queryset_hook).select_related(*detail_schema.SELECT_RELATED).prefetch_related(*detail_schema.PREFETCH_RELATED) + qs = plan_queryset(qs, select_related, prefetch) + obj = qs.filter(pk=test_id).first() + if obj is None: + msg = f"Test {test_id} not found" + raise not_found_problem(msg) + fields = parse_fields(request.GET.get("fields"), allowed_field_names(detail_schema)) + return json_response(apply_fields(serialize(obj, detail_schema, expand_tree), fields)) + + @router.post("/tests", response=detail_schema, url_name="tests_create") + def create_test(request: HttpRequest, payload: TestWrite): + data = payload.dict() + tags = data.pop("tags") + engagement_id = data.pop("engagement") + # Mirror check_post_permission(request, Engagement, "engagement", "add"): 404 if the target + # engagement doesn't exist, 403 if the user can't add tests to it. + engagement = get_object_or_none(Engagement, pk=engagement_id) + if engagement is None: + msg = f"Engagement {engagement_id} not found" + raise not_found_problem(msg) + if not user_has_permission(request.user, engagement, Permissions.Test_Add): + raise PermissionDenied + + instance = Test(engagement=engagement) + _apply_relations_and_scalars(request, instance, {k: v for k, v in data.items() if v is not None}) + try: + instance.save() + except DjangoValidationError as exc: + raise _validation_from_django(exc) from exc + if tags is not None: + instance.tags = tags + instance.save() + return json_response(serialize(instance, detail_schema, {}), status=201) + + @router.patch("/tests/{int:test_id}", response=detail_schema, url_name="tests_update") + def update_test(request: HttpRequest, test_id: int, payload: TestUpdate): + instance = _base_queryset(request, queryset_hook).filter(pk=test_id).first() + if instance is None: + msg = f"Test {test_id} not found" + raise not_found_problem(msg) # 404: unknown or unauthorized-to-view + if not user_has_permission(request.user, instance, Permissions.Test_Edit): + raise PermissionDenied # 403: visible but not editable + + data = payload.dict(exclude_unset=True) + tags = data.pop("tags", _UNSET) + _apply_relations_and_scalars(request, instance, data) + if tags is not _UNSET: + instance.tags = tags if tags is not None else [] + try: + instance.save() + except DjangoValidationError as exc: + raise _validation_from_django(exc) from exc + return json_response(serialize(instance, detail_schema, {})) + + @router.put("/tests/{int:test_id}", response=detail_schema, url_name="tests_replace") + def replace_test(request: HttpRequest, test_id: int, payload: TestReplace): + # Full replace (PUT). Validates against the create-shaped TestReplace (required + # test_type/target_start/target_end, extra="forbid") and applies payload.dict() WITHOUT + # exclude_unset, so omitted optionals reset to their schema defaults (§4.11). ``engagement`` + # is not in the replace schema -- it is editable=False and never reassigned on update, so + # PUT mirrors PATCH exactly (§12). Permission ladder identical to PATCH: authorized-view + # resolve (404) then object Edit (403); a supplied api_scan_configuration is view-gated. + instance = _base_queryset(request, queryset_hook).filter(pk=test_id).first() + if instance is None: + msg = f"Test {test_id} not found" + raise not_found_problem(msg) # 404: unknown or unauthorized-to-view + if not user_has_permission(request.user, instance, Permissions.Test_Edit): + raise PermissionDenied # 403: visible but not editable + + data = payload.dict() + tags = data.pop("tags") + _apply_relations_and_scalars(request, instance, data) + instance.tags = tags if tags is not None else [] + try: + instance.save() + except DjangoValidationError as exc: + raise _validation_from_django(exc) from exc + return json_response(serialize(instance, detail_schema, {})) + + @router.delete("/tests/{int:test_id}", url_name="tests_delete") + def delete_test(request: HttpRequest, test_id: int): + instance = _base_queryset(request, queryset_hook).filter(pk=test_id).first() + if instance is None: + msg = f"Test {test_id} not found" + raise not_found_problem(msg) + if not user_has_permission(request.user, instance, Permissions.Test_Delete): + raise PermissionDenied + _destroy(instance) + response = HttpResponse(status=204) + response["X-API-Status"] = settings.API_V3_STATUS + return response + + return router diff --git a/dojo/test/api_v3/schemas.py b/dojo/test/api_v3/schemas.py new file mode 100644 index 00000000000..43bd77dddac --- /dev/null +++ b/dojo/test/api_v3/schemas.py @@ -0,0 +1,244 @@ +""" +Test response + write schemas for API v3 (§4.5, §4.11, OS3b). + +``TestSlim`` (plus ``TestTypeSlim`` / ``EnvironmentSlim``) is the canonical Test slim -- relocated +here from ``dojo/finding/api_v3`` (where OS1 first defined it for finding ``?expand=``); the finding +module now re-exports these copies so there is exactly one class per model (verified is-identity in +the tests -- see §12). ``TestDetail`` adds the documented heavier read fields (§4.5). +``test_type``/``engagement``/``asset``/``organization``/``environment``/``lead`` are expandable +relations (§4.6). Per D11 the Product/Product_Type models are exposed on the wire as +``asset``/``organization`` (ref keys + expand keys); the models/DB/module paths are not renamed (§12). + +Write schemas mirror the v2 ``TestCreateSerializer`` (create) and ``TestSerializer`` (update): the +model requires ``engagement``, ``test_type``, ``target_start`` and ``target_end``. ``engagement`` is +``editable=False`` on the model, so it is writable **only on create** (mirrors v2 -- the update +serializer treats it as read-only). Relations are referenced by integer id (§4.11); server-managed / +``editable=False`` fields (``id``, ``created``, ``updated``, ``notes``, ``files``) are never +writable; unknown fields are rejected (``extra="forbid"``). +""" +from __future__ import annotations + +from datetime import datetime # noqa: TC003 -- runtime import: pydantic resolves the schema field types +from typing import ClassVar + +from ninja import Schema + +from dojo.api_v3.expand import ExpandRel +from dojo.api_v3.refs import Ref, to_ref +from dojo.engagement.api_v3.schemas import EngagementSlim +from dojo.models import Development_Environment, Test, Test_Type +from dojo.product.api_v3.schemas import AssetSlim +from dojo.product_type.api_v3.schemas import OrganizationSlim +from dojo.user.api_v3.schemas import UserSlim + + +class TestTypeSlim(Schema): + django_model: ClassVar = Test_Type + SELECT_RELATED: ClassVar[tuple] = () + PREFETCH_RELATED: ClassVar[tuple] = () + EXPANDABLE: ClassVar[dict[str, ExpandRel]] = {} + + id: int + name: str + active: bool | None + + +class EnvironmentSlim(Schema): + django_model: ClassVar = Development_Environment + SELECT_RELATED: ClassVar[tuple] = () + PREFETCH_RELATED: ClassVar[tuple] = () + EXPANDABLE: ClassVar[dict[str, ExpandRel]] = {} + + id: int + name: str + + +class TestSlim(Schema): + django_model: ClassVar = Test + SELECT_RELATED: ClassVar[tuple] = ("test_type", "engagement__product__prod_type", "environment", "lead") + PREFETCH_RELATED: ClassVar[tuple] = ("tags",) + EXPANDABLE: ClassVar[dict[str, ExpandRel]] = {} + + id: int + name: str | None + test_type: Ref + engagement: Ref + asset: Ref + organization: Ref + environment: Ref | None + lead: Ref | None + target_start: datetime | None + target_end: datetime | None + percent_complete: int | None + tags: list[str] + created: datetime | None + updated: datetime | None + + @staticmethod + def resolve_name(obj) -> str | None: + return obj.title + + @staticmethod + def resolve_test_type(obj) -> dict | None: + return to_ref(obj.test_type) + + @staticmethod + def resolve_engagement(obj) -> dict | None: + return to_ref(obj.engagement) + + @staticmethod + def resolve_asset(obj) -> dict | None: + return to_ref(obj.engagement.product) + + @staticmethod + def resolve_organization(obj) -> dict | None: + return to_ref(obj.engagement.product.prod_type) + + @staticmethod + def resolve_environment(obj) -> dict | None: + return to_ref(obj.environment) + + @staticmethod + def resolve_lead(obj) -> dict | None: + return to_ref(obj.lead) + + @staticmethod + def resolve_tags(obj) -> list[str]: + return [t.name for t in obj.tags.all()] + + +TestSlim.EXPANDABLE = { + "test_type": ExpandRel(attr="test_type", path="test_type", schema=TestTypeSlim), + "engagement": ExpandRel(attr="engagement", path="engagement", schema=EngagementSlim), + "asset": ExpandRel(attr="engagement.product", path="engagement__product", schema=AssetSlim), + "organization": ExpandRel(attr="engagement.product.prod_type", path="engagement__product__prod_type", schema=OrganizationSlim), + "lead": ExpandRel(attr="lead", path="lead", schema=UserSlim), + "environment": ExpandRel(attr="environment", path="environment", schema=EnvironmentSlim), +} + + +class TestDetail(TestSlim): + + """ + Slim + the documented heavier read fields (§4.5). Retrieve returns detail; list returns slim. + + ``deduplication_algorithm`` / ``hash_code_fields`` expose the effective finding-matching policy + for this test as **read-only, computed** fields (they surface which dedupe/reimport algorithm and + hashcode fields matching uses, resolved from the per-scanner settings, without a user reading + ``settings.dist.py``). They reuse the v2 helper verbatim -- the ``Test.deduplication_algorithm`` / + ``Test.hash_code_fields`` model properties (``dojo/test/models.py``), the same lookup the v2 + ``TestSerializer`` reads -- so the lookup logic is never duplicated. They are computed from + ``test_type`` (already ``select_related`` in ``TestSlim.SELECT_RELATED``) and ``scan_type`` (a + concrete column), so no extra query is issued on a detail fetch. They are **not** on any write + schema (``TestWrite``/``TestUpdate``/``TestReplace``, all ``extra="forbid"``), so any attempt to + write them is rejected with a 400 -- a deliberate hardening over v2, which silently ignores writes + to them (§12). + + ``DETAIL_FIELD_COLUMNS``: on a LIST, ``scan_type`` is deferred by default (Part B). When either + computed field is requested via ``?fields=`` on a list, the kernel un-defers ``scan_type`` (the + column the resolvers read) so the query count stays constant -- ``test_type`` is already joined by + the slim base (§4.7 / §12 fields-opt-up). + """ + + DETAIL_FIELD_COLUMNS: ClassVar[dict[str, tuple[str, ...]]] = { + "deduplication_algorithm": ("scan_type",), + "hash_code_fields": ("scan_type",), + } + + description: str | None + scan_type: str | None + version: str | None + build_id: str | None + commit_hash: str | None + branch_tag: str | None + deduplication_algorithm: str + hash_code_fields: list[str] | None + + @staticmethod + def resolve_deduplication_algorithm(obj) -> str: + # Reuse the v2 model property (dojo/test/models.py) -- never duplicate the settings lookup. + return obj.deduplication_algorithm + + @staticmethod + def resolve_hash_code_fields(obj) -> list[str] | None: + # Reuse the v2 model property (dojo/test/models.py) -- never duplicate the settings lookup. + return obj.hash_code_fields + + +class TestWrite(Schema): + + """Create payload (POST). ``engagement``/``test_type``/``target_start``/``target_end`` required.""" + + model_config = {"extra": "forbid"} + + engagement: int + test_type: int + target_start: datetime + target_end: datetime + title: str | None = None + description: str | None = None + scan_type: str | None = None + lead: int | None = None + percent_complete: int | None = None + environment: int | None = None + version: str | None = None + build_id: str | None = None + commit_hash: str | None = None + branch_tag: str | None = None + api_scan_configuration: int | None = None + tags: list[str] | None = None + + +class TestUpdate(Schema): + + """Partial update payload (PATCH). ``engagement`` is not writable (editable=False, mirrors v2).""" + + model_config = {"extra": "forbid"} + + test_type: int | None = None + target_start: datetime | None = None + target_end: datetime | None = None + title: str | None = None + description: str | None = None + scan_type: str | None = None + lead: int | None = None + percent_complete: int | None = None + environment: int | None = None + version: str | None = None + build_id: str | None = None + commit_hash: str | None = None + branch_tag: str | None = None + api_scan_configuration: int | None = None + tags: list[str] | None = None + + +class TestReplace(Schema): + + """ + Full-replace payload (PUT). A dedicated Replace schema is required because ``TestWrite`` cannot + serve full-replace semantics: ``engagement`` is ``editable=False`` on the model and therefore + **not** writable on update (mirrors PATCH / v2's ``TestSerializer`` which treats it read-only), + yet ``TestWrite`` requires it. So PUT, like PATCH, never reassigns the parent engagement (§12). + + ``test_type``/``target_start``/``target_end`` stay required (create-shaped); the rest are + optional and reset to ``None`` when omitted (all are nullable columns -- there is no non-null + default field in the writable subset). Applied without ``exclude_unset`` by the route. + """ + + model_config = {"extra": "forbid"} + + test_type: int + target_start: datetime + target_end: datetime + title: str | None = None + description: str | None = None + scan_type: str | None = None + lead: int | None = None + percent_complete: int | None = None + environment: int | None = None + version: str | None = None + build_id: str | None = None + commit_hash: str | None = None + branch_tag: str | None = None + api_scan_configuration: int | None = None + tags: list[str] | None = None diff --git a/dojo/test/services.py b/dojo/test/services.py index fbd5dbf3b59..062fc27b575 100644 --- a/dojo/test/services.py +++ b/dojo/test/services.py @@ -1,16 +1,36 @@ # # tests import logging +from crum import get_current_request from django.urls import reverse from django.utils.translation import gettext as _ from dojo.celery_dispatch import dojo_dispatch_task -from dojo.notifications.helper import create_notification +from dojo.notifications.helper import create_notification, process_tag_notifications from dojo.utils import calculate_grade logger = logging.getLogger(__name__) +def process_note_added(test, note, *, user): + """ + Fire the same side-effects as the v2 test notes ``@action`` create branch + (``dojo/test/api/views.py``) after a note is persisted and linked: @mention notifications only -- + the test @action has **no** JIRA comment sync and **no** ``last_reviewed`` stamping. Reuses the + exact v2 parsing/notification helper (``process_tag_notifications``); the request is read from crum + (see ``dojo/finding/services.py``), and mentions are skipped with no request. ``user`` is part of + the callback contract (I6) but unused here (no side-effect needs it). + """ + request = get_current_request() + if request is not None: + process_tag_notifications( + request=request, + note=note, + parent_url=request.build_absolute_uri(reverse("view_test", args=(test.id,))), + parent_title=f"Test: {test.title}", + ) + + def copy_test(test, engagement, user): """ Copy a test (and its findings) into the given engagement, recalculate the product diff --git a/dojo/urls.py b/dojo/urls.py index 063d65ef220..e26f02adf2f 100644 --- a/dojo/urls.py +++ b/dojo/urls.py @@ -209,6 +209,26 @@ ), ] +# API v3 (alpha) -- mounted conditionally on V3_FEATURE_LOCATIONS (D5/§4.1). With the flag off the +# whole /api/v3-alpha/ tree is absent. The prefix and version live in settings (single source). +if getattr(settings, "V3_FEATURE_LOCATIONS", False): + from dojo.api_v3.api import api_v3 + from dojo.api_v3.reference_docs import scalar_reference + + api_v2_urls += [ + # Scalar reference (CDN + SRI, §12) must be registered BEFORE the NinjaAPI catch-all + # prefix so /reference is not swallowed by the API's 404 handling. + re_path( + r"^{}{}/reference$".format(get_system_setting("url_prefix"), settings.API_V3_URL_PREFIX), + scalar_reference, + name="api_v3_reference", + ), + re_path( + r"^{}{}/".format(get_system_setting("url_prefix"), settings.API_V3_URL_PREFIX), + api_v3.urls, + ), + ] + urlpatterns = [] # sometimes urlpatterns needed be added from local_settings.py before other URLs of core dojo diff --git a/dojo/user/api_v3/__init__.py b/dojo/user/api_v3/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/dojo/user/api_v3/routes.py b/dojo/user/api_v3/routes.py new file mode 100644 index 00000000000..b6d302e6f66 --- /dev/null +++ b/dojo/user/api_v3/routes.py @@ -0,0 +1,301 @@ +""" +User (Dojo_User) CRUD routes for API v3 (§4.5, §4.9, §4.11, OS3a). + +``build_users_router()`` is a router factory (I5), same signature style as ``build_findings_router``. +Routes are thin (I6): authorize -> filter -> plan queryset -> serialize -> shape. + +RBAC (mirroring the v2 ``UsersViewSet``, read first): +- reads: every authenticated user gets the collaborator-scoped view via ``get_authorized_users`` + (I8). This is the OS user-visibility model (superusers/staff see all; others see users + sharing their authorized products/product-types, plus superusers). See §12 for why reads + are opened to authenticated users rather than gated behind the ``view_user`` config perm. +- writes: admin/superuser-only, mirroring ``UserHasConfigurationPermissionSuperuser`` (the Django + ``add_user``/``change_user``/``delete_user`` configuration permissions -- superusers pass + automatically). The ``UserSerializer.validate()`` rules are ported verbatim: only + superusers may add/edit superusers or staff; password is write-only and cannot be changed + via PATCH; a password is required on create when ``REQUIRE_PASSWORD_ON_USER`` is set; and + a user may not delete themselves (v2 ``destroy``). +""" +from __future__ import annotations + +from typing import TYPE_CHECKING + +from django.conf import settings +from django.contrib.auth.password_validation import validate_password +from django.core.exceptions import PermissionDenied +from django.core.exceptions import ValidationError as DjangoValidationError +from django.http import HttpResponse +from ninja import Router, Schema +from ninja.constants import NOT_SET + +from dojo.api_v3.csv_export import stream_csv_export +from dojo.api_v3.errors import json_response, not_found_problem, validation_problem +from dojo.api_v3.expand import ( + allowed_field_names, + apply_fields, + parse_fields, + plan, + plan_list_fields, + plan_queryset, + serialize, + serialize_list_row, +) +from dojo.api_v3.filtering import ( + FilterSpec, + apply_filters, + filter_field, + register_filter_spec, +) +from dojo.api_v3.include import apply_includes +from dojo.api_v3.pagination import paginate +from dojo.authorization.authorization import user_has_configuration_permission +from dojo.models import Dojo_User +from dojo.user.api_v3.schemas import UserDetail, UserSlim, UserUpdate, UserWrite +from dojo.user.queries import get_authorized_users + +if TYPE_CHECKING: + from collections.abc import Callable + + from django.db.models import QuerySet + from django.http import HttpRequest + +# --- User filter vocabulary (§4.9, minimal set) ----------------------------------------------- + +USER_FILTER_SPEC = register_filter_spec("user", FilterSpec( + model=Dojo_User, + filters={ + "id__in": filter_field("id", "in", "number"), + "username__icontains": filter_field("username", "icontains", "char"), + "first_name__icontains": filter_field("first_name", "icontains", "char"), + "last_name__icontains": filter_field("last_name", "icontains", "char"), + "email__icontains": filter_field("email", "icontains", "char"), + "is_active": filter_field("is_active", "exact", "bool"), + "is_superuser": filter_field("is_superuser", "exact", "bool"), + "date_joined__gte": filter_field("date_joined", "gte", "datetime"), + "date_joined__lte": filter_field("date_joined", "lte", "datetime"), + "last_login__gte": filter_field("last_login", "gte", "datetime"), + "last_login__lte": filter_field("last_login", "lte", "datetime"), + }, + orderings={ + "id": "id", + "username": "username", + "date_joined": "date_joined", + "last_login": "last_login", + }, + search_fields=["username", "first_name", "last_name", "email"], +)) + + +class UserListResponse(Schema): + + """OpenAPI documentation of the list envelope (I1); runtime serialization is manual.""" + + count: int + next: str | None + previous: str | None + results: list[UserSlim] + meta: dict | None = None + + +def _base_queryset(request: HttpRequest, queryset_hook: Callable | None) -> QuerySet: + # Mirror the v2 UsersViewSet ``view_user`` gate (§6 OS3 "read + self"; conservative per §10.2): + # - holders of the ``view_user`` configuration permission get the RBAC-scoped queryset + # (superusers/staff -> all; a non-staff holder -> the collaborator subset, which is <= v2's + # "return all users" exposure); + # - everyone else sees only themselves, so a plain user lists exactly their own record and + # ``GET /users/{own id}`` always resolves (never a 404 on your own record). + if user_has_configuration_permission(request.user, "auth.view_user"): + qs = get_authorized_users("view", user=request.user) + else: + qs = Dojo_User.objects.filter(pk=request.user.pk) + if queryset_hook is not None: + qs = queryset_hook(qs, request) + return qs + + +def _require_config_permission(request: HttpRequest, codename: str) -> None: + if not user_has_configuration_permission(request.user, codename): + raise PermissionDenied + + +def _enforce_superuser_staff_rules(request: HttpRequest, *, current: Dojo_User | None, data: dict) -> None: + """Port ``UserSerializer.validate()`` superuser/staff gating (§12 / D7 reconciliation).""" + acting_is_superuser = bool(getattr(request.user, "is_superuser", False)) + instance_is_superuser = bool(current.is_superuser) if current is not None else False + data_is_superuser = data.get("is_superuser", instance_is_superuser) + if not acting_is_superuser and (instance_is_superuser or data_is_superuser): + raise validation_problem({"is_superuser": ["Only superusers are allowed to add or edit superusers."]}) + + instance_is_staff = bool(current.is_staff) if current is not None else False + data_is_staff = data.get("is_staff", instance_is_staff) + if not acting_is_superuser and data_is_staff != instance_is_staff: + raise validation_problem({"is_staff": ["Only superusers are allowed to add or edit staff users."]}) + + +def _enforce_identity_field_rules(request: HttpRequest, *, current: Dojo_User | None, data: dict) -> None: + """ + Port ``UserSerializer.validate()`` identity-field gating (§12 OS3a; PR #15191 parity). + + A non-superuser may not change **another** account's identity fields (``email``/``username``). + Changing another user's email would enable account takeover via the email-based password-reset + flow, so the identity fields join ``is_superuser``/``is_staff`` under superuser-only control. + Editing one's own identity stays allowed, and superusers are unrestricted. On create there is no + existing account whose identity could be hijacked (``current is None``), so the rule is a no-op + there and applies to updates (PATCH/PUT) only. + """ + if bool(getattr(request.user, "is_superuser", False)): + return + if current is None or request.user.pk == current.pk: + return + for field in ("email", "username"): + if field in data and data[field] != getattr(current, field): + raise validation_problem( + {field: [f"Only superusers are allowed to change another user's {field}."]}, + ) + + +def _validate_password_or_400(password: str) -> None: + try: + validate_password(password) + except DjangoValidationError as exc: + raise validation_problem({"password": list(exc.messages)}) from exc + + +def build_users_router( + *, + schema: type = UserSlim, + detail_schema: type = UserDetail, + filter_spec: FilterSpec = USER_FILTER_SPEC, + queryset_hook: Callable | None = None, + auth=NOT_SET, +) -> Router: + """Build the users router (I5).""" + router = Router(tags=["users"], auth=auth) + + @router.get("/users", response=UserListResponse, url_name="users_list") + def list_users(request: HttpRequest): + filtered = apply_filters(request, _base_queryset(request, queryset_hook), filter_spec) + + expand_tree, select_related, prefetch = plan(schema, request.GET.get("expand")) + # ?fields= may opt up into the detail field set (§4.7 Part A); defer the heavy detail + # columns not requested (Part B). + fields = parse_fields(request.GET.get("fields"), allowed_field_names(detail_schema)) + fplan = plan_list_fields(schema, detail_schema, fields) + page_qs = filtered.select_related(*schema.SELECT_RELATED, *fplan.select_related).prefetch_related(*schema.PREFETCH_RELATED) + page_qs = plan_queryset(page_qs, select_related, prefetch) + if fplan.defer: + page_qs = page_qs.defer(*fplan.defer) + + def serialize_row(obj: object) -> dict: + return serialize_list_row(obj, fplan, expand_tree) + + envelope = paginate(request, count_qs=filtered, page_qs=page_qs, serialize=serialize_row, filter_spec=filter_spec) + include_meta = apply_includes(request, filtered, allowed=set()) + if include_meta: + envelope.setdefault("meta", {}).update(include_meta) + return json_response(envelope) + + # Literal ``export.csv`` cannot match the ``{int:user_id}`` detail route (int converter), so no + # collision. Same filter contract + RBAC-scoped queryset as the list (a plain user exports only + # their own record -- the documented self-visibility scope, §12 OS3a); streamed as CSV (§4.15). + @router.get("/users/export.csv", url_name="users_export_csv") + def export_users(request: HttpRequest): + filtered = apply_filters(request, _base_queryset(request, queryset_hook), filter_spec) + fields = parse_fields(request.GET.get("fields"), allowed_field_names(detail_schema)) + fplan = plan_list_fields(schema, detail_schema, fields) + page_qs = filtered.select_related(*schema.SELECT_RELATED, *fplan.select_related).prefetch_related(*schema.PREFETCH_RELATED) + if fplan.defer: + page_qs = page_qs.defer(*fplan.defer) + return stream_csv_export(request, resource="users", count_qs=filtered, page_qs=page_qs, plan=fplan) + + @router.get("/users/{int:user_id}", response=detail_schema, url_name="users_detail") + def get_user(request: HttpRequest, user_id: int): + obj = _base_queryset(request, queryset_hook).filter(pk=user_id).first() + if obj is None: + msg = f"User {user_id} not found" + raise not_found_problem(msg) + fields = parse_fields(request.GET.get("fields"), allowed_field_names(detail_schema)) + return json_response(apply_fields(serialize(obj, detail_schema, {}), fields)) + + @router.post("/users", response=detail_schema, url_name="users_create") + def create_user(request: HttpRequest, payload: UserWrite): + _require_config_permission(request, "auth.add_user") + data = payload.dict() + password = data.pop("password") + _enforce_superuser_staff_rules(request, current=None, data=data) + if password: + _validate_password_or_400(password) + elif settings.REQUIRE_PASSWORD_ON_USER: + raise validation_problem({"password": ["Passwords must be supplied for new users"]}) + if Dojo_User.objects.filter(username=data["username"]).exists(): + raise validation_problem({"username": ["A user with that username already exists."]}) + + instance = Dojo_User(**data) + if password: + instance.set_password(password) + else: + instance.set_unusable_password() + instance.save() + return json_response(serialize(instance, detail_schema, {}), status=201) + + @router.patch("/users/{int:user_id}", response=detail_schema, url_name="users_update") + def update_user(request: HttpRequest, user_id: int, payload: UserUpdate): + # Resolve against the authorized view queryset first (404 for unknown-or-unauthorized). + instance = _base_queryset(request, queryset_hook).filter(pk=user_id).first() + if instance is None: + msg = f"User {user_id} not found" + raise not_found_problem(msg) + _require_config_permission(request, "auth.change_user") + data = payload.dict(exclude_unset=True) + if "password" in data: + raise validation_problem({"password": ["Update of password though API is not allowed"]}) + _enforce_superuser_staff_rules(request, current=instance, data=data) + _enforce_identity_field_rules(request, current=instance, data=data) + for key, value in data.items(): + setattr(instance, key, value) + instance.save() + return json_response(serialize(instance, detail_schema, {})) + + @router.put("/users/{int:user_id}", response=detail_schema, url_name="users_replace") + def replace_user(request: HttpRequest, user_id: int, payload: UserWrite): + # Full replace (PUT). Validates against the create-shaped UserWrite (username/email + # required, extra="forbid") and applies payload.dict() WITHOUT exclude_unset, so omitted + # optionals reset to their schema defaults (first_name/last_name -> "", is_active -> True, + # is_staff/is_superuser -> False; all model defaults). UserWrite serves full-replace as-is: + # its non-null booleans already default to the model defaults. The UserSerializer-ported + # guards are kept identical to PATCH: resolve via the authorized queryset first (404), + # require auth.change_user, reject any password set via update, enforce the superuser/staff + # gating. Password is never settable via update (mirrors PATCH), so a supplied password is a + # 400 -- a full replace simply omits it. + instance = _base_queryset(request, queryset_hook).filter(pk=user_id).first() + if instance is None: + msg = f"User {user_id} not found" + raise not_found_problem(msg) # 404: unknown or unauthorized-to-view + _require_config_permission(request, "auth.change_user") + data = payload.dict() + password = data.pop("password", None) + if password is not None: + raise validation_problem({"password": ["Update of password though API is not allowed"]}) + _enforce_superuser_staff_rules(request, current=instance, data=data) + _enforce_identity_field_rules(request, current=instance, data=data) + for key, value in data.items(): + setattr(instance, key, value) + instance.save() + return json_response(serialize(instance, detail_schema, {})) + + @router.delete("/users/{int:user_id}", url_name="users_delete") + def delete_user(request: HttpRequest, user_id: int): + instance = _base_queryset(request, queryset_hook).filter(pk=user_id).first() + if instance is None: + msg = f"User {user_id} not found" + raise not_found_problem(msg) + _require_config_permission(request, "auth.delete_user") + if request.user.pk == instance.pk: + # Mirror v2 UsersViewSet.destroy: users may not delete themselves. + raise validation_problem({"non_field_errors": ["Users may not delete themselves"]}) + instance.delete() + response = HttpResponse(status=204) + response["X-API-Status"] = settings.API_V3_STATUS + return response + + return router diff --git a/dojo/user/api_v3/schemas.py b/dojo/user/api_v3/schemas.py new file mode 100644 index 00000000000..a1aadca2669 --- /dev/null +++ b/dojo/user/api_v3/schemas.py @@ -0,0 +1,81 @@ +""" +User (Dojo_User) response + write schemas for API v3 (§4.5, §4.11, OS3a). + +``UserSlim`` is the canonical user slim (relocated here from ``dojo/finding/api_v3`` where OS1 first +defined it -- see §12; the finding module now re-exports this copy). Every field in §4.5's UserSlim +was verified against the model. ``UserDetail`` adds ``is_staff`` and ``date_joined`` so the +superuser/staff write rules operate over documented read fields. + +Write schemas mirror the v2 ``UserSerializer``: ``username``/``email`` required on create, +``password`` write-only. The superuser/staff/self-delete/password-on-PATCH rules are enforced in +the route (ported from ``UserSerializer.validate()``). Server-managed fields (``id``, +``date_joined``, ``last_login``) are never writable; unknown fields are rejected (``extra="forbid"``). +``configuration_permissions`` is intentionally out of the alpha write surface (see §12). +""" +from __future__ import annotations + +from datetime import datetime # noqa: TC003 -- runtime import: pydantic resolves the schema field types +from typing import TYPE_CHECKING, ClassVar + +from ninja import Schema + +from dojo.models import Dojo_User + +if TYPE_CHECKING: + from dojo.api_v3.expand import ExpandRel + + +class UserSlim(Schema): + django_model: ClassVar = Dojo_User + SELECT_RELATED: ClassVar[tuple] = () + PREFETCH_RELATED: ClassVar[tuple] = () + EXPANDABLE: ClassVar[dict[str, ExpandRel]] = {} + + id: int + username: str + first_name: str + last_name: str + email: str + is_active: bool + is_superuser: bool + last_login: datetime | None + + +class UserDetail(UserSlim): + + """Slim + documented heavier read fields (§4.5).""" + + is_staff: bool + date_joined: datetime | None + + +class UserWrite(Schema): + + """Create payload (POST). ``username``/``email`` required, mirroring the v2 serializer (§6 OS3).""" + + model_config = {"extra": "forbid"} + + username: str + email: str + first_name: str = "" + last_name: str = "" + is_active: bool = True + is_staff: bool = False + is_superuser: bool = False + password: str | None = None + + +class UserUpdate(Schema): + + """Partial update payload (PATCH). Every field optional; ``password`` is rejected here (§12).""" + + model_config = {"extra": "forbid"} + + username: str | None = None + email: str | None = None + first_name: str | None = None + last_name: str | None = None + is_active: bool | None = None + is_staff: bool | None = None + is_superuser: bool | None = None + password: str | None = None diff --git a/requirements.txt b/requirements.txt index 59242da3e8d..391e3b50717 100644 --- a/requirements.txt +++ b/requirements.txt @@ -13,6 +13,7 @@ django-filter==26.1 django-htmx==1.28.0 django-imagekit==6.1.0 django-multiselectfield==1.0.1 +django-ninja==1.6.2 # API v3 framework (alpha); pulls pydantic v2 (already present transitively) django-polymorphic==4.11.6 django-crispy-forms==2.6 django_extensions==4.1 diff --git a/unittests/api_v3/__init__.py b/unittests/api_v3/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/unittests/api_v3/base.py b/unittests/api_v3/base.py new file mode 100644 index 00000000000..f715ff040ea --- /dev/null +++ b/unittests/api_v3/base.py @@ -0,0 +1,70 @@ +""" +Base test case for API v3 (§6 preamble). + +All v3 tests use the standard in-process Django test client (via ``DojoAPITestCase``) so the full +URLconf/middleware/auth/CSRF stack runs, server-side exceptions propagate as real tracebacks, +``assertNumQueries`` works and the test transaction is shared. ``ninja.testing.TestClient`` is +reserved for kernel-internal unit tests only. +""" +from __future__ import annotations + +import json +from unittest import skipUnless + +from django.conf import settings +from rest_framework.authtoken.models import Token +from rest_framework.test import APIClient + +from dojo.models import User +from dojo.utils import get_system_setting +from unittests.dojo_test_case import DojoAPITestCase + + +@skipUnless( + settings.V3_FEATURE_LOCATIONS, + "API v3 is mounted only when V3_FEATURE_LOCATIONS is enabled (D5); with the flag off the " + "endpoints do not exist, so these contract tests are not applicable. The CI unit-test matrix " + "runs a flag-off leg, hence the guard. Flag-independent kernel unit tests (OpenAPI render, " + "static authz tripwire, expand cycle guard) use SimpleTestCase directly and still run.", +) +class ApiV3TestCase(DojoAPITestCase): + + """Shared helpers: v3 URL prefix, token + session/CSRF clients, JSON assertions.""" + + # v3 requires V3_FEATURE_LOCATIONS=True, under which the legacy Endpoint model raises on load; + # use the locations-aware fixture (same products/engagements/tests/findings, no Endpoint rows). + fixtures = ["dojo_testdata_locations.json"] + + def setUp(self): + super().setUp() + self.admin = User.objects.get(username="admin") + self.token, _ = Token.objects.get_or_create(user=self.admin) + # Default client authenticates with a token (mirrors login_as_admin()). + self.client = self.token_client() + + # --- URL helper ------------------------------------------------------------------------- + def v3_url(self, path: str = "") -> str: + prefix = get_system_setting("url_prefix") + return f"/{prefix}{settings.API_V3_URL_PREFIX}/{path.lstrip('/')}" + + # --- client helpers --------------------------------------------------------------------- + def token_client(self, *, user: User | None = None) -> APIClient: + token = self.token if user is None else Token.objects.get_or_create(user=user)[0] + client = APIClient() + client.credentials(HTTP_AUTHORIZATION="Token " + token.key) + return client + + def session_client(self, *, user: User | None = None, enforce_csrf: bool = False) -> APIClient: + client = APIClient(enforce_csrf_checks=enforce_csrf) + client.force_login(user or self.admin) + return client + + def anonymous_client(self) -> APIClient: + return APIClient() + + # --- assertion helpers ------------------------------------------------------------------ + def get_json(self, path: str, *, client: APIClient | None = None, data: dict | None = None, expected: int = 200): + client = client or self.client + response = client.get(self.v3_url(path), data or {}) + self.assertEqual(expected, response.status_code, response.content[:1000]) + return json.loads(response.content) if response.content else None diff --git a/unittests/api_v3/import_corpus_shim.py b/unittests/api_v3/import_corpus_shim.py new file mode 100644 index 00000000000..8e026c9ab11 --- /dev/null +++ b/unittests/api_v3/import_corpus_shim.py @@ -0,0 +1,163 @@ +""" +Dual-endpoint adapter for running the v2 import/reimport test corpus against API v3 (§10 port, +backlog priority 1). + +The v2 corpus (``unittests/test_import_reimport.py``) drives its scenarios through two helper +methods -- ``import_scan_with_params`` / ``reimport_scan_with_params`` -- and inspects results with +the shared ``DojoAPITestCase`` DB/finding helpers. This module supplies a mixin that overrides ONLY +those two helpers so the *identical* scenarios and assertions run against the consolidated +``POST /api/v3-alpha/import`` endpoint instead of the v2 ``importscan``/``reimportscan`` endpoints. + +Mapping performed by the shim (v2 wire -> v3 wire, D11 §12): + * ``product_name`` -> ``asset_name`` + * ``product_type_name`` -> ``organization_name`` + * import vs reimport -> a single POST with ``mode=import|reimport|auto`` + (``reimport`` with no ``test`` id and the auto-create fields resolves to ``mode=auto``) + * v3 response ``{mode_resolved, test:{id,name}, statistics:{...}, close_old_findings}`` + -> the v2-shaped dict the corpus reads: ``{"test": , ...}``. + +The endpoint→location count redirect (``db_endpoint_count`` / ``db_endpoint_status_count``) is +inherited unchanged from ``ImportReimportTestAPI`` -- under ``V3_FEATURE_LOCATIONS=True`` it already +delegates to the location counters, exactly as this shim needs. + +This file is intentionally NOT named ``test_*`` so Django's ``test*.py`` discovery does not load it +as a test module; it only holds the reusable mixin. +""" +from __future__ import annotations + +from pathlib import Path +from unittest import skipUnless + +from django.conf import settings + +from dojo.utils import get_system_setting + + +def _bool_str(value: object) -> str: + """Render a Python bool (or truthy value) the way the v3 multipart form expects it.""" + return "true" if bool(value) else "false" + + +@skipUnless( + settings.V3_FEATURE_LOCATIONS, + "API v3 import endpoint is mounted only when V3_FEATURE_LOCATIONS is enabled (D5); the CI " + "unit-test matrix runs a flag-off leg where /api/v3-alpha/ does not exist. Guarding the shim " + "skips every corpus test that mixes it in (they hit POST /api/v3-alpha/import).", +) +class ApiV3ImportShim: + + """ + Overrides ``import_scan_with_params`` / ``reimport_scan_with_params`` to hit ``POST /import``. + + Mix in *before* the v2 corpus class so these overrides win in the MRO, e.g.:: + + class Corpus(ApiV3ImportShim, ImportReimportTestAPI): + ... + """ + + # --- v3 URL helper (mirrors unittests.api_v3.base.ApiV3TestCase.v3_url) ------------------ + def v3_url(self, path: str = "") -> str: + prefix = get_system_setting("url_prefix") + return f"/{prefix}{settings.API_V3_URL_PREFIX}/{path.lstrip('/')}" + + # --- core POST + response translation --------------------------------------------------- + def _post_v3_import(self, payload: dict, filename, *, expected: int) -> dict: + with Path(filename).open(encoding="utf-8") as scan_file: + payload = {**payload, "file": scan_file} + response = self.client.post(self.v3_url("import"), payload, format="multipart") + self.assertEqual(expected, response.status_code, response.content[:1500]) + body = response.json() + if expected not in {200, 201}: + # error path: hand the raw problem+json back to the caller unchanged + return body + # Translate the v3 envelope into the v2-shaped dict the corpus assertions read. + return { + "test": body["test"]["id"], + "test_id": body["test"]["id"], + "mode_resolved": body.get("mode_resolved"), + "statistics": body.get("statistics"), + "close_old_findings": body.get("close_old_findings"), + } + + def _skip_unsupported(self, *, push_to_jira, group_by=None, endpoint_to_add=None) -> None: + if push_to_jira is not None: + self.skipTest("push_to_jira is deferred to the JIRA work stream (§4.13 / §12).") + if group_by is not None: + self.skipTest("group_by is not on the v3 ImportForm (no ported scenario requires it; §12).") + if endpoint_to_add is not None: + self.skipTest("endpoint_to_add / legacy Endpoint wiring is out of v3 scope (§4.13).") + + def _base_payload(self, *, mode, scan_type, minimum_severity) -> dict: + # version is sent unconditionally by the v2 helpers; keep parity. + return {"scan_type": scan_type, "mode": mode, "version": "1.0.1", "minimum_severity": minimum_severity} + + @staticmethod + def _apply_common(payload: dict, *, active, verified, close_old_findings, scan_date, service, tags, + test_title, product_name, product_type_name, engagement_name, auto_create_context) -> None: + if active is not None: + payload["active"] = _bool_str(active) + if verified is not None: + payload["verified"] = _bool_str(verified) + if close_old_findings is not None: + payload["close_old_findings"] = _bool_str(close_old_findings) + if scan_date is not None: + payload["scan_date"] = scan_date + if service is not None: + payload["service"] = service + if tags is not None: + payload["tags"] = ",".join(tags) if isinstance(tags, (list, tuple)) else tags + if test_title is not None: + payload["test_title"] = test_title + if product_name: + payload["asset_name"] = product_name + if product_type_name: + payload["organization_name"] = product_type_name + if engagement_name: + payload["engagement_name"] = engagement_name + if auto_create_context: + payload["auto_create_context"] = _bool_str(auto_create_context) + + # --- overridden corpus helpers ---------------------------------------------------------- + def import_scan_with_params(self, filename, scan_type="ZAP Scan", engagement=1, minimum_severity="Low", *, + active=True, verified=False, push_to_jira=None, endpoint_to_add=None, tags=None, + close_old_findings=None, group_by=None, engagement_name=None, product_name=None, + product_type_name=None, auto_create_context=None, expected_http_status_code=201, + test_title=None, scan_date=None, service=None, force_active=True, force_verified=True): + self._skip_unsupported(push_to_jira=push_to_jira, group_by=group_by, endpoint_to_add=endpoint_to_add) + mode = "auto" if auto_create_context else "import" + payload = self._base_payload(mode=mode, scan_type=scan_type, minimum_severity=minimum_severity) + if mode == "import" and engagement is not None: + payload["engagement"] = engagement + self._apply_common( + payload, active=active, verified=verified, close_old_findings=close_old_findings, + scan_date=scan_date, service=service, tags=tags, test_title=test_title, + product_name=product_name, product_type_name=product_type_name, + engagement_name=engagement_name, auto_create_context=auto_create_context, + ) + expected = 200 if expected_http_status_code == 201 else expected_http_status_code + return self._post_v3_import(payload, filename, expected=expected) + + def reimport_scan_with_params(self, test_id, filename, scan_type="ZAP Scan", engagement=1, minimum_severity="Low", *, + active=True, verified=False, push_to_jira=None, tags=None, close_old_findings=None, + group_by=None, engagement_name=None, scan_date=None, service=None, product_name=None, + product_type_name=None, auto_create_context=None, expected_http_status_code=201, + test_title=None): + self._skip_unsupported(push_to_jira=push_to_jira, group_by=group_by) + mode = "reimport" if test_id is not None else "auto" + payload = self._base_payload(mode=mode, scan_type=scan_type, minimum_severity=minimum_severity) + if test_id is not None: + payload["test"] = test_id + elif engagement is not None and not product_name: + # auto-create/resolve path: honour an explicit engagement id only when the caller + # is NOT resolving by asset/engagement name. When asset_name is supplied the name-based + # lookup is authoritative (mirrors the v2 serializer, which ignores the numeric default); + # sending both a mismatched engagement id and asset_name trips the v3 consistency check. + payload["engagement"] = engagement + self._apply_common( + payload, active=active, verified=verified, close_old_findings=close_old_findings, + scan_date=scan_date, service=service, tags=tags, test_title=test_title, + product_name=product_name, product_type_name=product_type_name, + engagement_name=engagement_name, auto_create_context=auto_create_context, + ) + expected = 200 if expected_http_status_code == 201 else expected_http_status_code + return self._post_v3_import(payload, filename, expected=expected) diff --git a/unittests/api_v3/query_report.py b/unittests/api_v3/query_report.py new file mode 100644 index 00000000000..26d5d8f2b8f --- /dev/null +++ b/unittests/api_v3/query_report.py @@ -0,0 +1,111 @@ +""" +Query-capture harness for API v3 (§7 of the plan / OS6 verification). + +Captures every SQL query a v3 request executes and detects the N+1 signature: the same +*normalized* query shape executed many times within one request. Complements the per-list +``assertNumQueries`` tests (which pin totals) by identifying *which* query repeats when a +regression appears, and by sweeping the whole mounted surface in one place +(``test_apiv3_query_report.py``) so new resources are covered automatically. + +Also usable interactively from a shell (``manage.py shell`` + test client) to profile an +endpoint; ``format_report`` renders the capture as markdown. +""" +from __future__ import annotations + +import re +from collections import Counter +from dataclasses import dataclass, field + +from django.db import connection +from django.test.utils import CaptureQueriesContext + +# Normalization: collapse literals so structurally-identical queries group together. +_NORMALIZERS = ( + (re.compile(r"'(?:[^']|'')*'"), "'?'"), # string literals + (re.compile(r"\b\d+(?:\.\d+)?\b"), "N"), # numeric literals + (re.compile(r"IN \([^)]*\)", re.IGNORECASE), "IN (...)"), # IN lists (prefetch batches) + (re.compile(r"\s+"), " "), # whitespace +) + +# Shapes that legitimately repeat and are never an N+1 signal. +_IGNORED_SHAPES = ( + re.compile(r"^(SAVEPOINT|RELEASE SAVEPOINT|ROLLBACK TO SAVEPOINT)", re.IGNORECASE), +) + + +def normalize_sql(sql: str) -> str: + for pattern, replacement in _NORMALIZERS: + sql = pattern.sub(replacement, sql) + return sql.strip() + + +@dataclass +class EndpointCapture: + + """One request's query profile.""" + + label: str + path: str + status_code: int + query_count: int + result_rows: int | None + shapes: Counter = field(default_factory=Counter) + + def repeated_shapes(self, threshold: int) -> list[tuple[str, int]]: + """Normalized shapes executed >= threshold times, excluding known-benign ones.""" + flagged = [] + for shape, count in self.shapes.most_common(): + if count < threshold: + break + if any(p.search(shape) for p in _IGNORED_SHAPES): + continue + flagged.append((shape, count)) + return flagged + + +def capture_request(client, label: str, path: str) -> EndpointCapture: + """Execute a GET through the in-process client and capture its query profile.""" + with CaptureQueriesContext(connection) as ctx: + response = client.get(path) + shapes = Counter(normalize_sql(q["sql"]) for q in ctx.captured_queries) + rows = None + # Streaming responses (e.g. the files-download FileResponse) have no `.content`; skip body + # parsing for them -- they are single-object endpoints, not paginated lists. + if response.status_code == 200 and not getattr(response, "streaming", False): + try: + body = response.json() + rows = len(body["results"]) if isinstance(body, dict) and "results" in body else None + except ValueError: + rows = None + return EndpointCapture( + label=label, + path=path, + status_code=response.status_code, + query_count=len(ctx.captured_queries), + result_rows=rows, + shapes=shapes, + ) + + +def format_report(captures: list[EndpointCapture], threshold: int) -> str: + """Render captures as a markdown report (written to /tmp by the sweep test).""" + lines = [ + "# API v3 query report", + "", + f"N+1 flag threshold: same normalized query shape >= {threshold}x in one request.", + "", + "| endpoint | status | queries | rows | flagged shapes |", + "|---|---|---:|---:|---|", + ] + for cap in captures: + flagged = cap.repeated_shapes(threshold) + flag_text = "; ".join(f"{count}x `{shape[:80]}...`" for shape, count in flagged) if flagged else "-" + lines.append( + f"| {cap.label} | {cap.status_code} | {cap.query_count} " + f"| {cap.result_rows if cap.result_rows is not None else '-'} | {flag_text} |", + ) + lines.append("") + for cap in captures: + for shape, count in cap.repeated_shapes(threshold): + lines += [f"## {cap.label}: {count}x", "", "```sql", shape, "```", ""] + return "\n".join(lines) diff --git a/unittests/api_v3/snapshots/filters.json b/unittests/api_v3/snapshots/filters.json new file mode 100644 index 00000000000..b8f050113ef --- /dev/null +++ b/unittests/api_v3/snapshots/filters.json @@ -0,0 +1,198 @@ +{ + "asset": { + "model": "Product", + "orderings": [ + "created", + "id", + "name", + "updated" + ], + "params": [ + "created__gte", + "created__lte", + "id__in", + "name__icontains", + "organization", + "organization__in", + "updated__gte", + "updated__lte" + ], + "search_fields": [ + "description", + "name" + ] + }, + "engagement": { + "model": "Engagement", + "orderings": [ + "created", + "id", + "name", + "target_start", + "updated" + ], + "params": [ + "asset", + "asset__in", + "created__gte", + "created__lte", + "engagement_type", + "id__in", + "lead", + "name__icontains", + "organization", + "status", + "target_end__gte", + "target_end__lte", + "target_start__gte", + "target_start__lte", + "updated__gte", + "updated__lte" + ], + "search_fields": [ + "description", + "name" + ] + }, + "finding": { + "model": "Finding", + "orderings": [ + "created", + "date", + "id", + "severity", + "title", + "updated" + ], + "params": [ + "active", + "asset", + "asset__in", + "created__gte", + "created__lte", + "cwe", + "cwe__in", + "date__gte", + "date__lte", + "duplicate", + "engagement", + "false_p", + "id__in", + "is_mitigated", + "organization", + "out_of_scope", + "reporter", + "risk_accepted", + "severity", + "severity__in", + "tags__in", + "test", + "title__icontains", + "updated__gte", + "updated__lte", + "verified" + ], + "search_fields": [ + "description", + "title" + ] + }, + "location": { + "model": "Location", + "orderings": [ + "id", + "name" + ], + "params": [ + "asset", + "name__icontains", + "type" + ], + "search_fields": [ + "location_value" + ] + }, + "organization": { + "model": "Product_Type", + "orderings": [ + "created", + "id", + "name", + "updated" + ], + "params": [ + "created__gte", + "created__lte", + "id__in", + "name__icontains", + "updated__gte", + "updated__lte" + ], + "search_fields": [ + "description", + "name" + ] + }, + "test": { + "model": "Test", + "orderings": [ + "created", + "id", + "target_start", + "title", + "updated" + ], + "params": [ + "asset", + "asset__in", + "created__gte", + "created__lte", + "engagement", + "engagement__in", + "environment", + "id__in", + "lead", + "organization", + "target_end__gte", + "target_end__lte", + "target_start__gte", + "target_start__lte", + "test_type", + "title__icontains", + "updated__gte", + "updated__lte" + ], + "search_fields": [ + "description", + "title" + ] + }, + "user": { + "model": "Dojo_User", + "orderings": [ + "date_joined", + "id", + "last_login", + "username" + ], + "params": [ + "date_joined__gte", + "date_joined__lte", + "email__icontains", + "first_name__icontains", + "id__in", + "is_active", + "is_superuser", + "last_login__gte", + "last_login__lte", + "last_name__icontains", + "username__icontains" + ], + "search_fields": [ + "email", + "first_name", + "last_name", + "username" + ] + } +} diff --git a/unittests/api_v3/test_apiv3_assets.py b/unittests/api_v3/test_apiv3_assets.py new file mode 100644 index 00000000000..55df5ceef6a --- /dev/null +++ b/unittests/api_v3/test_apiv3_assets.py @@ -0,0 +1,265 @@ +"""Asset CRUD + RBAC + contract tests for API v3 (OS3a; D11 wire rename product -> asset).""" +from __future__ import annotations + +from unittest import mock + +from django.db import connection +from django.test.utils import CaptureQueriesContext + +from dojo.models import Dojo_User, Product, Product_Type, User + +from .base import ApiV3TestCase + +_SLIM_KEYS = {"id", "name", "description", "organization", "lifecycle", "tags", "created", "updated"} + + +class TestApiV3AssetsRead(ApiV3TestCase): + + def test_list_envelope_and_slim_shape(self): + body = self.get_json("assets") + self.assertEqual({"count", "next", "previous", "results"}, set(body) - {"meta"}) + self.assertGreater(body["count"], 0) + row = body["results"][0] + self.assertEqual(_SLIM_KEYS, set(row)) + self.assertEqual({"id", "name"}, set(row["organization"])) + self.assertIsInstance(row["tags"], list) + + def test_detail_adds_heavy_fields(self): + product = Product.objects.first() + detail = self.get_json(f"assets/{product.id}") + for key in ("business_criticality", "platform", "origin", "asset_manager", "technical_contact", "team_manager"): + self.assertIn(key, detail) + + def test_detail_unknown_is_404_problem(self): + response = self.client.get(self.v3_url("assets/99999999")) + self.assertEqual(404, response.status_code) + self.assertEqual("application/problem+json", response["Content-Type"]) + + def test_expand_organization_inlines_slim(self): + row = self.get_json("assets", data={"expand": "organization"})["results"][0] + # ref swapped for the organization slim (carries description, timestamps, not just id/name). + self.assertIn("description", row["organization"]) + self.assertIn("critical_product", row["organization"]) + + def test_expand_unknown_relation_is_400(self): + self.get_json("assets", data={"expand": "not_a_relation"}, expected=400) + + +class TestApiV3AssetsFilters(ApiV3TestCase): + + def test_filter_organization(self): + pt_id = Product.objects.first().prod_type_id + body = self.get_json("assets", data={"organization": pt_id, "limit": 250}) + self.assertGreater(body["count"], 0) + for row in body["results"]: + self.assertEqual(pt_id, row["organization"]["id"]) + + def test_filter_name_icontains(self): + name = Product.objects.first().name + body = self.get_json("assets", data={"name__icontains": name[:5]}) + self.assertGreater(body["count"], 0) + + def test_ordering_by_name(self): + names = [r["name"] for r in self.get_json("assets", data={"o": "name", "limit": 250})["results"]] + self.assertEqual(names, sorted(names)) + + def test_unknown_filter_param_is_400(self): + self.get_json("assets", data={"not_a_filter": "x"}, expected=400) + + +class TestApiV3AssetsPagination(ApiV3TestCase): + + def test_limit_next_previous(self): + body = self.get_json("assets", data={"limit": 2, "offset": 2}) + self.assertLessEqual(len(body["results"]), 2) + self.assertIsNotNone(body["previous"]) + + +class TestApiV3AssetsQueryCount(ApiV3TestCase): + + def _bulk(self, count: int, start: int) -> None: + pt = Product_Type.objects.first() + Product.objects.bulk_create([ + Product(name=f"qcount asset {start + i}", description="x", prod_type=pt, sla_configuration_id=1) + for i in range(count) + ]) + + def _query_count(self, params: dict) -> int: + with CaptureQueriesContext(connection) as ctx: + response = self.client.get(self.v3_url("assets"), params) + self.assertEqual(200, response.status_code, response.content[:500]) + return len(ctx.captured_queries) + + def test_query_count_is_independent_of_row_count(self): + self._bulk(10, 0) + first = self._query_count({"limit": 250}) + first_expand = self._query_count({"limit": 250, "expand": "organization"}) + self._bulk(90, 100) + second = self._query_count({"limit": 250}) + second_expand = self._query_count({"limit": 250, "expand": "organization"}) + self.assertEqual(first, second, f"query count grew (no expand): {first} -> {second}") + self.assertEqual(first_expand, second_expand, f"query count grew (expand): {first_expand} -> {second_expand}") + + +class TestApiV3AssetsWrite(ApiV3TestCase): + + def test_create_happy_path(self): + pt = Product_Type.objects.first() + response = self.client.post( + self.v3_url("assets"), + {"name": "v3 created asset", "description": "made by v3", "organization": pt.id, + "lifecycle": "production", "tags": ["pci", "v3"]}, + format="json", + ) + self.assertEqual(201, response.status_code, response.content[:500]) + body = response.json() + self.assertEqual("v3 created asset", body["name"]) + self.assertEqual(pt.id, body["organization"]["id"]) + created = Product.objects.get(name="v3 created asset") + self.assertEqual("production", created.lifecycle) + self.assertEqual({"pci", "v3"}, {t.name for t in created.tags.all()}) + + def test_create_missing_required_is_400(self): + response = self.client.post(self.v3_url("assets"), {"name": "no organization"}, format="json") + self.assertEqual(400, response.status_code) + self.assertEqual("application/problem+json", response["Content-Type"]) + + def test_create_unknown_field_is_400(self): + pt = Product_Type.objects.first() + response = self.client.post( + self.v3_url("assets"), + {"name": "x", "description": "y", "organization": pt.id, "bogus": 1}, + format="json", + ) + self.assertEqual(400, response.status_code) + + def test_create_nonexistent_organization_is_404(self): + response = self.client.post( + self.v3_url("assets"), + {"name": "orphan", "description": "y", "organization": 99999999}, + format="json", + ) + self.assertEqual(404, response.status_code) + + def test_patch_partial_update(self): + pt = Product_Type.objects.first() + product = Product.objects.create(name="v3 patch asset", description="old", prod_type=pt, sla_configuration_id=1) + response = self.client.patch( + self.v3_url(f"assets/{product.id}"), {"description": "new"}, format="json", + ) + self.assertEqual(200, response.status_code, response.content[:500]) + product.refresh_from_db() + self.assertEqual("new", product.description) + self.assertEqual("v3 patch asset", product.name) + + def test_delete(self): + pt = Product_Type.objects.first() + product = Product.objects.create(name="v3 delete asset", description="d", prod_type=pt, sla_configuration_id=1) + response = self.client.delete(self.v3_url(f"assets/{product.id}")) + self.assertEqual(204, response.status_code) + self.assertFalse(Product.objects.filter(pk=product.id).exists()) + + +class TestApiV3AssetsReplace(ApiV3TestCase): + + """PUT full-replace: AssetReplace (required name/description/organization); omitted optionals reset.""" + + def _make_asset(self, **kwargs): + pt = Product_Type.objects.first() + defaults = {"name": "v3 put asset", "description": "old", "prod_type": pt, "sla_configuration_id": 1} + defaults.update(kwargs) + return Product.objects.create(**defaults), pt + + def test_put_full_replace_resets_omitted_optionals(self): + product, pt = self._make_asset(lifecycle="production", external_audience=True) + # PUT without lifecycle / external_audience -> nullable resets to None, non-null bool to False. + response = self.client.put( + self.v3_url(f"assets/{product.id}"), + {"name": "v3 put asset renamed", "description": "replaced", "organization": pt.id}, + format="json", + ) + self.assertEqual(200, response.status_code, response.content[:500]) + product.refresh_from_db() + self.assertEqual("v3 put asset renamed", product.name) + self.assertIsNone(product.lifecycle) # nullable -> reset to None + self.assertFalse(product.external_audience) # NOT NULL bool -> reset to model default False + + def test_put_reassign_organization(self): + product, _ = self._make_asset() + other = Product_Type.objects.create(name="v3 put asset other org") + response = self.client.put( + self.v3_url(f"assets/{product.id}"), + {"name": product.name, "description": "d", "organization": other.id}, + format="json", + ) + self.assertEqual(200, response.status_code, response.content[:500]) + product.refresh_from_db() + self.assertEqual(other.id, product.prod_type_id) + + def test_put_missing_required_is_400(self): + product, _ = self._make_asset() + response = self.client.put( + self.v3_url(f"assets/{product.id}"), {"name": "no org or description"}, format="json", + ) + self.assertEqual(400, response.status_code) + self.assertEqual("application/problem+json", response["Content-Type"]) + + def test_put_unknown_field_is_400(self): + product, pt = self._make_asset() + response = self.client.put( + self.v3_url(f"assets/{product.id}"), + {"name": "x", "description": "y", "organization": pt.id, "bogus": 1}, + format="json", + ) + self.assertEqual(400, response.status_code) + + def test_put_unauthorized_is_404(self): + product, pt = self._make_asset() + limited = User.objects.create_user(username="v3_asset_put_limited", password="x") # noqa: S106 + client = self.token_client(user=limited) + response = client.put( + self.v3_url(f"assets/{product.id}"), + {"name": "x", "description": "y", "organization": pt.id}, format="json", + ) + self.assertEqual(404, response.status_code) + + def test_put_visible_but_not_editable_is_403(self): + product, pt = self._make_asset() + with mock.patch("dojo.product.api_v3.routes.user_has_permission", return_value=False): + response = self.client.put( + self.v3_url(f"assets/{product.id}"), + {"name": "x", "description": "y", "organization": pt.id}, format="json", + ) + self.assertEqual(403, response.status_code, response.content[:300]) + + +class TestApiV3AssetsRbac(ApiV3TestCase): + + def setUp(self): + super().setUp() + self.limited = User.objects.create_user(username="v3_asset_limited", password="x") # noqa: S106 + # authorized_users M2M targets Dojo_User (the proxy), so the member must be a Dojo_User. + self.member = Dojo_User.objects.create_user(username="v3_asset_member", password="x") # noqa: S106 + self.product = Product.objects.first() + self.product.authorized_users.add(self.member) + + def test_unauthorized_read_is_404(self): + client = self.token_client(user=self.limited) + self.assertEqual(0, self.get_json("assets", client=client)["count"]) + self.get_json(f"assets/{self.product.id}", client=client, expected=404) + + def test_create_without_organization_add_is_403(self): + client = self.token_client(user=self.limited) + response = client.post( + self.v3_url("assets"), + {"name": "v3 rbac nope", "description": "d", "organization": self.product.prod_type_id}, + format="json", + ) + self.assertEqual(403, response.status_code, response.content[:300]) + self.assertEqual("application/problem+json", response["Content-Type"]) + + def test_member_can_view_but_delete_is_403(self): + client = self.token_client(user=self.member) + self.get_json(f"assets/{self.product.id}", client=client) + response = client.delete(self.v3_url(f"assets/{self.product.id}")) + self.assertEqual(403, response.status_code, response.content[:300]) diff --git a/unittests/api_v3/test_apiv3_auth.py b/unittests/api_v3/test_apiv3_auth.py new file mode 100644 index 00000000000..7420c7dd354 --- /dev/null +++ b/unittests/api_v3/test_apiv3_auth.py @@ -0,0 +1,49 @@ +"""Auth contract tests for API v3 (D8 / §4.2): token AND session+CSRF, both on the same endpoint.""" +from __future__ import annotations + +from .base import ApiV3TestCase + + +class TestApiV3Auth(ApiV3TestCase): + + def test_token_auth_get(self): + """A v2 token authenticates a v3 GET (same token store).""" + response = self.token_client().get(self.v3_url("findings")) + self.assertEqual(200, response.status_code, response.content[:500]) + self.assertEqual("alpha", response["X-API-Status"]) + + def test_session_auth_get(self): + """Django session authenticates a v3 GET (safe method needs no CSRF).""" + response = self.session_client().get(self.v3_url("findings")) + self.assertEqual(200, response.status_code, response.content[:500]) + + def test_anonymous_is_401_problem_json(self): + """No credentials -> 401 problem+json (not 403, not a redirect).""" + response = self.anonymous_client().get(self.v3_url("findings")) + self.assertEqual(401, response.status_code, response.content[:500]) + self.assertEqual("application/problem+json", response["Content-Type"]) + body = response.json() + self.assertEqual(401, body["status"]) + self.assertIn("type", body) + self.assertIn("title", body) + + def test_both_auth_modes_on_same_endpoint(self): + """The same endpoint accepts both token and session auth.""" + self.assertEqual(200, self.token_client().get(self.v3_url("findings")).status_code) + self.assertEqual(200, self.session_client().get(self.v3_url("findings")).status_code) + + def test_token_bypasses_csrf_on_unsafe_method(self): + """Header (token) auth needs no CSRF: an unsafe POST reaches the handler (400, not 401/403).""" + response = self.token_client().post( + self.v3_url("import"), {"scan_type": "ZAP Scan", "mode": "import"}, format="multipart", + ) + self.assertNotIn(response.status_code, (401, 403), response.content[:500]) + self.assertEqual(400, response.status_code, response.content[:500]) + + def test_session_csrf_enforced_on_unsafe_method(self): + """Cookie (session) auth on an unsafe method without a CSRF token -> 403.""" + client = self.session_client(enforce_csrf=True) + response = client.post( + self.v3_url("import"), {"scan_type": "ZAP Scan", "mode": "import"}, format="multipart", + ) + self.assertEqual(403, response.status_code, response.content[:500]) diff --git a/unittests/api_v3/test_apiv3_authz_static.py b/unittests/api_v3/test_apiv3_authz_static.py new file mode 100644 index 00000000000..b8c70bc0ad4 --- /dev/null +++ b/unittests/api_v3/test_apiv3_authz_static.py @@ -0,0 +1,273 @@ +""" +Structural authorization tripwire for API v3 (§5 I8, §10). + +A **source-scan** (no DB, no HTTP) over the v3 route layer that converts "someone forgot the RBAC +check" from a silent runtime hole into a compile-time-ish failure with a named location. It +complements the behavioural sweep (``test_apiv3_authz_sweep.py``): the sweep proves the *current* +routes deny by default; this test proves every route is *structurally wired* to an authorization +primitive, so a new route that skips the check fails CI even before anyone writes a request for it. + +Scanned files: ``dojo/*/api_v3/routes.py`` (every resource) + ``dojo/api_v3/import_routes.py`` + +``dojo/api_v3/subresources.py``. + +Two rules: + +**Rule A -- ban direct model-manager access.** Any source line containing ``.objects.`` fails unless +its ``Model.objects.method`` pattern is in ``OBJECTS_ALLOWLIST`` for that file, each entry carrying a +one-line justification. Reads MUST flow through the ``get_authorized_*`` querysets (I8); the only +legitimate ``.objects.`` uses are (a) edge reads from an already-authorized parent, (b) writes / +uniqueness pre-checks that run *after* an authorization gate, and (c) the users self-scope queryset +which *is* the RBAC mechanism. The allowlist is audited, never blanket -- and stale entries (allowing +a pattern that no longer occurs) fail too, so it cannot rot into a rubber stamp. + +**Rule B -- require an authorization reference.** (1) Every scanned file must reference at least one +authorization primitive. (2) Every function *registered as a route operation* (decorated with +``@router.`` or registered via ``router.(path, ...)(fn)``) must reference an authorization +primitive **in its own body, or in a helper it calls within the same module** (one level of +indirection -- ``_base_queryset`` / ``_detail_object`` / ``_require`` / the import ``_resolve_*`` / +``_check_auto_permission`` helpers count). Failures name ``file::function``. + +The authorization primitives (``AUTHZ_TOKENS``) are this codebase's real RBAC vocabulary: the +``get_authorized_*`` queryset helpers (``dojo//queries.py``), the ``user_has_*`` permission +helpers (``dojo/authorization/authorization.py`` -- object / global / configuration), the route-local +``_require*`` gate helpers, and the ``is_superuser`` gate that faithfully mirrors v2's ``IsSuperUser`` +class (§12 OS4). +""" +from __future__ import annotations + +import ast +from pathlib import Path + +from django.test import SimpleTestCase + +# --- what to scan ----------------------------------------------------------------------------- +# Repo root: unittests/api_v3/ -> parents[2]. +_REPO_ROOT = Path(__file__).resolve().parents[2] + + +def _scanned_files() -> list[Path]: + """Every v3 route module (resources are globbed so a new resource is auto-included).""" + files = sorted((_REPO_ROOT / "dojo").glob("*/api_v3/routes.py")) + files.append(_REPO_ROOT / "dojo" / "api_v3" / "import_routes.py") + files.append(_REPO_ROOT / "dojo" / "api_v3" / "subresources.py") + return files + + +# --- Rule A: the audited .objects. allowlist -------------------------------------------------- +# Keys are (repo-relative posix path, "Model.objects.method"). Every entry was verified by reading +# the code: each such call either reads from an ALREADY-authorized parent, or runs a write / +# uniqueness check AFTER an authorization gate, or IS the RBAC scoping itself. Nothing here is an +# unauthorized object read. A ".objects." line whose pattern is absent here fails the test. +OBJECTS_ALLOWLIST: dict[tuple[str, str], str] = { + ("dojo/location/api_v3/routes.py", "LocationFindingReference.objects.filter"): + "edge rows read from a parent finding ALREADY resolved via get_authorized_findings " + "(parent-inherited authz); mirrors the v2 LocationFindingReference viewset.", + ("dojo/location/api_v3/routes.py", "LocationProductReference.objects.filter"): + "edge rows read from a parent asset ALREADY resolved via get_authorized_products " + "(parent-inherited authz); mirrors the v2 LocationProductReference viewset.", + ("dojo/api_v3/subresources.py", "NoteHistory.objects.create"): + "writes the NoteHistory row for a note just created on a parent that was authorized and " + "permission-checked; a write of an owned sub-object, not an object read.", + ("dojo/api_v3/subresources.py", "FileUpload.objects.filter"): + "global title-uniqueness pre-check (mirrors DRF UniqueValidator -> 400) AFTER the parent " + "authz + add-permission gate; not object disclosure.", + ("dojo/user/api_v3/routes.py", "Dojo_User.objects.filter"): + "the RBAC scoping itself: the self-only fallback queryset for users lacking auth.view_user " + "(line 100) and the username-uniqueness check AFTER the auth.add_user gate (line 181).", +} + +# --- Rule B: authorization primitives --------------------------------------------------------- +# Substrings that mark an authorization reference. `user_has_` covers the object / global / +# configuration permission helpers; `get_authorized_` the RBAC querysets; `_require` the route-local +# gate helpers (_require / _require_permission / _require_config_permission); `is_superuser` the +# faithful v2 IsSuperUser mirror (§12 OS4). +AUTHZ_TOKENS = ("get_authorized_", "user_has_", "_require", "is_superuser") + +_ROUTE_VERBS = {"get", "post", "put", "patch", "delete"} + + +def _rel(path: Path) -> str: + return path.relative_to(_REPO_ROOT).as_posix() + + +def _is_router_verb_attr(node: ast.AST) -> bool: + """True for an attribute access ``router.`` (get/post/put/patch/delete).""" + return ( + isinstance(node, ast.Attribute) + and node.attr in _ROUTE_VERBS + and isinstance(node.value, ast.Name) + and node.value.id == "router" + ) + + +def _is_router_decorator(dec: ast.AST) -> bool: + """True for ``@router.(...)``.""" + return isinstance(dec, ast.Call) and _is_router_verb_attr(dec.func) + + +def _own_nodes(func: ast.FunctionDef): + """ + Yield the descendant nodes of ``func``'s body WITHOUT descending into nested function/lambda + definitions -- so a helper is credited only for authorization references in its *own* logic, + never for tokens that live in a closure it merely encloses. + """ + stack = list(func.body) + while stack: + node = stack.pop() + yield node + for child in ast.iter_child_nodes(node): + if isinstance(child, (ast.FunctionDef, ast.AsyncFunctionDef, ast.Lambda)): + continue + stack.append(child) + + +def _own_identifiers(func: ast.FunctionDef) -> set[str]: + """All ``Name``/``Attribute`` identifiers used directly in ``func``'s own body.""" + idents: set[str] = set() + for node in _own_nodes(func): + if isinstance(node, ast.Name): + idents.add(node.id) + elif isinstance(node, ast.Attribute): + idents.add(node.attr) + return idents + + +def _own_call_targets(func: ast.FunctionDef) -> set[str]: + """Names of same-module functions ``func`` calls directly (for one-level indirection).""" + targets: set[str] = set() + for node in _own_nodes(func): + if isinstance(node, ast.Call) and isinstance(node.func, ast.Name): + targets.add(node.func.id) + return targets + + +def _has_authz_token(idents: set[str]) -> bool: + return any(any(tok in ident for tok in AUTHZ_TOKENS) for ident in idents) + + +def _manually_registered_names(tree: ast.AST) -> set[str]: + """ + Function names registered via ``router.(path, ...)(fn)`` (subresources.py registers its + handlers manually so each gets a unique __name__ / operationId). The outer node is a Call whose + ``func`` is itself the ``router.(...)`` Call; its positional Name args are the handlers. + """ + names: set[str] = set() + for node in ast.walk(tree): + if ( + isinstance(node, ast.Call) + and isinstance(node.func, ast.Call) + and _is_router_verb_attr(node.func.func) + ): + names.update(arg.id for arg in node.args if isinstance(arg, ast.Name)) + return names + + +def _route_operations(tree: ast.AST) -> list[ast.FunctionDef]: + """Every function registered as a route operation (decorated OR manually registered).""" + registered = _manually_registered_names(tree) + ops: list[ast.FunctionDef] = [] + for node in ast.walk(tree): + if not isinstance(node, ast.FunctionDef): + continue + if any(_is_router_decorator(dec) for dec in node.decorator_list) or node.name in registered: + ops.append(node) + return ops + + +def _authz_helper_names(tree: ast.AST) -> set[str]: + """Same-module functions whose OWN body references an authorization primitive.""" + return { + node.name + for node in ast.walk(tree) + if isinstance(node, ast.FunctionDef) and _has_authz_token(_own_identifiers(node)) + } + + +class TestApiV3AuthzStatic(SimpleTestCase): + + """Source-scan tripwire: no raw manager access; every route operation is authz-wired.""" + + def test_scanned_files_exist(self): + """Guard the glob: the resource route modules must actually be found and scanned.""" + files = _scanned_files() + rels = {_rel(f) for f in files} + self.assertTrue(all(f.exists() for f in files), f"missing scanned file(s): {sorted(rels)}") + # A representative subset must be present (catches a broken glob / moved module). + for expected in ( + "dojo/finding/api_v3/routes.py", + "dojo/location/api_v3/routes.py", + "dojo/api_v3/import_routes.py", + "dojo/api_v3/subresources.py", + ): + self.assertIn(expected, rels, f"{expected} not in the scan set -- fix _scanned_files()") + + def test_no_unjustified_objects_manager_access(self): + """Rule A: every ``.objects.`` line must be in the audited allowlist (with a justification).""" + failures: list[str] = [] + seen_allowlist_keys: set[tuple[str, str]] = set() + for path in _scanned_files(): + rel = _rel(path) + for lineno, line in enumerate(path.read_text(encoding="utf-8").splitlines(), start=1): + if ".objects." not in line: + continue + pattern = self._objects_pattern(line) + key = (rel, pattern) + if key in OBJECTS_ALLOWLIST: + seen_allowlist_keys.add(key) + continue + failures.append( + f"{rel}:{lineno}: direct model-manager access `{pattern}` -- reads must go " + f"through get_authorized_* (I8). If this is a genuine exception, add " + f"('{rel}', '{pattern}') to OBJECTS_ALLOWLIST with a one-line justification.", + ) + self.assertFalse(failures, "unjustified .objects. access:\n" + "\n".join(failures)) + + # Keep the allowlist honest: no entry may allow a pattern that no longer occurs. + stale = set(OBJECTS_ALLOWLIST) - seen_allowlist_keys + self.assertFalse( + stale, + f"stale OBJECTS_ALLOWLIST entr(y/ies) that match nothing anymore -- remove: {sorted(stale)}", + ) + + def test_every_file_references_an_authz_primitive(self): + """Rule B(1): each scanned file references at least one authorization primitive.""" + failures: list[str] = [] + for path in _scanned_files(): + source = path.read_text(encoding="utf-8") + if not any(tok in source for tok in AUTHZ_TOKENS): + failures.append(f"{_rel(path)}: no authorization primitive {AUTHZ_TOKENS} anywhere in file") + self.assertFalse(failures, "file with no authorization reference:\n" + "\n".join(failures)) + + def test_every_route_operation_is_authz_wired(self): + """Rule B(2): every route operation references an authz primitive directly or one helper deep.""" + failures: list[str] = [] + for path in _scanned_files(): + rel = _rel(path) + tree = ast.parse(path.read_text(encoding="utf-8")) + authz_helpers = _authz_helper_names(tree) + operations = _route_operations(tree) + # Sanity: the heuristic must find at least one route operation per file, otherwise a + # broken detector would vacuously "pass" the whole file. + self.assertTrue( + operations, + f"{rel}: no route operation detected -- the @router. / router.(...)(fn) " + f"heuristic may be broken (or the file no longer defines routes).", + ) + for op in operations: + direct = _has_authz_token(_own_identifiers(op)) + indirect = bool(_own_call_targets(op) & authz_helpers) + if not (direct or indirect): + failures.append( + f"{rel}::{op.name} (line {op.lineno}): route operation has no authorization " + f"reference in its body or in a same-module helper it calls -- it must flow " + f"through get_authorized_* / user_has_* / _require* / is_superuser (I8).", + ) + self.assertFalse(failures, "route operation missing an authorization check:\n" + "\n".join(failures)) + + @staticmethod + def _objects_pattern(line: str) -> str: + """Extract the ``Model.objects.method`` token from a line (fallback: the stripped line).""" + import re # noqa: PLC0415 -- localized to this helper + + match = re.search(r"(\w+)\.objects\.(\w+)", line) + return f"{match.group(1)}.objects.{match.group(2)}" if match else line.strip() diff --git a/unittests/api_v3/test_apiv3_authz_sweep.py b/unittests/api_v3/test_apiv3_authz_sweep.py new file mode 100644 index 00000000000..8422f77a01e --- /dev/null +++ b/unittests/api_v3/test_apiv3_authz_sweep.py @@ -0,0 +1,438 @@ +""" +Deny-by-default authorization sweep for API v3 (§5 I8, §10). + +Enforces the RBAC invariant (I8) *structurally* across the **whole mounted surface**: every +operation in ``api_v3.get_openapi_schema()`` -- all methods, all paths -- is probed twice, and a +completeness gate (identical in spirit to ``test_apiv3_query_report.py``) fails the moment an +operation exists in the schema without a registered probe, so the sweep can never silently fall +behind a new endpoint. + +Two probes per operation: + +1. **Anonymous** -> must be **401**. Ninja runs authentication in ``Operation._run_checks`` *before* + it parses path/query/body (``_get_values``), so an anonymous request never reaches request + validation and needs no payload (confirmed by ``test_apiv3_auth``: anonymous GET -> 401 problem+ + json, v3 is in ``LOGIN_EXEMPT_URLS`` so there is no login redirect). + +2. **Zero-permission user** -- a freshly created, authenticated user with **no** memberships, roles + or configuration permissions -- must **NEVER receive data**: + + * ``GET`` list -> 200 with ``count == 0`` / ``results == []`` (RBAC-scoped empty); + ``/findings`` additionally checked with ``?include=counts`` (all totals must be 0); + ``/locations`` is the superuser-gated exception -> **403**; + ``/users`` is the documented self-visibility exception -> 200 with **exactly the caller's own + record** (§12 OS3a: a user always sees themselves and nobody else -- not a leak). + * ``GET`` detail / sub-resource ``GET`` -> **404** (unknown-or-unauthorized, never leak + existence, §4.10); ``/locations/{id}`` -> **403** (superuser gate). + * writes (POST/PATCH/PUT/DELETE) -> **403 or 404**. Each write probe carries a *minimal valid + payload* referencing fixture ids that exist but are NOT authorized for the zero-perm user, so + it gets past ninja request-validation and genuinely reaches the authorization gate. **A + 400/422 on a write probe is a sweep FAILURE** -- it means validation, not authz, produced the + denial (the gate was never exercised). The registry below reaches the gate for every write, so + there are zero validation-blocked probes. + +Because the whole thing is registry-driven and gated for completeness, any future endpoint fails +``test_sweep_covers_every_operation`` until a probe (request + expected outcome) is added for it. +""" +from __future__ import annotations + +import csv +import io +from dataclasses import dataclass +from typing import TYPE_CHECKING + +from django.conf import settings +from django.core.files.uploadedfile import SimpleUploadedFile + +from dojo.api_v3.api import api_v3 +from dojo.models import ( + Dojo_User, + Engagement, + Finding, + Product, + Product_Type, + Test, + Test_Type, +) + +from .base import ApiV3TestCase + +if TYPE_CHECKING: + from rest_framework.test import APIClient + +_HTTP_METHODS = {"get", "post", "put", "patch", "delete"} + + +@dataclass +class Probe: + + """One operation's representative request + its expected deny outcomes.""" + + method: str # HTTP verb, upper-case + path: str # OpenAPI mount-relative template, e.g. "/findings/{finding_id}" + url: str # concrete request URL (fixture ids substituted) + zero_status: int # expected status for the zero-permission user + payload: object = None # request body for the zero-perm write probe (dict) or None + fmt: str = "json" # "json" | "multipart" + list_kind: str | None = None # "empty" | "findings" | "users_self" -> zero-perm list assertion + note: str = "" # documentation / justification for the chosen outcome + + @property + def key(self) -> tuple[str, str]: + return (self.method, self.path) + + +class TestApiV3AuthzSweep(ApiV3TestCase): + + """Whole-surface deny-by-default sweep: anonymous -> 401, zero-perm -> never any data.""" + + def setUp(self): + super().setUp() + # A brand-new authenticated user with zero authorization: no product/type membership, no + # role, no configuration permission. Mirrors the "limited"/"member-before-grant" users the + # other v3 RBAC tests rely on (test_apiv3_finding_writes / test_apiv3_expand_rbac), which + # confirm a freshly created user sees nothing. + self.zero = Dojo_User.objects.create_user(username="v3_authz_zero", password="x") # noqa: S106 + self.zero_client: APIClient = self.token_client(user=self.zero) + + # Fixture objects the zero-perm user is NOT authorized for. They exist (so the write probes + # reach the authorization gate rather than 404-ing on a non-existent id where the outcome + # would be ambiguous) but are invisible to the zero-perm user's authorized querysets. + self.finding = Finding.objects.first() + self.test = Test.objects.first() + self.engagement = Engagement.objects.first() + self.asset = Product.objects.first() + self.organization = Product_Type.objects.first() + self.test_type = Test_Type.objects.first() + + # --- schema introspection ----------------------------------------------------------------- + def _schema_operations(self) -> set[tuple[str, str]]: + """Every (METHOD, mount-relative path) in the live OpenAPI schema (the completeness set).""" + schema = api_v3.get_openapi_schema() + ops: set[tuple[str, str]] = set() + for path, item in schema["paths"].items(): + # Schema paths carry the mount prefix (/api/v3-alpha/...); compare mount-relative, + # exactly as test_apiv3_query_report does. + rel = "/" + path.split(settings.API_V3_URL_PREFIX, 1)[-1].lstrip("/") + for method in item: + if method.lower() in _HTTP_METHODS: + ops.add((method.upper(), rel)) + return ops + + # --- probe registry ----------------------------------------------------------------------- + def _file_payload(self) -> dict: + """Fresh multipart body for a file/import write probe (a tiny scan file).""" + return { + "title": "authz-sweep-file", + "file": SimpleUploadedFile("authz-sweep.txt", b"authz sweep body", content_type="text/plain"), + } + + def _sub_probes(self, resource: str, parent_id: int, *, notes: bool, files: bool, tags: bool) -> list[Probe]: + """ + Sub-resource probes (§4.12). Every one is parent-inherited: the zero-perm user cannot see + the parent, so the parent resolves to None through its authorized-view queryset -> 404, + before any note/file/tag is touched (never leak existence, never mutate). + """ + base = f"/{resource}/{{parent_id}}" + probes: list[Probe] = [] + if notes: + probes += [ + Probe("GET", f"{base}/notes", self.v3_url(f"{resource}/{parent_id}/notes"), 404), + Probe("POST", f"{base}/notes", self.v3_url(f"{resource}/{parent_id}/notes"), 404, + payload={"entry": "authz sweep"}), + ] + if files: + probes += [ + Probe("GET", f"{base}/files", self.v3_url(f"{resource}/{parent_id}/files"), 404), + Probe("POST", f"{base}/files", self.v3_url(f"{resource}/{parent_id}/files"), 404, + payload=self._file_payload(), fmt="multipart"), + # file_id is irrelevant: parent-view 404 fires before the file lookup. + Probe("GET", f"{base}/files/{{file_id}}/download", + self.v3_url(f"{resource}/{parent_id}/files/1/download"), 404), + ] + if tags: + probes += [ + Probe("GET", f"{base}/tags", self.v3_url(f"{resource}/{parent_id}/tags"), 404), + Probe("PUT", f"{base}/tags", self.v3_url(f"{resource}/{parent_id}/tags"), 404, + payload={"tags": ["authz"]}), + Probe("POST", f"{base}/tags", self.v3_url(f"{resource}/{parent_id}/tags"), 404, + payload={"tags": ["authz"]}), + Probe("DELETE", f"{base}/tags/{{tag}}", + self.v3_url(f"{resource}/{parent_id}/tags/authz"), 404), + ] + return probes + + def _probes(self) -> list[Probe]: + f, e, t = self.finding.id, self.engagement.id, self.test.id + a, o, tt = self.asset.id, self.organization.id, self.test_type.id + admin_id = self.admin.id + p: list[Probe] = [] + + # ---- findings (list supports include=counts) ----------------------------------------- + p += [ + Probe("GET", "/findings", self.v3_url("findings"), 200, list_kind="findings"), + Probe("GET", "/findings/{finding_id}", self.v3_url(f"findings/{f}"), 404), + Probe("POST", "/findings", self.v3_url("findings"), 403, + payload={"test": t, "title": "authz sweep", "severity": "High", + "description": "x", "active": True, "verified": False}, + note="existing but unauthorized test -> Finding_Add gate -> 403"), + Probe("PATCH", "/findings/{finding_id}", self.v3_url(f"findings/{f}"), 404, + payload={"severity": "Low"}, note="finding invisible to zero-perm queryset -> 404"), + Probe("PUT", "/findings/{finding_id}", self.v3_url(f"findings/{f}"), 404, + payload={"title": "authz sweep", "severity": "High", "description": "x", + "active": True, "verified": False}, + note="full replace: finding invisible to zero-perm queryset -> 404 before edit gate"), + Probe("DELETE", "/findings/{finding_id}", self.v3_url(f"findings/{f}"), 404), + Probe("GET", "/findings/{finding_id}/locations", self.v3_url(f"findings/{f}/locations"), 404), + Probe("GET", "/findings/export.csv", self.v3_url("findings/export.csv"), 200, list_kind="csv_empty", + note="CSV export: zero-perm -> header-only CSV over the RBAC-scoped empty queryset"), + ] + p += self._sub_probes("findings", f, notes=True, files=True, tags=True) + + # ---- organizations (product_type) ---------------------------------------------------- + p += [ + Probe("GET", "/organizations", self.v3_url("organizations"), 200, list_kind="empty"), + Probe("GET", "/organizations/{organization_id}", self.v3_url(f"organizations/{o}"), 404), + Probe("POST", "/organizations", self.v3_url("organizations"), 403, + payload={"name": "authz sweep org"}, + note="no global add permission -> 403"), + Probe("PATCH", "/organizations/{organization_id}", self.v3_url(f"organizations/{o}"), 404, + payload={"name": "x"}), + Probe("PUT", "/organizations/{organization_id}", self.v3_url(f"organizations/{o}"), 404, + payload={"name": "x"}, note="full replace: organization invisible -> 404"), + Probe("DELETE", "/organizations/{organization_id}", self.v3_url(f"organizations/{o}"), 404), + Probe("GET", "/organizations/export.csv", self.v3_url("organizations/export.csv"), 200, + list_kind="csv_empty", note="CSV export: zero-perm -> header-only CSV (RBAC-scoped empty)"), + ] + + # ---- assets (product) ---------------------------------------------------------------- + p += [ + Probe("GET", "/assets", self.v3_url("assets"), 200, list_kind="empty"), + Probe("GET", "/assets/{asset_id}", self.v3_url(f"assets/{a}"), 404), + Probe("POST", "/assets", self.v3_url("assets"), 403, + payload={"name": "authz sweep asset", "description": "x", "organization": o}, + note="existing but unauthorized organization -> Product_Type_Add_Product gate -> 403"), + Probe("PATCH", "/assets/{asset_id}", self.v3_url(f"assets/{a}"), 404, payload={"name": "x"}), + Probe("PUT", "/assets/{asset_id}", self.v3_url(f"assets/{a}"), 404, + payload={"name": "x", "description": "y", "organization": o}, + note="full replace: asset invisible -> 404 before edit gate"), + Probe("DELETE", "/assets/{asset_id}", self.v3_url(f"assets/{a}"), 404), + Probe("GET", "/assets/{asset_id}/locations", self.v3_url(f"assets/{a}/locations"), 404), + Probe("GET", "/assets/export.csv", self.v3_url("assets/export.csv"), 200, list_kind="csv_empty", + note="CSV export: zero-perm -> header-only CSV (RBAC-scoped empty)"), + ] + p += self._sub_probes("assets", a, notes=False, files=False, tags=True) + + # ---- engagements --------------------------------------------------------------------- + p += [ + Probe("GET", "/engagements", self.v3_url("engagements"), 200, list_kind="empty"), + Probe("GET", "/engagements/{engagement_id}", self.v3_url(f"engagements/{e}"), 404), + Probe("POST", "/engagements", self.v3_url("engagements"), 403, + payload={"asset": a, "target_start": "2026-01-01", "target_end": "2026-01-02"}, + note="existing but unauthorized asset -> Engagement_Add gate -> 403"), + Probe("PATCH", "/engagements/{engagement_id}", self.v3_url(f"engagements/{e}"), 404, + payload={"name": "x"}), + Probe("PUT", "/engagements/{engagement_id}", self.v3_url(f"engagements/{e}"), 404, + payload={"asset": a, "target_start": "2026-01-01", "target_end": "2026-01-02"}, + note="full replace: engagement invisible -> 404 before edit gate"), + Probe("DELETE", "/engagements/{engagement_id}", self.v3_url(f"engagements/{e}"), 404), + Probe("GET", "/engagements/export.csv", self.v3_url("engagements/export.csv"), 200, + list_kind="csv_empty", note="CSV export: zero-perm -> header-only CSV (RBAC-scoped empty)"), + ] + p += self._sub_probes("engagements", e, notes=True, files=True, tags=True) + + # ---- tests --------------------------------------------------------------------------- + p += [ + Probe("GET", "/tests", self.v3_url("tests"), 200, list_kind="empty"), + Probe("GET", "/tests/{test_id}", self.v3_url(f"tests/{t}"), 404), + Probe("POST", "/tests", self.v3_url("tests"), 403, + payload={"engagement": e, "test_type": tt, + "target_start": "2026-01-01T00:00:00Z", "target_end": "2026-01-02T00:00:00Z"}, + note="existing but unauthorized engagement -> Test_Add gate -> 403"), + Probe("PATCH", "/tests/{test_id}", self.v3_url(f"tests/{t}"), 404, payload={"title": "x"}), + Probe("PUT", "/tests/{test_id}", self.v3_url(f"tests/{t}"), 404, + payload={"test_type": tt, "target_start": "2026-01-01T00:00:00Z", + "target_end": "2026-01-02T00:00:00Z"}, + note="full replace: test invisible -> 404 before edit gate"), + Probe("DELETE", "/tests/{test_id}", self.v3_url(f"tests/{t}"), 404), + Probe("GET", "/tests/export.csv", self.v3_url("tests/export.csv"), 200, list_kind="csv_empty", + note="CSV export: zero-perm -> header-only CSV (RBAC-scoped empty)"), + ] + p += self._sub_probes("tests", t, notes=True, files=True, tags=True) + + # ---- users (self-visibility exception on list/detail; writes admin-only) ------------- + p += [ + Probe("GET", "/users", self.v3_url("users"), 200, list_kind="users_self", + note="a user always sees exactly their own record and no other (§12 OS3a) -- not a leak"), + # Target the admin's id: invisible to the zero-perm user's self-only queryset -> 404. + Probe("GET", "/users/{user_id}", self.v3_url(f"users/{admin_id}"), 404), + Probe("POST", "/users", self.v3_url("users"), 403, + payload={"username": "authz_sweep_new", "email": "authz@example.com"}, + note="no auth.add_user configuration permission -> 403"), + Probe("PATCH", "/users/{user_id}", self.v3_url(f"users/{admin_id}"), 404, + payload={"first_name": "x"}, note="admin invisible to zero-perm self-scope -> 404"), + Probe("PUT", "/users/{user_id}", self.v3_url(f"users/{admin_id}"), 404, + payload={"username": "authz_sweep_put", "email": "authzput@example.com"}, + note="full replace: admin invisible to zero-perm self-scope -> 404"), + Probe("DELETE", "/users/{user_id}", self.v3_url(f"users/{admin_id}"), 404), + Probe("GET", "/users/export.csv", self.v3_url("users/export.csv"), 200, list_kind="csv_users_self", + note="CSV export: self-visibility scope -> exactly the caller's own record, no other (§12 OS3a)"), + ] + + # ---- locations (superuser-gated: 403 before any object lookup) ----------------------- + p += [ + Probe("GET", "/locations", self.v3_url("locations"), 403, + note="v2 LocationViewSet is IsSuperUser -> mirrored 403 for non-superusers (§12 OS4)"), + Probe("GET", "/locations/{location_id}", self.v3_url("locations/1"), 403, + note="superuser gate fires before the id lookup"), + Probe("GET", "/locations/export.csv", self.v3_url("locations/export.csv"), 403, + note="CSV export inherits the /locations superuser gate -> 403 before streaming (§12 OS4)"), + ] + + # ---- consolidated import ------------------------------------------------------------- + import_payload = self._file_payload() + import_payload.update({"scan_type": "ZAP Scan", "mode": "import", "engagement": e}) + p += [ + Probe("POST", "/import", self.v3_url("import"), 403, + payload=import_payload, fmt="multipart", + note="mode=import + existing but unauthorized engagement -> UserHasImportPermission -> 403"), + ] + return p + + # --- dispatch ----------------------------------------------------------------------------- + def _dispatch(self, client: APIClient, probe: Probe, *, with_payload: bool): + fn = getattr(client, probe.method.lower()) + if with_payload and probe.payload is not None: + return fn(probe.url, probe.payload, format=probe.fmt) + return fn(probe.url) + + @staticmethod + def _read_csv(response) -> list[list[str]]: + """Consume a streaming CSV export response into a list of rows (header first).""" + content = b"".join(response.streaming_content).decode("utf-8") + return list(csv.reader(io.StringIO(content))) + + def _assert_zero_list(self, probe: Probe, response) -> None: + label = f"{probe.method} {probe.path}" + if probe.list_kind in {"csv_empty", "csv_users_self"}: + # CSV export projection of the deny-by-default invariant: a zero-perm user gets a valid + # CSV whose data body is empty (header only) -- except /users, the documented + # self-visibility scope, where the export contains exactly the caller's own record and no + # other (identical filter contract + RBAC queryset as the /users list, §12 OS3a). + rows = self._read_csv(response) + self.assertGreaterEqual(len(rows), 1, f"{label}: a CSV export must always emit a header row") + header, data_rows = rows[0], rows[1:] + self.assertIn("id", header, f"{label}: CSV header must include the id column") + if probe.list_kind == "csv_empty": + self.assertEqual([], data_rows, f"{label}: zero-perm CSV export must be header-only (no data rows)") + return + self.assertEqual( + 1, len(data_rows), f"{label}: users CSV export must return exactly the caller's own record", + ) + self.assertEqual( + str(self.zero.id), data_rows[0][header.index("id")], + f"{label}: users CSV export leaked a record other than the caller's own", + ) + return + body = response.json() + if probe.list_kind == "users_self": + # The one documented deviation: a zero-perm user sees exactly their own record and no + # other user's data. Anything else (count 0, or a foreign row) would be a regression. + self.assertEqual(1, body["count"], f"{label}: users list must return exactly the caller's own record") + self.assertEqual( + [self.zero.id], [row["id"] for row in body["results"]], + f"{label}: users list leaked a record other than the caller's own", + ) + return + # Every other list must be RBAC-scoped empty for a zero-permission user. + self.assertEqual(0, body["count"], f"{label}: expected an empty (RBAC-scoped) list, got count={body['count']}") + self.assertEqual([], body["results"], f"{label}: expected results == [] for a zero-permission user") + if probe.list_kind == "findings": + # include=counts must aggregate over the (empty) authorized queryset -> all zero. + counts = self.get_json( + "findings", client=self.zero_client, data={"include": "counts"}, + )["meta"]["counts"] + self.assertEqual(0, counts["total"], f"{label}: include=counts total must be 0 for a zero-perm user") + for sev, n in counts["severity"].items(): + self.assertEqual(0, n, f"{label}: include=counts severity[{sev}] must be 0, got {n}") + + # --- tests -------------------------------------------------------------------------------- + def test_sweep_covers_every_operation(self): + """Completeness gate: every schema operation must have exactly one registered probe.""" + registered = {probe.key for probe in self._probes()} + schema_ops = self._schema_operations() + + missing = schema_ops - registered + self.assertFalse( + missing, + f"operation(s) {sorted(missing)} have no entry in the authorization sweep -- add a " + f"representative request + expected outcome to _probes() (deliberate: every new " + f"endpoint must be authorization-probed before it can ship).", + ) + extra = registered - schema_ops + self.assertFalse( + extra, + f"authorization sweep probes {sorted(extra)} no longer match any schema operation -- " + f"remove or fix the stale probe(s).", + ) + # Guard against accidental duplicate probes silently masking a gap. + keys = [probe.key for probe in self._probes()] + self.assertEqual(len(keys), len(set(keys)), "duplicate probe keys in _probes()") + + def test_anonymous_is_401_on_every_operation(self): + """Probe (a): an unauthenticated request is 401 on every operation (auth before body parse).""" + failures = [] + for probe in self._probes(): + response = self._dispatch(self.anonymous_client(), probe, with_payload=False) + if response.status_code != 401: + failures.append(f"{probe.method} {probe.path} -> {response.status_code} (expected 401)") + self.assertFalse(failures, "anonymous requests must be 401:\n" + "\n".join(failures)) + + def test_zero_permission_user_never_receives_data(self): + """Probe (b): a fully authenticated but zero-permission user never receives/mutates data.""" + failures = [] + for probe in self._probes(): + response = self._dispatch(self.zero_client, probe, with_payload=True) + label = f"{probe.method} {probe.path}" + # A write probe that 400/422s never reached the authz gate -- that is a sweep failure. + if probe.method != "GET" and response.status_code in {400, 422}: + failures.append( + f"{label} -> {response.status_code}: write probe blocked at request VALIDATION, " + f"not authorization -- fix the probe payload so it reaches the authz gate " + f"(body: {response.content[:200]!r})", + ) + continue + if response.status_code != probe.zero_status: + failures.append( + f"{label} -> {response.status_code} (expected {probe.zero_status})" + f"{' [' + probe.note + ']' if probe.note else ''} " + f"body: {response.content[:200]!r}", + ) + continue + if response.status_code == 200 and probe.list_kind is not None: + self._assert_zero_list(probe, response) + self.assertFalse( + failures, + "zero-permission user received an unexpected outcome (potential authorization gap):\n" + + "\n".join(failures), + ) + + def test_zero_perm_lists_empty_but_admin_sees_data(self): + """ + Sanity contrast proving the zero-perm empties are *authorization*, not an empty DB: the + admin (superuser) sees rows on the same lists where the zero-perm user sees none. + """ + for resource in ("findings", "organizations", "assets", "engagements", "tests"): + admin_body = self.get_json(resource) + self.assertGreater( + admin_body["count"], 0, + f"fixture has no {resource} for the admin -- the empty-list contrast is meaningless", + ) + zero_body = self.get_json(resource, client=self.zero_client) + self.assertEqual(0, zero_body["count"], f"zero-perm user unexpectedly sees {resource}") + # Locations: admin (superuser) 200, zero-perm 403 (superuser gate). + self.get_json("locations") + self.get_json("locations", client=self.zero_client, expected=403) + # Users: admin sees many; zero-perm sees only itself. + self.assertGreater(self.get_json("users")["count"], 1) + self.assertEqual(1, self.get_json("users", client=self.zero_client)["count"]) diff --git a/unittests/api_v3/test_apiv3_benchmark.py b/unittests/api_v3/test_apiv3_benchmark.py new file mode 100644 index 00000000000..c379aed9515 --- /dev/null +++ b/unittests/api_v3/test_apiv3_benchmark.py @@ -0,0 +1,191 @@ +r""" +In-process latency + query-count benchmark: v2 ``?prefetch=`` vs v3 ``?expand=`` (OS6, §11). + +**CI-EXCLUDED.** Gated behind ``DD_API_V3_BENCH=1`` so the normal suite never pays for it. Run: + + docker compose exec -e DD_API_V3_BENCH=1 uwsgi \\ + python manage.py test unittests.api_v3.test_apiv3_benchmark -v2 --keepdb + +It seeds ~1000 findings on ONE product/test, then times ``N=30`` repeated in-process test-client +GETs against each endpoint and captures the per-request SQL count. The methodology + caveats and +the results table are written **verbatim** to ``.claude/os6-benchmark.md`` (repo root is bind-mounted +into the container) and echoed to stdout. Numbers are **never fabricated**. + +Honesty caveats (also written into the report): these are IN-PROCESS numbers -- no HTTP layer, no +uwsgi worker, no network, no connection-pool warmup; single-threaded; the requests share one test +transaction against the (kept) test DB. They are therefore **directional** (they isolate the +serialization + ORM cost that the query-count gap predicts), not a substitute for a wall-clock +latency benchmark against a running uwsgi stack (the procedure for which is in +``.claude/os1-gate-report.md``). The query counts, by contrast, are exact and environment-independent. +""" +from __future__ import annotations + +import math +import os +import statistics +import time +from pathlib import Path +from unittest import skipUnless + +from django.db import connection +from django.test.utils import CaptureQueriesContext + +from dojo.models import Finding, Test +from dojo.utils import get_system_setting + +from .base import ApiV3TestCase + +_BENCH_ON = os.environ.get("DD_API_V3_BENCH") == "1" +_SEED = 1000 # findings bulk-created on one test/product +_N = 30 # timed repetitions per endpoint +_LIMIT = 100 # page size under test +_REPO_ROOT = Path(__file__).resolve().parents[2] +_REPORT = _REPO_ROOT / ".claude" / "os6-benchmark.md" + + +def _percentile(values: list[float], pct: float) -> float: + """Nearest-rank percentile (e.g. p95 of 30 samples -> the 29th smallest).""" + ordered = sorted(values) + idx = max(0, math.ceil(pct / 100 * len(ordered)) - 1) + return ordered[idx] + + +@skipUnless(_BENCH_ON, "benchmark is CI-excluded; set DD_API_V3_BENCH=1 to run") +class TestApiV3Benchmark(ApiV3TestCase): + + def _v2_url(self, query: str) -> str: + prefix = get_system_setting("url_prefix") + return f"/{prefix}api/v2/findings/{query}" + + def _time_endpoint(self, path: str) -> tuple[dict, int, int, int]: + """Warm up once, then time N GETs. Returns (stats, status, query_count, rows).""" + # Warmup (populates any per-request caches; excluded from timings). + warm = self.client.get(path) + # One capture for the exact query count. + with CaptureQueriesContext(connection) as ctx: + captured = self.client.get(path) + query_count = len(ctx.captured_queries) + rows = None + if captured.status_code == 200: + body = captured.json() + rows = len(body["results"]) if isinstance(body, dict) and "results" in body else None + + samples: list[float] = [] + for _ in range(_N): + start = time.perf_counter() + self.client.get(path) + samples.append((time.perf_counter() - start) * 1000.0) # ms + stats = { + "median_ms": statistics.median(samples), + "p95_ms": _percentile(samples, 95), + "min_ms": min(samples), + "max_ms": max(samples), + } + return stats, warm.status_code, query_count, rows + + def test_benchmark(self): + test = Test.objects.first() + self.assertIsNotNone(test, "fixture must provide at least one test") + Finding.objects.bulk_create([ + Finding( + title=f"bench finding {i}", severity="High", numerical_severity="S1", + description="benchmark seed", test=test, reporter=self.admin, + active=True, verified=False, + ) + for i in range(_SEED) + ]) + total_findings = Finding.objects.count() + + scenarios = [ + ("v2 list ?prefetch=test", self._v2_url(f"?limit={_LIMIT}&prefetch=test")), + ("v2 list (no prefetch)", self._v2_url(f"?limit={_LIMIT}")), + ("v3 list (slim, no expand)", self.v3_url(f"findings?limit={_LIMIT}")), + ("v3 list ?expand=test.engagement", self.v3_url(f"findings?limit={_LIMIT}&expand=test.engagement")), + ] + + results = [] + for label, path in scenarios: + stats, status, query_count, rows = self._time_endpoint(path) + results.append((label, path, status, query_count, rows, stats)) + + self._write_report(total_findings, results) + + # Assertions turn the harness into a real (if CI-excluded) test: everything responds 200 and + # the v3 query count is dramatically lower than v2's (the headline claim), constant vs rows. + by_label = {label: (status, qc) for label, _, status, qc, _, _ in results} + for label, (status, _qc) in by_label.items(): + self.assertEqual(200, status, f"{label} returned {status}") + self.assertLess( + by_label["v3 list (slim, no expand)"][1], + by_label["v2 list ?prefetch=test"][1], + "v3 slim must issue far fewer queries than v2 prefetch", + ) + + def _write_report(self, total_findings: int, results: list) -> None: + lines = [ + "# API v3 — OS6 latency & query-count benchmark", + "", + ("**Generated by** `unittests/api_v3/test_apiv3_benchmark.py` " + "(`DD_API_V3_BENCH=1`, CI-excluded). Numbers are real captures, never fabricated."), + "", + "## Methodology", + "", + (f"- Seeded **{_SEED}** findings on a single product/test via `bulk_create`; " + f"total findings in DB at measurement time: **{total_findings}**."), + (f"- Page size `limit={_LIMIT}`. For each endpoint: one warmup GET (discarded), one " + f"capture GET for the exact SQL count, then **N={_N} timed** GETs."), + ("- Timing via `time.perf_counter()` around the in-process Django test client " + "(`APIClient`, token auth). Median and nearest-rank p95 over the N samples."), + "- Query count via `CaptureQueriesContext` (one representative request).", + "", + "## CAVEATS (read before quoting the latency numbers)", + "", + ("These are **in-process** measurements. They deliberately isolate the ORM + " + "serialization cost, but they are **NOT** production latency:"), + "", + ("- **No HTTP / uwsgi / network layer** — no WSGI worker, no request parsing over a " + "socket, no gzip, no reverse proxy. The v3 wire win from gzip'd repeated refs (D3) is " + "not captured here at all."), + ("- **Single-threaded**, no connection-pool warmup, no concurrent load — the regime where " + "the v2 per-row query fan-out hurts most (connection contention) is absent."), + ("- **Shared test transaction** against the kept test DB; planner stats and cache state " + "differ from production."), + ("- Therefore: treat **latency** as *directional* (it should track the query-count gap) " + "and treat the **query counts** as the load-bearing, environment-independent evidence."), + ("- The honest wall-clock procedure against a running uwsgi stack is recorded in " + "`.claude/os1-gate-report.md` ('Latency numbers: PENDING') and remains the way to get " + "quotable production latency."), + "", + "## Results", + "", + "| Scenario | Status | Queries | Rows | Median (ms) | p95 (ms) | Min (ms) | Max (ms) |", + "|---|---:|---:|---:|---:|---:|---:|---:|", + ] + for label, _path, status, query_count, rows, stats in results: + lines.append( + f"| {label} | {status} | {query_count} | {rows if rows is not None else '-'} " + f"| {stats['median_ms']:.2f} | {stats['p95_ms']:.2f} " + f"| {stats['min_ms']:.2f} | {stats['max_ms']:.2f} |", + ) + lines += [ + "", + "## Interpretation", + "", + ("The query counts are the headline: v3's slim/expand paths issue a **constant** number " + "of queries independent of row count (the N+1 fix, D3), whereas v2's post-serialization " + "`?prefetch=` issues per-row queries. The in-process median/p95 should move in the same " + "direction as the query gap; quote them only with the caveats above."), + "", + ] + report = "\n".join(lines) + # Echo to stdout FIRST so the numbers are captured even if the file write is not permitted + # (the container user may not own .claude/); the markdown between the markers is verbatim. + print("\n===== OS6 BENCHMARK REPORT (verbatim) =====") # noqa: T201 + print(report) # noqa: T201 + print("===== END OS6 BENCHMARK REPORT =====") # noqa: T201 + try: + _REPORT.parent.mkdir(parents=True, exist_ok=True) + _REPORT.write_text(report, encoding="utf-8") + print(f"[benchmark] wrote {_REPORT}") # noqa: T201 + except OSError as exc: # best-effort: .claude/ may be host-owned/read-only for this uid + print(f"[benchmark] could not write {_REPORT} ({exc}); use the verbatim block above") # noqa: T201 diff --git a/unittests/api_v3/test_apiv3_csv_export.py b/unittests/api_v3/test_apiv3_csv_export.py new file mode 100644 index 00000000000..4f5c31db26b --- /dev/null +++ b/unittests/api_v3/test_apiv3_csv_export.py @@ -0,0 +1,332 @@ +""" +CSV export contract tests for API v3 (§4.15, D6 "one filter contract, many projections"). + +``GET //export.csv`` takes the identical filter contract as the list (filters, ``o=``, +``q=``, ``?fields=`` incl. the detail opt-up) and streams the whole filtered, authorized set as CSV. +No pagination/expand/include (those 400). Rows are flattened generically: refs -> ``_id`` / +``_name`` (+ ``_type`` for location refs), ``tags`` -> a semicolon-joined column, datetimes +ISO-8601 ``Z``. The filtered count is capped (``API_V3_EXPORT_MAX_ROWS``); over-cap is a 400, never +a silent truncation. Cell values that could be read as spreadsheet formulas are quote-prefixed. +""" +from __future__ import annotations + +import csv +import io +import json + +from django.db import connection +from django.test import override_settings +from django.test.utils import CaptureQueriesContext +from django.utils import timezone + +from dojo.api_v3.csv_export import _harden # noqa: PLC2701 -- focused kernel unit test of the injection guard +from dojo.models import ( + Dojo_User, + Engagement, + Finding, + Finding_CWE, + Product, + Product_Type, + Test, + Test_Type, + Vulnerability_Id, +) + +from .base import ApiV3TestCase + +# The flattened finding CSV header for the default (no ?fields=) export: slim shape with refs fanned +# out into id/name pairs and tags as one joined column (§4.15). +_FINDING_DEFAULT_COLUMNS = [ + "id", "title", "severity", "active", "verified", "false_p", "duplicate", "risk_accepted", + "out_of_scope", "is_mitigated", "date", "cwe", "cwes", "vulnerability_ids", + "test_id", "test_name", "engagement_id", "engagement_name", "asset_id", "asset_name", + "organization_id", "organization_name", "reporter_id", "reporter_name", + "locations_count", "tags", "created", "updated", +] + + +class _CsvExportTestCase(ApiV3TestCase): + + """Shared helpers: fetch an export, parse the streamed CSV, index rows by id.""" + + def _get(self, path: str, *, client=None, **params): + client = client or self.client + return client.get(self.v3_url(path), params) + + def _rows(self, response) -> list[list[str]]: + """Assert a well-formed CSV attachment and return its rows (header first).""" + self.assertEqual(200, response.status_code) + self.assertEqual("text/csv; charset=utf-8", response["Content-Type"]) + content = b"".join(response.streaming_content).decode("utf-8") + return list(csv.reader(io.StringIO(content))) + + @staticmethod + def _by_id(rows: list[list[str]]) -> dict[str, dict[str, str]]: + header = rows[0] + return {r[header.index("id")]: dict(zip(header, r, strict=True)) for r in rows[1:]} + + def _problem(self, response) -> dict: + self.assertEqual(400, response.status_code, response.content[:500]) + return json.loads(response.content) + + +class TestApiV3CsvExportHappyPath(_CsvExportTestCase): + + def test_header_and_all_rows_with_flattened_refs(self): + response = self._get("findings/export.csv") + rows = self._rows(response) + # Header = the generic flattening of FindingSlim (refs fanned out, tags one column). + self.assertEqual(_FINDING_DEFAULT_COLUMNS, rows[0]) + # The ref field keys never appear un-flattened. + for collapsed in ("test", "engagement", "asset", "organization", "reporter"): + self.assertNotIn(collapsed, rows[0]) + # Every finding the admin can see is exported (whole filtered set, no pagination). + total = self.get_json("findings", data={"limit": 250})["count"] + self.assertGreater(total, 0) + self.assertEqual(total, len(rows) - 1, "export must contain one row per finding, no page limit") + + def test_ref_columns_and_tags_join(self): + finding = Finding.objects.first() + finding.tags = ["alpha", "beta"] + finding.save() + rows = self._get("findings/export.csv") + indexed = self._by_id(self._rows(rows)) + row = indexed[str(finding.id)] + # Ref flattened to id + name (name = the test's ref label). + self.assertEqual(str(finding.test_id), row["test_id"]) + self.assertEqual(finding.test.title or str(finding.test.test_type), row["test_name"]) + # Tags joined with ';' (tagulous force-lowercases; order preserved). + self.assertEqual("alpha;beta", row["tags"]) + + def test_vulnerability_ids_and_cwes_columns_render_semicolon_joined(self): + finding = Finding.objects.first() + Vulnerability_Id.objects.create(finding=finding, vulnerability_id="CVE-2020-1234") + Vulnerability_Id.objects.create(finding=finding, vulnerability_id="GHSA-aaaa-bbbb-cccc") + Finding_CWE.objects.create(finding=finding, cwe="CWE-79") + Finding_CWE.objects.create(finding=finding, cwe="CWE-89") + row = self._by_id(self._rows(self._get("findings/export.csv")))[str(finding.id)] + # List fields flatten to one semicolon-joined column each (§4.15). + self.assertEqual("CVE-2020-1234;GHSA-aaaa-bbbb-cccc", row["vulnerability_ids"]) + self.assertEqual("79;89", row["cwes"]) + + def test_datetime_columns_are_iso_z(self): + finding = Finding.objects.first() + finding.save() # populate created/updated + row = self._by_id(self._rows(self._get("findings/export.csv")))[str(finding.id)] + if row["created"]: + self.assertTrue(row["created"].endswith("Z"), row["created"]) + + def test_response_headers(self): + response = self._get("findings/export.csv") + self.assertEqual(200, response.status_code) + self.assertEqual("text/csv; charset=utf-8", response["Content-Type"]) + self.assertEqual('attachment; filename="findings-export.csv"', response["Content-Disposition"]) + self.assertEqual("alpha", response["X-API-Status"]) + + def test_export_csv_does_not_collide_with_int_detail_route(self): + # "export.csv" cannot match the {int:finding_id} detail route: the export returns CSV and the + # numeric detail route still returns JSON. + self.assertEqual("text/csv; charset=utf-8", self._get("findings/export.csv")["Content-Type"]) + detail = self._get(f"findings/{Finding.objects.first().id}") + self.assertEqual(200, detail.status_code) + self.assertEqual("application/json", detail["Content-Type"]) + + +class TestApiV3CsvExportFilterContract(_CsvExportTestCase): + + def test_filters_apply(self): + rows = self._by_id(self._rows(self._get("findings/export.csv", severity="Critical"))) + for row in rows.values(): + self.assertEqual("Critical", row["severity"]) + # Contrast: the unfiltered export has at least as many rows. + all_rows = self._by_id(self._rows(self._get("findings/export.csv"))) + self.assertGreaterEqual(len(all_rows), len(rows)) + + def test_ordering_applies(self): + rows = self._rows(self._get("findings/export.csv", o="-id")) + ids = [int(r[0]) for r in rows[1:]] + self.assertEqual(sorted(ids, reverse=True), ids, "o=-id must sort rows by descending id") + + def test_free_text_q_applies(self): + finding = Finding.objects.first() + finding.title = "csv-export-needle-xyz" + finding.save() + with_q = self._by_id(self._rows(self._get("findings/export.csv", q="needle-xyz"))) + without_q = self._by_id(self._rows(self._get("findings/export.csv"))) + self.assertIn(str(finding.id), with_q, "the matching finding must be exported") + self.assertLess(len(with_q), len(without_q), "q= must narrow the exported set") + + def test_fields_narrowing(self): + rows = self._rows(self._get("findings/export.csv", fields="id,title")) + self.assertEqual(["id", "title"], rows[0]) + + def test_fields_detail_opt_up_adds_impact_column(self): + finding = Finding.objects.first() + finding.impact = "csv-impact-value" + finding.save() + rows = self._get("findings/export.csv", fields="id,title,impact") + parsed = self._rows(rows) + self.assertIn("impact", parsed[0], "?fields= must opt up into the detail column set") + self.assertEqual("csv-impact-value", self._by_id(parsed)[str(finding.id)]["impact"]) + + def test_impact_absent_from_default_export(self): + # The default export is the slim projection; a detail-only column is not present unless asked. + self.assertNotIn("impact", self._rows(self._get("findings/export.csv"))[0]) + + def test_unknown_field_is_400(self): + problem = self._problem(self._get("findings/export.csv", fields="id,not_a_field")) + self.assertTrue(problem["type"].endswith("/fields")) + + +class TestApiV3CsvExportReservedParams(_CsvExportTestCase): + + """expand/include/limit/offset/pagination/cursor are not applicable to an export -> 400.""" + + def test_expand_is_400(self): + self.assertTrue(self._problem(self._get("findings/export.csv", expand="test"))["type"].endswith("/export")) + + def test_include_is_400(self): + self.assertTrue(self._problem(self._get("findings/export.csv", include="counts"))["type"].endswith("/export")) + + def test_limit_is_400(self): + self.assertTrue(self._problem(self._get("findings/export.csv", limit="5"))["type"].endswith("/export")) + + def test_offset_is_400(self): + self.assertTrue(self._problem(self._get("findings/export.csv", offset="0"))["type"].endswith("/export")) + + def test_pagination_mode_is_400(self): + self.assertTrue(self._problem(self._get("findings/export.csv", pagination="cursor"))["type"].endswith("/export")) + + def test_cursor_is_400(self): + self.assertTrue(self._problem(self._get("findings/export.csv", cursor="abc"))["type"].endswith("/export")) + + +class TestApiV3CsvExportCap(_CsvExportTestCase): + + @override_settings(API_V3_EXPORT_MAX_ROWS=2) + def test_over_cap_is_400_not_truncation(self): + # There are more than 2 findings in the fixture, so a cap of 2 rejects the export. + self.assertGreater(self.get_json("findings", data={"limit": 250})["count"], 2) + problem = self._problem(self._get("findings/export.csv")) + self.assertEqual(400, problem["status"]) + self.assertTrue(problem["type"].endswith("/export")) + self.assertIn("cap", problem["detail"].lower()) + + @override_settings(API_V3_EXPORT_MAX_ROWS=100000) + def test_under_cap_streams(self): + self.assertEqual(200, self._get("findings/export.csv").status_code) + + +class TestApiV3CsvExportInjectionHardening(_CsvExportTestCase): + + def test_harden_prefixes_every_formula_trigger(self): + for trigger in ("=", "+", "-", "@", "\t"): + self.assertEqual(f"'{trigger}cmd", _harden(f"{trigger}cmd"), f"cell starting with {trigger!r} must be quoted") + # Safe values are untouched; empty stays empty. + self.assertEqual("Critical", _harden("Critical")) + self.assertEqual("", _harden("")) + + def test_dangerous_title_is_quote_prefixed_in_export(self): + test = Test.objects.first() + finding = Finding.objects.create( + title="=SUM(1+1)", severity="High", numerical_severity="S1", description="x", + test=test, reporter=self.admin, active=True, verified=False, + date=timezone.now().date(), + ) + row = self._by_id(self._rows(self._get("findings/export.csv")))[str(finding.id)] + # The model lowercases the title; the load-bearing assertion is the injection-defense prefix. + self.assertTrue(row["title"].startswith("'="), f"a formula-like title must be quote-prefixed: {row['title']!r}") + self.assertIn("sum(1+1)", row["title"].lower()) + + +class TestApiV3CsvExportRbac(_CsvExportTestCase): + + def test_zero_permission_user_export_is_header_only(self): + zero = Dojo_User.objects.create_user(username="v3_csv_zero", password="x") # noqa: S106 + rows = self._rows(self._get("findings/export.csv", client=self.token_client(user=zero))) + self.assertEqual(_FINDING_DEFAULT_COLUMNS, rows[0], "a header row is always emitted") + self.assertEqual(1, len(rows), "zero-permission user exports no rows (RBAC-scoped empty)") + + def test_unauthorized_rows_absent_from_member_export(self): + # A member authorized on product A only must never see product B's findings in the export. + member = Dojo_User.objects.create_user(username="v3_csv_member", password="x") # noqa: S106 + prod_type = Product_Type.objects.create(name="csv-rbac-pt") + tt = Test_Type.objects.create(name="csv-rbac-tt") + product_a = Product.objects.create(name="csv-A", description="x", prod_type=prod_type, sla_configuration_id=1) + product_b = Product.objects.create(name="csv-B", description="x", prod_type=prod_type, sla_configuration_id=1) + product_a.authorized_users.add(member) + ids_a, ids_b = [], [] + for product, bucket in ((product_a, ids_a), (product_b, ids_b)): + eng = Engagement.objects.create( + product=product, name=f"{product.name}-eng", + target_start=timezone.now().date(), target_end=timezone.now().date(), + ) + test = Test.objects.create( + engagement=eng, test_type=tt, + target_start=timezone.now(), target_end=timezone.now(), + ) + for i in range(2): + bucket.append(Finding.objects.create( + title=f"{product.name}-f{i}", severity="High", numerical_severity="S1", + description="x", test=test, reporter=self.admin, active=True, verified=False, + date=timezone.now().date(), + ).id) + exported = set(self._by_id(self._rows(self._get("findings/export.csv", client=self.token_client(user=member))))) + for fid in ids_a: + self.assertIn(str(fid), exported, "product A finding must be in the member's export") + for fid in ids_b: + self.assertNotIn(str(fid), exported, "product B finding must NOT leak into the member's export") + + +class TestApiV3CsvExportQueryCount(_CsvExportTestCase): + + """The export query count is independent of the number of rows streamed (§4.15).""" + + def _bulk_create_findings(self, count: int, test: Test) -> None: + today = timezone.now().date() + Finding.objects.bulk_create([ + Finding( + title=f"csv-qcount {i}", severity="High", numerical_severity="S1", + description="x", test=test, reporter=self.admin, active=True, verified=False, date=today, + ) + for i in range(count) + ]) + + def _export_query_count(self) -> int: + with CaptureQueriesContext(connection) as ctx: + response = self.client.get(self.v3_url("findings/export.csv")) + self.assertEqual(200, response.status_code) + b"".join(response.streaming_content) # consume so the chunked iterator + prefetch run in-context + return len(ctx.captured_queries) + + def test_query_count_is_independent_of_row_count(self): + test = Test.objects.first() + self._bulk_create_findings(10, test) + queries_10 = self._export_query_count() + self._bulk_create_findings(90, test) + queries_100 = self._export_query_count() + self.assertGreaterEqual(self.get_json("findings", data={"limit": 250})["count"], 100) + self.assertEqual( + queries_10, queries_100, + f"CSV export query count must not grow with row count: {queries_10} vs {queries_100}", + ) + + +class TestApiV3CsvExportAssets(_CsvExportTestCase): + + """A non-finding resource for uniformity: the same generic kernel handles assets (§4.15).""" + + def test_assets_export_flattens_organization_ref(self): + rows = self._rows(self._get("assets/export.csv")) + header = rows[0] + self.assertIn("organization_id", header) + self.assertIn("organization_name", header) + self.assertNotIn("organization", header) + self.assertIn("tags", header) + total = self.get_json("assets", data={"limit": 250})["count"] + self.assertEqual(total, len(rows) - 1) + + def test_assets_export_headers(self): + response = self._get("assets/export.csv") + self.assertEqual('attachment; filename="assets-export.csv"', response["Content-Disposition"]) + self.assertEqual("alpha", response["X-API-Status"]) diff --git a/unittests/api_v3/test_apiv3_cursor.py b/unittests/api_v3/test_apiv3_cursor.py new file mode 100644 index 00000000000..a9e5396b0be --- /dev/null +++ b/unittests/api_v3/test_apiv3_cursor.py @@ -0,0 +1,282 @@ +""" +Cursor (keyset) pagination contract tests for API v3 (D4 / §4.3). + +Covers the forward-only, opaque-signed cursor mode added on top of the offset default: +full-walk correctness, the keyset-safe ordering variants + per-resource allowed sets, tamper/ +ordering-mismatch rejection, composition with filters/include/fields/expand, and constant +(count-free) query cost per page. Offset mode is exercised unchanged by ``test_apiv3_findings``. +""" +from __future__ import annotations + +import datetime +import json +from urllib.parse import unquote, urlsplit + +from django.db import connection +from django.test.utils import CaptureQueriesContext + +from dojo.authorization.roles_permissions import Permissions +from dojo.finding.queries import get_authorized_findings +from dojo.models import Finding, Product, Product_Type, Test +from dojo.product.queries import get_authorized_products + +from .base import ApiV3TestCase + +_FANOUT = 30 +_PAGE = 10 + + +class _CursorWalkMixin: + + """Follow ``next`` cursors to exhaustion, asserting the forward-only envelope invariants.""" + + def _follow(self, next_url: str) -> dict: + parts = urlsplit(next_url) + path = parts.path + (f"?{parts.query}" if parts.query else "") + response = self.client.get(path) + self.assertEqual(200, response.status_code, response.content[:500]) + return json.loads(response.content) + + def _walk(self, resource: str, params: dict) -> tuple[list[int], int]: + """Walk every cursor page; return (collected ids in page order, page count).""" + limit = int(params["limit"]) + body = self.get_json(resource, data={**params, "pagination": "cursor"}) + collected: list[int] = [] + pages = 0 + while True: + pages += 1 + # Every cursor page: count null, previous null (forward-only, GitLab-style). + self.assertIsNone(body["count"], f"page {pages} count not null") + self.assertIsNone(body["previous"], f"page {pages} previous not null") + collected.extend(row["id"] for row in body["results"]) + if body["next"] is None: + break + # A non-final page is exactly full. + self.assertEqual(limit, len(body["results"]), f"non-final page {pages} not full") + body = self._follow(body["next"]) + return collected, pages + + +class TestApiV3CursorFindingsWalk(_CursorWalkMixin, ApiV3TestCase): + + def setUp(self): + super().setUp() + test = Test.objects.first() + self.new_ids = [ + Finding.objects.create( + title=f"cursor walk {i}", severity="High", numerical_severity="S1", + description="x", test=test, reporter=self.admin, active=True, verified=(i % 2 == 0), + ).pk + for i in range(_FANOUT) + ] + # Give the new rows distinct, deliberately shuffled created/updated (via .update() so the + # auto_now/auto_now_add fields are bypassed) so ordering by created/updated is genuinely + # different from ordering by id. + base = datetime.datetime(2020, 1, 1, tzinfo=datetime.UTC) + for offset, pk in enumerate(self.new_ids): + created = base + datetime.timedelta(minutes=_FANOUT - offset) # decreasing vs id + updated = base + datetime.timedelta(minutes=offset) # increasing vs id + Finding.objects.filter(pk=pk).update(created=created, updated=updated) + + def _expected(self, *order: str) -> list[int]: + qs = get_authorized_findings(Permissions.Finding_View, user=self.admin) + return list(qs.order_by(*order).values_list("id", flat=True)) + + def test_full_walk_visits_every_id_exactly_once(self): + expected = self._expected("id") + collected, pages = self._walk("findings", {"limit": _PAGE}) + self.assertEqual(expected, collected) # default cursor order = id asc + self.assertEqual(len(collected), len(set(collected))) # no id twice + self.assertEqual(-(-len(expected) // _PAGE), pages) # ceil(total/page) pages + + def test_default_cursor_ordering_is_id_asc(self): + body = self.get_json("findings", data={"pagination": "cursor", "limit": 250}) + ids = [r["id"] for r in body["results"]] + self.assertEqual(sorted(ids), ids) + + def test_walk_o_created_ascending(self): + collected, _ = self._walk("findings", {"limit": _PAGE, "o": "created"}) + self.assertEqual(self._expected("created", "id"), collected) + + def test_walk_o_updated_descending(self): + collected, _ = self._walk("findings", {"limit": _PAGE, "o": "-updated"}) + self.assertEqual(self._expected("-updated", "-id"), collected) + + def test_walk_o_id_descending(self): + collected, _ = self._walk("findings", {"limit": _PAGE, "o": "-id"}) + self.assertEqual(self._expected("-id"), collected) + + def test_non_keyset_ordering_is_400(self): + # title is a valid offset-mode ordering, but not keyset-safe -> 400 in cursor mode. + self.get_json("findings", data={"pagination": "cursor", "o": "title"}, expected=400) + # severity (a computed ordering) is likewise rejected. + self.get_json("findings", data={"pagination": "cursor", "o": "severity"}, expected=400) + + def test_multi_field_ordering_is_400(self): + self.get_json("findings", data={"pagination": "cursor", "o": "created,id"}, expected=400) + + +class TestApiV3CursorTamper(ApiV3TestCase): + + def _problem(self, params: dict): + response = self.client.get(self.v3_url("findings"), params) + self.assertEqual(400, response.status_code, response.content[:300]) + self.assertEqual("application/problem+json", response["Content-Type"]) + body = json.loads(response.content) + self.assertTrue(body["type"].endswith("/errors/pagination"), body["type"]) + return body + + def test_garbage_cursor_is_400(self): + self._problem({"pagination": "cursor", "cursor": "not-a-real-cursor"}) + + def test_tampered_cursor_is_400(self): + first = self.get_json("findings", data={"pagination": "cursor", "limit": 1}) + self.assertIsNotNone(first["next"]) + cursor = dict(x.split("=", 1) for x in urlsplit(first["next"]).query.split("&"))["cursor"] + # Flip the final character of the (url-decoded) token to break the signature. + cursor = unquote(cursor) + tampered = cursor[:-1] + ("A" if cursor[-1] != "A" else "B") + self._problem({"pagination": "cursor", "cursor": tampered}) + + def test_cursor_from_a_different_ordering_is_400(self): + # Mint a cursor under o=created, then present it under o=-updated -> ordering mismatch 400. + first = self.get_json("findings", data={"pagination": "cursor", "limit": 1, "o": "created"}) + self.assertIsNotNone(first["next"]) + cursor = unquote(dict(x.split("=", 1) for x in urlsplit(first["next"]).query.split("&"))["cursor"]) + self._problem({"pagination": "cursor", "o": "-updated", "cursor": cursor}) + + +class TestApiV3CursorComposition(_CursorWalkMixin, ApiV3TestCase): + + def setUp(self): + super().setUp() + test = Test.objects.first() + # A distinctive severity subset to prove the filter composes with the keyset walk. + self.low_ids = [ + Finding.objects.create( + title=f"cursor low {i}", severity="Low", numerical_severity="S3", + description="x", test=test, reporter=self.admin, active=True, verified=False, + ).pk + for i in range(_FANOUT) + ] + + def test_cursor_plus_severity_filter_walks_only_matching_rows(self): + collected, _ = self._walk("findings", {"limit": _PAGE, "severity": "Low"}) + qs = get_authorized_findings(Permissions.Finding_View, user=self.admin).filter(severity="Low") + self.assertEqual(list(qs.order_by("id").values_list("id", flat=True)), collected) + self.assertTrue(set(self.low_ids).issubset(collected)) + + def test_cursor_plus_include_counts_keeps_count_null(self): + body = self.get_json("findings", data={"pagination": "cursor", "include": "counts", "limit": 5}) + self.assertIsNone(body["count"]) + counts = body["meta"]["counts"] + for key in ("total", "active", "verified", "duplicate", "severity"): + self.assertIn(key, counts) + self.assertGreaterEqual(counts["severity"]["Low"], _FANOUT) + + def test_cursor_plus_fields_projects_rows(self): + body = self.get_json("findings", data={"pagination": "cursor", "fields": "id,title", "limit": 5}) + self.assertIsNone(body["count"]) + for row in body["results"]: + self.assertEqual({"id", "title"}, set(row)) + + def test_cursor_plus_expand_inlines_slim(self): + body = self.get_json("findings", data={"pagination": "cursor", "expand": "test.engagement", "limit": 5}) + self.assertIsNone(body["count"]) + row = body["results"][0] + self.assertIn("test_type", row["test"]) + self.assertIn("engagement", row["test"]) + + +class TestApiV3CursorQueryCost(ApiV3TestCase): + + """Cursor mode issues a constant number of queries per page -- one fewer than offset (no COUNT).""" + + # Offset mode is 7 queries (capped count + rows + tags/vulnerability_id_set/finding_cwe_set + # prefetches); cursor drops the count query -> exactly 6, constant per page regardless of + # row/page count. (Was 4; +2 for the FindingSlim vulnerability_ids/cwes prefetches added + # alongside tags -- fixed in-batch prefetches, not per-row.) + EXPECTED_CURSOR_QUERIES = 6 + + def setUp(self): + super().setUp() + test = Test.objects.first() + Finding.objects.bulk_create([ + Finding(title=f"cursor qcount {i}", severity="High", numerical_severity="S1", + description="x", test=test, reporter=self.admin, active=True, verified=False) + for i in range(_FANOUT) + ]) + + def _count_queries(self, params: dict) -> tuple[int, dict]: + with CaptureQueriesContext(connection) as ctx: + response = self.client.get(self.v3_url("findings"), params) + self.assertEqual(200, response.status_code, response.content[:500]) + return len(ctx.captured_queries), json.loads(response.content) + + def test_cursor_is_one_fewer_query_than_offset(self): + offset_q, _ = self._count_queries({"limit": _PAGE}) + cursor_q, body = self._count_queries({"pagination": "cursor", "limit": _PAGE}) + # No COUNT query in cursor mode -> exactly one fewer than offset (which counts + estimates). + self.assertEqual(offset_q - 1, cursor_q, f"offset={offset_q} cursor={cursor_q}") + self.assertEqual(self.EXPECTED_CURSOR_QUERIES, cursor_q) + # Page 2 (following the cursor) costs the same as page 1: constant per page. + page2_q, _ = self._count_queries( + {"pagination": "cursor", "limit": _PAGE, "cursor": _cursor_of(body)}, + ) + self.assertEqual(cursor_q, page2_q, f"page1={cursor_q} page2={page2_q}") + + +class TestApiV3CursorAssetsWalk(_CursorWalkMixin, ApiV3TestCase): + + """A non-finding resource cursor walk proves the mode is uniform across the list routes (I5).""" + + def setUp(self): + super().setUp() + prod_type = Product_Type.objects.first() + for i in range(_FANOUT): + Product.objects.create(name=f"cursor asset {i}", description="x", prod_type=prod_type) + + def test_assets_full_walk(self): + expected = list( + get_authorized_products(Permissions.Product_View, user=self.admin) + .order_by("id").values_list("id", flat=True), + ) + collected, pages = self._walk("assets", {"limit": 5}) + self.assertEqual(expected, collected) + self.assertEqual(len(collected), len(set(collected))) + self.assertEqual(-(-len(expected) // 5), pages) + + +class TestApiV3CursorPerResourceOrderings(ApiV3TestCase): + + """The keyset-safe ordering set is derived per FilterSpec (id always; created/updated if declared).""" + + def test_users_allow_only_id(self): + # users declare id/username/date_joined/last_login orderings, but only `id` is keyset-safe. + self.get_json("users", data={"pagination": "cursor", "limit": 5}) # id -> ok + self.get_json("users", data={"pagination": "cursor", "o": "created"}, expected=400) # not keyset-safe + self.get_json("users", data={"pagination": "cursor", "o": "username"}, expected=400) + + def test_organizations_allow_created_updated(self): + for order in ("id", "created", "-updated"): + self.get_json("organizations", data={"pagination": "cursor", "o": order, "limit": 5}) + self.get_json("organizations", data={"pagination": "cursor", "o": "name"}, expected=400) + + +class TestApiV3CursorSubResourceUnsupported(ApiV3TestCase): + + """Parent-scoped edge sub-resource lists have no FilterSpec -> cursor mode is a 400 there.""" + + def test_finding_locations_cursor_is_400(self): + finding_id = Finding.objects.first().pk + response = self.client.get( + self.v3_url(f"findings/{finding_id}/locations"), {"pagination": "cursor"}, + ) + self.assertEqual(400, response.status_code, response.content[:300]) + self.assertEqual("application/problem+json", response["Content-Type"]) + + +def _cursor_of(body: dict) -> str: + query = urlsplit(body["next"]).query + raw = dict(x.split("=", 1) for x in query.split("&"))["cursor"] + return unquote(raw) diff --git a/unittests/api_v3/test_apiv3_engagements.py b/unittests/api_v3/test_apiv3_engagements.py new file mode 100644 index 00000000000..82b8dae8e90 --- /dev/null +++ b/unittests/api_v3/test_apiv3_engagements.py @@ -0,0 +1,341 @@ +"""Engagement CRUD + RBAC + contract tests for API v3 (OS3b).""" +from __future__ import annotations + +import datetime +from unittest import mock + +from django.db import connection +from django.test.utils import CaptureQueriesContext + +from dojo.engagement.api_v3.schemas import EngagementSlim +from dojo.finding.api_v3 import schemas as finding_schemas +from dojo.models import Dojo_User, Engagement, Product, User + +from .base import ApiV3TestCase + +_SLIM_KEYS = { + "id", "name", "asset", "organization", "lead", "status", "engagement_type", + "target_start", "target_end", "active", "tags", "created", "updated", +} + + +class TestApiV3EngagementsRelocation(ApiV3TestCase): + + def test_engagement_slim_is_canonical_single_class(self): + # OS3b relocated EngagementSlim out of the finding module; the finding module now re-exports + # the one canonical class (is-identity, mirroring the OS3a relocation pattern -- §12). + self.assertIs(EngagementSlim, finding_schemas.EngagementSlim) + + +class TestApiV3EngagementsRead(ApiV3TestCase): + + def test_list_envelope_and_slim_shape(self): + body = self.get_json("engagements") + self.assertEqual({"count", "next", "previous", "results"}, set(body) - {"meta"}) + self.assertGreater(body["count"], 0) + row = body["results"][0] + self.assertEqual(_SLIM_KEYS, set(row)) + self.assertEqual({"id", "name"}, set(row["asset"])) + self.assertEqual({"id", "name"}, set(row["organization"])) + self.assertIsInstance(row["tags"], list) + + def test_detail_adds_heavy_fields(self): + engagement = Engagement.objects.first() + detail = self.get_json(f"engagements/{engagement.id}") + for key in ("description", "version", "first_contacted", "threat_model", "deduplication_on_engagement"): + self.assertIn(key, detail) + + def test_detail_unknown_is_404_problem(self): + response = self.client.get(self.v3_url("engagements/99999999")) + self.assertEqual(404, response.status_code) + self.assertEqual("application/problem+json", response["Content-Type"]) + + def test_expand_asset_inlines_slim(self): + row = self.get_json("engagements", data={"expand": "asset"})["results"][0] + self.assertIn("description", row["asset"]) + self.assertIn("lifecycle", row["asset"]) + + def test_expand_lead_and_organization(self): + eng = Engagement.objects.exclude(lead__isnull=True).first() + if eng is None: + eng = Engagement.objects.first() + eng.lead = self.admin + eng.save() + row = self.get_json("engagements", data={"expand": "lead,organization", "id__in": eng.id})["results"][0] + self.assertIn("username", row["lead"]) + self.assertIn("critical_product", row["organization"]) + + def test_expand_unknown_relation_is_400(self): + self.get_json("engagements", data={"expand": "not_a_relation"}, expected=400) + + +class TestApiV3EngagementsFieldsOptUp(ApiV3TestCase): + + """`?fields=` opt-up into detail fields is uniform across resources (§4.7 Part A) -- engagements.""" + + def test_description_absent_from_default_list(self): + row = self.get_json("engagements")["results"][0] + self.assertEqual(_SLIM_KEYS, set(row)) + self.assertNotIn("description", row) + + def test_fields_opts_up_into_description(self): + engagement = Engagement.objects.first() + engagement.description = "opt-up-engagement-description" + engagement.save(update_fields=["description"]) + row = next( + r for r in self.get_json("engagements", data={"fields": "id,name,description", "limit": 250})["results"] + if r["id"] == engagement.id + ) + self.assertEqual({"id", "name", "description"}, set(row)) + self.assertEqual("opt-up-engagement-description", row["description"]) + + +class TestApiV3EngagementsFilters(ApiV3TestCase): + + def test_filter_asset(self): + product_id = Engagement.objects.first().product_id + body = self.get_json("engagements", data={"asset": product_id, "limit": 250}) + self.assertGreater(body["count"], 0) + for row in body["results"]: + self.assertEqual(product_id, row["asset"]["id"]) + + def test_filter_status(self): + eng = Engagement.objects.first() + body = self.get_json("engagements", data={"status": eng.status, "limit": 250}) + for row in body["results"]: + self.assertEqual(eng.status, row["status"]) + + def test_ordering_by_id(self): + ids = [r["id"] for r in self.get_json("engagements", data={"o": "id", "limit": 250})["results"]] + self.assertEqual(ids, sorted(ids)) + + def test_unknown_filter_param_is_400(self): + self.get_json("engagements", data={"not_a_filter": "x"}, expected=400) + + +class TestApiV3EngagementsPagination(ApiV3TestCase): + + def test_limit_next_previous(self): + body = self.get_json("engagements", data={"limit": 2, "offset": 2}) + self.assertLessEqual(len(body["results"]), 2) + self.assertIsNotNone(body["previous"]) + + +class TestApiV3EngagementsQueryCount(ApiV3TestCase): + + def _bulk(self, count: int, start: int) -> None: + product = Product.objects.first() + day = datetime.date(2024, 1, 1) + Engagement.objects.bulk_create([ + Engagement(name=f"qcount engagement {start + i}", product=product, target_start=day, target_end=day) + for i in range(count) + ]) + + def _query_count(self, params: dict) -> int: + with CaptureQueriesContext(connection) as ctx: + response = self.client.get(self.v3_url("engagements"), params) + self.assertEqual(200, response.status_code, response.content[:500]) + return len(ctx.captured_queries) + + def test_query_count_is_independent_of_row_count(self): + self._bulk(10, 0) + first = self._query_count({"limit": 250}) + first_expand = self._query_count({"limit": 250, "expand": "asset.organization"}) + self._bulk(90, 100) + second = self._query_count({"limit": 250}) + second_expand = self._query_count({"limit": 250, "expand": "asset.organization"}) + self.assertEqual(first, second, f"query count grew (no expand): {first} -> {second}") + self.assertEqual(first_expand, second_expand, f"query count grew (expand): {first_expand} -> {second_expand}") + + +class TestApiV3EngagementsWrite(ApiV3TestCase): + + def test_create_happy_path(self): + product = Product.objects.first() + response = self.client.post( + self.v3_url("engagements"), + {"name": "v3 created engagement", "asset": product.id, + "target_start": "2024-01-01", "target_end": "2024-02-01", + "status": "In Progress", "tags": ["v3"]}, + format="json", + ) + self.assertEqual(201, response.status_code, response.content[:500]) + body = response.json() + self.assertEqual("v3 created engagement", body["name"]) + self.assertEqual(product.id, body["asset"]["id"]) + created = Engagement.objects.get(name="v3 created engagement") + self.assertEqual({"v3"}, {t.name for t in created.tags.all()}) + + def test_create_target_start_after_end_is_400(self): + product = Product.objects.first() + response = self.client.post( + self.v3_url("engagements"), + {"name": "bad dates", "asset": product.id, + "target_start": "2024-02-01", "target_end": "2024-01-01"}, + format="json", + ) + self.assertEqual(400, response.status_code) + self.assertEqual("application/problem+json", response["Content-Type"]) + + def test_create_missing_required_is_400(self): + response = self.client.post(self.v3_url("engagements"), {"name": "no asset"}, format="json") + self.assertEqual(400, response.status_code) + + def test_create_unknown_field_is_400(self): + product = Product.objects.first() + response = self.client.post( + self.v3_url("engagements"), + {"asset": product.id, "target_start": "2024-01-01", "target_end": "2024-02-01", "bogus": 1}, + format="json", + ) + self.assertEqual(400, response.status_code) + + def test_create_nonexistent_asset_is_404(self): + response = self.client.post( + self.v3_url("engagements"), + {"asset": 99999999, "target_start": "2024-01-01", "target_end": "2024-02-01"}, + format="json", + ) + self.assertEqual(404, response.status_code) + + def test_patch_partial_update(self): + product = Product.objects.first() + eng = Engagement.objects.create( + name="v3 patch engagement", product=product, + target_start=datetime.date(2024, 1, 1), target_end=datetime.date(2024, 2, 1), + ) + response = self.client.patch( + self.v3_url(f"engagements/{eng.id}"), {"name": "renamed"}, format="json", + ) + self.assertEqual(200, response.status_code, response.content[:500]) + eng.refresh_from_db() + self.assertEqual("renamed", eng.name) + + def test_delete(self): + product = Product.objects.first() + eng = Engagement.objects.create( + name="v3 delete engagement", product=product, + target_start=datetime.date(2024, 1, 1), target_end=datetime.date(2024, 2, 1), + ) + response = self.client.delete(self.v3_url(f"engagements/{eng.id}")) + self.assertEqual(204, response.status_code) + self.assertFalse(Engagement.objects.filter(pk=eng.id).exists()) + + +class TestApiV3EngagementsReplace(ApiV3TestCase): + + """PUT full-replace: EngagementReplace (required asset/target_start/target_end); optionals reset.""" + + def _make_engagement(self, **kwargs): + product = Product.objects.first() + defaults = { + "name": "v3 put engagement", "product": product, + "target_start": datetime.date(2024, 1, 1), "target_end": datetime.date(2024, 2, 1), + } + defaults.update(kwargs) + return Engagement.objects.create(**defaults), product + + def test_put_full_replace_resets_omitted_optionals(self): + eng, product = self._make_engagement(description="old", threat_model=False) + # PUT without description / threat_model -> nullable resets to None, non-null bool to model + # default True. + response = self.client.put( + self.v3_url(f"engagements/{eng.id}"), + {"asset": product.id, "target_start": "2024-01-01", "target_end": "2024-02-01"}, + format="json", + ) + self.assertEqual(200, response.status_code, response.content[:500]) + eng.refresh_from_db() + self.assertIsNone(eng.description) # nullable -> reset to None + self.assertTrue(eng.threat_model) # NOT NULL bool -> reset to model default True + + def test_put_reassign_asset(self): + eng, _ = self._make_engagement() + other = Product.objects.exclude(pk=eng.product_id).first() + response = self.client.put( + self.v3_url(f"engagements/{eng.id}"), + {"asset": other.id, "target_start": "2024-01-01", "target_end": "2024-02-01"}, + format="json", + ) + self.assertEqual(200, response.status_code, response.content[:500]) + eng.refresh_from_db() + self.assertEqual(other.id, eng.product_id) + + def test_put_target_start_after_end_is_400(self): + eng, product = self._make_engagement() + response = self.client.put( + self.v3_url(f"engagements/{eng.id}"), + {"asset": product.id, "target_start": "2024-02-01", "target_end": "2024-01-01"}, + format="json", + ) + self.assertEqual(400, response.status_code) + + def test_put_missing_required_is_400(self): + eng, _ = self._make_engagement() + response = self.client.put(self.v3_url(f"engagements/{eng.id}"), {"name": "no asset"}, format="json") + self.assertEqual(400, response.status_code) + self.assertEqual("application/problem+json", response["Content-Type"]) + + def test_put_unknown_field_is_400(self): + eng, product = self._make_engagement() + response = self.client.put( + self.v3_url(f"engagements/{eng.id}"), + {"asset": product.id, "target_start": "2024-01-01", "target_end": "2024-02-01", "bogus": 1}, + format="json", + ) + self.assertEqual(400, response.status_code) + + def test_put_unauthorized_is_404(self): + eng, product = self._make_engagement() + limited = User.objects.create_user(username="v3_eng_put_limited", password="x") # noqa: S106 + client = self.token_client(user=limited) + response = client.put( + self.v3_url(f"engagements/{eng.id}"), + {"asset": product.id, "target_start": "2024-01-01", "target_end": "2024-02-01"}, + format="json", + ) + self.assertEqual(404, response.status_code) + + def test_put_visible_but_not_editable_is_403(self): + eng, product = self._make_engagement() + with mock.patch("dojo.engagement.api_v3.routes.user_has_permission", return_value=False): + response = self.client.put( + self.v3_url(f"engagements/{eng.id}"), + {"asset": product.id, "target_start": "2024-01-01", "target_end": "2024-02-01"}, + format="json", + ) + self.assertEqual(403, response.status_code, response.content[:300]) + + +class TestApiV3EngagementsRbac(ApiV3TestCase): + + def setUp(self): + super().setUp() + self.limited = User.objects.create_user(username="v3_eng_limited", password="x") # noqa: S106 + self.member = Dojo_User.objects.create_user(username="v3_eng_member", password="x") # noqa: S106 + self.engagement = Engagement.objects.first() + self.product = self.engagement.product + self.product.authorized_users.add(self.member) + + def test_unauthorized_read_is_404(self): + client = self.token_client(user=self.limited) + self.assertEqual(0, self.get_json("engagements", client=client)["count"]) + self.get_json(f"engagements/{self.engagement.id}", client=client, expected=404) + + def test_create_without_asset_add_is_403(self): + client = self.token_client(user=self.limited) + response = client.post( + self.v3_url("engagements"), + {"asset": self.product.id, "target_start": "2024-01-01", "target_end": "2024-02-01"}, + format="json", + ) + self.assertEqual(403, response.status_code, response.content[:300]) + self.assertEqual("application/problem+json", response["Content-Type"]) + + def test_member_can_view_but_delete_is_403(self): + # OS legacy RBAC: membership grants view+edit+add, but delete is staff-only -- so the + # "authorized-to-see-but-not-modify" 403 is demonstrated on delete (§12). + client = self.token_client(user=self.member) + self.get_json(f"engagements/{self.engagement.id}", client=client) + response = client.delete(self.v3_url(f"engagements/{self.engagement.id}")) + self.assertEqual(403, response.status_code, response.content[:300]) diff --git a/unittests/api_v3/test_apiv3_errors.py b/unittests/api_v3/test_apiv3_errors.py new file mode 100644 index 00000000000..e6fb3d1af86 --- /dev/null +++ b/unittests/api_v3/test_apiv3_errors.py @@ -0,0 +1,126 @@ +""" +Error-contract sweep for API v3 (D9 / §4.10, §6 OS2 / I9). + +Every v3-raised failure path must return an RFC 9457 ``application/problem+json`` body with the +correct ``type`` URI, ``status`` and ``title``. This module asserts that for each failure kind. +(Django's default 500 handler is out of scope -- only v3-raised paths.) +""" +from __future__ import annotations + +from typing import ClassVar + +from django.test import SimpleTestCase, override_settings +from ninja import Schema + +from dojo.api_v3.errors import ProblemDetail +from dojo.api_v3.expand import ExpandRel, plan +from dojo.models import Finding + +from .base import ApiV3TestCase + + +class TestApiV3ErrorContract(ApiV3TestCase): + + def _assert_problem(self, path, *, status, type_suffix, client=None, data=None): + client = client or self.client + response = client.get(self.v3_url(path), data or {}) + self.assertEqual(status, response.status_code, response.content[:500]) + self.assertEqual("application/problem+json", response["Content-Type"], response.content[:200]) + body = response.json() + self.assertEqual(status, body["status"]) + self.assertIn("title", body) + self.assertTrue( + body["type"].endswith("#error-" + type_suffix), + f"expected type ...#error-{type_suffix}, got {body['type']}", + ) + return body + + # --- 400: expand ----------------------------------------------------------------------- + def test_unknown_expand_relation_is_problem(self): + self._assert_problem("findings", status=400, type_suffix="expand", data={"expand": "nope"}) + + def test_expand_budget_exceeded_is_problem(self): + with override_settings(API_V3_EXPAND_BUDGET=1): + self._assert_problem( + "findings", status=400, type_suffix="expand", + data={"expand": "test.engagement,reporter"}, + ) + + def test_expand_further_on_special_is_problem(self): + # `locations` is a leaf special renderer; drilling into it is rejected. + self._assert_problem( + "findings", status=400, type_suffix="expand", data={"expand": "locations.location"}, + ) + + # --- 400: fields ----------------------------------------------------------------------- + def test_unknown_field_is_problem(self): + self._assert_problem("findings", status=400, type_suffix="fields", data={"fields": "id,nope"}) + + # --- 400: filter ----------------------------------------------------------------------- + def test_unknown_filter_param_is_problem(self): + self._assert_problem("findings", status=400, type_suffix="filter", data={"not_a_filter": "x"}) + + def test_invalid_filter_value_is_problem(self): + # `cwe` is numeric; a non-numeric value fails django-filter validation. + self._assert_problem("findings", status=400, type_suffix="filter", data={"cwe": "abc"}) + + def test_unknown_ordering_is_problem(self): + self._assert_problem("findings", status=400, type_suffix="filter", data={"o": "nope"}) + + # --- 400: pagination ------------------------------------------------------------------- + def test_bad_limit_is_problem(self): + self._assert_problem("findings", status=400, type_suffix="pagination", data={"limit": "-1"}) + + def test_non_integer_limit_is_problem(self): + self._assert_problem("findings", status=400, type_suffix="pagination", data={"limit": "abc"}) + + def test_unknown_pagination_mode_is_problem(self): + self._assert_problem( + "findings", status=400, type_suffix="pagination", data={"pagination": "bogus"}, + ) + + def test_cursor_non_keyset_ordering_is_problem(self): + # Cursor mode is implemented; a non-keyset-safe ordering is a pagination problem. + self._assert_problem( + "findings", status=400, type_suffix="pagination", + data={"pagination": "cursor", "o": "title"}, + ) + + # --- 401 / 404 ------------------------------------------------------------------------- + def test_anonymous_is_401_problem(self): + self._assert_problem( + "findings", status=401, type_suffix="unauthorized", client=self.anonymous_client(), + ) + + def test_unknown_or_unauthorized_detail_is_404_problem(self): + self._assert_problem("findings/99999999", status=404, type_suffix="not-found") + + +class _CyclicSchema(Schema): + + """A synthetic self-referential schema used only to exercise the expand cycle guard.""" + + django_model: ClassVar = Finding + EXPANDABLE: ClassVar[dict] = {} + + id: int + + +_CyclicSchema.EXPANDABLE = {"loop": ExpandRel(attr="x", path="x", schema=_CyclicSchema)} + + +class TestApiV3ExpandCycleGuard(SimpleTestCase): + + """ + Kernel-internal unit test (§6 preamble). + + The cycle guard is not reachable via the OS2 findings registry (it has no cyclic relation), + so it is tested directly against ``plan()``. + """ + + def test_cycle_is_rejected(self): + with self.assertRaises(ProblemDetail) as ctx: + plan(_CyclicSchema, "loop") + self.assertEqual(400, ctx.exception.status) + self.assertEqual("expand", ctx.exception.error_type) + self.assertIn("cycle", ctx.exception.detail.lower()) diff --git a/unittests/api_v3/test_apiv3_examples.py b/unittests/api_v3/test_apiv3_examples.py new file mode 100644 index 00000000000..85bb103c12e --- /dev/null +++ b/unittests/api_v3/test_apiv3_examples.py @@ -0,0 +1,518 @@ +r""" +Real request/response capture harness -> ``api_v3_examples.md`` (repo root) (OS6, §6). + +**CI-EXCLUDED.** Gated behind ``DD_API_V3_EXAMPLES=1`` so the normal suite never runs it. Run: + + docker compose exec -e DD_API_V3_EXAMPLES=1 uwsgi \\ + python manage.py test unittests.api_v3.test_apiv3_examples -v2 --keepdb + +Every example is a **real** request executed through the in-process Django test client against the +fixture data; the response bodies are rendered **verbatim** (only the caller's token is redacted and +long ``results`` lists are truncated to ~3 rows with an explicit marker, per the task). The rendered +markdown is written to ``/app/api_v3_examples.md`` (the repo root is bind-mounted, so it persists to +the host) and echoed to stdout between markers as a fallback. Nothing is hand-written or fabricated. +""" +from __future__ import annotations + +import datetime +import json +import os +from pathlib import Path +from unittest import skipUnless +from urllib.parse import urlsplit + +from dojo.finding.helper import save_cwes, save_vulnerability_ids +from dojo.location.models import Location, LocationFindingReference +from dojo.models import Finding, Product, Product_Type +from dojo.utils import get_system_setting + +from .base import ApiV3TestCase + +_EXAMPLES_ON = os.environ.get("DD_API_V3_EXAMPLES") == "1" +_REPO_ROOT = Path(__file__).resolve().parents[2] +_OUT = _REPO_ROOT / "api_v3_examples.md" +_TOKEN_PLACEHOLDER = "" # doc placeholder, not a secret +_MAX_ROWS = 3 + + +def _truncate(body: object) -> object: + """Truncate a list envelope's ``results`` to ~3 rows with an explicit marker (task rule).""" + if isinstance(body, dict) and isinstance(body.get("results"), list): + results = body["results"] + if len(results) > _MAX_ROWS: + trimmed = dict(body) + hidden = len(results) - _MAX_ROWS + trimmed["results"] = [*results[:_MAX_ROWS], f"... {hidden} more result(s) truncated"] + return trimmed + return body + + +def _pretty(body: object) -> str: + return json.dumps(body, indent=2, default=str) + + +@skipUnless(_EXAMPLES_ON, "examples harness is CI-excluded; set DD_API_V3_EXAMPLES=1 to run") +class TestApiV3Examples(ApiV3TestCase): + + def setUp(self): + super().setUp() + self.blocks: list[str] = [] + self.prefix = get_system_setting("url_prefix") + + def _v2_findings_url(self, query: str = "") -> str: + return f"/{self.prefix}api/v2/findings/{query}" + + # --- capture / render ----------------------------------------------------------------------- + def _record( + self, *, title: str, intro: str, method: str, url: str, response, + req_headers: list[str] | None = None, req_body: str | None = None, truncate: bool = False, + body_transform=None, + ) -> None: + headers = req_headers or [f"Authorization: Token {_TOKEN_PLACEHOLDER}"] + lines = [f"### {title}", "", intro, "", "**Request**", "", "```http", f"{method} {url}"] + lines += headers + if req_body is not None: + lines += ["", req_body] + lines += ["```", "", f"**Response** — `{response.status_code}`", ""] + if getattr(response, "streaming", False): + lines += ["```", "", "```", ""] + else: + try: + body = response.json() + if body_transform is not None: + body = body_transform(body) + if truncate: + body = _truncate(body) + rendered = _pretty(body) + except ValueError: + rendered = response.content.decode("utf-8", "replace") or "" + lines += ["```json", rendered, "```", ""] + self.blocks.append("\n".join(lines)) + + def _record_csv(self, *, title: str, intro: str, url: str, response, max_lines: int = 4) -> None: + """Render a streaming CSV export: request, the response headers, and the first ~4 lines.""" + lines = [ + f"### {title}", "", intro, "", "**Request**", "", "```http", f"GET {url}", + f"Authorization: Token {_TOKEN_PLACEHOLDER}", "```", "", + f"**Response** — `{response.status_code}`", "", "Response headers:", "", "```http", + ] + lines += [ + f"{header}: {response[header]}" + for header in ("Content-Type", "Content-Disposition", "X-API-Status") + if response.has_header(header) + ] + lines += ["```", "", f"Body (first {max_lines} lines):", "", "```csv"] + content = b"".join(response.streaming_content).decode("utf-8") + body_lines = content.splitlines() + lines += body_lines[:max_lines] + hidden = len(body_lines) - len(body_lines[:max_lines]) + if hidden > 0: + lines.append(f"... {hidden} more row(s) truncated") + lines += ["```", ""] + self.blocks.append("\n".join(lines)) + + # --- cursor helpers ------------------------------------------------------------------------- + @staticmethod + def _relativize(url: str) -> str: + """Strip scheme+host from an absolute ``next`` URL so the test client can follow it verbatim.""" + parts = urlsplit(url) + return parts.path + (f"?{parts.query}" if parts.query else "") + + @staticmethod + def _shorten_cursor(url: str) -> str: + """Truncate the opaque ``cursor=`` token in a URL for readability (marked as truncated).""" + marker = "cursor=" + idx = url.find(marker) + if idx == -1: + return url + start = idx + len(marker) + end = url.find("&", start) + if end == -1: + end = len(url) + value = url[start:end] + if len(value) > 24: + value = value[:20] + "..." + return url[:start] + value + url[end:] + + def _shorten_next_cursor(self, body): + """``body_transform`` for cursor envelopes: truncate the opaque cursor in ``next``.""" + if isinstance(body, dict) and isinstance(body.get("next"), str): + body = dict(body) + body["next"] = self._shorten_cursor(body["next"]) + return body + + # --- the harness ---------------------------------------------------------------------------- + def test_capture_examples(self): + finding = Finding.objects.select_related("test__engagement__product").first() + self.assertIsNotNone(finding) + + # Attach a couple of URL location edges so the expand=locations / sub-resource examples show + # real edge rows (locations are import-driven; seeding here only makes the doc illustrative). + for value, status in (("https://example.com/login", "Active"), ("https://example.com/admin", "Mitigated")): + loc = Location.objects.create(location_type="url", location_value=value) + LocationFindingReference.objects.create(location=loc, finding=finding, status=status) + + # Seed a few High+active findings so the filtered page-2 envelope shows next AND previous. + Finding.objects.bulk_create([ + Finding(title=f"example high active {i}", severity="High", numerical_severity="S1", + description="doc seed", test=finding.test, reporter=self.admin, active=True, verified=True) + for i in range(6) + ]) + + self._capture_findings(finding) + self._capture_assets() + self._write(_OUT) + + # --- findings (the complex entity) ---------------------------------------------------------- + def _capture_findings(self, finding: Finding) -> None: + fid = finding.id + # Populate a detail-only field (`impact`) plus the identity-row fields (scalar `cwe`, the + # `cwes` list incl. an extra CWE, `vulnerability_ids`, and `tags`) so the captured finding + # bodies show them with real values rather than empties (§4.5 / §12 2026-07-21). These ride on + # every slim row, so the list/cursor/CSV examples below display them without a detail fetch. + finding.impact = "Unauthorized disclosure of customer data if exploited." + finding.cwe = 79 + finding.unsaved_cwes = [89] + save_cwes(finding) # persists Finding_CWE rows: primary CWE-79 first, then CWE-89 + save_vulnerability_ids(finding, ["CVE-2024-3094", "GHSA-8r3f-844x-mdgq"]) # sets finding.cve too + finding.tags = ["internal", "tls"] + finding.save() + + self._record( + title="Finding — GET detail (slim + detail fields)", + intro="Retrieve a single finding. Relations render as closed `{id, name}` refs (§4.4); the " + "parent chain (`test`/`engagement`/`asset`/`organization`) is denormalized onto the " + "finding. `locations_count` is an annotation; the full list is a sub-resource (§4.14).", + method="GET", url=self.v3_url(f"findings/{fid}"), + response=self.client.get(self.v3_url(f"findings/{fid}")), + ) + + self._record( + title="Finding — GET detail with `?expand=test.engagement,locations`", + intro="`?expand=` swaps a ref for the target's slim object inline and drives real " + "`select_related`/`prefetch_related` (§4.6). `expand=locations` replaces " + "`locations_count` with edge rows `{location, status, audit_time, auditor}`.", + method="GET", url=self.v3_url(f"findings/{fid}?expand=test.engagement,locations"), + response=self.client.get(self.v3_url(f"findings/{fid}?expand=test.engagement,locations")), + ) + + self._record( + title="Finding — GET list, filtered (`severity=High&active=true`) + pagination page 2", + intro="The filter grammar is a documented, snapshot-tested vocabulary (§4.9). The list " + "envelope is always `{count, next, previous, results, meta?}` (I1); `next`/`previous` " + "are opaque URLs (D4). Here `limit=2&offset=2` is page 2, so both are non-null.", + method="GET", + url=self.v3_url("findings?severity=High&active=true&limit=2&offset=2"), + response=self.client.get(self.v3_url("findings?severity=High&active=true&limit=2&offset=2")), + truncate=True, + ) + + self._capture_cursor() + + self._record( + title="Finding — GET list with `?include=counts`", + intro="`?include=counts` adds severity/status totals computed over the *filtered, " + "authorized* queryset into `meta` in one aggregate query — no second round-trip (§4.8).", + method="GET", url=self.v3_url("findings?include=counts&limit=2"), + response=self.client.get(self.v3_url("findings?include=counts&limit=2")), + truncate=True, + ) + + self._record( + title="Finding — GET list with `?fields=` opting into a detail field (`impact`)", + intro="A list returns the slim shape by default. `?fields=` may name any **detail** field " + "(here `impact`, normally only on the detail endpoint) and it is returned on the list " + "with no second request (§4.7). Fields are row-columns, so this is a wider `SELECT` on " + "the same single query — never a per-row cost; the default list defers these heavy " + "columns entirely and requesting one un-defers exactly it.", + method="GET", + url=self.v3_url(f"findings?id__in={fid}&fields=id,title,severity,impact"), + response=self.client.get(self.v3_url(f"findings?id__in={fid}&fields=id,title,severity,impact")), + truncate=True, + ) + + self._capture_csv_export() + + # notes sub-resource: POST then GET + note_body = {"entry": "Reviewed with the security team; scheduled for the next sprint.", "private": False} + self._record( + title="Finding — POST a note (sub-resource)", + intro="Notes are one generic sub-resource across resources (§4.12). Authorization is " + "inherited from the parent finding.", + method="POST", url=self.v3_url(f"findings/{fid}/notes"), + req_headers=[f"Authorization: Token {_TOKEN_PLACEHOLDER}", "Content-Type: application/json"], + req_body=_pretty(note_body), + response=self.client.post(self.v3_url(f"findings/{fid}/notes"), note_body, format="json"), + ) + self._record( + title="Finding — GET notes (sub-resource)", + intro="List a finding's notes (paginated envelope). v2 parity: all notes are returned; " + "`private` is a label, not a per-user read filter (§12).", + method="GET", url=self.v3_url(f"findings/{fid}/notes"), + response=self.client.get(self.v3_url(f"findings/{fid}/notes")), truncate=True, + ) + + self._record( + title="Finding — GET locations (sub-resource, edge rows)", + intro="Finding↔Location is many-to-many with status on the edge (D5). Each row is a " + "location ref (carrying `type`) plus the edge `status`/`audit_time`/`auditor` (§4.14).", + method="GET", url=self.v3_url(f"findings/{fid}/locations"), + response=self.client.get(self.v3_url(f"findings/{fid}/locations")), truncate=True, + ) + + # PATCH a *separate* finding so the read examples above stay pristine. + patch_target = Finding.objects.create( + title="Example finding to patch", severity="Low", numerical_severity="S3", + description="before patch", test=finding.test, reporter=self.admin, active=True, verified=False, + ) + patch_body = {"severity": "Medium", "verified": True} + self._record( + title="Finding — PATCH (partial update)", + intro="Partial update (PATCH-only in alpha; §12). Write payloads reference relations by " + "integer id and only send changed fields; the response is the updated detail shape.", + method="PATCH", url=self.v3_url(f"findings/{patch_target.id}"), + req_headers=[f"Authorization: Token {_TOKEN_PLACEHOLDER}", "Content-Type: application/json"], + req_body=_pretty(patch_body), + response=self.client.patch(self.v3_url(f"findings/{patch_target.id}"), patch_body, format="json"), + ) + + self._capture_cwes_write(finding) + self._capture_put(finding) + + # POST /import (multipart). Described request + real JSON response. + self._capture_import() + + # --- cursor pagination (D4/§4.3) ------------------------------------------------------------ + def _capture_cursor(self) -> None: + query = "findings?pagination=cursor&severity=High&active=true&limit=2" + page1 = self.client.get(self.v3_url(query)) + self._record( + title="Finding — GET list, cursor pagination (`?pagination=cursor`) — page 1", + intro="Forward-only keyset mode for export/sync consumers (D4/§4.3). Same envelope, but " + "`count` and `previous` are always `null` and `next` carries an opaque, signed " + "`cursor=` token (truncated below). No `COUNT` query runs; the page is read as " + "`limit+1` rows to detect a next page. Keyset-safe orderings only (`id` default, plus " + "`created`/`updated` where a resource declares them); filters/`fields=`/`expand=`/" + "`include=` compose unchanged.", + method="GET", url=self.v3_url(query), + response=page1, truncate=True, body_transform=self._shorten_next_cursor, + ) + next_url = page1.json().get("next") + if next_url: + follow = self._relativize(next_url) + self._record( + title="Finding — GET list, cursor pagination — page 2 (following `next`)", + intro="Follow the page-1 `next` URL verbatim (opaque). The signed cursor encodes only the " + "ordering + last-row key position (never the filters), so the keyset predicate reads " + "exactly the rows after that position — no offset, no count. When `next` is `null` " + "the walk is complete.", + method="GET", url=self._shorten_cursor(follow), + response=self.client.get(follow), truncate=True, body_transform=self._shorten_next_cursor, + ) + + # --- CSV export (D6/§4.15) ------------------------------------------------------------------ + def _capture_csv_export(self) -> None: + query = ("findings/export.csv?severity=High&o=id" + "&fields=id,title,severity,asset,cwe,cwes,vulnerability_ids,tags") + self._record_csv( + title="Finding — GET export.csv (CSV export, the second projection of the filter contract)", + intro="`GET //export.csv` (§4.15) streams the **whole** filtered, authorized set as " + "CSV using the identical filter/`o=`/`q=`/`fields=` contract as the list — there is no " + "pagination (`expand`/`include`/`limit`/`offset`/`pagination`/`cursor` → 400). A `{id, " + "name}` ref flattens to `_id`/`_name` columns (a location ref adds " + "`_type`); list fields (`tags`, `cwes`, `vulnerability_ids`) join with `;`. Cells " + "starting with `= + - @` or TAB are quote-prefixed (spreadsheet formula-injection " + "defense). A zero-row export still emits the header row.", + url=self.v3_url(query), response=self.client.get(self.v3_url(query)), + ) + + # --- cwes on writes (§12, 2026-07-21) ------------------------------------------------------- + def _capture_cwes_write(self, finding: Finding) -> None: + target = Finding.objects.create( + title="Example finding for cwes write", severity="Low", numerical_severity="S3", + description="before cwes patch", test=finding.test, reporter=self.admin, + active=True, verified=False, + ) + patch_body = {"cwes": [79, 89, 352]} + self._record( + title="Finding — PATCH `cwes` (write a CWE list; scalar `cwe` mirrors the primary)", + intro="Finding writes accept a flat `cwes: list[int]` (§12, 2026-07-21) — symmetric with the " + "read shape and parallel to `vulnerability_ids`. The first entry is mirrored into the " + "scalar `cwe`; the `Finding_CWE` rows persist primary-first. An omitted `cwes` leaves " + "existing rows untouched; an explicit `[]` clears the extras.", + method="PATCH", url=self.v3_url(f"findings/{target.id}"), + req_headers=[f"Authorization: Token {_TOKEN_PLACEHOLDER}", "Content-Type: application/json"], + req_body=_pretty(patch_body), + response=self.client.patch(self.v3_url(f"findings/{target.id}"), patch_body, format="json"), + ) + self._record( + title="Finding — GET detail after the `cwes` PATCH (read-back)", + intro="Read-back confirms persistence: `cwes` returns the list in storage order and the " + "scalar `cwe` mirror is the primary (first) entry.", + method="GET", url=self.v3_url(f"findings/{target.id}"), + response=self.client.get(self.v3_url(f"findings/{target.id}")), + ) + + # --- PUT full replace (§4.11) --------------------------------------------------------------- + def _capture_put(self, finding: Finding) -> None: + target = Finding.objects.create( + title="Example finding to replace", severity="High", numerical_severity="S1", + description="original description", test=finding.test, reporter=self.admin, + active=True, verified=True, + mitigation="Upgrade the affected dependency to a patched release.", + impact="Remote code execution on the affected host.", + ) + put_body = { + "title": "Example finding replaced via PUT", + "severity": "Medium", + "description": "replaced description", + "active": True, + "verified": False, + } + self._record( + title="Finding — PUT (full replace)", + intro="`PUT` is a **full replace** (§4.11): it validates against the create-shaped schema " + "(required fields enforced, unknown fields → 400) and applies the body **without** " + "`exclude_unset`, so every omitted optional resets to its default — mirroring v2's " + "`update(partial=False)`. The finding had `mitigation`/`impact` set, but the PUT body " + "omits them, so they reset to `null`. The immutable parent `test` is not in the replace " + "schema and is never reassigned (like PATCH). Finding PUT still flows through the " + "service (JIRA / risk-acceptance / vuln-id side-effects), never route logic (I6).", + method="PUT", url=self.v3_url(f"findings/{target.id}"), + req_headers=[f"Authorization: Token {_TOKEN_PLACEHOLDER}", "Content-Type: application/json"], + req_body=_pretty(put_body), + response=self.client.put(self.v3_url(f"findings/{target.id}"), put_body, format="json"), + ) + + def _capture_import(self) -> None: + from unittests.dojo_test_case import get_unit_tests_scans_path # noqa: PLC0415 + + scan_path = get_unit_tests_scans_path("zap") / "0_zap_sample.xml" + # Current form (§4.13). Auto-create fields speak the v3 wire names `asset_name` / + # `organization_name` (D11); there is NO `background`/job field — v3 imports are synchronous. + import_desc = ( + "multipart/form-data fields (v3 imports are synchronous — there is no `background`/job field):\n" + " mode=import # auto | import | reimport (default auto)\n" + " scan_type=ZAP Scan\n" + " engagement=4 # import target; or test= (reimport); or\n" + " # asset_name + engagement_name (+ organization_name)\n" + " # + auto_create_context=true to auto-create the target\n" + " file=@0_zap_sample.xml\n" + " active=true\n" + " verified=true\n" + " push_to_jira=false # OR-ed with the JIRA project's push_all_issues\n" + " close_old_findings=false # destructive flags are never implied by mode\n" + " close_old_findings_product_scope=false" + ) + with scan_path.open(encoding="utf-8") as scan: + payload = {"scan_type": "ZAP Scan", "mode": "import", "engagement": 4, + "file": scan, "active": "true", "verified": "true"} + response = self.client.post(self.v3_url("import"), payload, format="multipart") + self._record( + title="Import — POST /import (consolidated import/reimport/auto)", + intro="One endpoint for import, reimport and auto-resolve (§4.13). Destructive flags " + "(`close_old_findings`, `close_old_findings_product_scope`) are never implied by mode; " + "the response echoes the resolved mode + effective flags. Auto-create fields use the " + "v3 wire names `asset_name`/`organization_name` (D11); imports are synchronous (no job " + "resource).", + method="POST", url=self.v3_url("import"), + req_headers=[f"Authorization: Token {_TOKEN_PLACEHOLDER}", + "Content-Type: multipart/form-data"], + req_body=import_desc, response=response, + ) + + # --- assets (the simple contrast) ----------------------------------------------------------- + def _capture_assets(self) -> None: + product = Product.objects.first() + self._record( + title="Asset — GET detail", + intro="A simple entity for contrast with findings: identity, `organization` ref, and the " + "documented heavier detail fields.", + method="GET", url=self.v3_url(f"assets/{product.id}"), + response=self.client.get(self.v3_url(f"assets/{product.id}")), + ) + self._record( + title="Asset — GET list", + intro="Same envelope and grammar as findings; slim rows only on list (§4.5).", + method="GET", url=self.v3_url("assets?limit=2"), + response=self.client.get(self.v3_url("assets?limit=2")), truncate=True, + ) + + pt = Product_Type.objects.first() + create_body = {"name": "Example v3 Asset", "description": "Created via the v3 API examples harness", + "organization": pt.id, "lifecycle": "production", "tags": ["pci", "example"]} + create_resp = self.client.post(self.v3_url("assets"), create_body, format="json") + self._record( + title="Asset — POST (create)", + intro="Create an asset. Relations are referenced by integer id (§4.11); unknown fields " + "are rejected (400). Response is the created detail shape (`201`).", + method="POST", url=self.v3_url("assets"), + req_headers=[f"Authorization: Token {_TOKEN_PLACEHOLDER}", "Content-Type: application/json"], + req_body=_pretty(create_body), response=create_resp, + ) + + new_id = create_resp.json().get("id") if create_resp.status_code == 201 else product.id + patch_body = {"description": "Updated description via PATCH"} + self._record( + title="Asset — PATCH (partial update)", + intro="Partial update; only the changed field is sent.", + method="PATCH", url=self.v3_url(f"assets/{new_id}"), + req_headers=[f"Authorization: Token {_TOKEN_PLACEHOLDER}", "Content-Type: application/json"], + req_body=_pretty(patch_body), + response=self.client.patch(self.v3_url(f"assets/{new_id}"), patch_body, format="json"), + ) + + # --- output --------------------------------------------------------------------------------- + def _header(self) -> str: + prefix = f"/{self.prefix}" if self.prefix else "/" + base = f"{prefix.rstrip('/')}/api/v3-alpha" + return "\n".join([ + "# DefectDojo API v3 (alpha) — worked examples", + "", + ("> **Auto-generated, do not hand-edit.** Every request/response below was captured by " + "`unittests/api_v3/test_apiv3_examples.py` (`DD_API_V3_EXAMPLES=1`, CI-excluded) making " + "**real** in-process requests against the test fixture. Tokens are redacted; long lists " + "are truncated to ~3 rows. Regenerate with the command in that file's docstring."), + "", + f"Captured: {datetime.datetime.now(tz=datetime.UTC).isoformat()}", + "", + "## Conventions (see `API_V3_PLAN.md` §4)", + "", + (f"- **Mount:** alpha lives at `{base}/` (moves to `/api/v3/` at beta — one migration, D1). " + "Every response carries `X-API-Status: alpha`."), + ("- **Auth (D8):** send an existing v2 token as `Authorization: Token ` (works " + "unchanged on v3), or a Django session cookie + `X-CSRFToken` on unsafe methods."), + ("- **Envelope & pagination (§4.3):** every list is `{count, next, previous, results, meta?}` " + "and nothing else (I1). `next`/`previous` are opaque URLs. Offset mode is the default " + "(`limit=25`, max `250`); `?pagination=cursor` opts into forward-only keyset paging " + "(`count`/`previous` null, opaque signed cursor in `next`)."), + ("- **Refs (§4.4):** relations render as `{id, name}` (locations add `type`). Write payloads " + "reference relations by integer id — the asymmetry is intentional (§4.11)."), + ("- **Writes (§4.11):** `PATCH` is a partial update (only supplied fields change); `PUT` is a " + "full replace (omitted optionals reset to their defaults). Both reject unknown fields (400)."), + ("- **CSV export (§4.15):** `GET //export.csv` streams the whole filtered, " + "authorized set as CSV using the identical filter/`o=`/`q=`/`fields=` contract (no " + "pagination); refs flatten to `_id`/`_name` columns, list fields join with `;`."), + ("- **`?expand=` (§4.6):** dotted paths swap refs for slim objects inline and drive the " + "queryset (the real N+1 fix). Budget-guarded."), + ("- **`?fields=` (§4.7):** comma-separated allowlist; `id` is always included. On a list it " + "may also request any detail field (a wider SELECT on one query, never per-row)."), + ("- **`?include=counts` (§4.8):** adds aggregate totals to `meta` over the filtered, " + "authorized queryset."), + ("- **Errors (§4.10):** RFC 9457 `application/problem+json` with a `fields` extension for " + "validation errors."), + "", + "---", + "", + ]) + + def _write(self, out: Path) -> None: + document = self._header() + "\n\n---\n\n".join(self.blocks) + "\n" + print("\n===== OS6 API V3 EXAMPLES (verbatim) =====") # noqa: T201 + print(document) # noqa: T201 + print("===== END OS6 API V3 EXAMPLES =====") # noqa: T201 + try: + out.write_text(document, encoding="utf-8") + print(f"[examples] wrote {out}") # noqa: T201 + except OSError as exc: + print(f"[examples] could not write {out} ({exc}); use the verbatim block above") # noqa: T201 diff --git a/unittests/api_v3/test_apiv3_expand_rbac.py b/unittests/api_v3/test_apiv3_expand_rbac.py new file mode 100644 index 00000000000..03d84c71f0b --- /dev/null +++ b/unittests/api_v3/test_apiv3_expand_rbac.py @@ -0,0 +1,230 @@ +""" +Expand / include / sub-resource RBAC sweep for API v3 (OS6, §11). + +Ports the *intent* of ``unittests/test_apiv2_prefetch_rbac.py`` to the v3 ref / expand / include / +sub-resource surface. The v2 test pinned that ``?prefetch=`` could not reach objects outside the +caller's authorized querysets; this file pins the equivalent guarantee for v3's replacements: + + A user must NEVER see an **expanded** object, an **included** count, a **denormalized parent + ref**, or a **sub-resource** row that was drawn from outside their authorized querysets. + +The v3 design makes this structurally simpler than v2: every projection (slim refs, ``?expand=`` +select/prefetch, ``?include=counts`` aggregate, and the ``/{id}/`` routes) is computed over +``get_authorized_findings(Permissions.Finding_View, user=request.user)`` -- so a finding the caller +cannot see never enters the pipeline, and nothing reachable *from* an authorized finding (its parent +hierarchy, reporter, locations, notes) can therefore belong to another product. These tests build a +two-product world, authorize a non-superuser on exactly one product, and assert the other product's +data never leaks through any v3 projection. + +Setup mirrors the OSS authorization model used by the other v3 RBAC tests +(``test_apiv3_assets.py`` / ``test_apiv3_subresources.py``): membership via the legacy +``product.authorized_users`` M2M, which the auth-filter plugin resolves into the finding queryset. +""" +from __future__ import annotations + +import datetime + +from dojo.location.models import Location, LocationFindingReference +from dojo.models import ( + Dojo_User, + Engagement, + Finding, + Notes, + Product, + Product_Type, + Test, + Test_Type, +) + +from .base import ApiV3TestCase + +_NUMERICAL = {"Critical": "S0", "High": "S1", "Medium": "S2", "Low": "S3", "Info": "S4"} + + +class _TwoProductWorld(ApiV3TestCase): + + """ + Build two isolated products (A authorized to ``member``, B not) with a known finding multiset, + so leakage assertions can name exact counts and identify B's objects by their distinctive data. + """ + + def setUp(self): + super().setUp() + self.prod_type = Product_Type.objects.create(name="v3-rbac-pt") + self.test_type = Test_Type.objects.create(name="v3-rbac-tt") + + # A non-superuser member authorized on product A only (authorized_users targets Dojo_User). + self.member = Dojo_User.objects.create_user(username="v3_expand_member", password="x") # noqa: S106 + # A distinctive reporter for product B's findings -- if this user ever appears in an + # expanded/denormalized ref seen by the member, a user-enumeration leak has occurred (the + # v2 sub-vector 4a case). + self.reporter_b = Dojo_User.objects.create_user(username="v3_secret_reporter_b", password="x") # noqa: S106 + + self.product_a, self.test_a = self._make_product("v3-rbac-product-A") + self.product_b, self.test_b = self._make_product("v3-rbac-product-B") + self.product_a.authorized_users.add(self.member) + + # Product A: 2 Critical + 1 High, reported by admin. Product B: 3 Low + 1 Medium, reported + # by the secret reporter. Distinct multisets make count-isolation assertions unambiguous. + self.findings_a = self._make_findings(self.test_a, ["Critical", "Critical", "High"], self.admin) + self.findings_b = self._make_findings(self.test_b, ["Low", "Low", "Low", "Medium"], self.reporter_b) + + def _make_product(self, name: str) -> tuple[Product, Test]: + product = Product.objects.create(name=name, description="x", prod_type=self.prod_type, sla_configuration_id=1) + engagement = Engagement.objects.create( + product=product, name=f"{name}-eng", + target_start=datetime.date(2026, 1, 1), target_end=datetime.date(2026, 1, 2), + ) + test = Test.objects.create( + engagement=engagement, test_type=self.test_type, + target_start=datetime.datetime(2026, 1, 1, tzinfo=datetime.UTC), + target_end=datetime.datetime(2026, 1, 2, tzinfo=datetime.UTC), lead=self.admin, + ) + return product, test + + def _make_findings(self, test: Test, severities: list[str], reporter: Dojo_User) -> list[Finding]: + findings = Finding.objects.bulk_create([ + Finding( + title=f"{test.engagement.product.name} f{i} {sev}", + severity=sev, numerical_severity=_NUMERICAL[sev], description="x", + test=test, reporter=reporter, active=True, verified=False, + ) + for i, sev in enumerate(severities) + ]) + return list(findings) + + def member_client(self): + return self.token_client(user=self.member) + + +class TestApiV3ExpandRbacPositive(_TwoProductWorld): + + """A user authorized on the finding's product CAN expand its relations (the positive case).""" + + def test_authorized_member_can_expand_test_engagement(self): + finding = self.findings_a[0] + detail = self.get_json( + f"findings/{finding.id}", client=self.member_client(), data={"expand": "test.engagement"}, + ) + # test ref swapped for the test slim, engagement inlined inside it, all belonging to product A. + self.assertIn("engagement", detail["test"]) + self.assertEqual(self.test_a.engagement.id, detail["test"]["engagement"]["id"]) + self.assertEqual(self.product_a.id, detail["test"]["engagement"]["asset"]["id"]) + + def test_authorized_member_list_expands_only_own_product(self): + body = self.get_json( + "findings", client=self.member_client(), + data={"expand": "test.engagement,asset,organization", "limit": 250}, + ) + self.assertEqual(len(self.findings_a), body["count"]) + for row in body["results"]: + # Denormalized parent refs AND expanded objects must all be product A. + self.assertEqual(self.product_a.id, row["asset"]["id"]) + self.assertEqual(self.product_a.id, row["test"]["engagement"]["asset"]["id"]) + + def test_authorized_member_include_counts_reflects_own_product(self): + body = self.get_json("findings", client=self.member_client(), data={"include": "counts"}) + counts = body["meta"]["counts"] + self.assertEqual(len(self.findings_a), counts["total"]) + self.assertEqual(2, counts["severity"]["Critical"]) + self.assertEqual(1, counts["severity"]["High"]) + + +class TestApiV3ExpandRbacCrossProduct(_TwoProductWorld): + + """Product B's data must never leak into the member's list / detail / expand / filter.""" + + def test_list_never_includes_other_product_rows(self): + body = self.get_json("findings", client=self.member_client(), data={"limit": 250}) + seen_products = {row["asset"]["id"] for row in body["results"]} + self.assertEqual({self.product_a.id}, seen_products) + self.assertNotIn(self.product_b.id, seen_products) + + def test_detail_of_other_product_finding_is_404_no_expansion(self): + # Even asking to expand cannot coerce disclosure of an unauthorized finding (§4.10 404). + self.get_json( + f"findings/{self.findings_b[0].id}", client=self.member_client(), + data={"expand": "test.engagement"}, expected=404, + ) + + def test_filter_by_other_product_returns_empty(self): + # A filter naming product B must still be intersected with the authorized queryset -> empty. + body = self.get_json("findings", client=self.member_client(), data={"asset": self.product_b.id}) + self.assertEqual(0, body["count"]) + self.assertEqual([], body["results"]) + + def test_expand_reporter_never_enumerates_other_product_users(self): + # The v2 sub-vector 4a analogue: expanding reporter must not surface a user attached only to + # findings the caller cannot see. Product B's secret reporter must never appear. + body = self.get_json( + "findings", client=self.member_client(), data={"expand": "reporter", "limit": 250}, + ) + # expand=reporter swaps the ref for the full UserSlim (keyed by `username`, §4.5). + reporter_names = {row["reporter"]["username"] for row in body["results"] if row["reporter"]} + self.assertNotIn(self.reporter_b.username, reporter_names) + self.assertEqual({self.admin.username}, reporter_names) + + def test_include_counts_never_counts_other_product(self): + body = self.get_json("findings", client=self.member_client(), data={"include": "counts"}) + counts = body["meta"]["counts"] + # Product B's 3 Low + 1 Medium must not inflate the member's aggregate. + self.assertEqual(0, counts["severity"]["Low"]) + self.assertEqual(0, counts["severity"]["Medium"]) + self.assertEqual(len(self.findings_a), counts["total"]) + + def test_superuser_counts_exceed_member_counts(self): + # Sanity contrast: the admin (who can see both products) counts strictly more than the member, + # proving the member's lower number is authorization, not an empty database. + admin_counts = self.get_json("findings", data={"include": "counts"})["meta"]["counts"] + member_counts = self.get_json( + "findings", client=self.member_client(), data={"include": "counts"}, + )["meta"]["counts"] + self.assertGreater(admin_counts["total"], member_counts["total"]) + + +class TestApiV3SubResourceRbacInheritance(_TwoProductWorld): + + """ + Sub-resource rows (locations, notes) are drawn from the parent's own managers; an unauthorized + parent is a 404 so no row from another product can ever be read (parent-inherited authorization). + """ + + def setUp(self): + super().setUp() + self.finding_a = self.findings_a[0] + self.finding_b = self.findings_b[0] + # A location edge + a note on a finding in EACH product. + for finding, marker in ((self.finding_a, "a"), (self.finding_b, "b")): + location = Location.objects.create(location_type="url", location_value=f"https://example.com/{marker}") + LocationFindingReference.objects.create(location=location, finding=finding, status="Active") + note = Notes.objects.create(entry=f"note-{marker}", author=self.admin) + finding.notes.add(note) + + def test_locations_subresource_authorized_parent_ok(self): + body = self.get_json(f"findings/{self.finding_a.id}/locations", client=self.member_client()) + self.assertEqual(1, body["count"]) + self.assertEqual("https://example.com/a", body["results"][0]["location"]["name"]) + + def test_locations_subresource_unauthorized_parent_is_404(self): + self.get_json(f"findings/{self.finding_b.id}/locations", client=self.member_client(), expected=404) + + def test_notes_subresource_authorized_parent_ok(self): + body = self.get_json(f"findings/{self.finding_a.id}/notes", client=self.member_client()) + self.assertIn("note-a", [n["entry"] for n in body["results"]]) + + def test_notes_subresource_unauthorized_parent_is_404(self): + self.get_json(f"findings/{self.finding_b.id}/notes", client=self.member_client(), expected=404) + + def test_note_create_on_unauthorized_parent_is_404(self): + # Write attempts on an unseen parent are 404 too (never leak existence, never mutate). + response = self.member_client().post( + self.v3_url(f"findings/{self.finding_b.id}/notes"), {"entry": "x"}, format="json", + ) + self.assertEqual(404, response.status_code, response.content[:300]) + + def test_expand_locations_on_unauthorized_finding_is_404(self): + # expand=locations edge rows must not disclose another product's location edges either. + self.get_json( + f"findings/{self.finding_b.id}", client=self.member_client(), + data={"expand": "locations"}, expected=404, + ) diff --git a/unittests/api_v3/test_apiv3_filter_contract.py b/unittests/api_v3/test_apiv3_filter_contract.py new file mode 100644 index 00000000000..337a6d3e7d7 --- /dev/null +++ b/unittests/api_v3/test_apiv3_filter_contract.py @@ -0,0 +1,87 @@ +r""" +Filter-contract snapshot test for API v3 (D6 / §4.9, §6 OS2 / I2). + +The per-object filter vocabulary (params + orderings + search fields) is a *tested artifact*: this +test renders every registered ``FilterSpec`` to ``unittests/api_v3/snapshots/filters.json`` and +fails on any drift, so a contract change can never land silently -- it must be an intentional +snapshot update. + +WHY the vocabulary is a locked, tested contract (not just a convenience test): + +1. **It closes a silent-failure mode.** v2 (django-filter / DRF) *ignores* query params it does not + recognise: ``GET /findings?severty=Critical`` (a typo for ``severity``) silently drops the filter + and returns EVERY finding, while the caller believes they are looking at criticals only -- a + dangerous illusion for a security product (you think you triaged all criticals; you triaged + nothing). v3 rejects any unknown param with a 400 ``filter`` problem+json (§12 OS2), which is only + safe if the accepted vocabulary is a *closed, reviewed set*. This snapshot is what keeps it closed: + the vocabulary cannot grow (or a typo-prone alias sneak in) without a visible, reviewed change. + +2. **One vocabulary drives many projections (D6).** The exact same params drive the list, the + ``?include=counts`` aggregate over the filtered/authorized queryset, and -- by design -- future + aggregation/chart endpoints, CSV/export, and the reserved ``POST //query`` saved-view + substrate. A param that silently changes meaning (or disappears) therefore doesn't break one + endpoint; it breaks persisted saved filters and dashboards months later, far from the change that + caused it. Pinning the vocabulary in one snapshot makes every consumer's contract move together. + +3. **The workflow makes drift reviewable.** Unintended drift fails CI here. A *deliberate* contract + change is made by regenerating the snapshot, so the change shows up as a reviewable diff in + ``snapshots/filters.json`` in the same PR -- the vocabulary change is reviewed as data, not buried + in a route diff. + +Regenerate the snapshot deliberately with:: + + DD_API_V3_UPDATE_SNAPSHOTS=1 ./run-unittest.sh --test-case \\ + unittests.api_v3.test_apiv3_filter_contract +""" +from __future__ import annotations + +import json +import os +from pathlib import Path + +# Importing the built API instance forces every resource route module to import, which registers +# its FilterSpec into the kernel registry (resource -> kernel dependency direction, I5). +import dojo.api_v3.api # noqa: F401 +from dojo.api_v3.filtering import iter_filter_specs + +from .base import ApiV3TestCase + +SNAPSHOT = Path(__file__).parent / "snapshots" / "filters.json" + + +def _render() -> dict: + return {name: spec.vocabulary() for name, spec in sorted(iter_filter_specs().items())} + + +class TestApiV3FilterContract(ApiV3TestCase): + + def test_filter_contract_matches_snapshot(self): + current = _render() + # Sanity: at least the findings vocabulary must be registered. + self.assertIn("finding", current, "no FilterSpec registered -- did the routers import?") + + serialized = json.dumps(current, indent=2, sort_keys=True) + "\n" + + if os.environ.get("DD_API_V3_UPDATE_SNAPSHOTS") or not SNAPSHOT.exists(): + SNAPSHOT.parent.mkdir(parents=True, exist_ok=True) + SNAPSHOT.write_text(serialized) + if not os.environ.get("DD_API_V3_UPDATE_SNAPSHOTS"): + # First-ever run establishes the baseline; nothing to compare against yet. + return + + saved = json.loads(SNAPSHOT.read_text()) + self.assertEqual( + saved, + current, + "API v3 filter-contract drift detected. The per-object filter vocabulary is a LOCKED, " + "tested contract (see this module's docstring): unknown params 400 in v3, so the accepted " + "set must stay closed, and the SAME params drive lists / include=counts / future " + "aggregations / saved views -- silent drift breaks persisted filters far from the change.\n" + " * If this change is UNINTENDED: revert it -- you have altered the filter/ordering " + "vocabulary.\n" + " * If this change is DELIBERATE: regenerate the snapshot so the vocabulary change lands " + f"as a reviewable diff in {SNAPSHOT.name}, by re-running with " + "DD_API_V3_UPDATE_SNAPSHOTS=1.\n" + f"expected: {json.dumps(saved, sort_keys=True)}\n" + f"actual: {json.dumps(current, sort_keys=True)}", + ) diff --git a/unittests/api_v3/test_apiv3_finding_writes.py b/unittests/api_v3/test_apiv3_finding_writes.py new file mode 100644 index 00000000000..ba12f233c90 --- /dev/null +++ b/unittests/api_v3/test_apiv3_finding_writes.py @@ -0,0 +1,455 @@ +""" +Finding write-path tests for API v3 (OS3b, PART 2/3). + +Exercises the ``dojo/finding/services.py`` extraction (D7) via the v3 routes: create/update/delete +happy paths, side-effect assertions with JIRA **mocked** (the service imports ``jira_services`` as +``dojo.finding.services.jira_services`` -- patched here), vulnerability-id persistence incl. the +``cve`` mirror field, CWE handling, the mitigated-edit rules in both ``EDITABLE_MITIGATED_DATA`` +states, risk-acceptance invariants, and RBAC (404 unauthorized, 403 not-modifiable). +""" +from __future__ import annotations + +from unittest.mock import patch + +from django.test import override_settings + +from dojo.models import Dojo_User, Finding, Finding_CWE, Test, User, Vulnerability_Id + +from .base import ApiV3TestCase + +_PUSH = "dojo.finding.services.jira_services.push" +_KEEP_IN_SYNC = "dojo.finding.services.jira_services.is_keep_in_sync" + + +def _finding_payload(test_id: int, **overrides) -> dict: + payload = { + "test": test_id, + "title": "v3 write finding", + "severity": "High", + "description": "created via api v3", + "active": True, + "verified": False, + } + payload.update(overrides) + return payload + + +class TestApiV3FindingCreate(ApiV3TestCase): + + def setUp(self): + super().setUp() + self.test = Test.objects.first() + + def test_create_happy_path(self): + response = self.client.post(self.v3_url("findings"), _finding_payload(self.test.id), format="json") + self.assertEqual(201, response.status_code, response.content[:500]) + body = response.json() + self.assertEqual("V3 Write Finding", body["title"]) # title-cased on save + self.assertEqual(self.test.id, body["test"]["id"]) + created = Finding.objects.get(pk=body["id"]) + self.assertEqual(self.admin, created.reporter) # reporter defaults to the request user + + def test_create_persists_vulnerability_ids_and_cve(self): + payload = _finding_payload(self.test.id, vulnerability_ids=["CVE-2020-1234", "CVE-2020-5678"]) + response = self.client.post(self.v3_url("findings"), payload, format="json") + self.assertEqual(201, response.status_code, response.content[:500]) + created = Finding.objects.get(pk=response.json()["id"]) + vids = set(Vulnerability_Id.objects.filter(finding=created).values_list("vulnerability_id", flat=True)) + self.assertEqual({"CVE-2020-1234", "CVE-2020-5678"}, vids) + self.assertEqual("CVE-2020-1234", created.cve) # first vuln id mirrored into cve + + def test_create_persists_cwe_row(self): + response = self.client.post(self.v3_url("findings"), _finding_payload(self.test.id, cwe=79), format="json") + self.assertEqual(201, response.status_code, response.content[:500]) + created = Finding.objects.get(pk=response.json()["id"]) + self.assertEqual(79, created.cwe) + self.assertTrue(Finding_CWE.objects.filter(finding=created, cwe="CWE-79").exists()) + + def test_create_with_cwes_list_mirrors_primary_and_persists_rows(self): + # `cwes` only (no scalar `cwe`): the first entry becomes the primary (mirrored into `cwe`, + # like vulnerability_ids[0]->cve); rows are persisted primary-first and read back symmetric. + payload = _finding_payload(self.test.id, cwes=[89, 79]) + response = self.client.post(self.v3_url("findings"), payload, format="json") + self.assertEqual(201, response.status_code, response.content[:500]) + body = response.json() + created = Finding.objects.get(pk=body["id"]) + self.assertEqual(89, created.cwe) # first list entry mirrored into the scalar primary + self.assertEqual( + {"CWE-89", "CWE-79"}, + set(Finding_CWE.objects.filter(finding=created).values_list("cwe", flat=True)), + ) + self.assertEqual([89, 79], body["cwes"]) # read-back symmetry: primary first, then the rest + + def test_create_scalar_and_cwes_scalar_wins_primary(self): + # Both scalar `cwe` and `cwes`: the explicit scalar is the primary; the list supplies the + # rest of the rows (primary first on read-back). + payload = _finding_payload(self.test.id, cwe=79, cwes=[89, 100]) + response = self.client.post(self.v3_url("findings"), payload, format="json") + self.assertEqual(201, response.status_code, response.content[:500]) + body = response.json() + created = Finding.objects.get(pk=body["id"]) + self.assertEqual(79, created.cwe) # explicit scalar stays primary + self.assertEqual([79, 89, 100], body["cwes"]) # primary first, then the list + self.assertEqual( + {"CWE-79", "CWE-89", "CWE-100"}, + set(Finding_CWE.objects.filter(finding=created).values_list("cwe", flat=True)), + ) + + def test_create_duplicate_active_invariant_is_400(self): + payload = _finding_payload(self.test.id, active=True, duplicate=True) + response = self.client.post(self.v3_url("findings"), payload, format="json") + self.assertEqual(400, response.status_code) + self.assertEqual("application/problem+json", response["Content-Type"]) + + def test_create_false_positive_verified_invariant_is_400(self): + payload = _finding_payload(self.test.id, verified=True, false_p=True) + response = self.client.post(self.v3_url("findings"), payload, format="json") + self.assertEqual(400, response.status_code) + + def test_create_bad_severity_is_400(self): + payload = _finding_payload(self.test.id, severity="Catastrophic") + response = self.client.post(self.v3_url("findings"), payload, format="json") + self.assertEqual(400, response.status_code) + + def test_create_push_to_jira_calls_service(self): + with patch(_PUSH) as push: + payload = _finding_payload(self.test.id, push_to_jira=True) + response = self.client.post(self.v3_url("findings"), payload, format="json") + self.assertEqual(201, response.status_code, response.content[:500]) + push.assert_called_once() + + +class TestApiV3FindingUpdate(ApiV3TestCase): + + def setUp(self): + super().setUp() + self.finding = Finding.objects.filter(risk_accepted=False).first() + + def test_update_scalar(self): + response = self.client.patch( + self.v3_url(f"findings/{self.finding.id}"), {"severity": "Low"}, format="json", + ) + self.assertEqual(200, response.status_code, response.content[:500]) + self.finding.refresh_from_db() + self.assertEqual("Low", self.finding.severity) + + def test_update_vulnerability_ids(self): + response = self.client.patch( + self.v3_url(f"findings/{self.finding.id}"), + {"vulnerability_ids": ["CVE-2019-9999"]}, format="json", + ) + self.assertEqual(200, response.status_code, response.content[:500]) + self.finding.refresh_from_db() + self.assertEqual("CVE-2019-9999", self.finding.cve) + self.assertTrue(Vulnerability_Id.objects.filter(finding=self.finding, vulnerability_id="CVE-2019-9999").exists()) + + def _seed_cwe_rows(self, primary: int, extras: list[int]) -> None: + self.finding.cwe = primary + self.finding.save() + Finding_CWE.objects.filter(finding=self.finding).delete() + for n in [primary, *extras]: + Finding_CWE.objects.create(finding=self.finding, cwe=f"CWE-{n}") + + def _cwe_rows(self) -> set[str]: + return set(Finding_CWE.objects.filter(finding=self.finding).values_list("cwe", flat=True)) + + def test_update_cwe_resyncs_finding_cwe(self): + response = self.client.patch( + self.v3_url(f"findings/{self.finding.id}"), {"cwe": 89}, format="json", + ) + self.assertEqual(200, response.status_code, response.content[:500]) + self.finding.refresh_from_db() + self.assertEqual(89, self.finding.cwe) + self.assertTrue(Finding_CWE.objects.filter(finding=self.finding, cwe="CWE-89").exists()) + + def test_update_cwes_list_replaces_rows(self): + # A `cwes` list drives the rows: old rows replaced, first entry mirrored into the scalar + # primary (no explicit scalar in the request), read-back symmetric. + self._seed_cwe_rows(11, [22, 33]) + response = self.client.patch( + self.v3_url(f"findings/{self.finding.id}"), {"cwes": [79, 89]}, format="json", + ) + self.assertEqual(200, response.status_code, response.content[:500]) + self.finding.refresh_from_db() + self.assertEqual(79, self.finding.cwe) # first list entry mirrored (no scalar supplied) + self.assertEqual({"CWE-79", "CWE-89"}, self._cwe_rows()) # old rows replaced + self.assertEqual([79, 89], response.json()["cwes"]) # read-back symmetry + + def test_update_scalar_cwe_only_still_resyncs_to_primary(self): + # Existing behavior unchanged: a scalar-only `cwe` change resyncs the rows to just that + # primary, wiping any extra rows (documented §12 OS3b divergence). + self._seed_cwe_rows(11, [22, 33]) + response = self.client.patch( + self.v3_url(f"findings/{self.finding.id}"), {"cwe": 89}, format="json", + ) + self.assertEqual(200, response.status_code, response.content[:500]) + self.finding.refresh_from_db() + self.assertEqual(89, self.finding.cwe) + self.assertEqual({"CWE-89"}, self._cwe_rows()) # extras wiped by the scalar resync + + def test_update_cwes_empty_clears_extras_keeps_primary(self): + # An explicit empty list clears the extra rows and resyncs to just the scalar-derived primary + # (an explicit empty is a statement, unlike omission). + self._seed_cwe_rows(11, [22, 33]) + response = self.client.patch( + self.v3_url(f"findings/{self.finding.id}"), {"cwes": []}, format="json", + ) + self.assertEqual(200, response.status_code, response.content[:500]) + self.finding.refresh_from_db() + self.assertEqual(11, self.finding.cwe) # scalar untouched + self.assertEqual({"CWE-11"}, self._cwe_rows()) # extras cleared, primary retained + + def test_update_omitting_cwe_and_cwes_leaves_rows_untouched(self): + # Omitting BOTH `cwe` and `cwes` leaves the rows exactly as they were (no-reset-on-omit, + # symmetric with vulnerability_ids/reporter). + self._seed_cwe_rows(11, [22, 33]) + response = self.client.patch( + self.v3_url(f"findings/{self.finding.id}"), {"severity": "Low"}, format="json", + ) + self.assertEqual(200, response.status_code, response.content[:500]) + self.assertEqual({"CWE-11", "CWE-22", "CWE-33"}, self._cwe_rows()) + + def test_update_scalar_and_cwes_scalar_wins_primary(self): + # Both supplied on PATCH: the explicit scalar stays primary, the list drives the extra rows + # (list wins over the scalar-only resync -- save_cwes runs once). + self._seed_cwe_rows(11, []) + response = self.client.patch( + self.v3_url(f"findings/{self.finding.id}"), {"cwe": 79, "cwes": [89, 100]}, format="json", + ) + self.assertEqual(200, response.status_code, response.content[:500]) + self.finding.refresh_from_db() + self.assertEqual(79, self.finding.cwe) + self.assertEqual([79, 89, 100], response.json()["cwes"]) + self.assertEqual({"CWE-79", "CWE-89", "CWE-100"}, self._cwe_rows()) + + def test_update_duplicate_active_invariant_is_400(self): + self.finding.active = True + self.finding.save() + response = self.client.patch( + self.v3_url(f"findings/{self.finding.id}"), {"duplicate": True}, format="json", + ) + self.assertEqual(400, response.status_code) + + def test_update_risk_accept_disabled_is_400(self): + product = self.finding.test.engagement.product + product.enable_simple_risk_acceptance = False + product.save() + response = self.client.patch( + self.v3_url(f"findings/{self.finding.id}"), + {"risk_accepted": True, "active": False}, format="json", + ) + self.assertEqual(400, response.status_code) + + def test_update_active_and_risk_accepted_is_400(self): + product = self.finding.test.engagement.product + product.enable_simple_risk_acceptance = True + product.save() + response = self.client.patch( + self.v3_url(f"findings/{self.finding.id}"), + {"risk_accepted": True, "active": True}, format="json", + ) + self.assertEqual(400, response.status_code) + + @override_settings(EDITABLE_MITIGATED_DATA=False) + def test_update_mitigated_blocked_when_disabled(self): + response = self.client.patch( + self.v3_url(f"findings/{self.finding.id}"), + {"mitigated": "2024-01-01T00:00:00Z"}, format="json", + ) + self.assertEqual(400, response.status_code) + + @override_settings(EDITABLE_MITIGATED_DATA=True) + def test_update_mitigated_allowed_for_superuser_when_enabled(self): + # admin is a superuser; can_edit_mitigated_data requires EDITABLE_MITIGATED_DATA + superuser. + response = self.client.patch( + self.v3_url(f"findings/{self.finding.id}"), + {"mitigated": "2024-01-01T00:00:00Z"}, format="json", + ) + self.assertEqual(200, response.status_code, response.content[:500]) + + def test_update_keep_in_sync_pushes_synchronously(self): + with patch(_KEEP_IN_SYNC, return_value=True), patch(_PUSH, return_value=(True, "ok")) as push: + response = self.client.patch( + self.v3_url(f"findings/{self.finding.id}"), {"severity": "Medium"}, format="json", + ) + self.assertEqual(200, response.status_code, response.content[:500]) + push.assert_called_once() + self.assertTrue(push.call_args.kwargs.get("force_sync")) + + def test_update_jira_push_failure_is_400(self): + with patch(_KEEP_IN_SYNC, return_value=False), patch(_PUSH, return_value=(False, "jira exploded")): + response = self.client.patch( + self.v3_url(f"findings/{self.finding.id}"), + {"severity": "Medium", "push_to_jira": True}, format="json", + ) + self.assertEqual(400, response.status_code) + + +class TestApiV3FindingReplace(ApiV3TestCase): + + """PUT full-replace: validates FindingReplace, resets omitted optionals, flows through the service.""" + + def setUp(self): + super().setUp() + self.finding = Finding.objects.filter(risk_accepted=False, is_mitigated=False).first() + + def _put(self, **overrides): + payload = { + "title": "v3 put finding", + "severity": "High", + "description": "replaced via put", + "active": True, + "verified": False, + } + payload.update(overrides) + return self.client.put(self.v3_url(f"findings/{self.finding.id}"), payload, format="json") + + def test_put_full_replace_resets_omitted_optionals(self): + # Set an optional via PATCH, then PUT without it -> it resets to the schema default. + self.client.patch( + self.v3_url(f"findings/{self.finding.id}"), {"impact": "temporary impact"}, format="json", + ) + self.finding.refresh_from_db() + self.assertEqual("temporary impact", self.finding.impact) + response = self._put() + self.assertEqual(200, response.status_code, response.content[:500]) + self.finding.refresh_from_db() + self.assertIsNone(self.finding.impact) # omitted from PUT -> reset to default (None) + self.assertEqual("V3 Put Finding", self.finding.title) # title-cased on save + + def test_put_non_null_boolean_resets_to_model_default(self): + # A non-null status bool set via PATCH resets to its model default (False), not None (§12). + self.client.patch( + self.v3_url(f"findings/{self.finding.id}"), {"out_of_scope": True}, format="json", + ) + self.finding.refresh_from_db() + self.assertTrue(self.finding.out_of_scope) + self.assertEqual(200, self._put().status_code) + self.finding.refresh_from_db() + self.assertFalse(self.finding.out_of_scope) + + def _cwe_rows(self) -> set[str]: + return set(Finding_CWE.objects.filter(finding=self.finding).values_list("cwe", flat=True)) + + def test_put_with_cwes_persists_rows_primary_first(self): + # PUT accepts a `cwes` list: first entry mirrored into the scalar primary (no explicit scalar + # in the payload), rows persisted primary-first, read-back symmetric. + response = self._put(cwes=[89, 79]) + self.assertEqual(200, response.status_code, response.content[:500]) + self.finding.refresh_from_db() + self.assertEqual(89, self.finding.cwe) + self.assertEqual({"CWE-89", "CWE-79"}, self._cwe_rows()) + self.assertEqual([89, 79], response.json()["cwes"]) + + def test_put_omitting_cwes_follows_scalar_resync(self): + # Omitting `cwes` on PUT does not run the list-driven replacement (no-reset-on-omit); the rows + # follow the existing scalar-`cwe` resync. Here PUT supplies scalar cwe=55 and no list, so the + # rows become just {CWE-55} -- the seeded extras are gone via the scalar resync of the full + # replace, not because the omitted `cwes` cleared them (§12). + Finding_CWE.objects.filter(finding=self.finding).delete() + Finding_CWE.objects.create(finding=self.finding, cwe="CWE-11") + Finding_CWE.objects.create(finding=self.finding, cwe="CWE-22") + response = self._put(cwe=55) + self.assertEqual(200, response.status_code, response.content[:500]) + self.finding.refresh_from_db() + self.assertEqual(55, self.finding.cwe) + self.assertEqual({"CWE-55"}, self._cwe_rows()) + + def test_put_missing_required_is_400(self): + response = self.client.put( + self.v3_url(f"findings/{self.finding.id}"), + {"title": "x", "description": "y", "active": True, "verified": False}, # no severity + format="json", + ) + self.assertEqual(400, response.status_code) + self.assertEqual("application/problem+json", response["Content-Type"]) + + def test_put_unknown_field_is_400(self): + # `test` is not writable on replace (editable=False) -> rejected as unknown (extra=forbid). + response = self._put(test=self.finding.test_id) + self.assertEqual(400, response.status_code) + + def test_put_side_effects_flow_through_service_jira(self): + with patch(_KEEP_IN_SYNC, return_value=True), patch(_PUSH, return_value=(True, "ok")) as push: + response = self._put() + self.assertEqual(200, response.status_code, response.content[:500]) + push.assert_called_once() + self.assertTrue(push.call_args.kwargs.get("force_sync")) + + def test_put_unauthorized_is_404(self): + limited = User.objects.create_user(username="v3_put_limited", password="x") # noqa: S106 + client = self.token_client(user=limited) + response = client.put( + self.v3_url(f"findings/{self.finding.id}"), + {"title": "x", "severity": "High", "description": "y", "active": True, "verified": False}, + format="json", + ) + self.assertEqual(404, response.status_code) + + def test_put_visible_but_not_editable_is_403(self): + # OS legacy authz can't express view-but-not-edit; exercise the 403 code path by failing the + # edit permission check while the object stays visible to the admin (§12, OS5 pattern). + with patch("dojo.finding.api_v3.routes.user_has_permission", return_value=False): + response = self._put() + self.assertEqual(403, response.status_code, response.content[:300]) + + +class TestApiV3FindingDelete(ApiV3TestCase): + + def test_delete(self): + finding = Finding.objects.first() + response = self.client.delete(self.v3_url(f"findings/{finding.id}")) + self.assertEqual(204, response.status_code) + self.assertFalse(Finding.objects.filter(pk=finding.id).exists()) + + def test_delete_runs_jira_sync_with_v2_default(self): + # D17 regression pin (API_V3_DIVERGENCE_ANALYSIS.md): v3 delete must pass the v2 tri-state + # default push_to_jira=None so finding_delete's JIRA close/reassign runs. A bare + # finding.delete() hits the model's suppress-sentinel and silently skips it. + # Use a fresh standalone finding: mocking finding_delete skips its dedup FK reassignment, + # so deleting a fixture finding referenced via duplicate_finding would violate the FK. + finding = Finding.objects.first() + finding.pk = None + finding.title = "d17 regression pin" + finding.save() + with patch("dojo.finding.helper.finding_delete") as mock_finding_delete: + response = self.client.delete(self.v3_url(f"findings/{finding.id}")) + self.assertEqual(204, response.status_code) + mock_finding_delete.assert_called_once() + self.assertIsNone(mock_finding_delete.call_args.kwargs.get("push_to_jira", "MISSING")) + + +class TestApiV3FindingWriteRbac(ApiV3TestCase): + + def setUp(self): + super().setUp() + self.limited = User.objects.create_user(username="v3_fw_limited", password="x") # noqa: S106 + self.member = Dojo_User.objects.create_user(username="v3_fw_member", password="x") # noqa: S106 + self.finding = Finding.objects.first() + self.product = self.finding.test.engagement.product + self.product.authorized_users.add(self.member) + + def test_unauthorized_finding_update_is_404(self): + client = self.token_client(user=self.limited) + response = client.patch(self.v3_url(f"findings/{self.finding.id}"), {"severity": "Low"}, format="json") + self.assertEqual(404, response.status_code) + + def test_unauthorized_finding_delete_is_404(self): + client = self.token_client(user=self.limited) + response = client.delete(self.v3_url(f"findings/{self.finding.id}")) + self.assertEqual(404, response.status_code) + + def test_create_on_unauthorized_test_is_403(self): + client = self.token_client(user=self.limited) + response = client.post( + self.v3_url("findings"), _finding_payload(self.finding.test_id), format="json", + ) + self.assertEqual(403, response.status_code, response.content[:300]) + + def test_member_can_view_but_delete_is_403(self): + # OS legacy RBAC: membership grants view+edit; delete is staff-only, so the not-modifiable + # 403 is demonstrated on delete (Finding_Edit is not staff-gated in OS legacy -- §12). + client = self.token_client(user=self.member) + self.get_json(f"findings/{self.finding.id}", client=client) + response = client.delete(self.v3_url(f"findings/{self.finding.id}")) + self.assertEqual(403, response.status_code, response.content[:300]) diff --git a/unittests/api_v3/test_apiv3_findings.py b/unittests/api_v3/test_apiv3_findings.py new file mode 100644 index 00000000000..21051fd66e5 --- /dev/null +++ b/unittests/api_v3/test_apiv3_findings.py @@ -0,0 +1,427 @@ +"""Findings read-path contract tests for API v3 (§4.3-§4.9, OS1).""" +from __future__ import annotations + +from django.db import connection +from django.test import override_settings +from django.test.utils import CaptureQueriesContext + +from dojo.location.models import Location, LocationFindingReference +from dojo.models import Finding, Finding_CWE, User, Vulnerability_Id + +from .base import ApiV3TestCase + +_SLIM_KEYS = { + "id", "title", "severity", "active", "verified", "false_p", "duplicate", "risk_accepted", + "out_of_scope", "is_mitigated", "date", "cwe", "cwes", "vulnerability_ids", "test", "engagement", + "asset", "organization", "reporter", "locations_count", "tags", "created", "updated", +} + + +class TestApiV3FindingsSlim(ApiV3TestCase): + + def test_list_envelope_shape(self): + body = self.get_json("findings") + self.assertEqual({"count", "next", "previous", "results"}, set(body) - {"meta"}) + self.assertIsInstance(body["count"], int) + self.assertIsInstance(body["results"], list) + self.assertGreater(body["count"], 0) + + def test_slim_shape_and_denormalized_parent_refs(self): + row = self.get_json("findings")["results"][0] + self.assertEqual(_SLIM_KEYS, set(row)) + # Refs are closed {id, name}; parent chain is denormalized onto the finding. + for key in ("test", "engagement", "asset", "organization"): + self.assertEqual({"id", "name"}, set(row[key]), key) + self.assertIsInstance(row["locations_count"], int) + self.assertIsInstance(row["tags"], list) + + def test_datetime_is_iso_z(self): + row = self.get_json("findings")["results"][0] + if row["created"]: + self.assertTrue(row["created"].endswith("Z"), row["created"]) + + def test_detail_adds_heavy_fields(self): + row = self.get_json("findings")["results"][0] + detail = self.get_json(f"findings/{row['id']}") + for key in ("description", "mitigation", "impact", "file_path", "line", "mitigated", "mitigated_by"): + self.assertIn(key, detail) + + def test_detail_unknown_or_unauthorized_is_404(self): + self.get_json("findings/99999999", expected=404) + + +class TestApiV3FindingsExpand(ApiV3TestCase): + + def test_expand_test_engagement_inlines_slim(self): + row = self.get_json("findings", data={"expand": "test.engagement"})["results"][0] + # test ref swapped for the test slim shape (has a title-derived name + its own refs). + self.assertIn("test_type", row["test"]) + self.assertIn("engagement", row["test"]) + # engagement nested inside test is itself a slim (carries name, not just id). + self.assertIn("name", row["test"]["engagement"]) + self.assertIn("asset", row["test"]["engagement"]) + + def test_expand_reporter(self): + row = self.get_json("findings", data={"expand": "reporter"})["results"][0] + if row["reporter"] is not None: + self.assertIn("username", row["reporter"]) + + def test_expand_unknown_relation_is_400(self): + body = self.get_json("findings", data={"expand": "not_a_relation"}, expected=400) + self.assertEqual(400, body["status"]) + + def test_expand_budget_exceeded_is_400(self): + with override_settings(API_V3_EXPAND_BUDGET=1): + self.get_json("findings", data={"expand": "test.engagement,reporter"}, expected=400) + + +class TestApiV3FindingsFields(ApiV3TestCase): + + def test_fields_subsets_output(self): + row = self.get_json("findings", data={"fields": "id,title"})["results"][0] + self.assertEqual({"id", "title"}, set(row)) + + def test_fields_always_includes_id(self): + row = self.get_json("findings", data={"fields": "title"})["results"][0] + self.assertIn("id", row) + + def test_unknown_field_is_400(self): + self.get_json("findings", data={"fields": "id,not_a_field"}, expected=400) + + +class TestApiV3FindingsFieldsOptUp(ApiV3TestCase): + + """`?fields=` on the LIST endpoint may opt UP into the detail field set (§4.7 Part A).""" + + def test_fields_opts_up_into_detail_fields(self): + # impact + references are detail-only fields; requesting them returns exactly the projection. + row = self.get_json("findings", data={"fields": "id,title,impact,references"})["results"][0] + self.assertEqual({"id", "title", "impact", "references"}, set(row)) + + def test_detail_fields_absent_from_default_list(self): + # The default (no fields=) list is the slim shape exactly -- detail fields never appear. + row = self.get_json("findings")["results"][0] + self.assertEqual(_SLIM_KEYS, set(row)) + self.assertNotIn("impact", row) + self.assertNotIn("references", row) + + def test_opt_up_value_matches_detail_get(self): + finding = Finding.objects.first() + finding.impact = "opt-up-impact-value" + finding.save(update_fields=["impact"]) + row = next( + r for r in self.get_json("findings", data={"fields": "id,impact", "limit": 250})["results"] + if r["id"] == finding.id + ) + self.assertEqual("opt-up-impact-value", row["impact"]) + # Identical to what the detail GET (which serializes with FindingDetail) returns. + self.assertEqual(self.get_json(f"findings/{finding.id}")["impact"], row["impact"]) + + def test_detail_only_relation_ref_via_fields(self): + # mitigated_by is a detail-only relation rendered as a {id, name} ref (fixed join, Part A). + finding = Finding.objects.first() + finding.mitigated_by = self.admin + finding.save(update_fields=["mitigated_by"]) + row = next( + r for r in self.get_json("findings", data={"fields": "id,mitigated_by", "limit": 250})["results"] + if r["id"] == finding.id + ) + self.assertEqual({"id", "mitigated_by"}, set(row)) + self.assertEqual({"id", "name"}, set(row["mitigated_by"])) + self.assertEqual(self.admin.username, row["mitigated_by"]["name"]) + + def test_fields_mixing_slim_detail_and_expandable_keys(self): + # title (slim) + impact (detail) + locations (an EXPANDABLE key, not a model field). + body = self.get_json( + "findings", data={"fields": "id,title,impact,locations", "expand": "locations"}, + ) + row = body["results"][0] + self.assertEqual({"id", "title", "impact", "locations"}, set(row)) + self.assertIsInstance(row["locations"], list) + + def test_unknown_field_still_400_under_opt_up_allowlist(self): + self.get_json("findings", data={"fields": "id,not_a_field"}, expected=400) + + +class TestApiV3FindingsDefer(ApiV3TestCase): + + """Part B: the LIST row query defers the heavy detail columns that were not requested.""" + + def _main_row_sql(self, params: dict) -> str: + with CaptureQueriesContext(connection) as ctx: + response = self.client.get(self.v3_url("findings"), params) + self.assertEqual(200, response.status_code, response.content[:500]) + # The main row query is the one selecting the finding table's slim `title` column. + rows = [q["sql"] for q in ctx.captured_queries if '"dojo_finding"."title"' in q["sql"]] + self.assertEqual(1, len(rows), f"expected exactly one main finding row query, got {len(rows)}") + return rows[0] + + def test_default_list_does_not_select_deferred_columns(self): + sql = self._main_row_sql({"limit": 250}) + self.assertIn('"dojo_finding"."title"', sql) # a slim column is always selected + self.assertNotIn('"dojo_finding"."impact"', sql) # heavy detail columns are deferred + self.assertNotIn('"dojo_finding"."description"', sql) + self.assertNotIn('"dojo_finding"."mitigation"', sql) + + def test_requesting_impact_un_defers_impact(self): + sql = self._main_row_sql({"limit": 250, "fields": "id,impact"}) + self.assertIn('"dojo_finding"."impact"', sql) # requested -> selected + # Other heavy detail columns stay deferred ("un-defers exactly impact"). + self.assertNotIn('"dojo_finding"."mitigation"', sql) + self.assertNotIn('"dojo_finding"."description"', sql) + + def test_relation_slim_fields_are_never_deferred(self): + # vulnerability_ids/cwes are resolver-backed relations (not concrete own-model columns), so + # the defer planner -- which only defers detail-only concrete columns -- can never defer them + # (plan_list_fields uses model._meta.concrete_fields, which excludes them automatically). + from dojo.api_v3.expand import plan_list_fields # noqa: PLC0415 -- kernel unit assertion + from dojo.finding.api_v3.schemas import FindingDetail, FindingSlim # noqa: PLC0415 + plan = plan_list_fields(FindingSlim, FindingDetail, requested=None) + self.assertNotIn("vulnerability_ids", plan.defer) + self.assertNotIn("cwes", plan.defer) + # They are slim fields, so the default list always includes them (never deferred away). + row = self.get_json("findings")["results"][0] + self.assertIn("vulnerability_ids", row) + self.assertIn("cwes", row) + + +class TestApiV3FindingsFilters(ApiV3TestCase): + + def test_filter_severity(self): + body = self.get_json("findings", data={"severity": "High"}) + for row in body["results"]: + self.assertEqual("High", row["severity"]) + + def test_filter_active_boolean(self): + body = self.get_json("findings", data={"active": "true"}) + for row in body["results"]: + self.assertTrue(row["active"]) + + def test_ordering(self): + body = self.get_json("findings", data={"o": "-id"}) + ids = [r["id"] for r in body["results"]] + self.assertEqual(ids, sorted(ids, reverse=True)) + + def test_unknown_ordering_is_400(self): + self.get_json("findings", data={"o": "not_orderable"}, expected=400) + + def test_severity_ordering_is_by_rank_not_alphabetical(self): + rank = {"Critical": 0, "High": 1, "Medium": 2, "Low": 3, "Info": 4} + severities = [r["severity"] for r in self.get_json("findings", data={"o": "severity", "limit": 250})["results"]] + ranks = [rank[s] for s in severities] + # Rank-sorted ascending -> Critical first (§4.9), never alphabetical. + self.assertEqual(ranks, sorted(ranks), severities) + # Alphabetical would place "Info" before "Medium"; rank ordering must not. + if "Medium" in severities and "Info" in severities: + self.assertLess(severities.index("Medium"), severities.index("Info"), severities) + + def test_severity_ordering_descending_reverses_rank(self): + rank = {"Critical": 0, "High": 1, "Medium": 2, "Low": 3, "Info": 4} + severities = [r["severity"] for r in self.get_json("findings", data={"o": "-severity", "limit": 250})["results"]] + ranks = [rank[s] for s in severities] + self.assertEqual(ranks, sorted(ranks, reverse=True), severities) + + def test_q_free_text_search(self): + body = self.get_json("findings", data={"q": "DUMMY", "limit": 250}) + self.assertGreater(body["count"], 0) + for row in body["results"]: + self.assertIn("dummy", row["title"].lower()) + + def test_q_no_match_returns_empty(self): + body = self.get_json("findings", data={"q": "zzz_no_such_finding_zzz"}) + self.assertEqual(0, body["count"]) + + def test_unknown_filter_param_is_400(self): + self.get_json("findings", data={"not_a_real_filter": "x"}, expected=400) + + +class TestApiV3FindingsInclude(ApiV3TestCase): + + def test_include_counts(self): + body = self.get_json("findings", data={"include": "counts"}) + counts = body["meta"]["counts"] + for key in ("total", "active", "verified", "duplicate", "severity"): + self.assertIn(key, counts) + self.assertEqual(body["count"], counts["total"]) + self.assertEqual( + {"Critical", "High", "Medium", "Low", "Info"}, set(counts["severity"]), + ) + + def test_unknown_include_is_400(self): + self.get_json("findings", data={"include": "bogus"}, expected=400) + + +class TestApiV3FindingsPagination(ApiV3TestCase): + + def test_limit_and_next(self): + body = self.get_json("findings", data={"limit": 2}) + self.assertLessEqual(len(body["results"]), 2) + if body["count"] > 2: + self.assertIsNotNone(body["next"]) + self.assertIsNone(body["previous"]) + + def test_offset_previous(self): + body = self.get_json("findings", data={"limit": 2, "offset": 2}) + self.assertIsNotNone(body["previous"]) + + def test_cursor_mode_smoke(self): + # Cursor mode is implemented (see test_apiv3_cursor for the full contract): count/previous + # are null and the envelope shape is otherwise unchanged. + body = self.get_json("findings", data={"pagination": "cursor", "limit": 2}) + self.assertIsNone(body["count"]) + self.assertIsNone(body["previous"]) + self.assertLessEqual(len(body["results"]), 2) + + def test_bad_limit_400(self): + self.get_json("findings", data={"limit": "-1"}, expected=400) + + def test_count_exact_below_cap(self): + body = self.get_json("findings") + # Default CAP is large; count is exact and no count_exact flag appears. + self.assertNotIn("count_exact", body.get("meta", {})) + + @override_settings(API_V3_COUNT_CAP=1) + def test_count_switches_to_estimate_above_cap(self): + body = self.get_json("findings") + # Above the (lowered) cap: estimate clamped to >= CAP+1, flagged count_exact false. + self.assertGreaterEqual(body["count"], 2) + self.assertIn("count_exact", body["meta"]) + self.assertFalse(body["meta"]["count_exact"]) + + +class TestApiV3FindingsLocationsExpand(ApiV3TestCase): + + def _attach_location(self, finding: Finding, value: str, status: str = "Active") -> None: + location = Location.objects.create(location_type="url", location_value=value) + LocationFindingReference.objects.create(location=location, finding=finding, status=status) + + def test_expand_locations_swaps_count_for_edge_rows(self): + finding = Finding.objects.first() + existing = finding.locations.count() # fixture may already attach some + self._attach_location(finding, "https://example.com/os2-a") + self._attach_location(finding, "https://example.com/os2-b", status="Mitigated") + + detail = self.get_json(f"findings/{finding.id}", data={"expand": "locations"}) + # `locations_count` is swapped for the edge rows (§4.6). + self.assertNotIn("locations_count", detail) + self.assertIn("locations", detail) + rows = detail["locations"] + self.assertEqual(existing + 2, len(rows)) + for row in rows: + # OS4 added `auditor` to the expand=locations edge rows (§12). + self.assertEqual({"location", "status", "audit_time", "auditor"}, set(row)) + self.assertEqual({"id", "name", "type"}, set(row["location"])) + by_name = {row["location"]["name"]: row for row in rows} + self.assertEqual("url", by_name["https://example.com/os2-a"]["location"]["type"]) + self.assertEqual("Active", by_name["https://example.com/os2-a"]["status"]) + self.assertEqual("Mitigated", by_name["https://example.com/os2-b"]["status"]) + + def test_slim_without_expand_keeps_locations_count(self): + row = self.get_json("findings")["results"][0] + self.assertIn("locations_count", row) + self.assertNotIn("locations", row) + + def test_expand_into_locations_is_400(self): + self.get_json("findings", data={"expand": "locations.location"}, expected=400) + + def test_expand_locations_query_count_constant(self): + test = Finding.objects.first().test + params = {"limit": 250, "expand": "locations"} + + def query_count() -> int: + with CaptureQueriesContext(connection) as ctx: + response = self.client.get(self.v3_url("findings"), params) + self.assertEqual(200, response.status_code, response.content[:500]) + return len(ctx.captured_queries) + + Finding.objects.bulk_create([ + Finding(title=f"loc qcount {i}", severity="High", numerical_severity="S1", + description="x", test=test, reporter=self.admin, active=True, verified=False) + for i in range(10) + ]) + first = query_count() + Finding.objects.bulk_create([ + Finding(title=f"loc qcount b{i}", severity="High", numerical_severity="S1", + description="x", test=test, reporter=self.admin, active=True, verified=False) + for i in range(90) + ]) + second = query_count() + self.assertEqual(first, second, f"expand=locations query count grew: {first} -> {second}") + + +class TestApiV3FindingsVulnIdsAndCwes(ApiV3TestCase): + + """`vulnerability_ids` (flat strings) and `cwes` (flat ints) render on list AND detail (§4.5).""" + + def _make_finding_with_identity_rows(self) -> Finding: + test = Finding.objects.first().test + finding = Finding.objects.create( + title="vuln+cwe render", severity="High", numerical_severity="S1", description="x", + test=test, reporter=self.admin, active=True, verified=False, cwe=79, + ) + # Two vuln ids: the first is the one mirrored into `cve` (storage order is preserved). + Vulnerability_Id.objects.create(finding=finding, vulnerability_id="CVE-2020-1234") + Vulnerability_Id.objects.create(finding=finding, vulnerability_id="GHSA-aaaa-bbbb-cccc") + # Two CWE rows stored as canonical "CWE-" strings (the primary cwe=79 is one of them, + # mirroring what save_cwes persists). + Finding_CWE.objects.create(finding=finding, cwe="CWE-79") + Finding_CWE.objects.create(finding=finding, cwe="CWE-89") + return finding + + def test_list_renders_both_lists(self): + finding = self._make_finding_with_identity_rows() + row = next( + r for r in self.get_json("findings", data={"id__in": str(finding.id)})["results"] + if r["id"] == finding.id + ) + self.assertEqual(["CVE-2020-1234", "GHSA-aaaa-bbbb-cccc"], row["vulnerability_ids"]) + self.assertEqual([79, 89], row["cwes"]) + + def test_detail_renders_both_lists(self): + finding = self._make_finding_with_identity_rows() + detail = self.get_json(f"findings/{finding.id}") + self.assertEqual(["CVE-2020-1234", "GHSA-aaaa-bbbb-cccc"], detail["vulnerability_ids"]) + self.assertEqual([79, 89], detail["cwes"]) + + def test_empty_lists_not_null_when_none(self): + test = Finding.objects.first().test + finding = Finding.objects.create( + title="no vuln no cwe", severity="Low", numerical_severity="S3", description="x", + test=test, reporter=self.admin, active=True, verified=False, cwe=0, + ) + row = next( + r for r in self.get_json("findings", data={"id__in": str(finding.id)})["results"] + if r["id"] == finding.id + ) + # Empty lists (not null) when the finding has no vuln-id / CWE rows. + self.assertEqual([], row["vulnerability_ids"]) + self.assertEqual([], row["cwes"]) + detail = self.get_json(f"findings/{finding.id}") + self.assertEqual([], detail["vulnerability_ids"]) + self.assertEqual([], detail["cwes"]) + + def test_fields_accepts_the_new_slim_names(self): + # ?fields= naturally accepts the new slim field names (they are slim fields, not relations + # requiring opt-up); requesting exactly them projects to exactly them. + finding = self._make_finding_with_identity_rows() + row = next( + r for r in self.get_json( + "findings", data={"id__in": str(finding.id), "fields": "id,cwes,vulnerability_ids"}, + )["results"] + if r["id"] == finding.id + ) + self.assertEqual({"id", "cwes", "vulnerability_ids"}, set(row)) + self.assertEqual(["CVE-2020-1234", "GHSA-aaaa-bbbb-cccc"], row["vulnerability_ids"]) + self.assertEqual([79, 89], row["cwes"]) + + +class TestApiV3FindingsRbac(ApiV3TestCase): + + def test_include_counts_and_list_respect_authorized_queryset(self): + """A user with no product access sees an empty list and zeroed counts (RBAC via querysets).""" + limited = User.objects.create_user(username="v3_limited", password="x") # noqa: S106 + client = self.token_client(user=limited) + body = self.get_json("findings", client=client, data={"include": "counts"}) + self.assertEqual(0, body["count"]) + self.assertEqual([], body["results"]) + self.assertEqual(0, body["meta"]["counts"]["total"]) diff --git a/unittests/api_v3/test_apiv3_findings_queries.py b/unittests/api_v3/test_apiv3_findings_queries.py new file mode 100644 index 00000000000..ba22b487cfe --- /dev/null +++ b/unittests/api_v3/test_apiv3_findings_queries.py @@ -0,0 +1,133 @@ +""" +Query-count regression test for API v3 findings (§4.6, §6 OS1) -- the headline guarantee. + +The number of SQL queries for a findings list must be **independent of the number of rows +returned**, both with and without ``?expand=``. This is what v3's slim+ref+expand model buys over +v2's post-serialization ``?prefetch=`` (which issues per-row-per-field queries). +""" +from __future__ import annotations + +from django.db import connection +from django.test.utils import CaptureQueriesContext +from django.utils import timezone + +from dojo.models import Finding, Finding_CWE, Test, Vulnerability_Id + +from .base import ApiV3TestCase + + +class TestApiV3FindingsQueryCount(ApiV3TestCase): + + def _bulk_create_findings(self, count: int, test: Test) -> None: + today = timezone.now().date() + Finding.objects.bulk_create([ + Finding( + title=f"qcount finding {i}", + severity="High", + numerical_severity="S1", + description="query-count fixture finding", + test=test, + reporter=self.admin, + active=True, + verified=False, + date=today, + ) + for i in range(count) + ]) + + def _query_count(self, params: dict) -> int: + with CaptureQueriesContext(connection) as ctx: + response = self.client.get(self.v3_url("findings"), params) + self.assertEqual(200, response.status_code, response.content[:500]) + return len(ctx.captured_queries) + + def test_query_count_is_independent_of_row_count(self): + test = Test.objects.first() + + self._bulk_create_findings(10, test) + queries_10 = self._query_count({"limit": 250}) + queries_10_expand = self._query_count({"limit": 250, "expand": "test.engagement"}) + + self._bulk_create_findings(90, test) # now 100+ of our findings plus the fixture rows + queries_100 = self._query_count({"limit": 250}) + queries_100_expand = self._query_count({"limit": 250, "expand": "test.engagement"}) + + # Confirm we actually returned ~100+ rows so the assertion is meaningful. + self.assertGreaterEqual(self.get_json("findings", data={"limit": 250})["count"], 100) + + self.assertEqual( + queries_10, queries_100, + f"query count must not grow with row count (no expand): {queries_10} vs {queries_100}", + ) + self.assertEqual( + queries_10_expand, queries_100_expand, + f"query count must not grow with row count (expand): {queries_10_expand} vs {queries_100_expand}", + ) + + def test_query_count_constant_with_beyond_slim_fields(self): + """ + `?fields=` opting up into detail fields stays row-count-independent (§4.7 Part A/B). + + Includes ``mitigated_by`` (a detail-only relation ref) to prove its extra ``select_related`` + is a fixed join, not a per-row query. + """ + test = Test.objects.first() + params = {"limit": 250, "fields": "id,title,impact,references,mitigated_by"} + + self._bulk_create_findings(10, test) + queries_10 = self._query_count(params) + + self._bulk_create_findings(90, test) + queries_100 = self._query_count(params) + + self.assertGreaterEqual(self.get_json("findings", data={"limit": 250})["count"], 100) + self.assertEqual( + queries_10, queries_100, + f"opt-up query count must not grow with row count: {queries_10} vs {queries_100}", + ) + + def test_query_count_constant_with_vuln_ids_and_cwes(self): + """ + The vulnerability_id_set / finding_cwe_set prefetches are fixed IN-batches, not per-row. + + Attaching a vuln-id + CWE row to every finding would surface a per-row query (COUNT/SELECT + per finding) if these fields were resolved lazily; the count must stay row-independent. + """ + test = Test.objects.first() + + def _attach_identity_rows() -> None: + for finding in Finding.objects.all(): + Vulnerability_Id.objects.get_or_create( + finding=finding, vulnerability_id=f"CVE-2020-{finding.id}", + ) + Finding_CWE.objects.get_or_create(finding=finding, cwe="CWE-79") + + self._bulk_create_findings(10, test) + _attach_identity_rows() + queries_10 = self._query_count({"limit": 250}) + + self._bulk_create_findings(90, test) + _attach_identity_rows() + queries_100 = self._query_count({"limit": 250}) + + self.assertGreaterEqual(self.get_json("findings", data={"limit": 250})["count"], 100) + self.assertEqual( + queries_10, queries_100, + f"vuln-id/cwe prefetch query count must not grow with row count: {queries_10} vs {queries_100}", + ) + + def test_default_list_query_count_unchanged_by_defer(self): + """ + Deferring the heavy detail columns changes the SELECT column list, never the query count. + + The default list is still the same constant number of queries with defer active (Part B). + """ + test = Test.objects.first() + self._bulk_create_findings(50, test) + default_queries = self._query_count({"limit": 250}) + # Requesting a detail column un-defers it but issues no additional query (row-column, not per-row). + opt_up_queries = self._query_count({"limit": 250, "fields": "id,title,impact"}) + self.assertEqual( + default_queries, opt_up_queries, + f"opting up into a detail column must not add queries: {default_queries} vs {opt_up_queries}", + ) diff --git a/unittests/api_v3/test_apiv3_import.py b/unittests/api_v3/test_apiv3_import.py new file mode 100644 index 00000000000..42e2904eb64 --- /dev/null +++ b/unittests/api_v3/test_apiv3_import.py @@ -0,0 +1,119 @@ +""" +Import equivalence tests for API v3 (§4.13, §6 OS1). + +The consolidated ``POST /import`` (import/reimport/auto) must reproduce the v2 endpoints' DB state +for identical payloads, including ``close_old_findings``. Both paths run in the shared test +transaction so DB-state assertions are exact. +""" +from __future__ import annotations + +from collections import Counter + +from django.urls import reverse + +from dojo.models import Finding, Test + +from .base import ApiV3TestCase + +_ZAP = "ZAP Scan" + + +def _finding_multiset(test_id: int) -> Counter: + return Counter( + (f.title, f.severity, f.active, f.is_mitigated) + for f in Finding.objects.filter(test_id=test_id) + ) + + +class TestApiV3Import(ApiV3TestCase): + + def _scan(self, name: str): + from unittests.dojo_test_case import get_unit_tests_scans_path # noqa: PLC0415 + + return (get_unit_tests_scans_path("zap") / name).open(encoding="utf-8") + + def _v2_import(self, engagement: int, name: str = "0_zap_sample.xml", **extra) -> dict: + with self._scan(name) as scan: + payload = {"scan_type": _ZAP, "engagement": engagement, "file": scan, + "active": "true", "verified": "true", **extra} + response = self.client.post(reverse("importscan-list"), payload) + self.assertEqual(201, response.status_code, response.content[:1000]) + return response.json() + + def _v2_reimport(self, test_id: int, name: str, **extra) -> dict: + with self._scan(name) as scan: + payload = {"scan_type": _ZAP, "test": test_id, "file": scan, + "active": "true", "verified": "true", **extra} + response = self.client.post(reverse("reimportscan-list"), payload) + self.assertEqual(201, response.status_code, response.content[:1000]) + return response.json() + + def _v3_import(self, name: str = "0_zap_sample.xml", *, mode: str = "import", expected: int = 200, **extra) -> dict: + with self._scan(name) as scan: + payload = {"scan_type": _ZAP, "mode": mode, "file": scan, + "active": "true", "verified": "true", **extra} + response = self.client.post(self.v3_url("import"), payload, format="multipart") + self.assertEqual(expected, response.status_code, response.content[:1000]) + return response.json() + + # --- import equivalence ----------------------------------------------------------------- + def test_import_creates_same_findings_as_v2(self): + v2 = self._v2_import(engagement=1) + v3 = self._v3_import(mode="import", engagement=4) + v2_test = v2.get("test_id") or v2.get("test") + v3_test = v3["test"]["id"] + self.assertEqual(_finding_multiset(v2_test), _finding_multiset(v3_test)) + self.assertGreater(sum(_finding_multiset(v3_test).values()), 0) + + def test_import_response_shape(self): + v3 = self._v3_import(mode="import", engagement=4) + self.assertEqual("import", v3["mode_resolved"]) + self.assertEqual({"id", "name"}, set(v3["test"])) + self.assertEqual({"new", "reactivated", "closed", "untouched"}, set(v3["statistics"])) + self.assertIn("close_old_findings", v3) + # New import: statistics.new equals the number of findings created. + self.assertEqual(v3["statistics"]["new"], Finding.objects.filter(test_id=v3["test"]["id"]).count()) + + # --- reimport equivalence incl. close_old_findings -------------------------------------- + def test_reimport_close_old_findings_equivalence(self): + # v2: import full sample, then reimport a subset with close_old_findings -> some closed. + v2 = self._v2_import(engagement=1) + v2_test = v2.get("test_id") or v2.get("test") + self._v2_reimport(v2_test, "1_zap_sample_0_and_new_absent.xml", close_old_findings="true") + + # v3: same sequence via the consolidated endpoint. + v3 = self._v3_import(mode="import", engagement=4) + v3_test = v3["test"]["id"] + v3_re = self._v3_import("1_zap_sample_0_and_new_absent.xml", mode="reimport", test=v3_test, + close_old_findings="true") + + self.assertEqual("reimport", v3_re["mode_resolved"]) + # Final DB state matches v2 exactly (active + mitigated multisets). + self.assertEqual(_finding_multiset(v2_test), _finding_multiset(v3_test)) + # And the reimport reported the closures. + self.assertGreaterEqual(v3_re["statistics"]["closed"], 0) + self.assertTrue(v3_re["close_old_findings"]) + + def test_reimport_default_close_old_findings_is_true(self): + v3 = self._v3_import(mode="import", engagement=4) + v3_re = self._v3_import("0_zap_sample.xml", mode="reimport", test=v3["test"]["id"]) + # ReImport default for close_old_findings is True (mirrors v2), echoed in the response. + self.assertTrue(v3_re["close_old_findings"]) + + # --- auto mode -------------------------------------------------------------------------- + def test_auto_mode_creates_then_reuses(self): + created = self._v3_import( + mode="auto", asset_name="v3 Auto Product", engagement_name="v3 Auto Eng", + organization_name="v3 Auto PT", auto_create_context="true", + ) + self.assertEqual("import", created["mode_resolved"]) + first_test = created["test"]["id"] + self.assertTrue(Test.objects.filter(pk=first_test).exists()) + + # Auto again with the same identifiers resolves the existing test -> reimport. + reused = self._v3_import( + mode="auto", asset_name="v3 Auto Product", engagement_name="v3 Auto Eng", + organization_name="v3 Auto PT", auto_create_context="true", + ) + self.assertEqual("reimport", reused["mode_resolved"]) + self.assertEqual(first_test, reused["test"]["id"]) diff --git a/unittests/api_v3/test_apiv3_import_corpus.py b/unittests/api_v3/test_apiv3_import_corpus.py new file mode 100644 index 00000000000..726903ef6d9 --- /dev/null +++ b/unittests/api_v3/test_apiv3_import_corpus.py @@ -0,0 +1,51 @@ +""" +API v3 import/reimport corpus (ported from ``unittests/test_import_reimport.py``, §10 backlog #1). + +Strategy (architect-recorded dual-endpoint adapter): the v2 ``ImportReimportTestAPI`` scenarios and +assertions run **unchanged** against the consolidated ``POST /api/v3-alpha/import`` endpoint by +mixing in :class:`ApiV3ImportShim`, which overrides only the two endpoint helper methods +(``import_scan_with_params`` / ``reimport_scan_with_params``). Everything else -- the finding-list +DB assertions, the endpoint→location count redirect, ``block_execution`` for synchronous +dedupe, the ``@versioned_fixtures`` locations fixture -- is inherited from the v2 class. Both APIs +therefore prove the same import DB state. + +The v2 base class is referenced via the ``_v2corpus`` module attribute (never bound as a module-level +name) so Django's ``test*.py`` discovery does not also collect the v2 class inside this v3 package. + +Skipped scenarios (honest, enumerated in the port report): + * the two ``*_statistics`` tests -- they assert the v2 before/after per-severity ``statistics`` + envelope, which v3 deliberately does not emit (v3 returns the delta shape + ``{new, reactivated, closed, untouched}``; §4.13). + * the two ``*_additional_endpoint`` tests -- they use ``endpoint_to_add`` (legacy Endpoint + wiring), which is out of v3 scope (§4.13). +""" +from __future__ import annotations + +from unittest import skip + +import unittests.test_import_reimport as _v2corpus + +from .import_corpus_shim import ApiV3ImportShim + + +class ApiV3ImportReimportCorpus(ApiV3ImportShim, _v2corpus.ImportReimportTestAPI): + + """The v2 ImportReimportTestAPI corpus (mixin scenarios + API-only scenarios) bound to v3.""" + + # --- v2-only response shape: v3 emits the delta statistics, not the before/after envelope --- + @skip("v3 returns delta statistics {new,reactivated,closed,untouched}, not the v2 before/after envelope (§4.13).") + def test_import_0_reimport_1_active_verified_reimport_0_active_verified_statistics(self): + ... + + @skip("v3 returns delta statistics {new,reactivated,closed,untouched}, not the v2 before/after envelope (§4.13).") + def test_import_0_reimport_1_active_verified_reimport_0_active_verified_statistics_no_history(self): + ... + + # --- legacy Endpoint param (endpoint_to_add) is out of v3 scope (§4.13) --- + @skip("endpoint_to_add / legacy Endpoint wiring is out of v3 scope (§4.13).") + def test_import_param_close_old_findings_with_additional_endpoint(self): + ... + + @skip("endpoint_to_add / legacy Endpoint wiring is out of v3 scope (§4.13).") + def test_import_param_close_old_findings_default_with_additional_endpoint(self): + ... diff --git a/unittests/api_v3/test_apiv3_import_options_corpus.py b/unittests/api_v3/test_apiv3_import_options_corpus.py new file mode 100644 index 00000000000..f39026996eb --- /dev/null +++ b/unittests/api_v3/test_apiv3_import_options_corpus.py @@ -0,0 +1,182 @@ +""" +API v3 port of the remaining import corpus siblings (§10 backlog #1): + + * ``unittests/test_apiv2_scan_import_options.py`` (``ScanImportOptionsTest``) -- empty/full ZAP + upload scenarios. Ported by subclassing the v2 class and overriding ONLY its ``import_zap_scan`` + helper to hit ``POST /api/v3-alpha/import``; the scenarios + assertions run unchanged. + * ``unittests/test_importers_closeold.py`` (``TestDojoCloseOld``) -- these are importer *unit* + tests that call ``DefaultImporter`` directly (no HTTP endpoint), so they cannot be adapted by a + helper override. The close-old behaviours that are observable through the consolidated endpoint + are re-expressed here as endpoint-level tests, which additionally exercise the new + ``close_old_findings_product_scope`` ImportForm field end-to-end. Two originals are intentionally + NOT ported (the query-column-optimization spy is importer-internal, not endpoint-observable; + enumerated in the port report). + +The v2 base class is referenced via a module attribute so Django's ``test*.py`` discovery does not +also collect it inside this v3 package. +""" +from __future__ import annotations + +from pathlib import Path + +from django.core.files.uploadedfile import SimpleUploadedFile +from django.utils import timezone + +import unittests.test_apiv2_scan_import_options as _v2opts +from dojo.models import Engagement, Product, Product_Type, Test +from unittests.dojo_test_case import get_unit_tests_scans_path + +from .base import ApiV3TestCase +from .import_corpus_shim import ApiV3ImportShim + + +class ApiV3ScanImportOptions(ApiV3ImportShim, _v2opts.ScanImportOptionsTest): + + """``ScanImportOptionsTest`` scenarios bound to ``POST /import`` (only the helper is overridden).""" + + def import_zap_scan(self, *, upload_empty_scan=False): + with Path("tests/zap_sample.xml").open(encoding="utf-8") as file: + if upload_empty_scan: + tested_file = SimpleUploadedFile("zap_sample.xml", self.EMPTY_ZAP_SCAN.encode("utf-8")) + else: + tested_file = file + self.payload = { + "engagement": 1, + "scan_type": "ZAP Scan", + "mode": "import", + "file": tested_file, + } + test_ids = list(_v2opts.Test.objects.values_list("id", flat=True)) + r = self.client.post(self.v3_url("import"), self.payload, format="multipart") + self.assertEqual(200, r.status_code, r.content[:1000]) + return _v2opts.Test.objects.exclude(id__in=test_ids).get() + + +class ApiV3CloseOldFindingsEndpoint(ApiV3ImportShim, ApiV3TestCase): + + """ + Endpoint-level ports of the close-old importer behaviours (incl. product-scope). + + Assertions read the v3 delta statistics (``new`` == importer new count, ``closed`` == importer + closed count), which are the exact tuple positions the v2 unit tests assert on. + """ + + ACUNETIX = "Acunetix Scan" + SEMGREP = "Semgrep JSON Report" + + def _make_product(self, pt_name: str, product_name: str) -> Product: + product_type, _ = Product_Type.objects.get_or_create(name=pt_name) + product, _ = Product.objects.get_or_create(name=product_name, description="Test", prod_type=product_type) + return product + + def _make_engagement(self, name: str, product: Product) -> Engagement: + engagement, _ = Engagement.objects.get_or_create( + name=name, product=product, target_start=timezone.now(), target_end=timezone.now(), + ) + return engagement + + def _import(self, engagement_id: int, scan_path, *, scan_type: str, cof: bool, cofps: bool = False) -> dict: + payload = { + "scan_type": scan_type, + "mode": "import", + "minimum_severity": "Info", + "engagement": engagement_id, + "active": "true", + "verified": "false", + "close_old_findings": "true" if cof else "false", + } + if cofps: + payload["close_old_findings_product_scope"] = "true" + return self._post_v3_import(payload, scan_path, expected=200) + + def test_close_old_findings_engagement_scope(self): + """Port of ``test_close_old_same_engagement`` (close_old_findings, engagement scope).""" + product = self._make_product("closeold", "TestDojoCloseOldImporter1") + engagement = self._make_engagement("Close Old Same Engagement", product) + many = get_unit_tests_scans_path("acunetix") / "many_findings.xml" + one = get_unit_tests_scans_path("acunetix") / "one_finding.xml" + + r1 = self._import(engagement.id, many, scan_type=self.ACUNETIX, cof=False) + self.assertEqual(4, r1["statistics"]["new"]) + self.assertEqual(0, r1["statistics"]["closed"]) + + r2 = self._import(engagement.id, many, scan_type=self.ACUNETIX, cof=True) + self.assertEqual(4, r2["statistics"]["new"]) + self.assertEqual(0, r2["statistics"]["closed"]) + + r3 = self._import(engagement.id, one, scan_type=self.ACUNETIX, cof=True) + self.assertEqual(1, r3["statistics"]["new"]) + self.assertEqual(8, r3["statistics"]["closed"]) + + def test_close_old_findings_product_scope(self): + """Port of ``test_close_old_same_product_scan`` (exercises close_old_findings_product_scope).""" + product = self._make_product("test2", "TestDojoCloseOldImporter2") + eng1 = self._make_engagement("Close Old Same Product 1", product) + eng2 = self._make_engagement("Close Old Same Product 2", product) + eng3 = self._make_engagement("Close Old Same Product 3", product) + many = get_unit_tests_scans_path("acunetix") / "many_findings.xml" + one = get_unit_tests_scans_path("acunetix") / "one_finding.xml" + + r1 = self._import(eng1.id, many, scan_type=self.ACUNETIX, cof=False, cofps=True) + self.assertEqual(4, r1["statistics"]["new"]) + self.assertEqual(0, r1["statistics"]["closed"]) + + r2 = self._import(eng2.id, many, scan_type=self.ACUNETIX, cof=True, cofps=True) + self.assertEqual(4, r2["statistics"]["new"]) + self.assertEqual(0, r2["statistics"]["closed"]) + + r3 = self._import(eng3.id, one, scan_type=self.ACUNETIX, cof=True, cofps=True) + self.assertEqual(1, r3["statistics"]["new"]) + self.assertEqual(8, r3["statistics"]["closed"]) + + def test_close_old_findings_product_scope_matching_unique_id(self): + """Port of ``test_close_old_same_product_scan_matching_with_unique_id_from_tool``.""" + product = self._make_product("test2", "TestDojoCloseOldImporter3") + eng1 = self._make_engagement("Close Old Same Product 1", product) + eng2 = self._make_engagement("Close Old Same Product 2", product) + eng3 = self._make_engagement("Close Old Same Product 3", product) + semgrep = get_unit_tests_scans_path("semgrep") + + r1 = self._import(eng1.id, semgrep / "close_old_findings_report_line31.json", + scan_type=self.SEMGREP, cof=False, cofps=True) + self.assertEqual(1, r1["statistics"]["new"]) + self.assertEqual(0, r1["statistics"]["closed"]) + + r2 = self._import(eng2.id, semgrep / "close_old_findings_report_second_run_line24.json", + scan_type=self.SEMGREP, cof=True, cofps=True) + self.assertEqual(1, r2["statistics"]["new"]) + self.assertEqual(0, r2["statistics"]["closed"]) + + r3 = self._import(eng3.id, semgrep / "close_old_findings_report_third_run_different_unique_id.json", + scan_type=self.SEMGREP, cof=True, cofps=True) + self.assertEqual(1, r3["statistics"]["new"]) + self.assertEqual(1, r3["statistics"]["closed"]) + + def test_close_old_closes_risk_accepted_findings(self): + """Port of ``test_close_old_closes_risk_accepted_findings`` (close_old removes risk acceptance).""" + import dojo.risk_acceptance.helper as ra_helper # noqa: PLC0415 + + product = self._make_product("closeold_risk", "TestCloseOldRiskAccepted") + product.enable_simple_risk_acceptance = True + product.save() + engagement = self._make_engagement("Close Old Risk Accepted", product) + many = get_unit_tests_scans_path("acunetix") / "many_findings.xml" + one = get_unit_tests_scans_path("acunetix") / "one_finding.xml" + + r1 = self._import(engagement.id, many, scan_type=self.ACUNETIX, cof=False) + self.assertEqual(4, r1["statistics"]["new"]) + self.assertEqual(0, r1["statistics"]["closed"]) + + finding_to_accept = Test.objects.get(id=r1["test"]).finding_set.first() + ra_helper.simple_risk_accept(self.admin, finding_to_accept) + finding_to_accept.refresh_from_db() + self.assertTrue(finding_to_accept.risk_accepted) + self.assertFalse(finding_to_accept.active) + + r2 = self._import(engagement.id, one, scan_type=self.ACUNETIX, cof=True) + self.assertEqual(1, r2["statistics"]["new"]) + self.assertGreaterEqual(r2["statistics"]["closed"], 3) + + finding_to_accept.refresh_from_db() + self.assertTrue(finding_to_accept.is_mitigated, "Risk-accepted finding should be mitigated when fixed") + self.assertFalse(finding_to_accept.risk_accepted, "Risk acceptance should be removed when fixed") diff --git a/unittests/api_v3/test_apiv3_jira_push.py b/unittests/api_v3/test_apiv3_jira_push.py new file mode 100644 index 00000000000..3d86d5cf53a --- /dev/null +++ b/unittests/api_v3/test_apiv3_jira_push.py @@ -0,0 +1,208 @@ +""" +JIRA push flow tests for API v3 (§10 backlog #2 -- JIRA push intent ported from +``unittests/test_jira_import_and_pushing_api.py``). + +The v2 corpus is a VCR/cassette suite (``DojoVCRAPITestCase``) that records *real* JIRA HTTP +traffic and is deeply entangled with v2-only surfaces the v3 alpha deliberately lacks -- finding +**groups** (``group_by``), **epics**, **webhooks**, and the UI ``/finding/bulk`` flow. A dual-endpoint +shim over it is not economical (most scenarios pin those absent surfaces, and the cassettes are keyed +to the v2 request sequence). So per the §10 directive these are targeted v3 ports of the *key flows*, +mocking JIRA at the same layer the existing v3 tests do: + +* import path -> ``dojo.jira.services.push`` (what ``default_importer`` and + ``finding.helper.post_process_findings_batch`` both call -- see the coverage map in the debrief); +* finding write path -> ``dojo.finding.services.jira_services.push`` (as in + ``test_apiv3_finding_writes.py``). + +Never contacts a real JIRA. ``block_execution=True`` forces the importer's async post-processing to +run in-process so the push is observable within the request (mirrors the v2 corpus ``setUp``). + +Coverage map (corpus scenario -> here / skip+reason) is in the module-level docstring of the debrief +and API_V3_PLAN.md §12. +""" +from __future__ import annotations + +from unittest.mock import patch + +from dojo.models import Finding, JIRA_Instance, JIRA_Project, UserContactInfo +from unittests.dojo_test_case import get_unit_tests_scans_path + +from .base import ApiV3TestCase + +# Import path: default_importer (grouped) and finding.helper post-processing (ungrouped) both call +# ``jira_services.push`` == this attribute; patching it catches every import-time push. +_IMPORT_PUSH = "dojo.jira.services.push" +# Finding write path: the service reference the v3 finding-write tests already patch. +_SVC_PUSH = "dojo.finding.services.jira_services.push" +_SVC_KEEP_IN_SYNC = "dojo.finding.services.jira_services.is_keep_in_sync" + +_ZAP = "ZAP Scan" +_SCAN = "0_zap_sample.xml" + + +class _JiraV3Base(ApiV3TestCase): + + def setUp(self): + super().setUp() + self.system_settings(enable_jira=True) + # Force async post-processing (jira push, grading, dedup) to run in-process so the push is + # observable within the import request -- mirrors the v2 corpus setUp. + UserContactInfo.objects.update_or_create(user=self.admin, defaults={"block_execution": True}) + + def _scan_file(self): + return (get_unit_tests_scans_path("zap") / _SCAN).open(encoding="utf-8") + + def _import(self, *, engagement: int = 1, expected: int = 200, **extra) -> dict: + with self._scan_file() as scan: + payload = {"scan_type": _ZAP, "mode": "import", "engagement": engagement, "file": scan, + "active": "true", "verified": "true", **extra} + response = self.client.post(self.v3_url("import"), payload, format="multipart") + self.assertEqual(expected, response.status_code, response.content[:1000]) + return response.json() + + def _reimport(self, test_id: int, *, expected: int = 200, **extra) -> dict: + with self._scan_file() as scan: + payload = {"scan_type": _ZAP, "mode": "reimport", "test": test_id, "file": scan, + "active": "true", "verified": "true", **extra} + response = self.client.post(self.v3_url("import"), payload, format="multipart") + self.assertEqual(expected, response.status_code, response.content[:1000]) + return response.json() + + @staticmethod + def _pushed_finding_ids(mock_push) -> set[int]: + return {call.args[0].id for call in mock_push.call_args_list if isinstance(call.args[0], Finding)} + + @staticmethod + def _test_finding_ids(test_id: int) -> set[int]: + return set(Finding.objects.filter(test_id=test_id).values_list("id", flat=True)) + + +class TestApiV3ImportJiraPush(_JiraV3Base): + + """Import/reimport push behaviour (corpus: ``test_import_*`` non-group scenarios).""" + + def setUp(self): + super().setUp() + # Keep every parsed finding (no dedup collapse) so "each finding is pushed" is exact. + self.system_settings(enable_deduplication=False) + + def test_import_with_push_to_jira_pushes_each_finding(self): + # Corpus: test_import_with_push_to_jira. Engagement 1 -> product 2 has an enabled JIRA project. + with patch(_IMPORT_PUSH, return_value=(True, "ok")) as push: + body = self._import(push_to_jira="true") + test_id = body["test"]["id"] + self.assertTrue(push.called, "push_to_jira=true must push") + # Each finding in the new test is pushed exactly to the JIRA layer (set-equality is robust to + # any double-dispatch and to call ordering). + self.assertEqual(self._test_finding_ids(test_id), self._pushed_finding_ids(push)) + self.assertGreater(len(self._test_finding_ids(test_id)), 0) + + def test_import_without_push_to_jira_does_not_push(self): + # Corpus: test_import_no_push_to_jira / test_import_with_push_to_jira_is_false. + with patch(_IMPORT_PUSH, return_value=(True, "ok")) as push: + self._import() # push_to_jira omitted -> defaults False + push.assert_not_called() + + def test_import_push_to_jira_false_does_not_push(self): + with patch(_IMPORT_PUSH, return_value=(True, "ok")) as push: + self._import(push_to_jira="false") + push.assert_not_called() + + def test_import_push_all_issues_forces_push(self): + # Corpus: test_import_no_push_to_jira_but_push_all. push_all_issues on the JIRA project forces + # the push even though the request omits push_to_jira (the importer OR-s it via is_keep_in_sync). + JIRA_Project.objects.filter(product__engagement__id=1).update(push_all_issues=True) + with patch(_IMPORT_PUSH, return_value=(True, "ok")) as push: + body = self._import() # no push_to_jira + test_id = body["test"]["id"] + self.assertTrue(push.called, "push_all_issues must force pushing on import") + self.assertEqual(self._test_finding_ids(test_id), self._pushed_finding_ids(push)) + + def test_reimport_with_push_to_jira_pushes(self): + # Corpus: test_import_no_push_to_jira_reimport_with_push_to_jira. + with patch(_IMPORT_PUSH, return_value=(True, "ok")) as push: + body = self._import() # import without push + push.assert_not_called() + test_id = body["test"]["id"] + with patch(_IMPORT_PUSH, return_value=(True, "ok")) as push: + self._reimport(test_id, push_to_jira="true") + self.assertTrue(push.called, "reimport push_to_jira=true must push") + + def test_reimport_without_push_to_jira_does_not_push(self): + # Corpus: test_import_no_push_to_jira_reimport_no_push_to_jira. + with patch(_IMPORT_PUSH, return_value=(True, "ok")): + body = self._import() + test_id = body["test"]["id"] + with patch(_IMPORT_PUSH, return_value=(True, "ok")) as push: + self._reimport(test_id, push_to_jira="false") + push.assert_not_called() + + def test_import_jira_push_failure_does_not_fail_import(self): + # Divergence from finding PATCH/PUT (which surface push failures as 400): the import path is + # fire-and-forget for JIRA -- a failed push is logged, not raised, so the import still returns + # 200 (mirrors v2, where the importer does not propagate JIRA errors). Documented in §12. + with patch(_IMPORT_PUSH, return_value=(False, "jira exploded")) as push: + body = self._import(push_to_jira="true", expected=200) + self.assertTrue(push.called) + self.assertIn("test", body) + + +class TestApiV3FindingWriteJiraProjectSetting(_JiraV3Base): + + """ + Finding PATCH/PUT push semantics vs the JIRA project setting -- the route OR-s the request's + ``push_to_jira`` with ``jira_project.push_all_issues`` (routes.py). The keep-in-sync / + push-failure-as-400 / explicit-push cases already live in ``test_apiv3_finding_writes.py``; + this class adds ONLY the ``push_all_issues`` OR-ing gap. + """ + + def setUp(self): + super().setUp() + # A finding whose product has an enabled JIRA project so ``jira_services.get_project`` in the + # route resolves it (engagement 1 -> product 2 -> project pk=2, product-level). + self.finding = Finding.objects.filter(test__engagement__id=1, risk_accepted=False).first() + self.assertIsNotNone(self.finding, "fixture must have a finding under engagement 1") + project = JIRA_Project.objects.filter(product=self.finding.test.engagement.product).first() + if project is None: + project = JIRA_Project.objects.create( + product=self.finding.test.engagement.product, + jira_instance=JIRA_Instance.objects.first(), project_key="TEST", + ) + self.project = project + + def test_patch_push_all_issues_forces_push(self): + self.project.push_all_issues = True + self.project.enabled = True + self.project.save() + with patch(_SVC_KEEP_IN_SYNC, return_value=False), patch(_SVC_PUSH, return_value=(True, "ok")) as push: + response = self.client.patch( + self.v3_url(f"findings/{self.finding.id}"), {"severity": "Low"}, format="json", + ) + self.assertEqual(200, response.status_code, response.content[:500]) + push.assert_called_once() # push_all_issues OR-ed in by the route despite no push_to_jira + self.assertTrue(push.call_args.kwargs.get("force_sync")) + + def test_patch_no_push_when_project_not_push_all(self): + self.project.push_all_issues = False + self.project.enabled = True + self.project.save() + with patch(_SVC_KEEP_IN_SYNC, return_value=False), patch(_SVC_PUSH, return_value=(True, "ok")) as push: + response = self.client.patch( + self.v3_url(f"findings/{self.finding.id}"), {"severity": "Low"}, format="json", + ) + self.assertEqual(200, response.status_code, response.content[:500]) + push.assert_not_called() + + def test_put_push_all_issues_forces_push(self): + self.project.push_all_issues = True + self.project.enabled = True + self.project.save() + payload = { + "title": self.finding.title, "severity": "High", "description": "put replace", + "active": True, "verified": False, + } + with patch(_SVC_KEEP_IN_SYNC, return_value=False), patch(_SVC_PUSH, return_value=(True, "ok")) as push: + response = self.client.put(self.v3_url(f"findings/{self.finding.id}"), payload, format="json") + self.assertEqual(200, response.status_code, response.content[:500]) + push.assert_called_once() + self.assertTrue(push.call_args.kwargs.get("force_sync")) diff --git a/unittests/api_v3/test_apiv3_locations.py b/unittests/api_v3/test_apiv3_locations.py new file mode 100644 index 00000000000..a2475cbcd5e --- /dev/null +++ b/unittests/api_v3/test_apiv3_locations.py @@ -0,0 +1,324 @@ +""" +Locations resource + finding/asset location sub-resources for API v3 (§4.14, OS4). + +Covers: the read-only ``/locations`` resource (slim/detail shapes incl. URL-subtype fields, filters, +orderings, pagination, the v2 superuser-only RBAC mirror, constant query count); the +``/findings/{id}/locations`` and ``/assets/{id}/locations`` edge sub-resources (edge shapes, +auditor ref, parent-inherited authorization 404, pagination, constant query count); the +``?fields=`` / ``?expand=`` interplay; and flag-off behaviour (whole /api/v3-alpha/ tree absent). +""" +from __future__ import annotations + +import importlib + +from django.db import connection +from django.test import override_settings +from django.test.utils import CaptureQueriesContext +from django.urls import Resolver404, clear_url_caches, resolve +from django.utils import timezone + +import dojo.urls +from dojo.location.models import Location, LocationFindingReference, LocationProductReference +from dojo.models import Finding, Product, User + +from .base import ApiV3TestCase + +_SLIM_KEYS = {"id", "name", "type", "tags"} +_DETAIL_KEYS = _SLIM_KEYS | {"protocol", "host", "port", "path", "query", "fragment"} +_FINDING_EDGE_KEYS = {"location", "status", "audit_time", "auditor"} +_ASSET_EDGE_KEYS = {"location", "status"} +_LOCATION_REF_KEYS = {"id", "name", "type"} + + +class TestApiV3LocationsRead(ApiV3TestCase): + + """The read-only /locations resource (admin is a superuser, so RBAC never hides rows here).""" + + def test_list_envelope_and_slim_shape(self): + body = self.get_json("locations") + self.assertEqual({"count", "next", "previous", "results"}, set(body) - {"meta"}) + self.assertGreater(body["count"], 0) + row = body["results"][0] + self.assertEqual(_SLIM_KEYS, set(row)) + self.assertIsInstance(row["tags"], list) + self.assertEqual("url", row["type"]) + + def test_detail_adds_url_subtype_fields(self): + # Fixture Location 1 is http://127.0.0.1/endpoint/420/edit/ (URL subtype pk 1). + detail = self.get_json("locations/1") + self.assertEqual(_DETAIL_KEYS, set(detail)) + self.assertEqual("url", detail["type"]) + self.assertEqual("http://127.0.0.1/endpoint/420/edit/", detail["name"]) + self.assertEqual("http", detail["protocol"]) + self.assertEqual("127.0.0.1", detail["host"]) + self.assertEqual(80, detail["port"]) + self.assertEqual("endpoint/420/edit/", detail["path"]) + self.assertEqual("", detail["query"]) + self.assertEqual("", detail["fragment"]) + + def test_detail_unknown_is_404_problem(self): + response = self.client.get(self.v3_url("locations/99999999")) + self.assertEqual(404, response.status_code) + self.assertEqual("application/problem+json", response["Content-Type"]) + + def test_filter_type(self): + body = self.get_json("locations", data={"type": "url", "limit": 250}) + self.assertGreater(body["count"], 0) + for row in body["results"]: + self.assertEqual("url", row["type"]) + # No non-url locations exist in alpha; a bogus type returns nothing. + self.assertEqual(0, self.get_json("locations", data={"type": "code"})["count"]) + + def test_filter_name_icontains(self): + body = self.get_json("locations", data={"name__icontains": "bar.foo", "limit": 250}) + self.assertGreater(body["count"], 0) + for row in body["results"]: + self.assertIn("bar.foo", row["name"]) + + def test_filter_asset(self): + # Fixture: LocationProductReference links asset (product) 1 -> location 6. + body = self.get_json("locations", data={"asset": 1, "limit": 250}) + ids = {row["id"] for row in body["results"]} + self.assertIn(6, ids) + + def test_ordering_by_name(self): + names = [r["name"] for r in self.get_json("locations", data={"o": "name", "limit": 250})["results"]] + self.assertEqual(names, sorted(names)) + + def test_ordering_by_id_desc(self): + ids = [r["id"] for r in self.get_json("locations", data={"o": "-id", "limit": 250})["results"]] + self.assertEqual(ids, sorted(ids, reverse=True)) + + def test_pagination_envelope(self): + body = self.get_json("locations", data={"limit": 2, "offset": 2}) + self.assertLessEqual(len(body["results"]), 2) + self.assertIsNotNone(body["previous"]) + + def test_unknown_filter_param_is_400(self): + self.get_json("locations", data={"not_a_filter": "x"}, expected=400) + + def test_unknown_field_is_400(self): + self.get_json("locations", data={"fields": "id,not_a_field"}, expected=400) + + +class TestApiV3LocationsRbac(ApiV3TestCase): + + """/locations mirrors v2 LocationViewSet exactly: superuser-only (IsSuperUser) -> 403 otherwise.""" + + def setUp(self): + super().setUp() + self.regular = User.objects.create_user(username="v3_loc_regular", password="x") # noqa: S106 + + def test_non_superuser_list_is_403(self): + client = self.token_client(user=self.regular) + response = client.get(self.v3_url("locations")) + self.assertEqual(403, response.status_code, response.content[:300]) + self.assertEqual("application/problem+json", response["Content-Type"]) + + def test_non_superuser_detail_is_403(self): + client = self.token_client(user=self.regular) + response = client.get(self.v3_url("locations/1")) + self.assertEqual(403, response.status_code, response.content[:300]) + + def test_superuser_can_read(self): + self.assertGreater(self.get_json("locations")["count"], 0) + self.get_json("locations/1") + + +class TestApiV3LocationsQueryCount(ApiV3TestCase): + + def _query_count(self, params: dict) -> int: + with CaptureQueriesContext(connection) as ctx: + response = self.client.get(self.v3_url("locations"), params) + self.assertEqual(200, response.status_code, response.content[:500]) + return len(ctx.captured_queries) + + def test_query_count_is_independent_of_row_count(self): + Location.objects.bulk_create([ + Location(location_type="url", location_value=f"https://qcount.example/{i}") for i in range(10) + ]) + first = self._query_count({"limit": 250}) + Location.objects.bulk_create([ + Location(location_type="url", location_value=f"https://qcount.example/b{i}") for i in range(90) + ]) + second = self._query_count({"limit": 250}) + self.assertEqual(first, second, f"query count grew with row count: {first} -> {second}") + + +class TestApiV3FindingLocationsSubResource(ApiV3TestCase): + + """GET /findings/{id}/locations edge rows: location ref + status/audit_time/auditor (§4.14).""" + + def _attach(self, finding, value, status="Active", auditor=None, audit_time=None): + location = Location.objects.create(location_type="url", location_value=value) + return LocationFindingReference.objects.create( + location=location, finding=finding, status=status, auditor=auditor, audit_time=audit_time, + ) + + def test_edge_shape_and_location_ref(self): + # Fixture finding 227 already has three location edges (5/6/7). + body = self.get_json("findings/227/locations") + self.assertEqual({"count", "next", "previous", "results"}, set(body) - {"meta"}) + self.assertEqual(3, body["count"]) + for row in body["results"]: + self.assertEqual(_FINDING_EDGE_KEYS, set(row)) + self.assertEqual(_LOCATION_REF_KEYS, set(row["location"])) + self.assertEqual("url", row["location"]["type"]) + self.assertIsNone(row["auditor"]) # fixture edges carry no auditor + + def test_status_values_pass_through(self): + finding = Finding.objects.get(pk=228) # fixture: one FalsePositive edge on location 5 + rows = self.get_json(f"findings/{finding.id}/locations")["results"] + self.assertEqual(["FalsePositive"], [r["status"] for r in rows]) + + def test_auditor_ref_rendered(self): + finding = Finding.objects.get(pk=2) + when = timezone.now() + self._attach(finding, "https://audited.example/a", auditor=self.admin, audit_time=when) + rows = self.get_json(f"findings/{finding.id}/locations")["results"] + audited = [r for r in rows if r["auditor"] is not None] + self.assertEqual(1, len(audited)) + self.assertEqual({"id", "name"}, set(audited[0]["auditor"])) + self.assertEqual(self.admin.id, audited[0]["auditor"]["id"]) + self.assertEqual(self.admin.username, audited[0]["auditor"]["name"]) + self.assertIsNotNone(audited[0]["audit_time"]) + + def test_unknown_parent_is_404(self): + response = self.client.get(self.v3_url("findings/99999999/locations")) + self.assertEqual(404, response.status_code) + self.assertEqual("application/problem+json", response["Content-Type"]) + + def test_unauthorized_parent_is_404(self): + # A user with no product access cannot see finding 227 -> parent-inherited 404 (not 403). + limited = User.objects.create_user(username="v3_loc_findlimited", password="x") # noqa: S106 + client = self.token_client(user=limited) + response = client.get(self.v3_url("findings/227/locations")) + self.assertEqual(404, response.status_code, response.content[:300]) + + def test_pagination_envelope(self): + finding = Finding.objects.get(pk=2) + for i in range(5): + self._attach(finding, f"https://page.example/{i}") + body = self.get_json(f"findings/{finding.id}/locations", data={"limit": 2, "offset": 2}) + self.assertLessEqual(len(body["results"]), 2) + self.assertIsNotNone(body["previous"]) + + def test_query_count_is_independent_of_edge_count(self): + finding = Finding.objects.get(pk=2) + + def query_count() -> int: + with CaptureQueriesContext(connection) as ctx: + response = self.client.get(self.v3_url(f"findings/{finding.id}/locations"), {"limit": 250}) + self.assertEqual(200, response.status_code, response.content[:500]) + return len(ctx.captured_queries) + + for i in range(5): + self._attach(finding, f"https://qc.example/{i}", auditor=self.admin) + first = query_count() + for i in range(20): + self._attach(finding, f"https://qc.example/b{i}", auditor=self.admin) + second = query_count() + self.assertEqual(first, second, f"sub-resource query count grew: {first} -> {second}") + + +class TestApiV3AssetLocationsSubResource(ApiV3TestCase): + + """GET /assets/{id}/locations edge rows: location ref + status only (no audit columns, §12).""" + + def _attach(self, product, value, status="Active"): + location = Location.objects.create(location_type="url", location_value=value) + return LocationProductReference.objects.create(location=location, product=product, status=status) + + def test_edge_shape_and_location_ref(self): + # Fixture: asset (product) 1 -> location 6 (Active). + body = self.get_json("assets/1/locations") + self.assertEqual({"count", "next", "previous", "results"}, set(body) - {"meta"}) + self.assertGreaterEqual(body["count"], 1) + row = next(r for r in body["results"] if r["location"]["id"] == 6) + self.assertEqual(_ASSET_EDGE_KEYS, set(row)) + self.assertEqual(_LOCATION_REF_KEYS, set(row["location"])) + self.assertEqual("url", row["location"]["type"]) + self.assertEqual("Active", row["status"]) + + def test_unknown_parent_is_404(self): + response = self.client.get(self.v3_url("assets/99999999/locations")) + self.assertEqual(404, response.status_code) + self.assertEqual("application/problem+json", response["Content-Type"]) + + def test_unauthorized_parent_is_404(self): + limited = User.objects.create_user(username="v3_loc_assetlimited", password="x") # noqa: S106 + client = self.token_client(user=limited) + response = client.get(self.v3_url("assets/1/locations")) + self.assertEqual(404, response.status_code, response.content[:300]) + + def test_query_count_is_independent_of_edge_count(self): + product = Product.objects.get(pk=1) + + def query_count() -> int: + with CaptureQueriesContext(connection) as ctx: + response = self.client.get(self.v3_url(f"assets/{product.id}/locations"), {"limit": 250}) + self.assertEqual(200, response.status_code, response.content[:500]) + return len(ctx.captured_queries) + + for i in range(5): + self._attach(product, f"https://pqc.example/{i}") + first = query_count() + for i in range(20): + self._attach(product, f"https://pqc.example/b{i}") + second = query_count() + self.assertEqual(first, second, f"sub-resource query count grew: {first} -> {second}") + + +class TestApiV3FieldsExpandInterplay(ApiV3TestCase): + + """The ?fields= allowlist = schema fields + registered EXPANDABLE keys (OS4 item 4, §12).""" + + def test_expand_key_accepted_in_fields(self): + # `locations` is an expandable key, not a model field; it must be nameable in ?fields=. + finding = Finding.objects.get(pk=227) + body = self.get_json( + "findings", data={"expand": "locations", "fields": "id,title,locations", "id__in": finding.id}, + ) + row = next(r for r in body["results"] if r["id"] == finding.id) + self.assertEqual({"id", "title", "locations"}, set(row)) + self.assertNotIn("locations_count", row) + self.assertGreater(len(row["locations"]), 0) + + def test_fields_without_expand_drops_expand_key_silently(self): + # Naming an expand key in ?fields= without ?expand= is accepted (no 400) but renders nothing. + row = self.get_json("findings", data={"fields": "id,title,locations"})["results"][0] + self.assertEqual({"id", "title"}, set(row)) + + def test_unknown_field_still_400(self): + self.get_json("findings", data={"fields": "id,definitely_not_a_field"}, expected=400) + + def test_unknown_field_alongside_expand_key_still_400(self): + self.get_json( + "findings", data={"expand": "locations", "fields": "id,locations,bogus_field"}, expected=400, + ) + + +class TestApiV3LocationsFlagOff(ApiV3TestCase): + + """With V3_FEATURE_LOCATIONS=False the entire /api/v3-alpha/ tree is unmounted (D5/§4.1).""" + + def test_flag_off_unmounts_entire_v3_tree(self): + # Flag is on in the test settings: the whole v3 tree resolves. + resolve(self.v3_url("locations")) + resolve(self.v3_url("findings")) + + try: + with override_settings(V3_FEATURE_LOCATIONS=False): + clear_url_caches() + importlib.reload(dojo.urls) + # Nothing under the v3 prefix resolves once the flag is off -- the mount is gone. + for path in ("locations", "findings", "assets", "import"): + with self.assertRaises(Resolver404): + resolve(self.v3_url(path), urlconf=dojo.urls) + finally: + # Restore the real (flag-on) URLconf for the rest of the suite. + clear_url_caches() + importlib.reload(dojo.urls) + + # Restored: the v3 tree resolves again. + resolve(self.v3_url("locations")) diff --git a/unittests/api_v3/test_apiv3_openapi.py b/unittests/api_v3/test_apiv3_openapi.py new file mode 100644 index 00000000000..69fab9e2105 --- /dev/null +++ b/unittests/api_v3/test_apiv3_openapi.py @@ -0,0 +1,78 @@ +""" +OpenAPI schema-generation guard for API v3 (§4.1, §6 OS2). + +A CI-facing unit test asserting ``api_v3.get_openapi_schema()`` renders and contains the expected +paths, components, per-resource tags and the alpha banner. This guards against a schema-generation +regression (a broken schema silently breaks client codegen and the interactive docs). +""" +from __future__ import annotations + +from unittest import skipUnless + +from django.conf import settings +from django.test import SimpleTestCase + +from dojo.api_v3.api import api_v3 + + +@skipUnless( + settings.V3_FEATURE_LOCATIONS, + "api_v3.get_openapi_schema() resolves the mounted namespace, which does not exist when " + "V3_FEATURE_LOCATIONS is off (D5); the CI unit-test matrix runs a flag-off leg.", +) +class TestApiV3OpenApi(SimpleTestCase): + + @classmethod + def setUpClass(cls): + super().setUpClass() + cls.schema = api_v3.get_openapi_schema() + + def test_info_version_and_alpha_banner(self): + self.assertEqual(settings.API_V3_VERSION, self.schema["info"]["version"]) + self.assertIn("alpha", self.schema["info"]["description"].lower()) + + def test_expected_paths_present(self): + # Paths carry the mount prefix (/api/v3-alpha/...); assert on the operation suffix. + # Per D11 the wire paths are /organizations and /assets (not /product_types, /products). + paths = set(self.schema["paths"]) + for suffix in ( + "/findings", "/findings/{finding_id}", "/import", + "/organizations", "/organizations/{organization_id}", + "/assets", "/assets/{asset_id}", "/assets/{asset_id}/locations", + ): + self.assertTrue( + any(p.endswith(suffix) for p in paths), + f"missing a path ending in {suffix}; have {sorted(paths)}", + ) + # D11 all-or-nothing guard: no legacy product/product_type token in any wire path segment. + offenders = [p for p in paths if "product" in p.lower()] + self.assertEqual([], offenders, f"legacy product token in wire paths: {offenders}") + + def test_expected_components_present(self): + # Per D11 the schema classes are Asset*/Organization* (not Product*/ProductType*). + components = set(self.schema["components"]["schemas"]) + for expected in ("Ref", "FindingSlim", "FindingDetail", "AssetSlim", "OrganizationSlim"): + self.assertIn(expected, components, f"missing component {expected}") + # D11 guard: no legacy product token in component *schema names*. (The scalar model-column + # properties critical_product/key_product are the deliberate documented residual -- D11 + # excludes DB columns -- so component *property* names are intentionally not checked; §12.) + name_offenders = [c for c in components if "product" in c.lower()] + self.assertEqual([], name_offenders, f"legacy product token in component names: {name_offenders}") + + def test_ref_component_is_closed_shape(self): + ref = self.schema["components"]["schemas"]["Ref"] + self.assertEqual({"id", "name"}, set(ref["properties"])) + + def test_per_resource_tags(self): + tags = { + tag + for path in self.schema["paths"].values() + for operation in path.values() + for tag in operation.get("tags", []) + } + self.assertLessEqual( + {"findings", "import", "organizations", "assets"}, tags, f"tags found: {sorted(tags)}", + ) + # D11 guard: no legacy product/product_type token in any OpenAPI tag. + tag_offenders = [t for t in tags if "product" in t.lower()] + self.assertEqual([], tag_offenders, f"legacy product token in tags: {tag_offenders}") diff --git a/unittests/api_v3/test_apiv3_organizations.py b/unittests/api_v3/test_apiv3_organizations.py new file mode 100644 index 00000000000..b1a9b3394a6 --- /dev/null +++ b/unittests/api_v3/test_apiv3_organizations.py @@ -0,0 +1,211 @@ +"""Organization CRUD + RBAC + contract tests for API v3 (OS3a; D11 wire rename product_type -> organization).""" +from __future__ import annotations + +from unittest import mock + +from django.db import connection +from django.test.utils import CaptureQueriesContext + +from dojo.models import Dojo_User, Product_Type, User + +from .base import ApiV3TestCase + +_SLIM_KEYS = {"id", "name", "description", "critical_product", "key_product", "created", "updated"} + + +class TestApiV3OrganizationsRead(ApiV3TestCase): + + def test_list_envelope_and_slim_shape(self): + body = self.get_json("organizations") + self.assertEqual({"count", "next", "previous", "results"}, set(body) - {"meta"}) + self.assertGreater(body["count"], 0) + self.assertEqual(_SLIM_KEYS, set(body["results"][0])) + + def test_detail_shape(self): + pt = Product_Type.objects.first() + detail = self.get_json(f"organizations/{pt.id}") + self.assertEqual(pt.id, detail["id"]) + self.assertEqual(pt.name, detail["name"]) + + def test_detail_unknown_is_404_problem(self): + response = self.client.get(self.v3_url("organizations/99999999")) + self.assertEqual(404, response.status_code) + self.assertEqual("application/problem+json", response["Content-Type"]) + + def test_fields_projection(self): + row = self.get_json("organizations", data={"fields": "id,name"})["results"][0] + self.assertEqual({"id", "name"}, set(row)) + + def test_unknown_field_is_400(self): + self.get_json("organizations", data={"fields": "id,nope"}, expected=400) + + +class TestApiV3OrganizationsFilters(ApiV3TestCase): + + def test_filter_name_icontains(self): + pt = Product_Type.objects.first() + body = self.get_json("organizations", data={"name__icontains": pt.name[:4]}) + self.assertGreater(body["count"], 0) + + def test_ordering_by_name(self): + Product_Type.objects.create(name="ZZZ v3 last type") + Product_Type.objects.create(name="AAA v3 first type") + names = [r["name"] for r in self.get_json("organizations", data={"o": "name", "limit": 250})["results"]] + self.assertEqual(names, sorted(names)) + + def test_unknown_filter_param_is_400(self): + self.get_json("organizations", data={"not_a_filter": "x"}, expected=400) + + def test_unknown_ordering_is_400(self): + self.get_json("organizations", data={"o": "nope"}, expected=400) + + +class TestApiV3OrganizationsPagination(ApiV3TestCase): + + def test_limit_and_next(self): + for i in range(4): + Product_Type.objects.create(name=f"v3 page pt {i}") + body = self.get_json("organizations", data={"limit": 2}) + self.assertLessEqual(len(body["results"]), 2) + self.assertIsNotNone(body["next"]) + self.assertIsNone(body["previous"]) + + +class TestApiV3OrganizationsQueryCount(ApiV3TestCase): + + def _query_count(self, params: dict) -> int: + with CaptureQueriesContext(connection) as ctx: + response = self.client.get(self.v3_url("organizations"), params) + self.assertEqual(200, response.status_code, response.content[:500]) + return len(ctx.captured_queries) + + def test_query_count_is_independent_of_row_count(self): + Product_Type.objects.bulk_create([Product_Type(name=f"qcount pt {i}") for i in range(10)]) + first = self._query_count({"limit": 250}) + Product_Type.objects.bulk_create([Product_Type(name=f"qcount pt b{i}") for i in range(90)]) + second = self._query_count({"limit": 250}) + self.assertEqual(first, second, f"query count grew with rows: {first} -> {second}") + + +class TestApiV3OrganizationsWrite(ApiV3TestCase): + + def test_create_happy_path(self): + response = self.client.post( + self.v3_url("organizations"), + {"name": "v3 created type", "description": "made by v3", "critical_product": True}, + format="json", + ) + self.assertEqual(201, response.status_code, response.content[:500]) + body = response.json() + self.assertEqual("v3 created type", body["name"]) + self.assertTrue(body["critical_product"]) + self.assertTrue(Product_Type.objects.filter(name="v3 created type").exists()) + + def test_create_missing_required_name_is_400(self): + response = self.client.post(self.v3_url("organizations"), {"description": "no name"}, format="json") + self.assertEqual(400, response.status_code) + self.assertEqual("application/problem+json", response["Content-Type"]) + + def test_create_unknown_field_is_400(self): + response = self.client.post( + self.v3_url("organizations"), {"name": "x", "bogus_field": 1}, format="json", + ) + self.assertEqual(400, response.status_code) + + def test_patch_partial_update(self): + pt = Product_Type.objects.create(name="v3 patch me", description="old") + response = self.client.patch( + self.v3_url(f"organizations/{pt.id}"), {"description": "new"}, format="json", + ) + self.assertEqual(200, response.status_code, response.content[:500]) + self.assertEqual("new", response.json()["description"]) + pt.refresh_from_db() + self.assertEqual("new", pt.description) + self.assertEqual("v3 patch me", pt.name) # untouched + + def test_delete(self): + pt = Product_Type.objects.create(name="v3 delete me") + response = self.client.delete(self.v3_url(f"organizations/{pt.id}")) + self.assertEqual(204, response.status_code) + self.assertFalse(Product_Type.objects.filter(pk=pt.id).exists()) + + +class TestApiV3OrganizationsReplace(ApiV3TestCase): + + """PUT full-replace: reuses the create-shaped OrganizationWrite; omitted optionals reset.""" + + def test_put_full_replace_resets_omitted_optionals(self): + pt = Product_Type.objects.create(name="v3 put org", description="old", critical_product=True) + # PUT without description / critical_product -> both reset to their schema defaults. + response = self.client.put( + self.v3_url(f"organizations/{pt.id}"), {"name": "v3 put org renamed"}, format="json", + ) + self.assertEqual(200, response.status_code, response.content[:500]) + body = response.json() + self.assertEqual("v3 put org renamed", body["name"]) + self.assertIsNone(body["description"]) # reset to default (None) + self.assertFalse(body["critical_product"]) # reset to default (False) + pt.refresh_from_db() + self.assertIsNone(pt.description) + self.assertFalse(pt.critical_product) + + def test_put_missing_required_name_is_400(self): + pt = Product_Type.objects.create(name="v3 put org missing") + response = self.client.put( + self.v3_url(f"organizations/{pt.id}"), {"description": "no name"}, format="json", + ) + self.assertEqual(400, response.status_code) + self.assertEqual("application/problem+json", response["Content-Type"]) + + def test_put_unknown_field_is_400(self): + pt = Product_Type.objects.create(name="v3 put org unknown") + response = self.client.put( + self.v3_url(f"organizations/{pt.id}"), {"name": "x", "bogus": 1}, format="json", + ) + self.assertEqual(400, response.status_code) + + def test_put_unauthorized_is_404(self): + limited = User.objects.create_user(username="v3_org_put_limited", password="x") # noqa: S106 + pt = Product_Type.objects.first() + client = self.token_client(user=limited) + response = client.put(self.v3_url(f"organizations/{pt.id}"), {"name": "x"}, format="json") + self.assertEqual(404, response.status_code) + + def test_put_visible_but_not_editable_is_403(self): + # OS legacy authz can't express view-but-not-edit; fail the edit check while the object stays + # visible to the admin (§12, OS5 pattern). + pt = Product_Type.objects.first() + with mock.patch("dojo.product_type.api_v3.routes.user_has_permission", return_value=False): + response = self.client.put(self.v3_url(f"organizations/{pt.id}"), {"name": "x"}, format="json") + self.assertEqual(403, response.status_code, response.content[:300]) + + +class TestApiV3OrganizationsRbac(ApiV3TestCase): + + def setUp(self): + super().setUp() + self.limited = User.objects.create_user(username="v3_org_limited", password="x") # noqa: S106 + # authorized_users M2M targets Dojo_User (the proxy), so the member must be a Dojo_User. + self.member = Dojo_User.objects.create_user(username="v3_org_member", password="x") # noqa: S106 + self.pt = Product_Type.objects.first() + self.pt.authorized_users.add(self.member) + + def test_unauthorized_read_is_404(self): + client = self.token_client(user=self.limited) + # Limited user has no organization membership -> empty list, detail 404. + self.assertEqual(0, self.get_json("organizations", client=client)["count"]) + self.get_json(f"organizations/{self.pt.id}", client=client, expected=404) + + def test_create_without_global_add_is_403(self): + client = self.token_client(user=self.limited) + response = client.post(self.v3_url("organizations"), {"name": "v3 nope"}, format="json") + self.assertEqual(403, response.status_code, response.content[:300]) + self.assertEqual("application/problem+json", response["Content-Type"]) + + def test_member_can_view_but_delete_is_403(self): + client = self.token_client(user=self.member) + # Member can view (200) ... + self.get_json(f"organizations/{self.pt.id}", client=client) + # ... but delete is staff-only for non-staff members (legacy model) -> 403 (not 404). + response = client.delete(self.v3_url(f"organizations/{self.pt.id}")) + self.assertEqual(403, response.status_code, response.content[:300]) diff --git a/unittests/api_v3/test_apiv3_query_report.py b/unittests/api_v3/test_apiv3_query_report.py new file mode 100644 index 00000000000..c73b6154fe8 --- /dev/null +++ b/unittests/api_v3/test_apiv3_query_report.py @@ -0,0 +1,184 @@ +""" +Whole-surface query sweep for API v3 (§7 / OS6 verification). + +Two guarantees on every mounted v3 GET endpoint: + +1. **No N+1 signature** — no normalized query shape repeats >= ``_THRESHOLD`` times within one + request (the per-list ``assertNumQueries`` tests pin totals; this test identifies the shape + when something regresses, across the whole surface at once). +2. **Completeness** — every GET path in the OpenAPI schema must have a representative request + below. Adding a resource without extending the sweep fails the test, so OS4/OS5+ endpoints + are covered by construction. + +The full capture is always written to ``/tmp/apiv3_query_report.md`` for review. +""" +from __future__ import annotations + +from pathlib import Path + +from django.conf import settings +from django.core.files.base import ContentFile + +from dojo.api_v3.api import api_v3 +from dojo.file_uploads.models import FileUpload +from dojo.location.models import Location, LocationFindingReference, LocationProductReference +from dojo.models import Engagement, Finding, Product, Product_Type, Test +from dojo.notes.models import Notes + +from .base import ApiV3TestCase +from .query_report import capture_request, format_report + +_THRESHOLD = 4 +_REPORT_PATH = Path("/tmp/apiv3_query_report.md") +# POST-only paths (no GET operation) are exempt from the completeness check. +_ROW_FANOUT = 15 # enough rows that any per-row query trips the threshold unmistakably + + +class TestApiV3QueryReport(ApiV3TestCase): + + """Sweep every mounted v3 GET route; flag N+1 shapes; enforce sweep completeness.""" + + def setUp(self): + super().setUp() # base gives self.client = admin token client: RBAC never hides rows here + self._fan_out_rows() + + def _fan_out_rows(self): + """Clone fixture rows so per-row queries repeat >= threshold and cannot hide.""" + finding = Finding.objects.first() + for i in range(_ROW_FANOUT): + clone = Finding.objects.get(pk=finding.pk) + clone.pk = None + clone.title = f"query-report fanout {i}" + clone.save() + prod_type = Product_Type.objects.first() + for i in range(_ROW_FANOUT): + Product_Type.objects.create(name=f"query-report pt {i}") + for i in range(_ROW_FANOUT): + Product.objects.create( + name=f"query-report product {i}", + description="query report fanout", + prod_type=prod_type, + ) + for model in (Engagement, Test): + template = model.objects.first() + if template is None: + continue + for _ in range(_ROW_FANOUT): + clone = model.objects.get(pk=template.pk) + clone.pk = None + clone.save() + # Fan out location edges so the sub-resource per-row queries (if any) trip the threshold. + self._loc_finding = Finding.objects.order_by("pk").first() + self._loc_product = Product.objects.order_by("pk").first() + for i in range(_ROW_FANOUT): + loc = Location.objects.create(location_type="url", location_value=f"query-report loc {i}") + LocationFindingReference.objects.create(location=loc, finding=self._loc_finding, status="Active") + LocationProductReference.objects.create(location=loc, product=self._loc_product, status="Active") + + # Fan out notes / files / tags so the notes|files|tags sub-resource lists (OS5) can't hide a + # per-row query. Capture one file id per parent for the download endpoint. + self._sub_finding = self._loc_finding + self._sub_engagement = Engagement.objects.order_by("pk").first() + self._sub_test = Test.objects.order_by("pk").first() + self._download_file_ids: dict[str, int] = {} + for key, parent in ( + ("findings", self._sub_finding), + ("engagements", self._sub_engagement), + ("tests", self._sub_test), + ): + for i in range(_ROW_FANOUT): + note = Notes.objects.create(entry=f"query-report note {i}", author=self.admin) + parent.notes.add(note) + upload = FileUpload.objects.create( + title=f"query-report {key} file {i}.txt", + file=ContentFile(b"query report file body", name=f"qr-{key}-{i}.txt"), + ) + parent.files.add(upload) + if i == 0: + self._download_file_ids[key] = upload.pk + parent.tags.add(f"qr-tag-{key}") + + def _representative_requests(self) -> dict[str, str]: + """OpenAPI GET path -> concrete request URL. Extend when a phase adds endpoints.""" + finding_id = Finding.objects.first().pk + product_id = Product.objects.first().pk + product_type_id = Product_Type.objects.first().pk + user_id = self.admin.pk + limit = f"limit={_ROW_FANOUT + 5}" + return { + "/findings": self.v3_url(f"findings?{limit}&expand=test.engagement,locations&include=counts"), + "/findings/{finding_id}": self.v3_url(f"findings/{finding_id}?expand=test.engagement"), # noqa: RUF027 + "/assets": self.v3_url(f"assets?{limit}&expand=organization"), + "/assets/{asset_id}": self.v3_url(f"assets/{product_id}"), + "/organizations": self.v3_url(f"organizations?{limit}"), + "/organizations/{organization_id}": self.v3_url(f"organizations/{product_type_id}"), + "/users": self.v3_url(f"users?{limit}"), + "/users/{user_id}": self.v3_url(f"users/{user_id}"), # noqa: RUF027 + "/engagements": self.v3_url(f"engagements?{limit}"), + "/engagements/{engagement_id}": self.v3_url(f"engagements/{Engagement.objects.first().pk}"), + "/tests": self.v3_url(f"tests?{limit}"), + "/tests/{test_id}": self.v3_url(f"tests/{Test.objects.first().pk}"), + "/locations": self.v3_url(f"locations?{limit}"), + "/locations/{location_id}": self.v3_url(f"locations/{Location.objects.order_by('pk').first().pk}"), + # CSV export endpoints (§4.15): streamed responses -- capture_request captures the + # synchronous work (auth + capped-count) without consuming the streamed body. The + # per-row query independence is pinned by the dedicated assertNumQueries test. + "/findings/export.csv": self.v3_url("findings/export.csv"), + "/assets/export.csv": self.v3_url("assets/export.csv"), + "/organizations/export.csv": self.v3_url("organizations/export.csv"), + "/engagements/export.csv": self.v3_url("engagements/export.csv"), + "/tests/export.csv": self.v3_url("tests/export.csv"), + "/users/export.csv": self.v3_url("users/export.csv"), + "/locations/export.csv": self.v3_url("locations/export.csv"), + "/findings/{finding_id}/locations": self.v3_url(f"findings/{self._loc_finding.pk}/locations?{limit}"), # noqa: RUF027 + "/assets/{asset_id}/locations": self.v3_url(f"assets/{self._loc_product.pk}/locations?{limit}"), + # OS5 notes / files / tags sub-resources (finding/engagement/test; tags also on asset). + "/findings/{parent_id}/notes": self.v3_url(f"findings/{self._sub_finding.pk}/notes?{limit}"), + "/engagements/{parent_id}/notes": self.v3_url(f"engagements/{self._sub_engagement.pk}/notes?{limit}"), + "/tests/{parent_id}/notes": self.v3_url(f"tests/{self._sub_test.pk}/notes?{limit}"), + "/findings/{parent_id}/files": self.v3_url(f"findings/{self._sub_finding.pk}/files?{limit}"), + "/engagements/{parent_id}/files": self.v3_url(f"engagements/{self._sub_engagement.pk}/files?{limit}"), + "/tests/{parent_id}/files": self.v3_url(f"tests/{self._sub_test.pk}/files?{limit}"), + "/findings/{parent_id}/files/{file_id}/download": self.v3_url(f"findings/{self._sub_finding.pk}/files/{self._download_file_ids['findings']}/download"), + "/engagements/{parent_id}/files/{file_id}/download": self.v3_url(f"engagements/{self._sub_engagement.pk}/files/{self._download_file_ids['engagements']}/download"), + "/tests/{parent_id}/files/{file_id}/download": self.v3_url(f"tests/{self._sub_test.pk}/files/{self._download_file_ids['tests']}/download"), + "/findings/{parent_id}/tags": self.v3_url(f"findings/{self._sub_finding.pk}/tags"), + "/engagements/{parent_id}/tags": self.v3_url(f"engagements/{self._sub_engagement.pk}/tags"), + "/tests/{parent_id}/tags": self.v3_url(f"tests/{self._sub_test.pk}/tags"), + "/assets/{parent_id}/tags": self.v3_url(f"assets/{self._loc_product.pk}/tags"), + } + + def _openapi_get_paths(self) -> set[str]: + schema = api_v3.get_openapi_schema() + # Schema paths carry the mount prefix (/api/v3-alpha/...); compare mount-relative. + return { + "/" + path.split(settings.API_V3_URL_PREFIX, 1)[-1].lstrip("/") + for path, ops in schema["paths"].items() + if "get" in ops + } + + def test_no_n_plus_one_across_surface(self): + requests = self._representative_requests() + + missing = self._openapi_get_paths() - set(requests) + self.assertFalse( + missing, + f"GET endpoint(s) {sorted(missing)} have no representative request in the query " + f"sweep — add one to _representative_requests() (this is deliberate: every new " + f"endpoint must be query-profiled).", + ) + + captures = [capture_request(self.client, path, url) for path, url in requests.items()] + _REPORT_PATH.write_text(format_report(captures, _THRESHOLD), encoding="utf-8") + + failures = [] + for cap in captures: + self.assertEqual(cap.status_code, 200, f"{cap.label} -> {cap.status_code}") + for shape, count in cap.repeated_shapes(_THRESHOLD): + failures.append(f"{cap.label}: {count}x {shape[:160]}") + self.assertFalse( + failures, + "N+1 signature detected (same query shape repeated within one request):\n" + + "\n".join(failures) + + f"\nFull report: {_REPORT_PATH}", + ) diff --git a/unittests/api_v3/test_apiv3_reference_docs.py b/unittests/api_v3/test_apiv3_reference_docs.py new file mode 100644 index 00000000000..018ece82488 --- /dev/null +++ b/unittests/api_v3/test_apiv3_reference_docs.py @@ -0,0 +1,58 @@ +""" +Scalar reference page (§12: npm-at-image-build — the bundle is a yarn-managed static asset, +no CDN at runtime, no vendored blob in git). + +The page is an HTML shell: the only executable content is the locally-served Scalar bundle, +installed by the existing components yarn step (exact pin in ``components/package.json``, +integrity via ``components/yarn.lock``) and exposed through ``STATICFILES_DIRS`` → +``components/node_modules``. These tests pin that contract: the static script URL, no CDN/SRI +remnants, the exact-pin in package.json, and the schema/docs URLs resolved by name (so the beta +URL move carries them along). +""" +from __future__ import annotations + +import json +from pathlib import Path + +from django.templatetags.static import static +from django.urls import reverse + +from dojo.api_v3.api import api_v3 +from dojo.api_v3.reference_docs import SCALAR_STATIC_PATH + +from .base import ApiV3TestCase + + +class TestApiV3ScalarReference(ApiV3TestCase): + + def _get(self): + return self.anonymous_client().get(reverse("api_v3_reference")) + + def test_page_serves_locally_hosted_bundle(self): + response = self._get() + self.assertEqual(200, response.status_code) + html = response.content.decode() + self.assertIn(f'', html) + # The npm-at-build decision: no runtime CDN, so no CDN host and no SRI attribute. + self.assertNotIn("cdn.jsdelivr.net", html) + self.assertNotIn("integrity=", html) + + def test_bundle_is_exact_pinned_in_components_package_json(self): + # Integrity is enforced by yarn (exact pin + lockfile hashes) at image build, not by a + # browser SRI attribute at runtime — so the pin itself is the contract to guard. + package_json = Path(__file__).parents[2] / "components" / "package.json" + deps = json.loads(package_json.read_text())["dependencies"] + version = deps["@scalar/api-reference"] + self.assertRegex(version, r"^\d+\.\d+\.\d+$", "Scalar must be pinned exactly (no ^/~/latest)") + + def test_page_points_at_v3_schema_and_swagger_fallback(self): + html = self._get().content.decode() + self.assertIn(f'data-url="{reverse("api_v3:openapi-json")}"', html) + # Swagger (framework-bundled assets) remains linked for noscript. + self.assertIn(reverse("api_v3:openapi-view"), html) + + def test_reference_page_is_not_an_api_operation(self): + # A plain Django view: it must NOT appear in the OpenAPI schema (and therefore places no + # obligations on the authz/query completeness gates, which walk the schema). + paths = api_v3.get_openapi_schema()["paths"] + self.assertFalse(any(p.endswith("/reference") for p in paths)) diff --git a/unittests/api_v3/test_apiv3_subresources.py b/unittests/api_v3/test_apiv3_subresources.py new file mode 100644 index 00000000000..cd2288880b2 --- /dev/null +++ b/unittests/api_v3/test_apiv3_subresources.py @@ -0,0 +1,451 @@ +""" +Generic notes / tags / files sub-resource tests for API v3 (§4.12, OS5). + +Covers the storage support matrix (notes/files: finding/engagement/test; tags: those + asset), +note privacy (v2 parity: private notes are returned, not per-user filtered), parent-authorization +inheritance (404 unknown-or-unauthorized parent, 403 write), multipart upload + streamed download +roundtrip, tag replace/append/delete semantics + normalization, the pagination envelope on the +list endpoints, and the MANDATORY constant-query guarantee for the finding notes/tags/files lists. +""" +from __future__ import annotations + +from unittest import mock + +from django.core.files.uploadedfile import SimpleUploadedFile +from django.db import connection +from django.test.utils import CaptureQueriesContext + +from dojo.models import Dojo_User, Engagement, Finding, Product, Test, User +from dojo.notes.models import Notes + +from .base import ApiV3TestCase + +_NOTE_KEYS = {"id", "entry", "author", "private", "edited", "created", "updated"} +_FILE_KEYS = {"id", "title", "size", "created"} +_ENVELOPE_KEYS = {"count", "next", "previous", "results"} + + +def _txt(name: str = "os5upload.txt", body: bytes = b"os5 file body") -> SimpleUploadedFile: + return SimpleUploadedFile(name, body, content_type="text/plain") + + +class _SubResourceBase(ApiV3TestCase): + + def setUp(self): + super().setUp() + self.finding = Finding.objects.first() + self.engagement = Engagement.objects.first() + self.test = Test.objects.first() + self.product = Product.objects.first() + self.note_file_parents = [ + ("findings", self.finding), + ("engagements", self.engagement), + ("tests", self.test), + ] + self.tag_parents = [*self.note_file_parents, ("assets", self.product)] + + +class TestApiV3SubresourcesNotes(_SubResourceBase): + + def test_notes_matrix_create_and_list_roundtrip(self): + for resource, parent in self.note_file_parents: + with self.subTest(resource=resource): + created = self.client.post( + self.v3_url(f"{resource}/{parent.pk}/notes"), + {"entry": f"note on {resource}", "private": False}, + format="json", + ) + self.assertEqual(201, created.status_code, created.content[:400]) + body = created.json() + self.assertEqual(_NOTE_KEYS, set(body)) + self.assertEqual(f"note on {resource}", body["entry"]) + self.assertFalse(body["private"]) + self.assertFalse(body["edited"]) + self.assertEqual(self.admin.pk, body["author"]["id"]) + self.assertEqual("admin", body["author"]["name"]) + self.assertIsNotNone(body["created"]) + + listing = self.get_json(f"{resource}/{parent.pk}/notes") + self.assertEqual(_ENVELOPE_KEYS, set(listing) - {"meta"}) + self.assertIn(body["id"], [n["id"] for n in listing["results"]]) + self.assertEqual(_NOTE_KEYS, set(listing["results"][0])) + + def test_note_created_persisted_and_linked_to_parent(self): + before = self.finding.notes.count() + self.client.post( + self.v3_url(f"findings/{self.finding.pk}/notes"), + {"entry": "persisted"}, format="json", + ) + self.assertEqual(before + 1, self.finding.notes.count()) + + def test_note_unknown_field_is_400(self): + response = self.client.post( + self.v3_url(f"findings/{self.finding.pk}/notes"), + {"entry": "x", "bogus": 1}, format="json", + ) + self.assertEqual(400, response.status_code) + self.assertEqual("application/problem+json", response["Content-Type"]) + + def test_note_missing_entry_is_400(self): + response = self.client.post( + self.v3_url(f"findings/{self.finding.pk}/notes"), {}, format="json", + ) + self.assertEqual(400, response.status_code) + + +class TestApiV3SubresourcesNotePrivacy(_SubResourceBase): + + def test_private_note_returned_and_flagged(self): + created = self.client.post( + self.v3_url(f"findings/{self.finding.pk}/notes"), + {"entry": "secret", "private": True}, format="json", + ).json() + self.assertTrue(created["private"]) + listing = self.get_json(f"findings/{self.finding.pk}/notes") + match = next(n for n in listing["results"] if n["id"] == created["id"]) + self.assertTrue(match["private"]) + + def test_private_note_visible_to_other_authorized_user_v2_parity(self): + """v2 parity: the notes endpoint returns *all* notes; `private` is not a per-user read filter.""" + member = Dojo_User.objects.create_user(username="v3_note_member", password="x") # noqa: S106 + self.finding.test.engagement.product.authorized_users.add(member) + created = self.client.post( + self.v3_url(f"findings/{self.finding.pk}/notes"), + {"entry": "private-but-visible", "private": True}, format="json", + ).json() + + member_view = self.get_json(f"findings/{self.finding.pk}/notes", client=self.token_client(user=member)) + self.assertIn(created["id"], [n["id"] for n in member_view["results"]]) + + +class TestApiV3SubresourcesFiles(_SubResourceBase): + + def test_files_matrix_upload_list_download_roundtrip(self): + for resource, parent in self.note_file_parents: + with self.subTest(resource=resource): + body_bytes = f"payload-{resource}".encode() + created = self.client.post( + self.v3_url(f"{resource}/{parent.pk}/files"), + {"title": f"os5 {resource} attachment", "file": _txt(body=body_bytes)}, + format="multipart", + ) + self.assertEqual(201, created.status_code, created.content[:400]) + fbody = created.json() + self.assertEqual(_FILE_KEYS, set(fbody)) + self.assertEqual(f"os5 {resource} attachment", fbody["title"]) + self.assertEqual(len(body_bytes), fbody["size"]) + self.assertIsNone(fbody["created"]) + + listing = self.get_json(f"{resource}/{parent.pk}/files") + self.assertEqual(_ENVELOPE_KEYS, set(listing) - {"meta"}) + self.assertIn(fbody["id"], [f["id"] for f in listing["results"]]) + + dl = self.client.get(self.v3_url(f"{resource}/{parent.pk}/files/{fbody['id']}/download")) + self.assertEqual(200, dl.status_code) + self.assertIn("attachment", dl["Content-Disposition"]) + self.assertEqual(body_bytes, b"".join(dl.streaming_content)) + + def test_upload_rejects_disallowed_extension(self): + response = self.client.post( + self.v3_url(f"findings/{self.finding.pk}/files"), + {"title": "os5 evil", "file": _txt(name="evil.exe")}, + format="multipart", + ) + self.assertEqual(400, response.status_code, response.content[:400]) + self.assertEqual("application/problem+json", response["Content-Type"]) + self.assertIn("file", response.json()["fields"]) + + def test_duplicate_title_is_400(self): + payload = {"title": "os5 dup title", "file": _txt()} + first = self.client.post(self.v3_url(f"findings/{self.finding.pk}/files"), payload, format="multipart") + self.assertEqual(201, first.status_code) + dup = self.client.post( + self.v3_url(f"tests/{self.test.pk}/files"), + {"title": "os5 dup title", "file": _txt()}, format="multipart", + ) + self.assertEqual(400, dup.status_code, dup.content[:400]) + + def test_download_missing_file_is_404(self): + response = self.client.get(self.v3_url(f"findings/{self.finding.pk}/files/9999999/download")) + self.assertEqual(404, response.status_code) + + +class TestApiV3SubresourcesTags(_SubResourceBase): + + def test_tags_get_shape_all_resources(self): + for resource, parent in self.tag_parents: + with self.subTest(resource=resource): + body = self.get_json(f"{resource}/{parent.pk}/tags") + self.assertEqual({"tags"}, set(body)) + self.assertIsInstance(body["tags"], list) + + def test_tags_replace_append_delete_semantics(self): + # Asset tags have clean semantics (no inheritance-sticky re-add); inheritance is off by + # default anyway, but assets are the crispest surface for the exact-set assertions. + for resource, parent in self.tag_parents: + with self.subTest(resource=resource): + base = self.v3_url(f"{resource}/{parent.pk}/tags") + # PUT replace + normalization (force_lowercase). + replaced = self.client.put(base, {"tags": ["PCI", "Sox"]}, format="json") + self.assertEqual(200, replaced.status_code, replaced.content[:400]) + self.assertEqual({"pci", "sox"}, set(replaced.json()["tags"])) + # POST append (dedup, case-insensitive). + appended = self.client.post(base, {"tags": ["Extra", "PCI"]}, format="json") + self.assertEqual(200, appended.status_code) + self.assertEqual({"pci", "sox", "extra"}, set(appended.json()["tags"])) + # DELETE one (case-insensitive match on the stored lowercase tag). + removed = self.client.delete(self.v3_url(f"{resource}/{parent.pk}/tags/PCI")) + self.assertEqual(204, removed.status_code) + self.assertEqual({"sox", "extra"}, set(self.get_json(f"{resource}/{parent.pk}/tags")["tags"])) + + def test_delete_absent_tag_is_404(self): + self.client.put(self.v3_url(f"findings/{self.finding.pk}/tags"), {"tags": ["keep"]}, format="json") + response = self.client.delete(self.v3_url(f"findings/{self.finding.pk}/tags/nope")) + self.assertEqual(404, response.status_code) + self.assertEqual("application/problem+json", response["Content-Type"]) + + def test_tags_unknown_body_field_is_400(self): + response = self.client.put( + self.v3_url(f"findings/{self.finding.pk}/tags"), + {"tags": ["a"], "bogus": 1}, format="json", + ) + self.assertEqual(400, response.status_code) + + +class TestApiV3SubresourcesSupportMatrix(_SubResourceBase): + + """Sub-resources are attached only where the model stores them; everything else is 404.""" + + def test_unsupported_notes_and_files_are_404(self): + pt_id = self.product.prod_type_id + for path in ( + f"organizations/{pt_id}/notes", + f"organizations/{pt_id}/files", + f"assets/{self.product.pk}/notes", # asset has tags but no notes/files + f"assets/{self.product.pk}/files", + f"users/{self.admin.pk}/notes", + f"users/{self.admin.pk}/files", + ): + with self.subTest(path=path): + self.assertEqual(404, self.client.get(self.v3_url(path)).status_code) + + def test_unsupported_tags_are_404(self): + # organization (product_type) / user have no TagField; location has one but is read-only + + # superuser-only with no v2 tag-mutation endpoint, so no tag sub-resource is attached (tags[] + # is on its read shape instead). + from dojo.location.models import Location # noqa: PLC0415 + location = Location.objects.first() + for path in ( + f"organizations/{self.product.prod_type_id}/tags", + f"users/{self.admin.pk}/tags", + *([f"locations/{location.pk}/tags"] if location else []), + ): + with self.subTest(path=path): + self.assertEqual(404, self.client.get(self.v3_url(path)).status_code) + + +class TestApiV3SubresourcesAuthInheritance(_SubResourceBase): + + """Authorization inherited from the parent: 404 unknown-or-unauthorized parent, 403 on write.""" + + def test_unknown_parent_is_404(self): + for path in ( + "findings/9999999/notes", + "engagements/9999999/files", + "tests/9999999/tags", + "assets/9999999/tags", + ): + with self.subTest(path=path): + self.assertEqual(404, self.client.get(self.v3_url(path)).status_code) + + def test_unauthorized_parent_is_404_not_403(self): + """A non-member cannot see the parent, so every sub-resource op is 404 (never leak existence).""" + limited = User.objects.create_user(username="v3_sub_limited", password="x") # noqa: S106 + client = self.token_client(user=limited) + self.assertEqual(404, client.get(self.v3_url(f"findings/{self.finding.pk}/notes")).status_code) + self.assertEqual(404, client.get(self.v3_url(f"findings/{self.finding.pk}/files")).status_code) + self.assertEqual(404, client.get(self.v3_url(f"findings/{self.finding.pk}/tags")).status_code) + post = client.post( + self.v3_url(f"findings/{self.finding.pk}/notes"), {"entry": "x"}, format="json", + ) + self.assertEqual(404, post.status_code) + + def test_write_forbidden_when_permission_fails_is_403(self): + """ + Parent visible (admin/superuser) but the write permission check fails -> 403. Under the OS + legacy model a viewing member also has edit/add, so this exercises the 403 code path via a + permission-check failure (see .claude/os5-report.md / §12). + """ + with mock.patch("dojo.api_v3.subresources.user_has_permission", return_value=False): + note = self.client.post( + self.v3_url(f"findings/{self.finding.pk}/notes"), {"entry": "x"}, format="json", + ) + self.assertEqual(403, note.status_code, note.content[:300]) + self.assertEqual("application/problem+json", note["Content-Type"]) + + tag = self.client.put( + self.v3_url(f"findings/{self.finding.pk}/tags"), {"tags": ["x"]}, format="json", + ) + self.assertEqual(403, tag.status_code) + + upload = self.client.post( + self.v3_url(f"findings/{self.finding.pk}/files"), + {"title": "os5 403", "file": _txt()}, format="multipart", + ) + self.assertEqual(403, upload.status_code) + + +class TestApiV3SubresourcesPagination(_SubResourceBase): + + def test_notes_list_pagination_envelope(self): + for i in range(5): + note = Notes.objects.create(entry=f"pag {i}", author=self.admin) + self.finding.notes.add(note) + body = self.get_json(f"findings/{self.finding.pk}/notes", data={"limit": 2}) + self.assertGreaterEqual(body["count"], 5) + self.assertEqual(2, len(body["results"])) + self.assertIsNotNone(body["next"]) + self.assertIsNone(body["previous"]) + + +class TestApiV3NoteSideEffectsFinding(_SubResourceBase): + + """ + v2 parity for finding note-create side-effects (mirrors ``dojo/finding/api/views.py`` notes + @action): ``last_reviewed`` stamping, JIRA comment sync on the linked issue / finding-group issue, + and @mention notifications. JIRA is mocked at the service seam (``dojo.finding.services + .jira_services.add_comment``); the mention path is exercised end-to-end through the real + ``process_tag_notifications`` parser, capturing ``create_notification``. + """ + + _ADD_COMMENT = "dojo.finding.services.jira_services.add_comment" + _CREATE_NOTIFICATION = "dojo.notifications.helper.create_notification" + + def _post_note(self, *, entry="side-effect note", parent=None): + parent = parent or self.finding + response = self.client.post( + self.v3_url(f"findings/{parent.pk}/notes"), {"entry": entry}, format="json", + ) + self.assertEqual(201, response.status_code, response.content[:400]) + return response.json() + + def test_finding_note_stamps_last_reviewed(self): + body = self._post_note() + note = Notes.objects.get(pk=body["id"]) + self.finding.refresh_from_db() + self.assertEqual(note.date, self.finding.last_reviewed) + self.assertEqual(self.admin.pk, self.finding.last_reviewed_by_id) + + def test_finding_note_jira_comment_when_linked(self): + with mock.patch.object(Finding, "has_jira_issue", new_callable=mock.PropertyMock, return_value=True), \ + mock.patch(self._ADD_COMMENT) as add_comment: + self._post_note() + add_comment.assert_called_once() + self.assertEqual(self.finding.pk, add_comment.call_args.args[0].pk) + + def test_finding_note_no_jira_comment_when_not_linked(self): + # The default fixture finding has neither a linked issue nor a finding-group issue. + self.assertFalse(self.finding.has_jira_issue) + self.assertFalse(self.finding.has_jira_group_issue) + with mock.patch(self._ADD_COMMENT) as add_comment: + self._post_note() + add_comment.assert_not_called() + + def test_finding_note_group_jira_comment_when_group_linked(self): + with mock.patch.object(Finding, "has_jira_issue", new_callable=mock.PropertyMock, return_value=False), \ + mock.patch.object(Finding, "has_jira_group_issue", new_callable=mock.PropertyMock, return_value=True), \ + mock.patch(self._ADD_COMMENT) as add_comment: + self._post_note() + add_comment.assert_called_once() + # The finding-group branch comments on the group, not the finding (v2 parity). + expected_pk = getattr(self.finding.finding_group, "pk", None) + self.assertEqual(expected_pk, getattr(add_comment.call_args.args[0], "pk", None)) + + def test_finding_note_mention_dispatches_user_mentioned_notification(self): + Dojo_User.objects.create_user(username="v3_mention_target", password="x") # noqa: S106 + with mock.patch(self._CREATE_NOTIFICATION) as create_notif: + self._post_note(entry="hey @v3_mention_target please review") + mention_calls = [c for c in create_notif.call_args_list if c.kwargs.get("event") == "user_mentioned"] + self.assertEqual(1, len(mention_calls)) + self.assertIn("v3_mention_target", mention_calls[0].kwargs["recipients"]) + + +class TestApiV3NoteSideEffectsEngagementTest(_SubResourceBase): + + """ + v2 parity for engagement/test note-create side-effects: @mention notifications **only** -- their + v2 @actions (``dojo/engagement/api/views.py``, ``dojo/test/api/views.py``) have no JIRA comment + sync and no ``last_reviewed`` stamping (neither model even has that field, so a 201 is itself + evidence the finding-only side-effects did not run). + """ + + _CREATE_NOTIFICATION = "dojo.notifications.helper.create_notification" + _JIRA_ADD_COMMENT = "dojo.jira.services.add_comment" + + def _assert_mention_notified(self, resource, parent, username): + Dojo_User.objects.create_user(username=username, password="x") # noqa: S106 + with mock.patch(self._CREATE_NOTIFICATION) as create_notif: + response = self.client.post( + self.v3_url(f"{resource}/{parent.pk}/notes"), {"entry": f"cc @{username}"}, format="json", + ) + self.assertEqual(201, response.status_code, response.content[:400]) + mention_calls = [c for c in create_notif.call_args_list if c.kwargs.get("event") == "user_mentioned"] + self.assertEqual(1, len(mention_calls)) + self.assertIn(username, mention_calls[0].kwargs["recipients"]) + + def test_engagement_note_mention_dispatches_notification(self): + self._assert_mention_notified("engagements", self.engagement, "v3_eng_mention") + + def test_test_note_mention_dispatches_notification(self): + self._assert_mention_notified("tests", self.test, "v3_test_mention") + + def test_engagement_and_test_notes_fire_no_jira_comment(self): + # Parity guard: unlike the finding @action, the engagement/test @actions never sync a JIRA + # comment. Patched at the source (dojo.jira.services.add_comment) so a mis-wired finding + # callback would also be caught here. + with mock.patch(self._JIRA_ADD_COMMENT) as add_comment: + eng = self.client.post( + self.v3_url(f"engagements/{self.engagement.pk}/notes"), {"entry": "no jira"}, format="json", + ) + tst = self.client.post( + self.v3_url(f"tests/{self.test.pk}/notes"), {"entry": "no jira"}, format="json", + ) + self.assertEqual(201, eng.status_code, eng.content[:400]) + self.assertEqual(201, tst.status_code, tst.content[:400]) + add_comment.assert_not_called() + + +class TestApiV3SubresourcesQueryCounts(_SubResourceBase): + + """MANDATORY: finding notes/tags/files list query counts are independent of row count.""" + + def _query_count(self, path: str) -> int: + with CaptureQueriesContext(connection) as ctx: + response = self.client.get(self.v3_url(path)) + self.assertEqual(200, response.status_code) + return len(ctx.captured_queries) + + def test_notes_list_query_count_constant(self): + path = f"findings/{self.finding.pk}/notes?limit=250" + for i in range(5): + self.finding.notes.add(Notes.objects.create(entry=f"n{i}", author=self.admin)) + first = self._query_count(path) + for i in range(50): + self.finding.notes.add(Notes.objects.create(entry=f"m{i}", author=self.admin)) + self.assertEqual(first, self._query_count(path)) + + def test_files_list_query_count_constant(self): + path = f"findings/{self.finding.pk}/files?limit=250" + for i in range(5): + self.finding.files.create(title=f"qc-a-{i}.txt", file=_txt()) + first = self._query_count(path) + for i in range(50): + self.finding.files.create(title=f"qc-b-{i}.txt", file=_txt()) + self.assertEqual(first, self._query_count(path)) + + def test_tags_list_query_count_constant(self): + path = f"findings/{self.finding.pk}/tags" + self.finding.tags.add("t1", "t2", "t3") + first = self._query_count(path) + self.finding.tags.add("t4", "t5", "t6", "t7", "t8", "t9", "t10") + self.assertEqual(first, self._query_count(path)) diff --git a/unittests/api_v3/test_apiv3_test_dedupe_policy.py b/unittests/api_v3/test_apiv3_test_dedupe_policy.py new file mode 100644 index 00000000000..9efebb0f6e8 --- /dev/null +++ b/unittests/api_v3/test_apiv3_test_dedupe_policy.py @@ -0,0 +1,253 @@ +""" +Test-detail dedupe/matching-policy read fields for API v3 (ports ``test_apiv2_test_dedupe_policy.py``). + +v3's Test detail exposes the effective finding-matching policy -- ``deduplication_algorithm`` and +``hash_code_fields`` -- as READ-ONLY computed fields (§4.5, §9.1). They reuse the v2 helper +verbatim: the ``Test.deduplication_algorithm`` / ``Test.hash_code_fields`` model properties +(``dojo/test/models.py``), the settings-driven per-scanner lookup the v2 ``TestSerializer`` reads. +The values mirror the per-scanner settings, so the assertions read the live settings dicts +dynamically (like the v2 test) rather than hardcoding the current config. + +Deliberate deviation, hardened over v2 (§12): the fields are NOT on any write schema (all +``extra="forbid"``), so a PATCH/PUT that supplies either is a **400** problem+json -- v2 silently +*ignores* writes to them. Silent-ignore is exactly the failure mode v3 rejects everywhere (the same +principle as unknown filter/expand/fields params). +""" +from __future__ import annotations + +import csv +import datetime +import io + +from django.conf import settings +from django.db import connection +from django.test.utils import CaptureQueriesContext + +from dojo.api_v3.expand import plan_list_fields +from dojo.models import Engagement, Test, Test_Type +from dojo.test.api_v3.schemas import TestDetail, TestSlim + +from .base import ApiV3TestCase + +# Two scan types with genuinely different policies (mirrors the v2 test's per-scanner assertions), +# plus a made-up scan type absent from both settings dicts for the default-fallback case. +_HASH_CODE_SCAN = "ZAP Scan" # -> hash_code + ["title", "cwe", "severity"] +_UNIQUE_ID_SCAN = "Checkmarx Scan detailed" # -> unique_id_from_tool + no hashcode fields (None) +_DEFAULT_SCAN = "V3 Nonexistent Custom Scan Type" # -> legacy (fallback) + None + + +def _now() -> datetime.datetime: + return datetime.datetime(2024, 1, 1, tzinfo=datetime.UTC) + + +class _DedupePolicyBase(ApiV3TestCase): + + def _make_test(self, scan_type: str) -> Test: + # test_type.name is what the model property checks first, then scan_type; set both equal (the + # real-world case for an imported test, and what the v2 test relied on). + engagement = Engagement.objects.first() + test_type, _ = Test_Type.objects.get_or_create(name=scan_type) + return Test.objects.create( + engagement=engagement, test_type=test_type, scan_type=scan_type, + target_start=_now(), target_end=_now(), + ) + + +class TestApiV3DedupePolicyRead(_DedupePolicyBase): + + """The Test detail exposes the effective matching policy, resolved from the per-scanner settings.""" + + def _assert_policy(self, scan_type: str) -> dict: + test = self._make_test(scan_type) + data = self.get_json(f"tests/{test.id}") + # Mirror v2: assert against the live settings so a config change never silently passes. + expected_algorithm = settings.DEDUPLICATION_ALGORITHM_PER_PARSER.get( + scan_type, settings.DEDUPE_ALGO_LEGACY) + expected_fields = settings.HASHCODE_FIELDS_PER_SCANNER.get(scan_type) + self.assertIn("deduplication_algorithm", data) + self.assertIn("hash_code_fields", data) + self.assertEqual(expected_algorithm, data["deduplication_algorithm"]) + self.assertEqual(expected_fields, data["hash_code_fields"]) + return data + + def test_hash_code_scan_type_exposes_algorithm_and_fields(self): + data = self._assert_policy(_HASH_CODE_SCAN) + # Concrete expectation documenting the "hash_code" policy for this scanner. + self.assertEqual("hash_code", data["deduplication_algorithm"]) + self.assertEqual(["title", "cwe", "severity"], data["hash_code_fields"]) + + def test_unique_id_scan_type_has_a_different_policy(self): + data = self._assert_policy(_UNIQUE_ID_SCAN) + # A genuinely different policy from _HASH_CODE_SCAN: unique_id_from_tool, no hashcode fields. + self.assertEqual("unique_id_from_tool", data["deduplication_algorithm"]) + self.assertIsNone(data["hash_code_fields"]) + + def test_two_scan_types_have_distinct_algorithms(self): + # Proves the field is genuinely computed per scan type, not a constant. + hash_code = self.get_json(f"tests/{self._make_test(_HASH_CODE_SCAN).id}") + unique_id = self.get_json(f"tests/{self._make_test(_UNIQUE_ID_SCAN).id}") + self.assertNotEqual( + hash_code["deduplication_algorithm"], unique_id["deduplication_algorithm"]) + + def test_unconfigured_scan_type_falls_back_to_legacy(self): + data = self._assert_policy(_DEFAULT_SCAN) + # The documented default when a scan type has no per-scanner configuration. + self.assertEqual(settings.DEDUPE_ALGO_LEGACY, data["deduplication_algorithm"]) + self.assertIsNone(data["hash_code_fields"]) + + def test_detail_computes_policy_without_extra_queries(self): + # The resolvers read only test_type (already select_related in TestSlim.SELECT_RELATED) and + # scan_type (a concrete column, never deferred on a detail fetch) -> zero extra queries. Uses + # the detail route's exact select_related shape (Test.objects stands in for the authorized + # queryset, which resolves the user via crum -- unset outside an HTTP request). + test = self._make_test(_HASH_CODE_SCAN) + obj = Test.objects.select_related(*TestDetail.SELECT_RELATED).get(pk=test.id) + with self.assertNumQueries(0): + self.assertEqual("hash_code", obj.deduplication_algorithm) + self.assertEqual(["title", "cwe", "severity"], obj.hash_code_fields) + + +class TestApiV3DedupePolicyWriteRejected(_DedupePolicyBase): + + """v3 REJECTS writes to the policy fields (400) where v2 silently IGNORES them (§12).""" + + def _assert_unknown_field_400(self, response, field: str) -> None: + self.assertEqual(400, response.status_code, response.content[:400]) + self.assertEqual("application/problem+json", response["Content-Type"]) + # The rejected field is named in the problem body (unknown-field, not a generic 400). + self.assertIn(field.encode(), response.content) + + def test_patch_deduplication_algorithm_is_400(self): + test = self._make_test(_HASH_CODE_SCAN) + response = self.client.patch( + self.v3_url(f"tests/{test.id}"), {"deduplication_algorithm": "hash_code"}, format="json") + self._assert_unknown_field_400(response, "deduplication_algorithm") + + def test_patch_hash_code_fields_is_400(self): + test = self._make_test(_HASH_CODE_SCAN) + response = self.client.patch( + self.v3_url(f"tests/{test.id}"), {"hash_code_fields": ["title"]}, format="json") + self._assert_unknown_field_400(response, "hash_code_fields") + + def test_put_deduplication_algorithm_is_400(self): + test = self._make_test(_HASH_CODE_SCAN) + payload = { + "test_type": test.test_type_id, + "target_start": "2024-01-01T00:00:00Z", + "target_end": "2024-01-02T00:00:00Z", + "deduplication_algorithm": "hash_code", + } + response = self.client.put(self.v3_url(f"tests/{test.id}"), payload, format="json") + self._assert_unknown_field_400(response, "deduplication_algorithm") + + def test_put_hash_code_fields_is_400(self): + test = self._make_test(_HASH_CODE_SCAN) + payload = { + "test_type": test.test_type_id, + "target_start": "2024-01-01T00:00:00Z", + "target_end": "2024-01-02T00:00:00Z", + "hash_code_fields": ["title"], + } + response = self.client.put(self.v3_url(f"tests/{test.id}"), payload, format="json") + self._assert_unknown_field_400(response, "hash_code_fields") + + def test_reject_is_atomic_v3_hardens_where_v2_ignores(self): + # The exact v2 scenario (test_matching_policy_ignored_on_write): a PATCH mixing the read-only + # policy fields with a real field. v2 returns 200 and silently ignores the policy fields; v3 + # rejects the whole request (400) and applies nothing -- the deliberate reject-vs-ignore + # deviation. The description must be unchanged after the rejection. + test = self._make_test(_HASH_CODE_SCAN) + original_description = test.description + response = self.client.patch( + self.v3_url(f"tests/{test.id}"), + {"deduplication_algorithm": "hash_code", "hash_code_fields": ["title"], "description": "updated"}, + format="json", + ) + self.assertEqual(400, response.status_code, response.content[:400]) + self.assertEqual("application/problem+json", response["Content-Type"]) + test.refresh_from_db() + self.assertEqual(original_description, test.description) + + +class TestApiV3DedupePolicyFieldsOptUp(_DedupePolicyBase): + + """``?fields=`` opt-up on a LIST: the computed detail field serializes via the detail resolver.""" + + def _bulk(self, count: int) -> None: + engagement = Engagement.objects.first() + test_type, _ = Test_Type.objects.get_or_create(name=_HASH_CODE_SCAN) + Test.objects.bulk_create([ + Test(engagement=engagement, test_type=test_type, scan_type=_HASH_CODE_SCAN, + target_start=_now(), target_end=_now()) + for _ in range(count) + ]) + + def _query_count(self, params: dict) -> int: + with CaptureQueriesContext(connection) as ctx: + response = self.client.get(self.v3_url("tests"), params) + self.assertEqual(200, response.status_code, response.content[:500]) + return len(ctx.captured_queries) + + def test_not_present_on_default_list_row(self): + # The default (no ?fields=) list is the slim shape -- the detail-only policy fields are absent. + self._make_test(_HASH_CODE_SCAN) + row = self.get_json("tests", data={"limit": 250})["results"][0] + self.assertNotIn("deduplication_algorithm", row) + self.assertNotIn("hash_code_fields", row) + + def test_fields_opt_up_returns_the_computed_field(self): + self._make_test(_HASH_CODE_SCAN) + body = self.get_json("tests", data={"fields": "id,name,deduplication_algorithm", "limit": 250}) + row = body["results"][0] + self.assertEqual({"id", "name", "deduplication_algorithm"}, set(row)) + self.assertIsInstance(row["deduplication_algorithm"], str) + + def test_fields_opt_up_value_matches_detail(self): + test = self._make_test(_HASH_CODE_SCAN) + row = next( + r for r in self.get_json( + "tests", data={"fields": "id,deduplication_algorithm,hash_code_fields", "limit": 250}, + )["results"] + if r["id"] == test.id + ) + self.assertEqual("hash_code", row["deduplication_algorithm"]) + self.assertEqual(["title", "cwe", "severity"], row["hash_code_fields"]) + + def test_computed_fields_never_enter_the_defer_set(self): + # Kernel-level: the policy fields are resolver-backed (not concrete columns), so they are + # never defer candidates; requesting deduplication_algorithm un-defers exactly ``scan_type`` + # (the concrete column its resolver reads) via TestDetail.DETAIL_FIELD_COLUMNS. + default_plan = plan_list_fields(TestSlim, TestDetail, None) + self.assertIn("scan_type", default_plan.defer) # deferred by default + self.assertNotIn("deduplication_algorithm", default_plan.defer) + self.assertNotIn("hash_code_fields", default_plan.defer) + + opt_up = plan_list_fields(TestSlim, TestDetail, {"id", "name", "deduplication_algorithm"}) + self.assertIn("deduplication_algorithm", opt_up.detail_extra) + self.assertNotIn("scan_type", opt_up.defer) # un-deferred: the resolver reads it + self.assertNotIn("deduplication_algorithm", opt_up.defer) + + def test_query_count_is_independent_of_row_count(self): + # GET /tests?fields=id,name,deduplication_algorithm must stay constant-query (no per-row lazy + # load of the deferred scan_type column). This is the headline claim for the opt-up. + self._bulk(10) + first = self._query_count({"limit": 250, "fields": "id,name,deduplication_algorithm"}) + self._bulk(90) + second = self._query_count({"limit": 250, "fields": "id,name,deduplication_algorithm"}) + self.assertEqual(first, second, f"query count grew with rows: {first} -> {second}") + + def test_csv_export_flattens_policy_columns(self): + test = self._make_test(_HASH_CODE_SCAN) + response = self.client.get( + self.v3_url("tests/export.csv"), + {"fields": "id,name,deduplication_algorithm,hash_code_fields"}, + ) + self.assertEqual(200, response.status_code) + rows = list(csv.reader(io.StringIO(b"".join(response.streaming_content).decode("utf-8")))) + header = rows[0] + self.assertIn("deduplication_algorithm", header) + self.assertIn("hash_code_fields", header) # a list field -> one semicolon-joined column + indexed = {r[header.index("id")]: dict(zip(header, r, strict=True)) for r in rows[1:]} + row = indexed[str(test.id)] + self.assertEqual("hash_code", row["deduplication_algorithm"]) + self.assertEqual("title;cwe;severity", row["hash_code_fields"]) diff --git a/unittests/api_v3/test_apiv3_tests.py b/unittests/api_v3/test_apiv3_tests.py new file mode 100644 index 00000000000..6db5009d2a0 --- /dev/null +++ b/unittests/api_v3/test_apiv3_tests.py @@ -0,0 +1,314 @@ +"""Test (scan run) CRUD + RBAC + contract tests for API v3 (OS3b).""" +from __future__ import annotations + +import datetime +from unittest import mock + +from django.db import connection +from django.test.utils import CaptureQueriesContext + +from dojo.finding.api_v3 import schemas as finding_schemas +from dojo.models import Dojo_User, Engagement, Test, Test_Type, User +from dojo.test.api_v3.schemas import EnvironmentSlim, TestSlim, TestTypeSlim + +from .base import ApiV3TestCase + +_SLIM_KEYS = { + "id", "name", "test_type", "engagement", "asset", "organization", "environment", + "lead", "target_start", "target_end", "percent_complete", "tags", "created", "updated", +} + + +class TestApiV3TestsRelocation(ApiV3TestCase): + + def test_test_slims_are_canonical_single_classes(self): + # OS3b relocated TestSlim/TestTypeSlim/EnvironmentSlim out of the finding module; the finding + # module now re-exports the one canonical class each (is-identity -- §12 relocation pattern). + self.assertIs(TestSlim, finding_schemas.TestSlim) + self.assertIs(TestTypeSlim, finding_schemas.TestTypeSlim) + self.assertIs(EnvironmentSlim, finding_schemas.EnvironmentSlim) + + +class TestApiV3TestsRead(ApiV3TestCase): + + def test_list_envelope_and_slim_shape(self): + body = self.get_json("tests") + self.assertEqual({"count", "next", "previous", "results"}, set(body) - {"meta"}) + self.assertGreater(body["count"], 0) + row = body["results"][0] + self.assertEqual(_SLIM_KEYS, set(row)) + self.assertEqual({"id", "name"}, set(row["test_type"])) + self.assertEqual({"id", "name"}, set(row["engagement"])) + self.assertIsInstance(row["tags"], list) + + def test_detail_adds_heavy_fields(self): + test = Test.objects.first() + detail = self.get_json(f"tests/{test.id}") + for key in ("description", "scan_type", "version", "build_id", "commit_hash", "branch_tag"): + self.assertIn(key, detail) + + def test_detail_unknown_is_404_problem(self): + response = self.client.get(self.v3_url("tests/99999999")) + self.assertEqual(404, response.status_code) + self.assertEqual("application/problem+json", response["Content-Type"]) + + def test_expand_engagement_and_test_type(self): + row = self.get_json("tests", data={"expand": "engagement,test_type"})["results"][0] + self.assertIn("status", row["engagement"]) + self.assertIn("active", row["test_type"]) + + def test_expand_unknown_relation_is_400(self): + self.get_json("tests", data={"expand": "not_a_relation"}, expected=400) + + +class TestApiV3TestsFilters(ApiV3TestCase): + + def test_filter_engagement(self): + engagement_id = Test.objects.first().engagement_id + body = self.get_json("tests", data={"engagement": engagement_id, "limit": 250}) + self.assertGreater(body["count"], 0) + for row in body["results"]: + self.assertEqual(engagement_id, row["engagement"]["id"]) + + def test_filter_test_type(self): + tt_id = Test.objects.first().test_type_id + body = self.get_json("tests", data={"test_type": tt_id, "limit": 250}) + for row in body["results"]: + self.assertEqual(tt_id, row["test_type"]["id"]) + + def test_ordering_by_id(self): + ids = [r["id"] for r in self.get_json("tests", data={"o": "id", "limit": 250})["results"]] + self.assertEqual(ids, sorted(ids)) + + def test_unknown_filter_param_is_400(self): + self.get_json("tests", data={"not_a_filter": "x"}, expected=400) + + +class TestApiV3TestsPagination(ApiV3TestCase): + + def test_limit_next_previous(self): + body = self.get_json("tests", data={"limit": 2, "offset": 2}) + self.assertLessEqual(len(body["results"]), 2) + self.assertIsNotNone(body["previous"]) + + +class TestApiV3TestsQueryCount(ApiV3TestCase): + + def _bulk(self, count: int, start: int) -> None: + engagement = Engagement.objects.first() + test_type = Test_Type.objects.first() + now = datetime.datetime(2024, 1, 1, tzinfo=datetime.UTC) + Test.objects.bulk_create([ + Test(engagement=engagement, test_type=test_type, target_start=now, target_end=now) + for _ in range(count) + ]) + + def _query_count(self, params: dict) -> int: + with CaptureQueriesContext(connection) as ctx: + response = self.client.get(self.v3_url("tests"), params) + self.assertEqual(200, response.status_code, response.content[:500]) + return len(ctx.captured_queries) + + def test_query_count_is_independent_of_row_count(self): + self._bulk(10, 0) + first = self._query_count({"limit": 250}) + first_expand = self._query_count({"limit": 250, "expand": "engagement.asset,test_type"}) + self._bulk(90, 100) + second = self._query_count({"limit": 250}) + second_expand = self._query_count({"limit": 250, "expand": "engagement.asset,test_type"}) + self.assertEqual(first, second, f"query count grew (no expand): {first} -> {second}") + self.assertEqual(first_expand, second_expand, f"query count grew (expand): {first_expand} -> {second_expand}") + + +class TestApiV3TestsWrite(ApiV3TestCase): + + def test_create_happy_path(self): + engagement = Engagement.objects.first() + test_type = Test_Type.objects.first() + response = self.client.post( + self.v3_url("tests"), + {"engagement": engagement.id, "test_type": test_type.id, + "target_start": "2024-01-01T00:00:00Z", "target_end": "2024-01-02T00:00:00Z", + "title": "v3 created test", "tags": ["v3"]}, + format="json", + ) + self.assertEqual(201, response.status_code, response.content[:500]) + body = response.json() + self.assertEqual("v3 created test", body["name"]) + self.assertEqual(engagement.id, body["engagement"]["id"]) + created = Test.objects.get(title="v3 created test") + self.assertEqual({"v3"}, {t.name for t in created.tags.all()}) + + def test_create_bad_test_type_is_400(self): + engagement = Engagement.objects.first() + response = self.client.post( + self.v3_url("tests"), + {"engagement": engagement.id, "test_type": 99999999, + "target_start": "2024-01-01T00:00:00Z", "target_end": "2024-01-02T00:00:00Z"}, + format="json", + ) + self.assertEqual(400, response.status_code) + + def test_create_missing_required_is_400(self): + response = self.client.post(self.v3_url("tests"), {"title": "no engagement"}, format="json") + self.assertEqual(400, response.status_code) + + def test_create_unknown_field_is_400(self): + engagement = Engagement.objects.first() + test_type = Test_Type.objects.first() + response = self.client.post( + self.v3_url("tests"), + {"engagement": engagement.id, "test_type": test_type.id, + "target_start": "2024-01-01T00:00:00Z", "target_end": "2024-01-02T00:00:00Z", "bogus": 1}, + format="json", + ) + self.assertEqual(400, response.status_code) + + def test_create_nonexistent_engagement_is_404(self): + test_type = Test_Type.objects.first() + response = self.client.post( + self.v3_url("tests"), + {"engagement": 99999999, "test_type": test_type.id, + "target_start": "2024-01-01T00:00:00Z", "target_end": "2024-01-02T00:00:00Z"}, + format="json", + ) + self.assertEqual(404, response.status_code) + + def test_patch_partial_update(self): + test = Test.objects.first() + response = self.client.patch( + self.v3_url(f"tests/{test.id}"), {"title": "renamed test"}, format="json", + ) + self.assertEqual(200, response.status_code, response.content[:500]) + test.refresh_from_db() + self.assertEqual("renamed test", test.title) + + def test_patch_engagement_is_rejected_as_unknown_field(self): + # engagement is editable=False on the model -> not part of the update schema (mirrors v2). + test = Test.objects.first() + other = Engagement.objects.exclude(pk=test.engagement_id).first() + response = self.client.patch( + self.v3_url(f"tests/{test.id}"), {"engagement": other.id}, format="json", + ) + self.assertEqual(400, response.status_code) + + def test_delete(self): + engagement = Engagement.objects.first() + test_type = Test_Type.objects.first() + test = Test.objects.create( + engagement=engagement, test_type=test_type, + target_start=datetime.datetime(2024, 1, 1, tzinfo=datetime.UTC), + target_end=datetime.datetime(2024, 1, 2, tzinfo=datetime.UTC), + ) + response = self.client.delete(self.v3_url(f"tests/{test.id}")) + self.assertEqual(204, response.status_code) + self.assertFalse(Test.objects.filter(pk=test.id).exists()) + + +class TestApiV3TestsReplace(ApiV3TestCase): + + """PUT full-replace: TestReplace (required test_type/target_start/target_end, no engagement).""" + + def _make_test(self, **kwargs): + engagement = Engagement.objects.first() + test_type = Test_Type.objects.first() + defaults = { + "engagement": engagement, "test_type": test_type, + "target_start": datetime.datetime(2024, 1, 1, tzinfo=datetime.UTC), + "target_end": datetime.datetime(2024, 1, 2, tzinfo=datetime.UTC), + } + defaults.update(kwargs) + return Test.objects.create(**defaults), test_type + + def _base_payload(self, test_type): + return { + "test_type": test_type.id, + "target_start": "2024-01-01T00:00:00Z", + "target_end": "2024-01-02T00:00:00Z", + } + + def test_put_full_replace_resets_omitted_optionals(self): + test, test_type = self._make_test(title="original title", version="9.9") + # PUT without title / version -> both reset to their schema defaults (None). + response = self.client.put( + self.v3_url(f"tests/{test.id}"), self._base_payload(test_type), format="json", + ) + self.assertEqual(200, response.status_code, response.content[:500]) + test.refresh_from_db() + self.assertIsNone(test.title) + self.assertIsNone(test.version) + + def test_put_does_not_reassign_engagement(self): + # engagement is editable=False -> not in TestReplace; PUT never touches it (mirrors PATCH). + test, test_type = self._make_test() + original_engagement_id = test.engagement_id + response = self.client.put( + self.v3_url(f"tests/{test.id}"), self._base_payload(test_type), format="json", + ) + self.assertEqual(200, response.status_code, response.content[:500]) + test.refresh_from_db() + self.assertEqual(original_engagement_id, test.engagement_id) + + def test_put_engagement_field_is_400(self): + # Supplying engagement (immutable) is an unknown field on the replace schema -> 400. + test, test_type = self._make_test() + payload = self._base_payload(test_type) + payload["engagement"] = test.engagement_id + response = self.client.put(self.v3_url(f"tests/{test.id}"), payload, format="json") + self.assertEqual(400, response.status_code) + + def test_put_missing_required_is_400(self): + test, _ = self._make_test() + response = self.client.put(self.v3_url(f"tests/{test.id}"), {"title": "no test_type"}, format="json") + self.assertEqual(400, response.status_code) + self.assertEqual("application/problem+json", response["Content-Type"]) + + def test_put_unauthorized_is_404(self): + test, test_type = self._make_test() + limited = User.objects.create_user(username="v3_test_put_limited", password="x") # noqa: S106 + client = self.token_client(user=limited) + response = client.put(self.v3_url(f"tests/{test.id}"), self._base_payload(test_type), format="json") + self.assertEqual(404, response.status_code) + + def test_put_visible_but_not_editable_is_403(self): + test, test_type = self._make_test() + with mock.patch("dojo.test.api_v3.routes.user_has_permission", return_value=False): + response = self.client.put( + self.v3_url(f"tests/{test.id}"), self._base_payload(test_type), format="json", + ) + self.assertEqual(403, response.status_code, response.content[:300]) + + +class TestApiV3TestsRbac(ApiV3TestCase): + + def setUp(self): + super().setUp() + self.limited = User.objects.create_user(username="v3_test_limited", password="x") # noqa: S106 + self.member = Dojo_User.objects.create_user(username="v3_test_member", password="x") # noqa: S106 + self.test = Test.objects.first() + self.product = self.test.engagement.product + self.product.authorized_users.add(self.member) + + def test_unauthorized_read_is_404(self): + client = self.token_client(user=self.limited) + self.assertEqual(0, self.get_json("tests", client=client)["count"]) + self.get_json(f"tests/{self.test.id}", client=client, expected=404) + + def test_create_without_engagement_add_is_403(self): + client = self.token_client(user=self.limited) + test_type = Test_Type.objects.first() + response = client.post( + self.v3_url("tests"), + {"engagement": self.test.engagement_id, "test_type": test_type.id, + "target_start": "2024-01-01T00:00:00Z", "target_end": "2024-01-02T00:00:00Z"}, + format="json", + ) + self.assertEqual(403, response.status_code, response.content[:300]) + self.assertEqual("application/problem+json", response["Content-Type"]) + + def test_member_can_view_but_delete_is_403(self): + # OS legacy RBAC: membership grants view+edit+add; delete is staff-only (§12). + client = self.token_client(user=self.member) + self.get_json(f"tests/{self.test.id}", client=client) + response = client.delete(self.v3_url(f"tests/{self.test.id}")) + self.assertEqual(403, response.status_code, response.content[:300]) diff --git a/unittests/api_v3/test_apiv3_users.py b/unittests/api_v3/test_apiv3_users.py new file mode 100644 index 00000000000..00231e20fec --- /dev/null +++ b/unittests/api_v3/test_apiv3_users.py @@ -0,0 +1,361 @@ +"""User (Dojo_User) CRUD + RBAC + contract tests for API v3 (OS3a).""" +from __future__ import annotations + +from django.contrib.auth.models import Permission +from django.db import connection +from django.test.utils import CaptureQueriesContext + +from dojo.models import Dojo_User, Product, Product_Type, User + +from .base import ApiV3TestCase + +_SLIM_KEYS = {"id", "username", "first_name", "last_name", "email", "is_active", "is_superuser", "last_login"} +_PASSWORD = "Zx9!qWtvB2mnLP4k" + + +class TestApiV3UsersRead(ApiV3TestCase): + + def test_list_envelope_and_slim_shape(self): + body = self.get_json("users") + self.assertEqual({"count", "next", "previous", "results"}, set(body) - {"meta"}) + self.assertGreater(body["count"], 0) + self.assertEqual(_SLIM_KEYS, set(body["results"][0])) + + def test_detail_adds_heavy_fields(self): + detail = self.get_json(f"users/{self.admin.id}") + self.assertIn("is_staff", detail) + self.assertIn("date_joined", detail) + self.assertEqual("admin", detail["username"]) + + def test_detail_unknown_is_404_problem(self): + response = self.client.get(self.v3_url("users/99999999")) + self.assertEqual(404, response.status_code) + self.assertEqual("application/problem+json", response["Content-Type"]) + + def test_fields_projection(self): + row = self.get_json("users", data={"fields": "id,username"})["results"][0] + self.assertEqual({"id", "username"}, set(row)) + + +class TestApiV3UsersFilters(ApiV3TestCase): + + def test_filter_username_icontains(self): + body = self.get_json("users", data={"username__icontains": "admin"}) + self.assertGreater(body["count"], 0) + + def test_filter_is_superuser(self): + body = self.get_json("users", data={"is_superuser": "true", "limit": 250}) + for row in body["results"]: + self.assertTrue(row["is_superuser"]) + + def test_ordering_by_username(self): + User.objects.create_user(username="v3_aaa_user", password=_PASSWORD) + User.objects.create_user(username="v3_zzz_user", password=_PASSWORD) + names = [r["username"] for r in self.get_json("users", data={"o": "username", "limit": 250})["results"]] + self.assertEqual(names, sorted(names)) + + def test_unknown_filter_param_is_400(self): + self.get_json("users", data={"not_a_filter": "x"}, expected=400) + + +class TestApiV3UsersQueryCount(ApiV3TestCase): + + def _query_count(self, params: dict) -> int: + with CaptureQueriesContext(connection) as ctx: + response = self.client.get(self.v3_url("users"), params) + self.assertEqual(200, response.status_code, response.content[:500]) + return len(ctx.captured_queries) + + def test_query_count_is_independent_of_row_count(self): + Dojo_User.objects.bulk_create([Dojo_User(username=f"qcount_user_{i}") for i in range(10)]) + first = self._query_count({"limit": 250}) + Dojo_User.objects.bulk_create([Dojo_User(username=f"qcount_user_b{i}") for i in range(90)]) + second = self._query_count({"limit": 250}) + self.assertEqual(first, second, f"query count grew with rows: {first} -> {second}") + + +class TestApiV3UsersWrite(ApiV3TestCase): + + def test_create_happy_path(self): + response = self.client.post( + self.v3_url("users"), + {"username": "v3_newuser", "email": "v3new@example.com", "first_name": "New", + "last_name": "User", "password": _PASSWORD}, + format="json", + ) + self.assertEqual(201, response.status_code, response.content[:500]) + body = response.json() + self.assertEqual("v3_newuser", body["username"]) + self.assertNotIn("password", body) # write-only, never echoed + created = Dojo_User.objects.get(username="v3_newuser") + self.assertTrue(created.check_password(_PASSWORD)) + + def test_create_missing_email_is_400(self): + response = self.client.post( + self.v3_url("users"), {"username": "v3_noemail", "password": _PASSWORD}, format="json", + ) + self.assertEqual(400, response.status_code) + self.assertEqual("application/problem+json", response["Content-Type"]) + + def test_create_missing_password_is_400_when_required(self): + # REQUIRE_PASSWORD_ON_USER defaults True -> password is mandatory on create (mirrors v2). + response = self.client.post( + self.v3_url("users"), {"username": "v3_nopass", "email": "np@example.com"}, format="json", + ) + self.assertEqual(400, response.status_code) + + def test_create_unknown_field_is_400(self): + response = self.client.post( + self.v3_url("users"), + {"username": "v3_extra", "email": "e@example.com", "password": _PASSWORD, "bogus": 1}, + format="json", + ) + self.assertEqual(400, response.status_code) + + def test_create_superuser_as_superuser_ok(self): + response = self.client.post( + self.v3_url("users"), + {"username": "v3_super", "email": "s@example.com", "password": _PASSWORD, "is_superuser": True}, + format="json", + ) + self.assertEqual(201, response.status_code, response.content[:500]) + self.assertTrue(Dojo_User.objects.get(username="v3_super").is_superuser) + + def test_patch_partial_update(self): + user = User.objects.create_user(username="v3_patch_user", password=_PASSWORD) + response = self.client.patch( + self.v3_url(f"users/{user.id}"), {"first_name": "Patched"}, format="json", + ) + self.assertEqual(200, response.status_code, response.content[:500]) + user.refresh_from_db() + self.assertEqual("Patched", user.first_name) + + def test_patch_password_is_rejected(self): + user = User.objects.create_user(username="v3_pwuser", password=_PASSWORD) + response = self.client.patch( + self.v3_url(f"users/{user.id}"), {"password": _PASSWORD}, format="json", + ) + self.assertEqual(400, response.status_code) + + def test_delete_other_user(self): + user = User.objects.create_user(username="v3_del_user", password=_PASSWORD) + response = self.client.delete(self.v3_url(f"users/{user.id}")) + self.assertEqual(204, response.status_code) + self.assertFalse(Dojo_User.objects.filter(pk=user.id).exists()) + + def test_delete_self_is_rejected(self): + response = self.client.delete(self.v3_url(f"users/{self.admin.id}")) + self.assertEqual(400, response.status_code) + self.assertTrue(Dojo_User.objects.filter(pk=self.admin.id).exists()) + + +class TestApiV3UsersReplace(ApiV3TestCase): + + """PUT full-replace: reuses the create-shaped UserWrite; password never settable via update.""" + + def test_put_full_replace_resets_omitted_optionals(self): + user = User.objects.create_user(username="v3_put_user", password=_PASSWORD, first_name="Original") + # PUT without first_name -> resets to its schema default (""). + response = self.client.put( + self.v3_url(f"users/{user.id}"), + {"username": "v3_put_user", "email": "put@example.com"}, + format="json", + ) + self.assertEqual(200, response.status_code, response.content[:500]) + body = response.json() + self.assertEqual("put@example.com", body["email"]) + self.assertEqual("", body["first_name"]) # omitted from PUT -> reset to default ("") + self.assertTrue(body["is_active"]) # default True + user.refresh_from_db() + self.assertEqual("", user.first_name) + + def test_put_password_is_rejected(self): + user = User.objects.create_user(username="v3_put_pw", password=_PASSWORD) + response = self.client.put( + self.v3_url(f"users/{user.id}"), + {"username": "v3_put_pw", "email": "pw@example.com", "password": _PASSWORD}, + format="json", + ) + self.assertEqual(400, response.status_code) + self.assertEqual("application/problem+json", response["Content-Type"]) + + def test_put_missing_required_email_is_400(self): + user = User.objects.create_user(username="v3_put_noemail", password=_PASSWORD) + response = self.client.put( + self.v3_url(f"users/{user.id}"), {"username": "v3_put_noemail"}, format="json", + ) + self.assertEqual(400, response.status_code) + + def test_put_unknown_field_is_400(self): + user = User.objects.create_user(username="v3_put_extra", password=_PASSWORD) + response = self.client.put( + self.v3_url(f"users/{user.id}"), + {"username": "v3_put_extra", "email": "e@example.com", "bogus": 1}, + format="json", + ) + self.assertEqual(400, response.status_code) + + def test_put_unauthorized_is_404(self): + # A plain user's self-only queryset hides the admin -> 404 before the write gate. + limited = User.objects.create_user(username="v3_put_u_limited", password=_PASSWORD) + client = self.token_client(user=limited) + response = client.put( + self.v3_url(f"users/{self.admin.id}"), + {"username": "admin", "email": "a@example.com"}, format="json", + ) + self.assertEqual(404, response.status_code) + + def test_put_visible_but_not_editable_is_403(self): + # A plain user can see their own record (404 never fires) but lacks auth.change_user -> 403. + limited = User.objects.create_user(username="v3_put_u_self", password=_PASSWORD) + client = self.token_client(user=limited) + response = client.put( + self.v3_url(f"users/{limited.id}"), + {"username": "v3_put_u_self", "email": "self@example.com"}, format="json", + ) + self.assertEqual(403, response.status_code, response.content[:300]) + + +class TestApiV3UsersRbac(ApiV3TestCase): + + def setUp(self): + super().setUp() + self.limited = User.objects.create_user(username="v3_user_limited", password=_PASSWORD) + + # --- plain user (no view_user perm): read is SELF-ONLY, self-read guaranteed --------------- + def test_plain_user_lists_only_self(self): + client = self.token_client(user=self.limited) + body = self.get_json("users", client=client) + self.assertEqual(1, body["count"]) + self.assertEqual([self.limited.id], [r["id"] for r in body["results"]]) + + def test_plain_user_can_always_read_own_record(self): + client = self.token_client(user=self.limited) + detail = self.get_json(f"users/{self.limited.id}", client=client) + self.assertEqual(self.limited.id, detail["id"]) + + def test_plain_user_cannot_read_other_record(self): + client = self.token_client(user=self.limited) + self.get_json(f"users/{self.admin.id}", client=client, expected=404) + + # --- view_user holder: RBAC-scoped visibility (<= v2's "all users" exposure) --------------- + def test_view_user_holder_gets_scoped_visibility(self): + self.limited.user_permissions.add( + Permission.objects.get(codename="view_user", content_type__app_label="auth", content_type__model="user"), + ) + client = self.token_client(user=self.limited) + # The scoped queryset (get_authorized_users) always surfaces superusers, so admin is visible + # to a view_user holder where a plain user (above) gets 404. + self.get_json(f"users/{self.admin.id}", client=client) + + # --- superuser: sees all ------------------------------------------------------------------- + def test_superuser_sees_all(self): + other = User.objects.create_user(username="v3_user_other", password=_PASSWORD) + ids = [r["id"] for r in self.get_json("users", data={"limit": 250})["results"]] + self.assertIn(other.id, ids) + self.assertIn(self.admin.id, ids) + + # --- writes stay admin/superuser-only (unchanged) ----------------------------------------- + def test_non_superuser_cannot_create(self): + client = self.token_client(user=self.limited) + response = client.post( + self.v3_url("users"), + {"username": "v3_denied", "email": "d@example.com", "password": _PASSWORD}, + format="json", + ) + self.assertEqual(403, response.status_code, response.content[:300]) + self.assertEqual("application/problem+json", response["Content-Type"]) + + def test_non_superuser_cannot_delete_other(self): + other = User.objects.create_user(username="v3_other", password=_PASSWORD) + client = self.token_client(user=self.limited) + response = client.delete(self.v3_url(f"users/{other.id}")) + # Self-only queryset -> `other` is invisible -> 404 (never reaches the delete perm check). + self.assertEqual(404, response.status_code, response.content[:300]) + + +class TestApiV3UsersIdentityFieldAuthz(ApiV3TestCase): + + """ + Parity with PR #15191: a non-superuser delegate holding the user-management configuration + permissions (``view_user`` + ``change_user``) may reach the write path for another visible + account, but must NOT be able to change that account's identity fields (``email``/``username``) + -- changing another user's email enables account takeover via the password-reset flow. The + delegate can still edit their OWN identity, and superusers remain unrestricted. The + ``configuration_permissions`` field itself is intentionally out of the v3 write surface (§12 + OS3a), so only the identity-field half of PR #15191 has a v3 attack surface. + """ + + def setUp(self): + super().setUp() + # Non-superuser delegate: holds view_user (so co-member accounts are visible via the RBAC + # queryset) + change_user (so the update route is reachable), nothing more. + self.delegate = User.objects.create_user(username="v3_identity_delegate", password=_PASSWORD) + self.delegate.user_permissions.add( + Permission.objects.get(codename="view_user", content_type__app_label="auth", content_type__model="user"), + Permission.objects.get(codename="change_user", content_type__app_label="auth", content_type__model="user"), + ) + self.target = User.objects.create_user( + username="v3_identity_target", email="target@example.com", password=_PASSWORD, + ) + # The view_user-scoped queryset returns co-members of the caller's authorized products + # (plus superusers) -- make delegate and target co-members of one product so the delegate + # can both see itself and resolve `target` (otherwise both are 404 before the write gate). + product_type = Product_Type.objects.create(name="v3_identity_pt") + product = Product.objects.create(name="v3_identity_prod", description="d", prod_type=product_type) + product.authorized_users.add(self.delegate.pk, self.target.pk) + self.delegate_client = self.token_client(user=self.delegate) + + def test_delegate_cannot_change_another_users_email(self): + response = self.delegate_client.patch( + self.v3_url(f"users/{self.target.id}"), {"email": "attacker@evil.example"}, format="json", + ) + self.assertEqual(400, response.status_code, response.content[:500]) + self.assertEqual("application/problem+json", response["Content-Type"]) + self.target.refresh_from_db() + self.assertEqual("target@example.com", self.target.email) + + def test_delegate_cannot_change_another_users_username(self): + response = self.delegate_client.patch( + self.v3_url(f"users/{self.target.id}"), {"username": "hijacked"}, format="json", + ) + self.assertEqual(400, response.status_code, response.content[:500]) + self.target.refresh_from_db() + self.assertEqual("v3_identity_target", self.target.username) + + def test_delegate_cannot_change_another_users_email_via_put(self): + response = self.delegate_client.put( + self.v3_url(f"users/{self.target.id}"), + {"username": "v3_identity_target", "email": "attacker@evil.example"}, format="json", + ) + self.assertEqual(400, response.status_code, response.content[:500]) + self.target.refresh_from_db() + self.assertEqual("target@example.com", self.target.email) + + def test_delegate_put_unchanged_identity_is_allowed(self): + # A PUT that re-sends the target's current identity is not a change -> not blocked by the + # identity guard (it fails/passes on other grounds, but never 400s on identity). + response = self.delegate_client.put( + self.v3_url(f"users/{self.target.id}"), + {"username": "v3_identity_target", "email": "target@example.com"}, format="json", + ) + self.assertEqual(200, response.status_code, response.content[:500]) + self.target.refresh_from_db() + self.assertEqual("target@example.com", self.target.email) + + def test_delegate_can_change_own_email(self): + response = self.delegate_client.patch( + self.v3_url(f"users/{self.delegate.id}"), {"email": "mynew@example.com"}, format="json", + ) + self.assertEqual(200, response.status_code, response.content[:500]) + self.delegate.refresh_from_db() + self.assertEqual("mynew@example.com", self.delegate.email) + + def test_superuser_can_change_another_users_email(self): + # Positive control: the admin (superuser, default client) is unrestricted. + response = self.client.patch( + self.v3_url(f"users/{self.target.id}"), {"email": "changed@example.com"}, format="json", + ) + self.assertEqual(200, response.status_code, response.content[:500]) + self.target.refresh_from_db() + self.assertEqual("changed@example.com", self.target.email)