Feature/add missing regulations#1
Open
tejas0077 wants to merge 543 commits into
Open
Conversation
tejas0077
added
enhancement
tejas0077
force-pushed
the
feature/add-missing-regulations
branch
2 times, most recently
from
f7b6264 to
f97e936
Compare
github-actions
Bot
added
docs
docker
helm
unittests
apiv2
ui
parser
and removed
New Migration
labels
…o-DefectDojo into report-css-fix
🎉 add mozilla foundation sec advice to vulnid
…nization-playbook docs: add CLAUDE.md with module reorganization playbook
fix css overflow issue - reports
…4708) * Add OSS subscriber for Open Source Messaging banner Fetches a markdown message from the DaaS-published GCS bucket, renders the bleached headline and optional expanded section through the existing additional_banners template loop. Cached for 1h; any fetch/parse failure silently yields no banner. No Django settings introduced — disabling the banner requires forking. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Enable nl2br in expanded markdown and fold module into dojo.announcement Single newlines in the expanded body now render as <br>, so authored markdown lays out multi-line. Module folded into the existing dojo/announcement/ app and test patch paths updated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Use <button> for banner toggle and clean focus styling Anchor-based toggle picked up Bootstrap alert link styles and a lingering focus outline after click, which showed as a stray glyph next to the caret. A plain <button type="button"> avoids link decoration entirely; focus outline and transition are also dropped so the caret flips instantly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Replace DD_CLOUD_BANNER with centralized additional_banners system Migrate all promotional messaging to the additional_banners context processor pattern. Product announcements now store banners in the session for rendering via the unified template loop. Each banner carries a source field (os, product_announcement) so downstream repos can filter by origin. - Remove DD_CREATE_CLOUD_BANNER setting and env var entirely - Repurpose ProductAnnouncementManager to use session-based banners - Remove evaluate_pro_proposition celery task and beat schedule - Remove create_announcement_banner from initialization command - Simplify announcement signal to remove cloud-specific logic - Add SHOW_PLG_LINK context variable for PLG menu item control - Rename os-banner-* CSS classes to generic banner-* classes - Add data-source attribute to banner template markup - Switch OS message bucket URL from dev to prod - Add 52 tests covering context processor and product announcements Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Remove unused import and add docstring to TestBannerDictSchema * Fix ruff FURB189: use UserDict instead of dict subclass Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Release: Merge release into master from: release/2.57.2
…ctDojo#14701) * perf: bulk-apply parser-supplied per-finding tags during import finding.tags.add() per finding calls tagulous's add() which does: - reload() → SELECT current tags (1 query) - _ensure_tags_in_db() → get_or_create per tag (T queries) - super().add() → INSERT through-table rows (1 query) - tag.increment() → UPDATE count per tag (T queries) For N findings with T parser-supplied tags: O(N·T) queries. Replace with bulk_apply_parser_tags() in tag_utils, which groups findings by tag name and calls bulk_add_tags_to_instances() once per unique tag: O(unique_tags) queries regardless of N. Tags are accumulated per batch and applied just before the post_process_findings_batch task is dispatched, so deduplication and rules tasks see the tags already written to the DB. Both default_importer and default_reimporter use the same approach. For the reimporter, finding_post_processing accepts an optional tag_accumulator list; when supplied, tags are accumulated rather than applied inline (backward-compatible for any direct callers). * chore: fix ruff linting errors in bulk-tag import code * improve bulk add tags for parsers * ruff * fix tag creation * fix tests * fix tests
* Add match-only deduplication helpers and close-old queryset extraction. Expose match_batch_* and match_batch_of_findings for read-only matching. Support unsaved findings in location/endpoint comparison and _is_candidate_older. Refactor default_importer close_old_findings to use get_close_old_findings_queryset. Restore batch deduplication debug logging. * Batch-refresh close_old_findings status fields to avoid N refresh_from_db queries. Replace per-finding refresh_from_db(false_p, risk_accepted, out_of_scope) with one values() query for all PKs and assign onto instances, falling back to refresh_from_db when a row is missing. * docs: cite DefectDojo#12291 for close_old_findings status refresh origin * perf: chunk close_old_findings status sync queries (1000 PKs per SELECT) * fix(parsers): use unsaved_tags instead of tags= in Finding constructor for performance Passing tags= directly to the Finding() constructor triggers expensive tagulous processing for every finding. Using finding.unsaved_tags instead bypasses this overhead and lets the import pipeline handle tags efficiently. Affected parsers: jfrog_xray_unified, dependency_check, cargo_audit, anchore_grype, threat_composer. Benchmark on 14,219 findings: 99s -> 7.97s (12x faster). * fix: resolve ruff D203 and COM812 lint errors from formatter conflict * fix: update tests to check unsaved_tags instead of tags * fix: correct unsaved_tags assertions to expect lists and fix tag ordering Update tests for dependency_check and jfrog_xray_unified parsers to match the actual list format returned by unsaved_tags, and fix the expected order of tags for the suppressed-without-notes case in dependency_check. * fix(reimport): do not update finding tags on reimport for matched findings Tags from the report were being appended to matched findings via tags.add(), causing tags to accumulate across reimports instead of being left unchanged. This aligns tag handling with how other finding fields are treated on reimport. Closes DefectDojo#14606 --------- Co-authored-by: Cody Maffucci <46459665+Maffooch@users.noreply.github.com>
….57.2-2.58.0-dev Release: Merge back 2.57.2 into dev from: master-into-dev/2.57.2-2.58.0-dev
…x/2.57.2-2.58.0-dev Release: Merge back 2.57.2 into bugfix from: master-into-bugfix/2.57.2-2.58.0-dev
Rename title to 'Open-Source Permissions'
….14.5-slim-trixie Update python:3.14.5-slim-trixie Docker digest from 3.14.5 to 3.14.5-slim-trixie (Dockerfile.integration-tests-debian)
…-18.x Update postgres Docker tag from 18.3 to v18.4 (docker-compose.yml)
…ctions-setup-minikube-2.x Update manusa/actions-setup-minikube action from v2.16.1 to v2.18.0 (.github/workflows/k8s-tests.yml)
…es-1.33.x Update dependency kubernetes from 1.33.11 to v1.33.12 (.github/workflows/k8s-tests.yml)
Fix SARIF parser crash on empty extensions
…4881) * perf(watson): prefetch relations + force async indexing Watson's SearchAdapter resolves __-separated relation paths via per-instance getattr, triggering an N+1 query storm during async indexing. For Finding (test__engagement__product__name + jira_issue__jira_key) and Vulnerability_Id (finding__test__engagement__product__name) on a 1000-row batch this adds thousands of extra queries per task. dojo/utils_watson_prefetch.py auto-derives select_related / prefetch_related paths from each adapter's fields/store by walking model._meta, then applies them in update_watson_search_index_for_model. Toggle: DD_WATSON_INDEX_PREFETCH_ENABLED (default True). On any error we log loudly and fall back to the plain queryset so indexing still completes. Also adds force_async=True to dojo_dispatch_task / we_want_async — keeps the watson indexer in the background even when the caller is a block_execution=True user, since index updates are slow and never need to be synchronous from the user's perspective. Tests: - unittests/test_watson_index_prefetch.py (10 tests) — path classification for Product/Finding/Vulnerability_Id/Endpoint, unknown-path drop, setting toggle, derivation-raise fallback with log assertion. - unittests/test_celery_dispatch_force_async.py (4 tests) — force_async precedence over sync=True and block_execution. * test(watson): fix CI failures from watson prefetch + force_async - test_tag_inheritance_perf: update V2/V3 import baselines (-52 each) to reflect adapter-derived select_related/prefetch_related in the async watson indexer running inline under CELERY_TASK_ALWAYS_EAGER. - test_watson_async_search_index: add CELERY_TASK_ALWAYS_EAGER=True to the threshold=0 case. force_async=True now always dispatches via apply_async; without eager mode the task never runs and the index stays empty. * perf(watson): intermediate flush + always-async index dispatch Wrap watson.search_context_manager.add_to_context with a size-based hook that drains the per-request context to async celery tasks as soon as it reaches WATSON_ASYNC_INDEX_UPDATE_BATCH_SIZE, instead of waiting for end-of-request. Bounds in-memory growth on long-running imports and lets celery workers start indexing batches earlier (parallel fanout). Hook installed once in dojo.apps.ready(). BATCH_SIZE doubles as threshold; set to 0/negative to disable the intermediate flush. Drop WATSON_ASYNC_INDEX_UPDATE_THRESHOLD: index dispatch is now unconditionally async. Removes the sub-threshold sync branch (which blocked the request on _bulk_save_search_entries) and the disable-async path. Consolidate _extract_tasks_for_async + _trigger_async_index_update + _dispatch_async_index_batches + _flush_search_context_intermediate into one helper `_drain_search_context_to_async` that groups, dispatches, and discards entries from the set in place. With the set drained, watson's end() bulk-saves an empty iterator — no explicit invalidate() needed. Tests: - test_watson_intermediate_flush: new — drain dispatches + clears, threshold-triggered hook, threshold=0 disables, invalid context skips. - test_watson_async_search_index: collapse three threshold-variant tests into one, class-level CELERY_TASK_ALWAYS_EAGER=True. - test_tag_inheritance_perf: reimport no-change baselines V2 69→74, V3 87→92 (always-async path adds 5 queries vs prior sub-threshold sync branch). * upgrade notes * test(watson): query-count assertion for prefetch helper Lock in the N+1 elimination claim directly with CaptureQueriesContext — previously only observed indirectly via the ZAP import perf test. * test(watson): supply Product.description in flush hook fixtures CI runs the V3_FEATURE_LOCATIONS=True matrix where BaseModel.save calls full_clean — Product.description is blank=False, so the bare fixture ValidationErrors out. Local default (V3 off) skips validation, masking this in the prior run.
Release 2.59.0: Merge Bugfix into Dev
Release: Merge release into master from: release/2.59.0
….59.0-2.60.0-dev Release: Merge back 2.59.0 into dev from: master-into-dev/2.59.0-2.60.0-dev
….2.0 (.github/workflows/release-x-manual-docker-containers.yml)
Bumps [drf-spectacular-sidecar](https://github.com/tfranzel/drf-spectacular-sidecar) from 2026.5.1 to 2026.6.1. - [Commits](tfranzel/drf-spectacular-sidecar@2026.5.1...2026.6.1) --- updated-dependencies: - dependency-name: drf-spectacular-sidecar dependency-version: 2026.6.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
…o#14957) Bumps [django-polymorphic](https://github.com/django-commons/django-polymorphic) from 4.11.3 to 4.11.5. - [Release notes](https://github.com/django-commons/django-polymorphic/releases) - [Commits](django-commons/django-polymorphic@v4.11.3...v4.11.5) --- updated-dependencies: - dependency-name: django-polymorphic dependency-version: 4.11.5 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ectDojo#14943) Populate fix_available from firstPatchedVersion in the GraphQL response.
…tatus (DefectDojo#14611) * Check statusCategory for Jira issue status * status category is mainly used to decide if a jira issue is active or not * if the category is undefined or an unknown status, fall back to resolution checking * the resolution object was compared to a string "None", this always returned False * provide unit tests for new functionality * removed trailing whitespace * Fix JIRA helper tests to comply with JIRA API specification - Remove test_issue_from_jira_is_active_with_unknown_status_and_none_resolution - Remove test_issue_from_jira_is_active_without_status_category_with_none_string_resolution These tests checked for resolution field as string 'None', which violates JIRA API spec. According to JIRA API, resolution is either an object with properties (id, name, etc) or null, never a string value. Remaining 12 tests verify correct behavior per the API spec. * fix import due to restructured modules * Align with previous changes for 14716 * Fix linter --------- Co-authored-by: Bernhard Willert <bernhard.willert@synaos.com> Co-authored-by: valentijnscholten <valentijnscholten@gmail.com>
…0 to v7.3.1 (.github/workflows/release-drafter.yml) (DefectDojo#14948) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
…ithub/workflows/validate_docs_build.yml) (DefectDojo#14947) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
* feat(parser): scaffold Alert Logic parser package
Empty __init__.py + stub parser.py with the 4 required methods returning
placeholder values. Sets up the package for TDD tests to import against
before the real implementation in Task 8.
Authored by T. Walker - DefectDojo
* test(parser): add synthetic Alert Logic CSV fixtures
Three fixtures matching the 26-column Alert Logic vulnerability export
shape (UTF-8 BOM, embedded CRLF in multi-line fields):
- no_vuln.csv — header only, 0 data rows
- one_vuln.csv — single Medium finding (HTTP/2 Rapid Reset)
- many_vulns.csv — 7 rows covering Info / Low / Medium / High / Critical,
with/without CVE, single & multi-IP (IPv4+IPv6),
CISA Known Exploited Yes/No, multi-line Description
and Resolution, a >500-char title for truncation test,
empty CVSS and empty Operating System edge cases.
All asset names, IPs, deployment names, and the customer account are
synthetic (reserved doc IP ranges 192.0.2.x / 198.51.100.x / 203.0.113.x;
.example.com hostnames; fictional AcmeCorp account). CVE identifiers and
their associated descriptions/resolutions are from public sources.
Authored by T. Walker - DefectDojo
* test(parser): add failing TDD scaffold for Alert Logic parser
Skeleton with 4 tests: get_scan_types, parse_no_findings, parse_one_finding,
parse_many_findings. The one/many assertions fail against the Task 3 stub
(which returns []) — that's the intended TDD red state. Full field-validation
tests will be appended in Task 9 after the parser implementation lands in
Task 8.
Authored by T. Walker - DefectDojo
* feat(parser): implement Alert Logic CSV parser
Parses Alert Logic vulnerability scan CSV exports (26 columns, UTF-8 with
BOM, multi-line quoted fields). Single-format, monolithic implementation
following the IriusRisk skeleton.
Field mapping:
- Vulnerability → title (truncated at 500 chars with ellipsis)
- Severity → severity (direct 1:1 Info/Low/Medium/High/Critical)
- CVSS Score → cvssv3_score (float, None if empty)
- Asset Name → component_name
- IP Address → unsaved_endpoints (comma-split IPv4/IPv6)
- Protocol/Port → endpoint protocol + port (port 0 → omitted)
- CVE → unsaved_vulnerability_ids
- Resolution → mitigation
- Vulnerability ID → unique_id_from_tool (stable native ID)
- Description, Evidence, OS,
Vuln Span ID, Vuln Key,
Asset Key/Type, Service,
Category, VPC/Network,
Deployment Name, Customer
Account, First Seen, Last
Scanned, Published Date,
Age (days), CISA KEV → description (markdown table)
- CISA Known Exploited = Yes → unsaved_tags: ["cisa-known-exploited"]
static_finding=True, dynamic_finding=False (infrastructure vulnerability
scanner pattern, matches Qualys VMDR).
All 7 fixture findings parse cleanly with correct severities, multi-IP
endpoint extraction (IPv4+IPv6), title truncation, CVE list, CVSS score,
and tags. endpoint.clean() passes on all 10 endpoints generated from the
many_vulns fixture.
Authored by T. Walker - DefectDojo
* test(parser): add field-validation tests for Alert Logic parser
Adds 28 new tests on top of the TDD scaffold, bringing total coverage to
32 tests. Categories covered:
- Scan-type metadata: get_label, get_description
- Basic fields: title, severity, component_name, unique_id_from_tool,
cvssv3_score, static/dynamic flags, mitigation content, description
structure
- Severity mapping: one test per source level (Info/Low/Medium/High/Critical)
- Title truncation: long (>500) gets [:497] + "...", short stays as-is
- unique_id_from_tool: distinct values per finding, matches source
- Endpoints: single IPv4, multi-IP (IPv4+IPv6), IPv6-only, port=0 omission,
endpoint.clean() on every endpoint
- CVE handling: present and absent
- CISA Known Exploited tag: added on "Yes", absent on "No"
- CVSS score: parsed when present, None when empty
- BOM handling: title resolves correctly (proves UTF-8 BOM is stripped)
- Multi-line field preservation in description
All 32 tests pass against the parser implementation from the previous
commit.
Authored by T. Walker - DefectDojo
* docs(parser): add Alert Logic parser documentation
Documents the Alert Logic CSV parser including:
- File-export workflow from the Alert Logic console
- Default deduplication strategy (unique_id_from_tool + hashcode fallback)
- Complete 26-column field mapping table (expandable)
- Additional Finding field settings (static/dynamic flags, active default)
- Special processing notes covering severity conversion, title truncation,
description construction, endpoint multi-IP / IPv6 / port-zero handling,
deduplication algorithm, CVE handling, CISA Known Exploited tagging,
and UTF-8 BOM + multi-line field handling
Authored by T. Walker - DefectDojo
* feat(parser): register Alert Logic deduplication configuration
Adds Alert Logic Scan entries to:
- HASHCODE_FIELDS_PER_SCANNER with ["title", "component_name", "vuln_id_from_tool"]
(fallback when Vulnerability ID is missing on a row)
- DEDUPLICATION_ALGORITHM_PER_PARSER as DEDUPE_ALGO_UNIQUE_ID_FROM_TOOL_OR_HASH_CODE
(uses Vulnerability ID as the stable native identifier with hashcode fallback)
Mirrors the Qualys VMDR dedup pattern (same field set, same algorithm).
Authored by T. Walker - DefectDojo
* fix(parser): support V3_FEATURE_LOCATIONS in Alert Logic parser
The Endpoint model is deprecated and raises NotImplementedError when V3_FEATURE_LOCATIONS is enabled. Build LocationData URL locations in that mode and fall back to Endpoint otherwise, matching the established parser migration pattern (e.g. Qualys VMDR). Endpoint tests now read via the get_unsaved_locations helper so they pass under both settings.
Authored by T. Walker - DefectDojo
Bumps [ruff](https://github.com/astral-sh/ruff) from 0.15.14 to 0.15.15. - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md) - [Commits](astral-sh/ruff@0.15.14...0.15.15) --- updated-dependencies: - dependency-name: ruff dependency-version: 0.15.15 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…v/drf-spectacular-sidecar-2026.6.1 chore(deps): bump drf-spectacular-sidecar from 2026.5.1 to 2026.6.1
…uild-push-action-7.x chore(deps): update docker/build-push-action action from v7.1.0 to v7.2.0 (.github/workflows/release-x-manual-docker-containers.yml)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When running frequent reimports, the
dojo_test_import_finding_actiontable grows infinitely causing significant database bloat (reported cases of 19GB+). This PR adds a configurablemax_import_historysetting similar to the existingmax_dupesfeature to automatically clean up old import history records.Changes made:
max_import_historyfield toSystem_SettingsmodelDD_IMPORT_HISTORY_MAX_PER_OBJECTsetting tosettings.dist.pyasync_import_history_cleanupcelery task intasks.pyto delete oldestTest_Importrecords when a test exceeds the configured limit0262_system_settings_max_import_history.pyCloses DefectDojo#13776
Test results
Manually verified the new field appears correctly in the System_Settings model. The cleanup task follows the same pattern as the existing
async_dupe_deletetask. Unit tests to be added in a follow-up if requested by maintainers.Documentation
No documentation changes needed. The setting is self-explanatory via the help_text in the model field, consistent with how
max_dupesis documented.Checklist
dev.dev.bugfixbranch.