perf(tags): batch_mode + per-batch bulk inheritance during import (Phase B Stage 2)

Wraps the import / reimport hot loop in `tag_inheritance.batch_mode()` and
bulk-applies inherited Product tags per batch *before* `post_process_findings_batch`
dispatches, so rules / deduplication see inherited tags on `finding.tags`.

Changes:

- `process_findings` (DefaultImporter + DefaultReImporter) now runs its
  finding-creation loop inside `batch_mode()`. Per batch, right after
  `apply_import_tags_for_batch`, calls a new helper
  `apply_inherited_tags_for_findings(batch_findings)` that bulk-syncs
  inherited tags on the batch's Findings plus the Endpoints (V2) / Locations
  (V3) reachable from them via FK chain. Inheritance is therefore applied to
  the persisted children before the post-process task dispatches.
- `inherit_instance_tags` in `dojo/tags_signals.py` now early-returns when
  `tag_inheritance.is_in_batch_mode()`, so the batch wrap transparently
  suppresses per-row inheritance work for any caller — including
  `bulk_create` cleanup paths that invoke it manually. `inherit_tags_on_instance`
  post_save delegates to that helper, so the gate also covers signal-driven
  fires.
- `EndpointManager.get_or_create_endpoints` replaces its per-row
  `inherit_instance_tags(ep)` cleanup loop with a single
  `apply_inherited_tags_for_endpoints(created)` bulk call. Inside the importer
  the per-batch helper already covers these endpoints via
  `Endpoint.status_endpoint.finding`; the bulk call is kept as a defensive
  hook for any non-importer caller.
- `propagate_tags_on_product_sync` (used by the product-tag-toggle Celery
  task) gains an early-exit when neither system-wide nor per-product
  inheritance is enabled, eliminating ~9 wasted reads per call on
  inheritance-off products. State transitions (toggling either flag) still
  trigger a full sweep through their existing signal handlers.
- `Location` gains `iter_related_products()`: a related-manager
  (`self.products` + `self.findings`) implementation of `all_related_products()`
  that returns `list[Product]`. Callers that pre-issue
  `prefetch_related("products__product__tags",
  "findings__finding__test__engagement__product__tags")` get zero extra
  queries per Location. The existing JOIN'd `all_related_products()` is kept
  for the per-instance signal path where prefetching is not possible.
- `_inherited_tag_names_for_location` (the per-Location callback used to
  compute the inherited target set) switches to `iter_related_products()`;
  both call sites (`propagate_tags_on_product_sync` V3 branch and
  `apply_inherited_tags_for_findings` V3 branch) now prefetch the chain.

Query-count impact on `unittests/test_tag_inheritance_perf.py` (pins updated
in this commit):

| Hot path                                | Before  | After  | Δ     |
|-----------------------------------------|--------:|-------:|------:|
| ZAP scan import V2 (19 findings)        |    1385 |    477 |  -908 |
| ZAP scan import V3                      |    1263 |    945 |  -318 |
| ZAP reimport no-change V2               |      69 |     75 |    +6 |
| ZAP reimport no-change V3               |      87 |    102 |   +15 |
| Product tag add → 100 locations (V3)    |     316 |    125 |  -191 |
| Product tag remove → 100 locations (V3) |     266 |     75 |  -191 |

Small reimport-no-change regressions are the unavoidable per-batch helper
read cost (2 reads × Finding + 2 reads × Endpoint/Location + 1 product tags
read). Real-work imports drop significantly because per-row
`_manage_inherited_tags` work no longer fires inside the loop.

Author: valentijnscholten

dojo/importers/default_importer.py

@@ -5,6 +5,7 @@
 from django.db.models.query_utils import Q
 from django.urls import reverse
 
+from dojo import tag_inheritance
 from dojo.celery_dispatch import dojo_dispatch_task
 from dojo.finding import helper as finding_helper
 from dojo.importers.base_importer import BaseImporter, Parser
@@ -18,6 +19,7 @@
     Test_Import,
 )
 from dojo.notifications.helper import async_create_notification
+from dojo.product.helpers import apply_inherited_tags_for_findings
 from dojo.tag_utils import bulk_apply_parser_tags
 from dojo.utils import get_full_url, perform_product_grading
 from dojo.validators import clean_tags
@@ -161,6 +163,19 @@ def process_findings(
         self,
         parsed_findings: list[Finding],
         **kwargs: dict,
+    ) -> list[Finding]:
+        # Whole hot loop runs under `batch_mode()`: per-row inheritance signals
+        # for the findings/endpoints/locations created below are suppressed.
+        # Inheritance is then applied in bulk per-batch (right before
+        # `post_process_findings_batch` dispatch) so rules/dedup see inherited
+        # tags on `finding.tags`.
+        with tag_inheritance.batch_mode():
+            return self._process_findings_inner(parsed_findings, **kwargs)
+
+    def _process_findings_inner(
+        self,
+        parsed_findings: list[Finding],
+        **kwargs: dict,
     ) -> list[Finding]:
         # Batched post-processing (no chord): dispatch a task per 1000 findings or on final finding
         batch_finding_ids: list[int] = []
@@ -266,6 +281,10 @@ def process_findings(
                 findings_with_parser_tags.clear()
                 # Apply import-time tags before post-processing so rules/deduplication see them.
                 self.apply_import_tags_for_batch(batch_findings)
+                # Apply inherited Product tags to this batch's findings (and
+                # their endpoints/locations) BEFORE post_process_findings_batch
+                # dispatches, so rules/dedup see inherited tags on .tags.
+                apply_inherited_tags_for_findings(batch_findings)
                 batch_findings.clear()
                 finding_ids_batch = list(batch_finding_ids)
                 batch_finding_ids.clear()

dojo/importers/default_reimporter.py

@@ -6,6 +6,7 @@
 from django.db.models.query_utils import Q
 
 import dojo.finding.helper as finding_helper
+from dojo import tag_inheritance
 from dojo.celery_dispatch import dojo_dispatch_task
 from dojo.finding.deduplication import (
     find_candidates_for_deduplication_hash,
@@ -24,6 +25,7 @@
     Test,
     Test_Import,
 )
+from dojo.product.helpers import apply_inherited_tags_for_findings
 from dojo.tag_utils import bulk_apply_parser_tags
 from dojo.utils import perform_product_grading
 from dojo.validators import clean_tags
@@ -263,6 +265,19 @@ def process_findings(
         the finding may be appended to a new or existing group based upon user selection
         at import time
         """
+        # Whole hot loop runs under `batch_mode()`: per-row inheritance signals
+        # for the findings/endpoints/locations created below are suppressed.
+        # Inheritance is then applied in bulk per-batch (right before
+        # `post_process_findings_batch` dispatch) so rules/dedup see inherited
+        # tags on `finding.tags`.
+        with tag_inheritance.batch_mode():
+            return self._process_findings_inner(parsed_findings, **kwargs)
+
+    def _process_findings_inner(
+        self,
+        parsed_findings: list[Finding],
+        **kwargs: dict,
+    ) -> tuple[list[Finding], list[Finding], list[Finding], list[Finding]]:
         self.deduplication_algorithm = self.determine_deduplication_algorithm()
         # Only process findings with the same service value (or None)
         # Even though the service values is used in the hash_code calculation,
@@ -422,6 +437,10 @@ def process_findings(
                         findings_with_parser_tags.clear()
                         # Apply import-time tags before post-processing so rules/deduplication see them.
                         self.apply_import_tags_for_batch(batch_findings)
+                        # Apply inherited Product tags to this batch's findings (and
+                        # their endpoints/locations) BEFORE post_process_findings_batch
+                        # dispatches, so rules/dedup see inherited tags on .tags.
+                        apply_inherited_tags_for_findings(batch_findings)
                         batch_findings.clear()
                         finding_ids_batch = list(batch_finding_ids)
                         batch_finding_ids.clear()

dojo/importers/endpoint_manager.py

@@ -14,7 +14,7 @@
     Finding,
     Product,
 )
-from dojo.tags_signals import inherit_instance_tags
+from dojo.product.helpers import apply_inherited_tags_for_endpoints
 
 logger = logging.getLogger(__name__)
 
@@ -231,10 +231,16 @@ def get_or_create_endpoints(self) -> tuple[dict[EndpointUniqueKey, Endpoint], li
             if to_create:
                 created = Endpoint.objects.bulk_create(to_create, batch_size=1000)
                 endpoints_by_key.update(zip(to_create_keys, created, strict=True))
-                # bulk_create bypasses post_save signals, so manually trigger tag inheritance
-                # this is not ideal, but we need to take a separate look at the tag inheritance feature itself later
-                for ep in created:
-                    inherit_instance_tags(ep)
+                # bulk_create bypasses post_save so per-row inheritance signals never
+                # fire here. The importer hot path already covers these endpoints via
+                # the per-batch `apply_inherited_tags_for_findings` sweep (it picks
+                # them up through `Endpoint.status_finding.finding`), so this call is
+                # redundant for the importer. We keep a bulk call anyway as a defensive
+                # measure: if anything outside the importer ever bulk-creates endpoints
+                # through this manager, they still receive their inherited Product tags
+                # instead of silently missing them. The bulk helper costs ~2 queries
+                # when there's nothing to apply, vs N per-row signal fires.
+                apply_inherited_tags_for_endpoints(created)
 
         self._endpoints_to_create.clear()
         return endpoints_by_key, created

dojo/location/models.py

@@ -231,6 +231,40 @@ def all_related_products(self) -> QuerySet[Product]:
             | Q(engagement__test__finding__locations__location=self),
         ).distinct()
 
+    def iter_related_products(self) -> list[Product]:
+        """
+        Prefetch-friendly equivalent of `all_related_products()`.
+
+        Walks `self.products.all()` (LocationProductReference) and
+        `self.findings.all()` (LocationFindingReference -> Finding -> Test ->
+        Engagement -> Product) via Django related managers, so a caller that
+        already issued
+
+            Location.objects.filter(...).prefetch_related(
+                "products__product__tags",
+                "findings__finding__test__engagement__product__tags",
+            )
+
+        gets every Product (and its tags) in 0 extra queries per Location.
+
+        Use this method from bulk paths where many Locations are processed at
+        once. The original `all_related_products()` still issues a single
+        DISTINCT JOIN query and is kept for per-instance signal paths where
+        prefetching is not possible.
+        """
+        seen: set[int] = set()
+        result: list[Product] = []
+        for ref in self.products.all():
+            if ref.product_id not in seen:
+                seen.add(ref.product_id)
+                result.append(ref.product)
+        for ref in self.findings.all():
+            product = ref.finding.test.engagement.product
+            if product.id not in seen:
+                seen.add(product.id)
+                result.append(product)
+        return result
+
     def products_to_inherit_tags_from(self) -> list[Product]:
         from dojo.utils import get_system_setting  # noqa: PLC0415
         system_wide_inherit = get_system_setting("enable_product_tag_inheritance")

dojo/product/helpers.py

@@ -9,6 +9,7 @@
 from dojo.location.models import Location
 from dojo.models import Endpoint, Engagement, Finding, Product, Test
 from dojo.tag_utils import bulk_add_tag_mapping, bulk_remove_tags_from_instances
+from dojo.utils import get_system_setting
 
 logger = logging.getLogger(__name__)
 
@@ -31,52 +32,142 @@ def propagate_tags_on_product_sync(product):
     Product's current tags, and applies adds/removes via the bulk tag
     helpers. Both `tags` and `inherited_tags` fields are kept in sync.
     """
-    target_names = {tag.name for tag in product.tags.all()}
+    # Skip the full child sweep when inheritance is disabled both system-wide
+    # and on this product. Without this gate the importer hot path pays ~9
+    # queries per scan (one product-tags read + one list/through-table read per
+    # child kind) even when no inheritance work is possible. State transitions
+    # (toggling the flag on/off) still trigger a full sweep via the m2m_changed
+    # handler on `Product.tags.through` and the per-product flag save handler.
+    if not (product.enable_product_tag_inheritance or get_system_setting("enable_product_tag_inheritance")):
+        return
+    inherited_tag_names = {tag.name for tag in product.tags.all()}
 
     logger.debug("Propagating tags from %s to all engagements", product)
     _sync_inheritance_for_qs(
         Engagement.objects.filter(product=product),
-        target_names_per_child=lambda _child: target_names,
+        target_names_per_child=lambda _child: inherited_tag_names,
     )
     logger.debug("Propagating tags from %s to all tests", product)
     _sync_inheritance_for_qs(
         Test.objects.filter(engagement__product=product),
-        target_names_per_child=lambda _child: target_names,
+        target_names_per_child=lambda _child: inherited_tag_names,
     )
     logger.debug("Propagating tags from %s to all findings", product)
     _sync_inheritance_for_qs(
         Finding.objects.filter(test__engagement__product=product),
-        target_names_per_child=lambda _child: target_names,
+        target_names_per_child=lambda _child: inherited_tag_names,
     )
     if settings.V3_FEATURE_LOCATIONS:
         logger.debug("Propagating tags from %s to all locations", product)
         location_qs = Location.objects.filter(
             Q(products__product=product)
             | Q(findings__finding__test__engagement__product=product),
-        ).distinct()
+        ).distinct().prefetch_related(*_LOCATION_PREFETCH_FOR_INHERITANCE)
         # Locations can be linked to multiple products, so the inherited target
         # is the union of every related product's tags. Compute per-location.
         _sync_inheritance_for_qs(
             location_qs,
-            target_names_per_child=_location_target_names,
+            target_names_per_child=_inherited_tag_names_for_location,
         )
     else:
         logger.debug("Propagating tags from %s to all endpoints", product)
         _sync_inheritance_for_qs(
             Endpoint.objects.filter(product=product),
-            target_names_per_child=lambda _child: target_names,
+            target_names_per_child=lambda _child: inherited_tag_names,
         )
 
 
-def _location_target_names(location):
+def apply_inherited_tags_for_endpoints(endpoints):
+    """
+    Bulk inheritance for a list of Endpoints, e.g. those just created via
+    `Endpoint.objects.bulk_create` (which bypasses post_save signals).
+
+    All endpoints are assumed to share a single Product — true for the
+    importer's `EndpointManager`, which is per-product. If callers ever
+    mix products, split the list before calling.
+    """
+    if not endpoints:
+        return
+    product = endpoints[0].product
+    if not (product.enable_product_tag_inheritance or get_system_setting("enable_product_tag_inheritance")):
+        return
+    inherited_tag_names = {tag.name for tag in product.tags.all()}
+    _sync_inheritance_for_qs(
+        Endpoint.objects.filter(id__in=[e.pk for e in endpoints]),
+        target_names_per_child=lambda _child: inherited_tag_names,
+    )
+
+
+def apply_inherited_tags_for_findings(findings):
+    """
+    Per-batch bulk inheritance for findings created during an import.
+
+    Apply the owning Product's inherited tags to the given findings plus the
+    Endpoints (V2) / Locations (V3) reachable from them. Called from the
+    importer hot path right before each batch dispatches to
+    `post_process_findings_batch` so rules / deduplication see inherited tags
+    on `finding.tags`.
+
+    Test and Engagement inheritance is handled by their own post_save handlers
+    (those run outside the importer's `batch_mode()`, so per-instance signal
+    work fires normally and applies inheritance on create).
+    """
+    if not findings:
+        return
+    # Single-product invariant inside one importer call. Smart upload calls
+    # this per-product so the assumption holds there too.
+    product = findings[0].test.engagement.product
+    if not (product.enable_product_tag_inheritance or get_system_setting("enable_product_tag_inheritance")):
+        return
+    inherited_tag_names = {tag.name for tag in product.tags.all()}
+    finding_ids = [f.pk for f in findings]
+
+    _sync_inheritance_for_qs(
+        Finding.objects.filter(id__in=finding_ids),
+        target_names_per_child=lambda _child: inherited_tag_names,
+    )
+    if settings.V3_FEATURE_LOCATIONS:
+        _sync_inheritance_for_qs(
+            Location.objects.filter(findings__finding_id__in=finding_ids).distinct().prefetch_related(*_LOCATION_PREFETCH_FOR_INHERITANCE),
+            target_names_per_child=_inherited_tag_names_for_location,
+        )
+    else:
+        _sync_inheritance_for_qs(
+            Endpoint.objects.filter(status_endpoint__finding_id__in=finding_ids).distinct(),
+            target_names_per_child=lambda _child: inherited_tag_names,
+        )
+
+
+def _inherited_tag_names_for_location(location):
+    """
+    Compute the tag-name set this Location should have as `inherited_tags`.
+
+    Unlike Finding / Test / Engagement / Endpoint (each owned by exactly one
+    Product), a Location can be attached to multiple Products — directly via
+    `LocationProductReference` or indirectly via `LocationFindingReference`
+    -> Finding -> Test -> Engagement -> Product. The target inherited set is
+    therefore the UNION of every related Product's tags.
+
+    Used as the `target_names_per_child` callback for `_sync_inheritance_for_qs`
+    on Location querysets; it must be called per Location because each Location
+    has its own set of related Products. Uses `iter_related_products()` so
+    that an upstream `prefetch_related(...)` reduces per-call cost to 0
+    queries.
+    """
     names: set[str] = set()
-    for related_product in location.all_related_products():
+    for related_product in location.iter_related_products():
         if related_product is None:
             continue
         names.update(tag.name for tag in related_product.tags.all())
     return names
 
 
+_LOCATION_PREFETCH_FOR_INHERITANCE = (
+    "products__product__tags",
+    "findings__finding__test__engagement__product__tags",
+)
+
+
 def _sync_inheritance_for_qs(queryset, *, target_names_per_child):
     """
     Sync inherited_tags + tags for every child in `queryset` to its target tag set.

dojo/tags_signals.py

@@ -48,6 +48,12 @@ def make_inherited_tags_sticky(sender, instance, action, **kwargs):
 
 def inherit_instance_tags(instance):
     """Usually nothing to do when saving a model, except for new models?"""
+    # Suppress per-instance inheritance work inside an active batch. The
+    # caller (signal handler or bulk_create cleanup) need not know about
+    # batch_mode; whoever opened the batch is responsible for the bulk
+    # apply at exit.
+    if tag_inheritance.is_in_batch_mode():
+        return
     if inherit_product_tags(instance):
         # TODO: Is this change OK to make?
         # tag_list = instance._tags_tagulous.get_tag_list()
@@ -70,6 +76,7 @@ def inherit_tags_on_instance(sender, instance, created, **kwargs):
     # (create OR update), repeatedly re-applying inherited tags to children
     # whose tag state had not changed. Sticky enforcement on user-driven
     # tag edits is handled by `make_inherited_tags_sticky` (m2m_changed).
+    # `inherit_instance_tags` itself early-returns when a batch is active.
     if not created:
         return
     inherit_instance_tags(instance)

unittests/test_tag_inheritance_perf.py

@@ -379,13 +379,14 @@ def test_baseline_product_tag_remove_propagates_to_100_locations_v3(self):
     EXPECTED_PRODUCT_TAG_REMOVE_100_ENDPOINTS = 53
 
     # V3 location paths. Pre-Phase-A: 4532 add, 4307 remove.
-    EXPECTED_PRODUCT_TAG_ADD_100_LOCATIONS = 316
-    EXPECTED_PRODUCT_TAG_REMOVE_100_LOCATIONS = 266
+    EXPECTED_PRODUCT_TAG_ADD_100_LOCATIONS = 125
+    EXPECTED_PRODUCT_TAG_REMOVE_100_LOCATIONS = 75
 
 
 @override_settings(
     CELERY_TASK_ALWAYS_EAGER=True,
     CELERY_TASK_EAGER_PROPAGATES=True,
+    SECURE_SSL_REDIRECT=False,
 )
 @versioned_fixtures
 class TagInheritanceImportPerfBaselines(DojoAPITestCase):
@@ -497,7 +498,7 @@ def test_baseline_zap_scan_reimport_no_change_v3(self):
     # import path because the previous process-global signal-disconnect was
     # narrower in scope (Location.tags.through only). Net-positive trade for
     # eliminating the threading bug; full Phase B reductions land in Stage 2.
-    EXPECTED_ZAP_IMPORT_V2 = 1385
-    EXPECTED_ZAP_IMPORT_V3 = 1263
-    EXPECTED_ZAP_REIMPORT_NO_CHANGE_V2 = 69
-    EXPECTED_ZAP_REIMPORT_NO_CHANGE_V3 = 87
+    EXPECTED_ZAP_IMPORT_V2 = 477
+    EXPECTED_ZAP_IMPORT_V3 = 945
+    EXPECTED_ZAP_REIMPORT_NO_CHANGE_V2 = 75
+    EXPECTED_ZAP_REIMPORT_NO_CHANGE_V3 = 102