Flatten q filter operands into a single WHERE where possible

Builds on the prior pk__in subquery rewrite. That change removed the
set-operations, but still wrapped every operand of a NOT/AND/OR in its
own pk__in subquery, so a q expression compiled to one subquery per
operand and the SQL grew a nesting level for every AND/OR/NOT -- up to 8
stacked subqueries at the complexity cap. This is the "deep nesting"
cost the reviewer asked about.

Introduce a hybrid Q-composition. Each operand can now report an
`as_q()`:

  - A leaf is Q-reducible only when it is a standard django-filter
    Filter (no custom .filter() override, no method=, no distinct=). It
    returns a plain `Q(field__lookup=value)` (negated for exclude
    filters). Opaque operands -- label filters, repository_version,
    href/prn/id resolvers, method filters, and ChoiceFilter (which
    overrides .filter() for null handling) -- return None.
  - NOT/AND/OR combine their children's Q objects with ~, &, | when all
    children are reducible.

`evaluate()` then folds every reducible operand into a single flat Q and
applies only the genuinely opaque operands as pk__in subqueries. An
all-reducible expression -- however deeply nested -- collapses to one
flat SELECT; mixed expressions keep one subquery per opaque operand.
Opaque operands are always evaluated against the small base queryset, so
they never re-embed an accumulated filter chain.

Results are unchanged and the existing q filter functional tests
(test_q_filter_valid / test_q_filter_invalid, 21 cases) pass without
modification.

Benchmarks (100k rows, steady-state, count(), best of 4), old set-ops
vs. this change; SELECTs = number of SQL SELECT statements emitted:

  Repositories (all-reducible name/number fields)
  Expression                          set-ops         this change
  ----------------------------------  --------------  --------------
  name AND name                        42 ms /  3      15 ms / 1
  (A AND B) OR (C AND D)               79 ms /  7      20 ms / 1
  (A AND (B OR C)) AND NOT D  (cx 8)  144 ms /  8      19 ms / 1
  name AND retain>=x AND retain<y      44 ms /  4      17 ms / 1
  (num range) OR num=0                124 ms /  5      15 ms / 1

  Tasks
  name AND NOT name                    68 ms /  4      17 ms / 1
  name AND (NOT state OR state)       119 ms /  6      90 ms / 5

  ContentGuards (name)
  A AND B                              56 ms /  3      19 ms / 1
  (A AND B) OR C                      155 ms /  5      20 ms / 1

Opaque-only expressions are unchanged: content repository_version
`X AND NOT Y` over 200k units stays ~2.45 s (199k pair) with no
regression, since both operands remain pk__in subqueries.

The speedup grows with nesting depth because the old plan added a
subquery per level while this one stays flat -- ~1.6x on a shallow OR up
to ~8x at the complexity-8 limit. The remaining opaque cases (e.g. task
`state`, a ChoiceFilter) could be reduced too in a follow-up.

Refs #7831

Assisted by: Claude Opus 4.8

Author: Andrew Cheng

CHANGES/7831.bugfix

@@ -1,3 +1,5 @@
-Optimized the ``q`` complex filter to compose ``NOT``/``AND``/``OR`` operands using ``pk__in``
-subqueries instead of the ``EXCEPT``/``INTERSECT``/``UNION`` set operations, avoiding full-table
-scans and disk-spilling sorts on large content sets.
+Optimized the `q` complex filter to compose `NOT`/`AND`/`OR` operands as flat `WHERE`
+conditions in a single SQL statement for standard field filters, falling back to `pk__in`
+subqueries only for opaque operands (label, `repository_version`, and href/prn/id resolvers).
+This replaces the previous `EXCEPT`/`INTERSECT`/`UNION` set operations, avoiding full-table
+scans and disk-spilling sorts, and keeps the query flat regardless of expression nesting depth.

pulpcore/filters.py

@@ -206,37 +206,100 @@ def __init__(self, filterset, tokens):
             self.value = form.cleaned_data[key]
             self.complexity = 1
 
+        def as_q(self):
+            # A leaf reduces to a plain WHERE condition (Q) only when it is a standard
+            # django-filter Filter: no custom .filter() override, no method, no .distinct().
+            # Anything else (href/prn resolvers, repository_version, method filters, ...) is
+            # opaque and must be applied via its own queryset transform in evaluate().
+            f = self.filter
+            if (
+                type(f).filter is not filters.Filter.filter
+                or getattr(f, "method", None) is not None
+                or getattr(f, "distinct", False)
+            ):
+                return None
+            if self.value in EMPTY_VALUES:
+                return models.Q()
+            q = models.Q(**{f"{f.field_name}__{f.lookup_expr}": self.value})
+            return ~q if f.exclude else q
+
         def evaluate(self, qs):
+            q = self.as_q()
+            if q is not None:
+                return qs.filter(q)
             return self.filter.filter(qs, self.value)
 
     class _NotAction:
         def __init__(self, tokens):
             self.expr = tokens[0][0]
             self.complexity = self.expr.complexity + 1
 
+        def as_q(self):
+            child = self.expr.as_q()
+            return None if child is None else ~child
+
         def evaluate(self, qs):
+            q = self.as_q()
+            if q is not None:
+                return qs.filter(q)
+            # Opaque operand: anti-join via a NOT IN subquery.
             return qs.exclude(pk__in=self.expr.evaluate(qs).values("pk"))
 
     class _AndAction:
         def __init__(self, tokens):
             self.exprs = tokens[0]
             self.complexity = sum((expr.complexity for expr in self.exprs)) + 1
 
+        def as_q(self):
+            children = [expr.as_q() for expr in self.exprs]
+            if any(child is None for child in children):
+                return None
+            query = models.Q()
+            for child in children:
+                query &= child
+            return query
+
         def evaluate(self, qs):
+            # Fold every Q-reducible operand into a single flat WHERE, and apply only the
+            # genuinely opaque operands as pk__in semi-join subqueries.
             result = qs
+            combined = models.Q()
+            has_q = False
             for expr in self.exprs:
-                result = result.filter(pk__in=expr.evaluate(qs).values("pk"))
+                child = expr.as_q()
+                if child is not None:
+                    combined &= child
+                    has_q = True
+                else:
+                    result = result.filter(pk__in=expr.evaluate(qs).values("pk"))
+            if has_q:
+                result = result.filter(combined)
             return result
 
     class _OrAction:
         def __init__(self, tokens):
             self.exprs = tokens[0]
             self.complexity = sum((expr.complexity for expr in self.exprs)) + 1
 
+        def as_q(self):
+            children = [expr.as_q() for expr in self.exprs]
+            if any(child is None for child in children):
+                return None
+            query = children[0]
+            for child in children[1:]:
+                query |= child
+            return query
+
         def evaluate(self, qs):
+            # Reducible operands stay as flat OR'd conditions; opaque operands contribute a
+            # pk__in subquery. Both are combined in a single WHERE clause.
             query = models.Q()
             for expr in self.exprs:
-                query |= models.Q(pk__in=expr.evaluate(qs).values("pk"))
+                child = expr.as_q()
+                if child is not None:
+                    query |= child
+                else:
+                    query |= models.Q(pk__in=expr.evaluate(qs).values("pk"))
             return qs.filter(query)
 
     def __init__(self, *args, **kwargs):