feat(cache): drop L2 (shared) tier for singleton settings cache; L1-only

The singleton settings cache (dojo/caching.py) becomes a single in-process L1
tier; the shared django.core.cache (L2) tier is removed. Cross-process/worker
freshness is provided entirely by resetting L1 at each request and task boundary
(middleware + DojoAsyncTask), so each request/task reads a singleton from the DB
at most once and never serves a value cached during a prior request/task.

Why no L2 for settings singletons:
- The dedupe correctness fix only needs each task to see current config, which
  the per-task L1 reset already guarantees by re-reading the DB -- a shared L2 +
  cross-process invalidation is not required for correctness.
- L2 added real cost and complexity for little gain on these tiny, hot
  singletons: a Redis dependency, pickled-vs-dict storage and version-keying,
  cross-process invalidation, and Redis-down fragility -- while the headline win
  (avoiding repeated reads within a bulk loop) comes from L1, not L2.
- Trade-off: one small indexed DB read per singleton per request/task instead of
  amortising across processes via Redis. Acceptable for low-cardinality config.

Removes SETTINGS_CACHE_L2_TTL and the L2 read/set/delete paths. DD_CACHE_URL and
the Redis default django cache backend stay (used by other framework caching).
Unit-test overrides drop the now-defunct L2 TTL var. test_caching rewritten to
the L1-only contract.

Author: valentijnscholten

docker-compose.override.unit_tests.yml

@@ -22,13 +22,11 @@ services:
       DD_DATABASE_USER: ${DD_DATABASE_USER:-defectdojo}
       DD_DATABASE_PASSWORD: ${DD_DATABASE_PASSWORD:-defectdojo}
       DD_CELERY_BROKER_URL: 'sqla+sqlite:///dojo.celerydb.sqlite'
-      # No shared cache backend in unit tests (no Redis/valkey here) -> LocMemCache.
+      # No Redis/valkey in unit tests -> default django cache is LocMemCache.
       DD_CACHE_URL: ''
-      # L1 (in-process, request/test-scoped) stays ON for deterministic
-      # assertNumQueries counts; reset per request (middleware) and per test
-      # (dojo_test_case setUp). L2 is OFF (no shared backend; would persist).
+      # In-process singleton cache (dojo/caching.py) stays ON for deterministic
+      # assertNumQueries counts; reset per request (middleware) and per test.
       DD_SETTINGS_CACHE_L1_TTL: '30'
-      DD_SETTINGS_CACHE_L2_TTL: '-1'
       DD_JIRA_EXTRA_ISSUE_TYPES: 'Vulnerability' # Shouldn't trigger a migration error
   celerybeat: !reset
   celeryworker: !reset

docker-compose.override.unit_tests_cicd.yml

@@ -23,14 +23,12 @@ services:
       DD_CELERY_BROKER_URL: 'sqla+sqlite:///dojo.celerydb.sqlite'
       DD_JIRA_EXTRA_ISSUE_TYPES: 'Vulnerability' # Shouldn't trigger a migration error
       DD_V3_FEATURE_LOCATIONS: ${DD_V3_FEATURE_LOCATIONS:-False}
-      # No shared cache backend in unit tests (no Redis/valkey here) -> LocMemCache.
+      # No Redis/valkey in unit tests -> default django cache is LocMemCache.
       DD_CACHE_URL: ''
-      # L1 (in-process, request/test-scoped) stays ON so a singleton is read once
-      # per request/test and assertNumQueries counts are deterministic; it is reset
-      # per request (middleware) and per test (dojo_test_case setUp). L2 is OFF
-      # (no shared backend here, and it would persist across tests/processes).
+      # In-process singleton cache (dojo/caching.py) stays ON: a singleton is read
+      # once per request/test (deterministic assertNumQueries), reset per request
+      # (middleware) and per test (dojo_test_case setUp).
       DD_SETTINGS_CACHE_L1_TTL: '30'
-      DD_SETTINGS_CACHE_L2_TTL: '-1'
   celerybeat: !reset
   celeryworker: !reset
   initializer: !reset

dojo/caching.py

@@ -1,32 +1,36 @@
 """
-Two-tier read-through cache for global, low-cardinality singleton config.
-
-One in-process **L1** tier sits on top of the shared **L2** tier
-(``django.core.cache``, Redis in deployments). A getter is resolved L1 → L2 → DB:
-the first tier with a value wins; a ``None`` anywhere means "not cached, compute
-it" and is never stored. This is deliberately simple — it is only for global,
-user-INDEPENDENT, signal-invalidated singletons (feature flags, system settings,
-and the like), never per-user or per-object data.
+In-process read-through cache for global, low-cardinality singleton config.
+
+A single per-thread **L1** tier resolves a getter L1 → DB: a hit wins; a ``None``
+result means "not cached, compute it" and is never stored. This is deliberately
+simple — it is only for global, user-INDEPENDENT, signal-invalidated singletons
+(feature flags, system settings, and the like), never per-user or per-object data.
+
+There is intentionally **no shared/cross-process (L2) tier**: freshness is provided
+by resetting L1 at every request and task boundary (middleware + the Celery task
+base), so each request/task reads the singleton from the DB at most once and never
+serves a value cached during a prior request/task (e.g. a since-changed
+``System_Settings``). This keeps the design free of a Redis dependency, pickled
+model graphs, and cross-process invalidation — at the cost of one DB read per
+singleton per request/task. (The default ``django.core.cache`` backend may still be
+Redis for other uses; this module no longer reads or writes it.)
 
 Values are stored as plain dicts/scalars (see ``model_to_cache_dict`` /
 ``cache_dict_to_model``), never pickled model instances.
 
-Two independent tiers, each turned off by setting its TTL to ``-1`` (both off makes
-the decorator a pass-through). Configuration (Django settings, wired from env in
-``settings.dist.py``):
+Configuration (Django setting, wired from env in ``settings.dist.py``):
 
 * ``SETTINGS_CACHE_L1_TTL`` — per-thread in-process freshness budget in seconds
-  (``-1`` disables L1). Keep it short — it bounds cross-process staleness. L1 is
-  reset at each request/task boundary, so it is effectively request/task scoped.
-* ``SETTINGS_CACHE_L2_TTL`` — L2 timeout in seconds (``-1`` disables L2).
+  (``-1`` disables the cache, making the decorator a pass-through). Keep it short.
+  L1 is reset at each request/task boundary, so it is effectively request/task
+  scoped.
 """
 
 import threading
 import time
 from functools import wraps
 
 from django.conf import settings
-from django.core.cache import cache
 
 
 class _L1Store:
@@ -88,34 +92,23 @@ def clear(self):
 
 def dojo_settings_cache(*, key: str):
     """
-    Read-through L1+L2 cache for a fixed-key singleton getter.
+    Read-through in-process (L1) cache for a fixed-key singleton getter.
 
-    Resolves L1 → L2 → wrapped function. A ``None`` result is treated as "no
-    value" and is not cached (so the next call retries). Becomes a pass-through
-    when both tiers are disabled (``SETTINGS_CACHE_L1_TTL=-1`` and
-    ``SETTINGS_CACHE_L2_TTL=-1``).
+    Resolves L1 → wrapped function. A ``None`` result is treated as "no value" and
+    is not cached (so the next call retries). Becomes a pass-through when L1 is
+    disabled (``SETTINGS_CACHE_L1_TTL=-1``). Freshness across processes comes from
+    resetting L1 each request/task (see ``reset_l1_cache``), not a shared tier.
     """
 
     def decorator(fn):
         @wraps(fn)
         def wrapper(*args, **kwargs):
-            l2_ttl = getattr(settings, "SETTINGS_CACHE_L2_TTL", 300)
-            l2_on = l2_ttl >= 0                      # SETTINGS_CACHE_L2_TTL == -1 disables L2
-
             value = _L1_STORE.get(key)              # ---- L1 ----
             if value is not None:
                 return value
 
-            if l2_on:                               # ---- L2 ----
-                value = cache.get(key)
-                if value is not None:
-                    _L1_STORE.set(key, value)
-                    return value
-
             value = fn(*args, **kwargs)             # ---- miss: compute ----
             if value is not None:
-                if l2_on:
-                    cache.set(key, value, timeout=l2_ttl)
                 _L1_STORE.set(key, value)
             return value
 
@@ -125,12 +118,12 @@ def wrapper(*args, **kwargs):
 
 
 def invalidate_dojo_settings_cache(key: str) -> None:
-    """Drop a cached singleton from L2 (all processes) and L1 (this process)."""
-    # Only touch L2 when it is enabled; when SETTINGS_CACHE_L2_TTL == -1 the L2
-    # tier is off and the backend (``cache``) may not even be reachable (e.g.
-    # unit tests run with no Redis configured), so skip the delete entirely.
-    if getattr(settings, "SETTINGS_CACHE_L2_TTL", 300) >= 0:
-        cache.delete(key)
+    """
+    Drop a cached singleton from L1 (this thread).
+
+    With no shared tier, other threads/processes self-heal at their next
+    request/task boundary (L1 reset), so there is nothing cross-process to drop.
+    """
     _L1_STORE.invalidate(key)
 
 

dojo/settings/settings.dist.py

@@ -280,16 +280,15 @@
     DD_V3_FEATURE_LOCATIONS=(bool, True),
     # Dictates if v3 org/asset relabeling (+url routing) will be enabled (on by default as of 3.0.0; set to False to restore Product/Product Type labels and URLs)
     DD_ENABLE_V3_ORGANIZATION_ASSET_RELABEL=(bool, True),
-    # Shared cache backend (django.core.cache). When set, used as the L2 tier for
-    # dojo/caching.py. MUST be a cross-process store (e.g. redis://valkey:6379/1)
-    # so cache invalidation propagates across uwsgi/celery processes; when empty
-    # Django falls back to per-process LocMemCache (single-process only).
+    # Shared cache backend (django.core.cache). When set, Django uses RedisCache
+    # (e.g. redis://valkey:6379/1); when empty it falls back to LocMemCache. Used
+    # by general framework caching; the singleton settings cache (dojo/caching.py)
+    # is in-process only and does not read or write this backend.
     DD_CACHE_URL=(str, ""),
-    # Two-tier read-through cache for global singleton getters (see dojo/caching.py).
-    # Per-thread in-process L1 freshness budget in seconds; -1 disables L1.
+    # In-process (L1) read-through cache for global singleton getters (see
+    # dojo/caching.py). Per-thread freshness budget in seconds; -1 disables it.
+    # Reset every request/task, so each request/task reads the singleton once.
     DD_SETTINGS_CACHE_L1_TTL=(int, 30),
-    # L2 (django.core.cache) timeout in seconds; -1 disables L2.
-    DD_SETTINGS_CACHE_L2_TTL=(int, 300),
     # Notification env-vars (SLA notify, alert refresh/counter/cap, system-level trump). Defined in dojo.notifications.settings.
     **NOTIFICATIONS_ENV_DEFAULTS,
 )
@@ -338,9 +337,9 @@ def generate_url(scheme, double_slashes, user, password, host, port, path, param
 # Raises django's ImproperlyConfigured exception if SECRET_KEY not in os.environ
 SECRET_KEY = env("DD_SECRET_KEY")
 
-# Shared cache backend. A cross-process store (Redis) is required for cache
-# invalidation to propagate across uwsgi/celery; otherwise Django defaults to
-# per-process LocMemCache (fine only single-process).
+# Default cache backend (django.core.cache). Redis when DD_CACHE_URL is set,
+# else per-process LocMemCache. General framework caching only; the singleton
+# settings cache (dojo/caching.py) is in-process and does not use this backend.
 if env("DD_CACHE_URL"):
     CACHES = {
         "default": {
@@ -349,9 +348,8 @@ def generate_url(scheme, double_slashes, user, password, host, port, path, param
         },
     }
 
-# Two-tier singleton cache (dojo/caching.py)
+# In-process singleton cache (dojo/caching.py)
 SETTINGS_CACHE_L1_TTL = env("DD_SETTINGS_CACHE_L1_TTL")
-SETTINGS_CACHE_L2_TTL = env("DD_SETTINGS_CACHE_L2_TTL")
 
 # Local time zone for this installation. Choices can be found here:
 # http://en.wikipedia.org/wiki/List_of_tz_zones_by_name

unittests/test_caching.py

@@ -1,8 +1,5 @@
-from unittest import mock
-
 from django.test import TestCase, override_settings
 
-from dojo import caching
 from dojo.caching import (
     _L1_STORE,  # noqa: PLC2701 test needs the in-process store
     cache_dict_to_model,
@@ -15,42 +12,20 @@
 from .dojo_test_case import DojoTestCase
 
 
-class _FakeCache:
-
-    """Dict-backed stand-in for ``django.core.cache.cache`` that counts access."""
-
-    def __init__(self):
-        self.store = {}
-        self.gets = 0
-        self.sets = 0
-        self.deletes = 0
-
-    def get(self, key, default=None):
-        self.gets += 1
-        return self.store.get(key, default)
-
-    def set(self, key, value, timeout=None):
-        self.sets += 1
-        self.store[key] = value
-
-    def delete(self, key):
-        self.deletes += 1
-        self.store.pop(key, None)
-
-
-@override_settings(SETTINGS_CACHE_L2_TTL=300, SETTINGS_CACHE_L1_TTL=30)
+@override_settings(SETTINGS_CACHE_L1_TTL=30)
 class DojoSettingsCacheTest(TestCase):
 
-    """Unit tests for the simplified L1+L2 read-through decorator (dojo/caching.py)."""
+    """
+    Unit tests for the in-process (L1) read-through decorator (dojo/caching.py).
+
+    There is no shared/L2 tier: the decorator only memoizes in-process and relies
+    on L1 reset at request/task boundaries for cross-process freshness.
+    """
 
     def setUp(self):
-        self.fake = _FakeCache()
-        self._patch = mock.patch.object(caching, "cache", self.fake)
-        self._patch.start()
         _L1_STORE.clear()
 
     def tearDown(self):
-        self._patch.stop()
         _L1_STORE.clear()
 
     def _build_getter(self, *, key="k", returns=1):
@@ -63,68 +38,40 @@ def getter():
 
         return getter, calls
 
-    @override_settings(SETTINGS_CACHE_L1_TTL=-1)
-    def test_l1_disabled_consults_l2_every_call(self):
-        getter, calls = self._build_getter()
-        self.assertEqual(getter(), 1)
-        self.assertEqual(getter(), 1)  # served from L2, fn not re-run
-        self.assertEqual(calls["n"], 1)        # computed once
-        self.assertEqual(self.fake.gets, 2)    # L2 consulted every call
-
-    def test_l1_serves_without_l2(self):
+    def test_l1_memoizes_in_process(self):
         getter, calls = self._build_getter()
         getter()
         getter()
         getter()
-        self.assertEqual(calls["n"], 1)
-        self.assertEqual(self.fake.gets, 1)    # only the first call reaches L2
+        self.assertEqual(calls["n"], 1)        # computed once, then served from L1
 
-    @override_settings(SETTINGS_CACHE_L2_TTL=-1)
-    def test_l2_ttl_negative_disables_l2(self):
-        getter, calls = self._build_getter()
-        getter()
-        getter()
-        self.assertEqual(self.fake.gets, 0)    # L2 never consulted
-        self.assertEqual(self.fake.sets, 0)
-        self.assertEqual(calls["n"], 1)        # but L1 still memoizes in-process
-
-    @override_settings(SETTINGS_CACHE_L1_TTL=-1, SETTINGS_CACHE_L2_TTL=-1)
-    def test_both_tiers_disabled_is_passthrough(self):
+    @override_settings(SETTINGS_CACHE_L1_TTL=-1)
+    def test_l1_disabled_is_passthrough(self):
         getter, calls = self._build_getter()
         getter()
         getter()
-        self.assertEqual(calls["n"], 2)        # recomputed every call
-        self.assertEqual(self.fake.gets, 0)    # L2 never consulted
+        self.assertEqual(calls["n"], 2)        # recomputed every call when L1 off
 
-    def test_reset_clears_l1(self):
+    def test_reset_recomputes(self):
         getter, calls = self._build_getter()
         getter()
         _L1_STORE.reset()
         getter()
-        self.assertEqual(calls["n"], 1)        # reset drops L1, but L2 still serves
-        self.assertEqual(self.fake.gets, 2)    # 2nd call re-reads L2 after reset
+        self.assertEqual(calls["n"], 2)        # reset drops L1, so it recomputes
 
     def test_none_result_is_not_cached(self):
         getter, calls = self._build_getter(returns=None)
         self.assertIsNone(getter())
         self.assertIsNone(getter())
         self.assertEqual(calls["n"], 2)        # None recomputed each call (never stored)
-        self.assertEqual(self.fake.sets, 0)
 
-    def test_invalidate_clears_l1_and_l2(self):
+    def test_invalidate_recomputes(self):
         getter, calls = self._build_getter(returns=7)
         self.assertEqual(getter(), 7)
         invalidate_dojo_settings_cache("k")
         getter()
         self.assertEqual(calls["n"], 2)        # recomputed after invalidation
 
-    @override_settings(SETTINGS_CACHE_L2_TTL=-1)
-    def test_invalidate_skips_l2_when_disabled(self):
-        # With L2 off the backend may be unreachable (no Redis in unit tests);
-        # invalidation must not call cache.delete.
-        invalidate_dojo_settings_cache("k")
-        self.assertEqual(self.fake.deletes, 0)
-
 
 class ModelDictRoundTripTest(DojoTestCase):
 

unittests/test_system_settings.py

@@ -93,15 +93,15 @@ def test_post_request_initializes_form_with_finding_instance(self):
         self.assertIn(response.status_code, [200, 302])
 
 
-@override_settings(SETTINGS_CACHE_L1_TTL=30, SETTINGS_CACHE_L2_TTL=-1)
+@override_settings(SETTINGS_CACHE_L1_TTL=30)
 class TestSystemSettingsMiddlewareIntegration(DojoTestCase):
 
     """
     Integration tests for DojoSettingsManagerMiddleware + System_Settings_Manager.
 
-    Caching lives in dojo.caching (decorator); the middleware only resets the
-    request-scoped L1 tier and surfaces a load error. These tests pin L1 on / L2
-    off via override_settings so they don't depend on the compose env.
+    Caching lives in dojo.caching (in-process L1 decorator); the middleware resets
+    the request-scoped L1 tier and surfaces a load error. These tests pin L1 on via
+    override_settings so they don't depend on the compose env.
     """
 
     def setUp(self):