feat(cache): NATS per-key TTL + init lock TTL on NATS by fatih-acar · Pull Request #9769 · opsmill/infrahub

fatih-acar · 2026-06-30T23:36:29Z

Summary

Builds on the Redis init-lock TTL work (this PR targets that branch). It modernizes the NATS cache backend and brings the global initialization-lock TTL safety net to NATS as well, so a worker that dies mid-initialization can no longer deadlock startup regardless of cache driver.

Key Changes

Commit 1 — feat(cache): switch NATS cache to per-key TTL

Bumps nats-py 2.7.2 → 2.15.0.
Replaces the per-bucket TTL workaround (one KV bucket per supported TTL, keys routed by prefix) with real per-message (per-key) TTL, available since NATS server 2.11.
A single KV bucket now applies the caller-supplied expiry directly via msg_ttl, so NATS finally honors the same per-key expiry Redis already did. Existing buckets are upgraded in place to enable per-message TTL.
The cache set(expires=…) value is now seconds (KVTTL members remain valid as they are integers). RedisCache aligned accordingly.
Test NATS image pinned to nats:2.11-alpine.

Commit 2 — fix(lock): enforce init lock TTL on the NATS backend

Fixes NATS-backed distributed locks: NATSLock.acquire() did not accept the token InfrahubLock passes, so acquiring any NATS lock raised a TypeError.
Global initialization locks acquired through NATS now carry the configurable TTL via the per-key cache expiry, matching the Redis behavior. A TTL'd lock is released only if we still own it, so an expired-and-reacquired lock is not clobbered.

Related Context

Follow-up to the Redis global-init-lock TTL (issue #9277). The TTL value/config (INFRAHUB_CACHE_INIT_LOCK_TTL_MINS, default 20 min) is defined on the base branch and reused here.

Test Plan

Validated against a real NATS server (nats:2.11-alpine, server v2.11.17) with the NATS cache driver:

INFRAHUB_CACHE_DRIVER=nats uv run pytest backend/tests/component/services/adapters/nats — cache per-key TTL (create + put paths), no-TTL persistence, list_keys; init lock auto-expiry, configured-TTL wiring, acquire/release cycle.
uv run pytest backend/tests/component/services/adapters/redis/test_redis.py and .../redis/test_lock_ttl.py — Redis path unchanged.
uv run pytest backend/tests/unit/test_lock.py backend/tests/unit/adapters/test_lock.py — local/Redis lock units.
ruff, ty clean on changed files.

Notes

NATS message-bus adapter API usage was checked for compatibility with the nats-py bump (publish/subscribe/add_stream/add_consumer signatures unchanged; publish gains an optional msg_ttl).
These NATS tests run in CI only when the suite is exercised with the NATS cache driver.

Summary by cubic

Switches the NATS cache to real per-key TTL and adds a configurable TTL to all global initialization locks so a crashed worker can’t block startup. Follow-up to #9277, aligning NATS with Redis behavior and defaulting TTL to 20 minutes via INFRAHUB_CACHE_INIT_LOCK_TTL_MINS.

New Features
- NATS uses per-message (per-key) TTL in a single KV bucket; existing buckets auto-upgrade to enable per-message TTL.
- Global init locks (global.init, global.taskmgr.init, global.worker.taskmgr.init) now auto-expire on both Redis and NATS; TTL is passed in seconds across caches.
- cache.set(expires=...) now takes seconds; KVTTL values still work since they’re ints. RedisCache and NATS both accept int seconds.
- Requires NATS server 2.11+; bumped nats-py to 2.15.0; tests pin nats:2.14.3-alpine.
Bug Fixes
- Fixed NATS lock acquire TypeError: NATSLock.acquire() now accepts an optional token to match InfrahubLock and keeps the timestamp::worker_id value.
- Releasing TTL’d locks is safe: no-op if the lock already expired or we no longer own it (Redis handles LockNotOwnedError; NATS checks the token).

^{Written for commit 36d0962. Summary will update on new commits.}

codspeed-hq · 2026-06-30T23:43:51Z

Merging this PR will degrade performance by 24.25%

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

❌ 2 regressed benchmarks
✅ 10 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Benchmark	`BASE`	`HEAD`	Efficiency
❌	`test_base_schema_duplicate_CoreProposedChange`	1.5 ms	2.2 ms	-32.72%
❌	`test_schemabranch_duplicate`	6.6 ms	7.7 ms	-14.71%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing fac/nats-perkey-ttl-init-locks (36d0962) with stable (0f9a929)}

cubic-dev-ai

No issues found across 12 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Shadow auto-approve: would require human review. Major dependency bump (nats-py 2.7->2.15) and core cache/lock refactor; requires human review.

Re-trigger cubic}

fatih-acar · 2026-07-08T15:56:14Z

Not an important one, we do not officially support NATS today anyway.
It was done in the background as bonus to stay consistent with the recent changes regard to init lock TTLs.

ajtmccarty · 2026-07-09T23:07:43Z

    def _tokenize_key_name(key: str) -> str:
        return key.replace(":", ".")

-    # FIXME: remove once NATS supports TTL for keys (2.11)


wow. a FIXME getting cleaned up. nice to see

personal affair to me, happy to clean that up 2 years later :D

ajtmccarty · 2026-07-09T23:10:44Z

-    async def set(self, key: str, value: str, expires: KVTTL | None = None, not_exists: bool = False) -> bool | None:
-        return await self.connection.set(name=key, value=value, ex=expires.value if expires else None, nx=not_exists)
+    async def set(self, key: str, value: str, expires: int | None = None, not_exists: bool = False) -> bool | None:
+        # int() normalizes both plain ints and KVTTL (IntEnum) members to a value redis-py can encode.


the comment makes it look like the typing should be expires: int | KVTTL | None = None, but I don't think that's right

yeah dumb agent, we should keep expires: KVTTL | None = None actually since all callers use KVTTL
good catch ty

I kept the shared set signature as expires: int | None (seconds) rather than KVTTL | None. It turns out not all callers use KVTTL: the NATS init-lock TTL passes a configurable seconds value (init_lock_ttl_mins * 60, default 1200), which isn't a KVTTL member. Since InfrahubCache.set is a single shared signature, narrowing only RedisCache to KVTTL while the base stays broader would be an LSP violation.

Existing cache callers still pass KVTTL (fine, it's an IntEnum). I removed the misleading comment; redis just coerces the IntEnum to a plain int for the driver. Happy to switch to a dedicated KVTTL-typed cache API + a separate seconds path for the lock if you'd prefer keeping expires strictly KVTTL.

Updated: typed the cache set expiry as KVTTL | int | None across the abstract and both adapters. Cache callers keep passing KVTTL; the NATS init lock passes its configurable seconds value as an int. (A single shared type is required — ty rejects a narrower KVTTL override of an int abstract as an LSP violation.)

Bump nats-py to 2.15.0 and rewrite the NATS cache to use per-message (per-key) TTL, available since NATS server 2.11. Previously the adapter maintained a separate KV bucket per supported TTL and routed keys by prefix into the matching bucket; the per-key expiry passed by callers was ignored on NATS. The adapter now uses a single KV bucket and applies the caller-supplied expiry directly via msg_ttl, bringing NATS in line with the Redis backend. Existing buckets created before per-message TTL are upgraded in place. The cache `set` expiry accepts a KVTTL or a plain int number of seconds. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

NATSLock.acquire() did not accept the token passed by InfrahubLock, so acquiring any NATS-backed distributed lock raised a TypeError. Accept the token and reuse it, so the NATS lock stores the same timestamp::worker_id value the Redis lock does (and the deadlock cleaner can parse it). With per-key TTL now available on the NATS cache, NATSLock applies the lock TTL via the cache expiry, so global initialization locks auto-expire on NATS exactly as they do on Redis. A TTL'd lock is released only if we still own it, so an expired-and-reacquired lock is not clobbered. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fatih-acar added type/feature New feature or request group/backend Issue related to the backend (API Server, Git Agent) labels Jun 30, 2026

cubic-dev-ai Bot reviewed Jun 30, 2026

View reviewed changes

fatih-acar force-pushed the fac/init-locks-ttl-ni508 branch from 3cb86ce to b118fff Compare July 1, 2026 15:40

fatih-acar force-pushed the fac/nats-perkey-ttl-init-locks branch from 9f350c8 to 74076cc Compare July 1, 2026 15:42

fatih-acar force-pushed the fac/init-locks-ttl-ni508 branch from b118fff to 0b3dfe5 Compare July 2, 2026 12:00

fatih-acar force-pushed the fac/nats-perkey-ttl-init-locks branch from 74076cc to cb948da Compare July 2, 2026 12:01

github-actions Bot added the type/documentation Improvements or additions to documentation label Jul 2, 2026

fatih-acar force-pushed the fac/nats-perkey-ttl-init-locks branch from cb948da to bf65ecd Compare July 2, 2026 21:37

github-actions Bot removed the type/documentation Improvements or additions to documentation label Jul 2, 2026

Base automatically changed from fac/init-locks-ttl-ni508 to stable July 7, 2026 08:11

fatih-acar force-pushed the fac/nats-perkey-ttl-init-locks branch from bf65ecd to ed03fb6 Compare July 8, 2026 15:21

fatih-acar marked this pull request as ready for review July 8, 2026 15:55

fatih-acar requested a review from a team as a code owner July 8, 2026 15:55

ajtmccarty approved these changes Jul 9, 2026

View reviewed changes

fatih-acar force-pushed the fac/nats-perkey-ttl-init-locks branch 2 times, most recently from 4102cb5 to 40eceb1 Compare July 10, 2026 06:58

fatih-acar and others added 2 commits July 10, 2026 07:17

fatih-acar force-pushed the fac/nats-perkey-ttl-init-locks branch from 40eceb1 to 36d0962 Compare July 10, 2026 07:18

fatih-acar merged commit afdb2f1 into stable Jul 10, 2026
62 of 63 checks passed

fatih-acar deleted the fac/nats-perkey-ttl-init-locks branch July 10, 2026 07:54

ajtmccarty mentioned this pull request Jul 14, 2026

docs(specs): IFC-2704 incremental merge regeneration: design documents #9886

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(cache): NATS per-key TTL + init lock TTL on NATS#9769

feat(cache): NATS per-key TTL + init lock TTL on NATS#9769
fatih-acar merged 2 commits into
stablefrom
fac/nats-perkey-ttl-init-locks

fatih-acar commented Jun 30, 2026 •

edited by cubic-dev-ai Bot

Loading

Uh oh!

codspeed-hq Bot commented Jun 30, 2026 •

edited

Loading

Uh oh!

cubic-dev-ai Bot left a comment

Uh oh!

fatih-acar commented Jul 8, 2026

Uh oh!

ajtmccarty Jul 9, 2026

Uh oh!

fatih-acar Jul 10, 2026

Uh oh!

ajtmccarty Jul 9, 2026

Uh oh!

fatih-acar Jul 10, 2026

Uh oh!

fatih-acar Jul 10, 2026

Uh oh!

fatih-acar Jul 10, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

fatih-acar commented Jun 30, 2026 • edited by cubic-dev-ai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Key Changes

Related Context

Test Plan

Notes

Summary by cubic

Uh oh!

codspeed-hq Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will degrade performance by 24.25%

Performance Changes

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

fatih-acar commented Jul 8, 2026

Uh oh!

ajtmccarty Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

fatih-acar Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

ajtmccarty Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

fatih-acar Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

fatih-acar Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

fatih-acar Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

fatih-acar commented Jun 30, 2026 •

edited by cubic-dev-ai Bot

Loading

codspeed-hq Bot commented Jun 30, 2026 •

edited

Loading