Skip to content

feat(BA-5837): ValkeyCache for AppConfigFragment merged-view reads#11298

Draft
jopemachine wants to merge 7 commits intoBA-5832from
BA-5837
Draft

feat(BA-5837): ValkeyCache for AppConfigFragment merged-view reads#11298
jopemachine wants to merge 7 commits intoBA-5832from
BA-5837

Conversation

@jopemachine
Copy link
Copy Markdown
Member

@jopemachine jopemachine commented Apr 25, 2026

📚 Stacked PRs

This PR is part of a 10-PR stack delivering BEP-1052. Merge in order:

  1. ⬇️ chore(BA-5822): drop legacy AppConfig layer (preparation for BEP-1052) #11265chore(BA-5822): drop legacy AppConfig layer
  2. ⬇️ feat(BA-5814): AppConfigPolicy foundation (data / repository / service / adapter, bulk-aware) #11266feat(BA-5814): AppConfigPolicy foundation
  3. ⬇️ feat(BA-5815): AppConfigPolicy GraphQL #11269feat(BA-5815): AppConfigPolicy GraphQL
  4. ⬇️ feat(BA-5844): AppConfigPolicy REST v2 surface #11312feat(BA-5844): AppConfigPolicy REST v2
  5. ⬇️ feat(BA-5827): AppConfigFragment foundation (DB + repository + errors) #11282feat(BA-5827): AppConfigFragment foundation
  6. ⬇️ feat(BA-5836): AppConfigFragment service vertical (service / DTO / adapter) #11296feat(BA-5836): AppConfigFragment service vertical
  7. ⬇️ feat(BA-5829): AppConfigFragment + AppConfig GraphQL #11285feat(BA-5829): AppConfigFragment + AppConfig GraphQL
  8. ⬇️ feat(BA-5830): AppConfigFragment + AppConfig REST v2 #11286feat(BA-5830): AppConfigFragment + AppConfig REST v2
  9. ⬇️ feat(BA-5832): AppConfig v2 SDK + CLI (BEP-1052) #11295feat(BA-5832): AppConfig v2 SDK + CLI
  10. 👉 feat(BA-5837): ValkeyCache for AppConfigFragment merged-view reads #11298feat(BA-5837): ValkeyCache for AppConfigFragment merged-view reads ← you are here

CI on intermediate PRs may show test churn since each one only lands a slice of the new layer. The full picture is guaranteed to build at the tip (#11298).

Closes #11297.

Summary

Re-introduces a Valkey-backed cache layer for the BEP-1052 merged
AppConfig view, replacing the cache source that was dropped together
with the legacy app_configs table in #11265 (BA-5822). The hot path
AppConfigFragmentRepository.app_config(user_id, name) — is the
WebUI bootstrap loop (BEP-1052 §6) and was previously a DB hit per
request.

What's in here

  • repositories/app_config_fragment/cache_source/cache_source.py:
    • get_merged_config(user_id, name) cache-aside read keyed by
      app_config:merged:{user_id}:{name}
    • set_merged_config(merged, domain_name=...) writes the deep-merged
      payload + indexes the key in app_config:user_keys:{user_id} and
      the user in app_config:domain_users:{domain_name} so invalidation
      runs without SCAN
    • invalidate_for_scope(scope_type, scope_id):
      • USER: drop the user's per-name set
      • DOMAIN / DOMAIN_USER_DEFAULTS: cascade through every member
        of the domain user set
      • PUBLIC: rely on TTL (broad invalidation TBD)
  • db_source.user_domain_name(user_id) lookup so the cache layer can
    tag merged-view entries with their owning domain.
  • AppConfigFragmentRepository.app_config(...) is now cache-aside
    (hit → cached payload + fresh DB fragments; miss → DB read +
    write-through).
  • AppConfigFragmentAdminRepository.{create,update,purge} invalidate
    the affected scope after each DB write.
  • repositories.py builds the cache source from
    args.valkey_stat_client and threads it into both repos.
  • All cache calls pass through suppress_with_log so cache failures
    log a warning and fall through to the DB.

Stack

main
 └─ BA-5822 #11265  legacy AppConfig drop  (removed the original cache source)
     └─ BA-5814 #11266  AppConfigPolicy foundation
         └─ BA-5815 #11269  AppConfigPolicy GQL + REST
             └─ BA-5827 #11282  AppConfigFragment foundation
                 └─ BA-5836 #11296  AppConfigFragment service vertical
                     └─ BA-5829 #11285  AppConfigFragment + AppConfig GQL
                         └─ BA-5830 #11286  AppConfigFragment + AppConfig REST
                             └─ BA-5832 #11295  v2 SDK + CLI  ← base of this PR
                                 └─ BA-5837        ValkeyCache  ← THIS PR

Test plan

  • Manager schema build OK; cache source loads cleanly.
  • E2E via ./bai:
    • Policy + my-bulk-create populates the user fragment.
    • First app-config my-get theme populates cache.
    • Second my-get returns the same payload.
    • my app-config bulk-update invalidates → next my-get returns the
      updated payload.
  • pants check clean on the touched files.
  • CI green.

📚 Documentation preview 📚: https://sorna--11298.org.readthedocs.build/en/11298/


📚 Documentation preview 📚: https://sorna-ko--11298.org.readthedocs.build/ko/11298/

Comment thread src/ai/backend/manager/repositories/app_config_fragment/db_source/db_source.py Outdated
@jopemachine
Copy link
Copy Markdown
Member Author

Please add unit tests

jopemachine and others added 7 commits April 27, 2026 11:47
…ew reads

Re-introduces a Valkey-backed cache layer for the merged AppConfig
view, replacing the cache source that was dropped together with the
legacy `app_configs` table in #11265 (BA-5822). The hot path —
`AppConfigFragmentRepository.app_config(user_id, name)` — is the
WebUI bootstrap loop (BEP-1052 §6) and was previously a DB hit per
request.

Mirrors the legacy `AppConfigCacheSource` shape on top of the
`(user, name)`-keyed merged view:

- `cache_source/cache_source.py`:
  - `get_merged_config(user_id, name)` cache-aside read keyed by
    `app_config:merged:{user_id}:{name}`
  - `set_merged_config(merged, domain_name=...)` writes the deep-merged
    payload + indexes the cache key in `app_config:user_keys:{user_id}`
    and the user in `app_config:domain_users:{domain_name}` so
    invalidation runs without `SCAN`
  - `invalidate_for_scope(scope_type, scope_id)`:
    - `USER`: drop the user's per-name set
    - `DOMAIN` / `DOMAIN_USER_DEFAULTS`: cascade through every member
      of the domain user set
    - `PUBLIC`: rely on TTL (broad invalidation TBD)
  - `invalidate_for_user(user_id)` convenience for self-service writes
- `db_source.user_domain_name(user_id)` single-column lookup so the
  cache layer can tag merged-view entries with their owning domain
- `AppConfigFragmentRepository.app_config(...)` is now cache-aside
  (cache hit returns the cached payload + fresh DB fragments; cache
  miss writes through)
- `AppConfigFragmentAdminRepository.{create,update,purge}` invalidate
  the affected scope after the DB write
- `repositories.py` builds the cache source from
  `args.valkey_stat_client` and threads it into both repos
- All cache calls pass through `suppress_with_log` — cache failures
  log a warning and fall through to the DB
…Cache surface

Carry the stack-wide `bulk_create_my` / `bulk_update_my` →
`my_bulk_create` / `my_bulk_update` rename through the cache_source
module added by this PR, plus the GQL / adapter / DTO touchpoints that
the rebase resolved with `-X theirs` (which kept stale identifiers).

Also strip the residual `BEP-1052 §X` references from the cache_source
docstring and the merged-view repository docstring. Only the news
fragments retain the BEP wording.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…attern

The Valkey cache wrapper added in this PR re-introduced the
`(key, spec)` and `(key)` signatures on `db_source.update` /
`db_source.purge`. Switch the cache-aware admin_repository over to
the Updater/Purger pattern from BA-5827 while keeping the merged-
view cache-invalidation hooks intact.

Also rename the residual `extra_config` / `extraConfig` references in
the GQL types and schema dump to `config` for consistency with the
column rename.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The lookup is a single-column fetch helper (not an entity getter), so
the explicit `get_` prefix matches sister DB-source helpers and reads
naturally at the call site. Update the only caller in
`AppConfigFragmentRepository.app_config` accordingly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: octodog <mu001@lablup.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:docs Documentations comp:common Related to Common component comp:manager Related to Manager component size:L 100~500 LoC

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant