feat(BA-6628): move fragment rank to the allow list with scope defaults (#12516)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Author: jopemachine

changes/12516.feature.md

@@ -0,0 +1 @@
+Move the app config merge `rank` from `app_config_fragments` to the admin-owned `app_config_allow_list` with per-scope-type defaults (public=100, domain=200, user=300)

proposals/BEP-1052-scoped-app-config-redesign.md

@@ -19,9 +19,9 @@ each managed independently.
 
 **Reads are the hot path.** A user's effective configuration for a
 `config_name` is the **deep merge of every fragment that applies to
-them**, taken
-straight from `app_config_fragments` in `rank` order — a single-table
-query with **no joins and no permission lookup**.
+them**, taken from `app_config_fragments` ordered by the `rank` of each
+fragment's allow-list entry — one indexed join, **no permission
+lookup**.
 
 **Write authorization is set up ahead of time.** Every config name is
 **explicitly registered** in `app_config_definitions`, and `app_config_allow_list`
@@ -68,18 +68,20 @@ Three scopes cover the use cases (`public` for the pre-login shell):
 - **Explicitly registered names.** Every config name lives in
   `app_config_definitions`; fragments and allow-list entries reference it by
   foreign key. No fragment may exist for an unregistered `config_name`.
-- **Reads are join-free and unconditional.** The merge reads
-  `app_config_fragments` alone, ordering the existing fragments by
-  `rank`. No allow-list or policy is consulted at read time — the hot
-  path stays a single indexed scan.
+- **Reads are unconditional.** The merge reads `app_config_fragments`
+  joined to `app_config_allow_list` only for each fragment's `rank` — an
+  indexed `(config_name, scope_type)` join. No permission or policy is
+  evaluated at read time.
 - **Allow-list = the write gate for every fragment.** `app_config_allow_list`
   holds **one record per `(config_name, scope_type)`**; a fragment at
   that scope may be created/updated/purged **only if** the record exists
   — through the admin mutations and the regular ones alike. What sets
   admins apart is that they alone manage the allow-list (and the
   `app_config_definitions`) itself. It governs **writes only** — never reads.
-- **`rank` lives on the fragment.** A fragment's `rank` is its merge
-  priority within a `config_name`; the read merge orders fragments by it.
+- **`rank` lives on the allow-list entry.** Merge priority is an
+  admin-owned policy: it sits on the (admin-managed) allow-list entry,
+  not on the fragment — a fragment owner editing their own fragment can
+  never re-order the merge (see §2).
 - **Single source-of-truth table.** One `app_config_fragments` table
   holds every scope; only the exposure layer is split.
 
@@ -109,10 +111,11 @@ Keyed by the natural composite `(scope_type, scope_id, config_name)`
 - `scope_id` — the scope's identifier (see convention below).
 - `config_name` — FK → `app_config_definitions.config_name`.
 - `config` — schema-less JSON payload.
-- `rank` — integer merge priority within the `config_name` (low → high;
-  higher wins). Assigned on create (see §2).
 - `created_at` / `updated_at`.
 
+The fragment carries no rank of its own — its merge priority is its
+allow-list entry's `rank` (see §2).
+
 ### `app_config_allow_list` — the per-`(config_name, scope_type)` write gate
 
 One row per `(config_name, scope_type)` (unique) — a normalized,
@@ -125,12 +128,14 @@ advance.
   (`public | domain | user`). A user-overridable config carries a
   `(config_name, user)` row; an admin-only value carries
   `(config_name, domain)` and/or `(config_name, public)`.
+- `rank` — the merge priority every fragment under this entry carries
+  (low → high; higher wins). Defaults per scope type (see §2); admins
+  may set it explicitly.
 - `created_at` / `updated_at`.
 
-A row's **presence** is the entire signal — there is no `rank` here,
-because the allow-list never participates in the merge. It gates **both**
-write paths; admins, unlike users, may also create/update/purge the
-allow-list rows themselves.
+A row's **presence** is the write grant, and its `rank` is the merge
+order of its fragments. It gates **both** write paths; admins, unlike
+users, may also create/update/purge the allow-list rows themselves.
 
 ### Scope-ID convention
 
@@ -185,7 +190,8 @@ boundary.
   merged value is the admin's (`public` / `domain` fragments only).
 - **Overridable**: the admin grants `(config_name, user)`. The admin
   sets the `domain` default; the user freely creates/updates/purges
-  their own `user` fragment, which (higher `rank`) wins on merge.
+  their own `user` fragment, which wins on merge (the `user` entry's
+  default `rank` is the highest).
 - **User-only**: the grant exists and no admin fragment is published.
 
 To promote a fixed value to user-customizable, the admin adds a single
@@ -198,19 +204,25 @@ is already stored).
 
 ## 2. `rank` — merge priority
 
-`rank` is the integer priority a **fragment** carries within a
-`config_name`; the read merge applies fragments in `rank` order (low →
-high, higher wins).
-
-- **Assignment.** A new fragment is placed after the existing ones for
-  the same `config_name`, so a later-created fragment outranks earlier ones by
-  default. Admins re-order by setting `rank` explicitly. (Publishing the
-  `domain` default before a user writes their own copy naturally gives
-  the `user` fragment the higher rank, so the user value wins.)
-- **No tier defaults.** Priority is the fragment's `rank`, not derived
-  from `scope_type` — a `user` fragment outranks a `domain` fragment
-  only because its `rank` is higher, not because `user` is "above"
-  `domain`.
+`rank` is the integer priority an **allow-list entry** carries; every
+fragment written under the entry merges at that priority (low → high,
+higher wins). It lives on the allow-list — not the fragment — because
+merge order is admin policy: fragment writes are partly user-owned, and
+a rank on the fragment would let a user re-order the merge by editing
+their own row.
+
+- **Scope-type defaults.** A new entry defaults its `rank` from its
+  `scope_type`: `public` = 100, `domain` = 200, `user` = 300. The
+  defaults order the scopes so a user's own fragment overrides the
+  domain default, which overrides the public value; the 100 gap leaves
+  room for custom placement in between.
+- **Admin override.** The admin may set `rank` explicitly when creating
+  the entry — e.g. a `domain` entry ranked above `user` yields a
+  domain-enforced value that user fragments cannot beat even where a
+  user grant exists.
+- **Per-caller totality.** For one caller and one `config_name`, at most
+  one fragment applies per scope type (§3), so the entry ranks totally
+  order every merge input.
 
 ---
 
@@ -219,7 +231,7 @@ high, higher wins).
 Two read shapes exist:
 
 - **`AppConfigFragment`** — one raw row, regardless of scope. Carries
-  `scopeType`, `scopeId`, `configName`, `rank`, and `config`. Callers
+  `scopeType`, `scopeId`, `configName`, and `config`. Callers
   disambiguate scope by reading `scopeType` (no per-scope wrapper
   types).
 - **`AppConfig`** — the merged per-user view for a `config_name`: the
@@ -235,10 +247,10 @@ those whose scope applies to them:
 - the user's `user` fragment (`scope_id = the user's id`).
 
 A single `app_config_fragments` query selects exactly those rows (the
-user's domain is known from the session — **no join, no allow-list
-lookup**), orders them by `rank` (low → high), and deep-merges: nested
-objects recurse, scalars and lists are wholesale-replaced, and the
-higher `rank` wins on conflict.
+user's domain is known from the session — no permission check), joins
+each to its allow-list entry for the `rank`, orders by it (low → high),
+and deep-merges: nested objects recurse, scalars and lists are
+wholesale-replaced, and the higher `rank` wins on conflict.
 
 **Null projection.** A stored `config` of `{}` reads back as `null`, and
 a merged `config` that is empty after combining every fragment is
@@ -276,7 +288,8 @@ likewise `null` — clients fall back to their built-in defaults.
   be written only after its scope's row exists.
 - **Pre-login public config** — anonymous read of `public` `theme`.
 - **Bootstrap after login** — read merged `AppConfig`s in one round of
-  queries; each is a single-table rank merge, no per-scope stitching.
+  queries; each is a rank merge over the fragments joined to their
+  allow-list entries, no per-scope stitching.
 - **User edits a config** — a regular create/update/purge on the
   caller's own `user` row (only where the grant exists); the response is
   the recomputed merge.
@@ -289,7 +302,8 @@ likewise `null` — clients fall back to their built-in defaults.
 - **Admin locks a value back down** — remove the `(config_name, user)`
   grant **and** purge existing `user` fragments (the grant gates writes,
   not reads).
-- **Admin reorders contributions** — adjust fragment `rank`s.
+- **Admin reorders contributions** — set the allow-list entries'
+  `rank`s (per `(config_name, scope_type)`, not per fragment).
 - **Admin retires a config name** — purge the fragments, then the
   allow-list entries, then the `app_config_definitions` row (purge is rejected
   while references remain).

src/ai/backend/common/data/app_config/types.py

@@ -41,3 +41,19 @@ def to_rbac_scope_id(self, scope_id: str) -> str:
         their own ``scope_id``.
         """
         return "" if self is AppConfigScopeType.PUBLIC else scope_id
+
+    def default_rank(self) -> int:
+        """Default merge rank for an allow-list entry at this scope type (BEP-1052).
+
+        The merge applies fragments in rank order (low → high; higher wins), so the
+        defaults order the scopes as ``public`` < ``domain`` < ``user`` — a user's own
+        fragment overrides the domain default, which overrides the public value. The
+        100 gap leaves room for admins to place custom ranks in between.
+        """
+        match self:
+            case AppConfigScopeType.PUBLIC:
+                return 100
+            case AppConfigScopeType.DOMAIN:
+                return 200
+            case AppConfigScopeType.USER:
+                return 300

src/ai/backend/manager/data/app_config_allow_list/types.py

@@ -9,11 +9,16 @@
 
 @dataclass(frozen=True)
 class AppConfigAllowListData:
-    """Domain data for one app config allow-list entry — a per-``(config_name, scope_type)`` write gate."""
+    """Domain data for one app config allow-list entry — a per-``(config_name, scope_type)`` write gate.
+
+    ``rank`` is the merge priority applied to every fragment written under the entry
+    (low → high; higher wins).
+    """
 
     id: AppConfigAllowListID
     config_name: str
     scope_type: AppConfigScopeType
+    rank: int
     created_at: datetime
     updated_at: datetime
 

src/ai/backend/manager/data/app_config_fragment/types.py

@@ -16,7 +16,6 @@ class AppConfigFragmentData:
     config_name: str
     scope_type: AppConfigScopeType
     scope_id: str
-    rank: int
     config: dict[str, Any]
     created_at: datetime
     updated_at: datetime

src/ai/backend/manager/models/alembic/versions/66d0f891ed20_move_fragment_rank_to_app_config_allow_list.py

@@ -0,0 +1,78 @@
+"""move fragment rank to app_config_allow_list
+
+Move the merge-priority ``rank`` from ``app_config_fragments`` to
+``app_config_allow_list`` (BEP-1052). Fragment writes are partly user-owned, so
+a rank on the fragment would let a fragment owner re-order the merge; the
+allow-list entry is admin-owned, making it the right carrier for merge
+priority. Existing allow-list rows are backfilled with the per-scope-type
+default ranks (public=100, domain=200, user=300).
+
+Revision ID: 66d0f891ed20
+Revises: ba6615a4d2f1
+Create Date: 2026-07-03
+
+"""
+
+import sqlalchemy as sa
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision = "66d0f891ed20"
+down_revision = "ba6615a4d2f1"
+# Part of: NEXT_RELEASE_VERSION
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    # Allow-list entries carry the merge rank; backfill by scope-type default.
+    op.add_column("app_config_allow_list", sa.Column("rank", sa.Integer(), nullable=True))
+    op.execute(
+        sa.text(
+            """
+            UPDATE app_config_allow_list
+            SET rank = CASE scope_type
+                WHEN 'public' THEN 100
+                WHEN 'domain' THEN 200
+                WHEN 'user' THEN 300
+            END
+            WHERE rank IS NULL
+            """
+        )
+    )
+    op.alter_column("app_config_allow_list", "rank", nullable=False)
+    op.drop_column("app_config_fragments", "rank")
+
+
+def downgrade() -> None:
+    op.add_column("app_config_fragments", sa.Column("rank", sa.Integer(), nullable=True))
+    # Restore each fragment's rank from its allow-list entry. Fragments whose entry
+    # was removed fall back to the scope-type default; fragments sharing a
+    # ``(config_name, scope_type)`` end up with equal ranks — the pre-move
+    # per-fragment next-value ranks are not recoverable.
+    op.execute(
+        sa.text(
+            """
+            UPDATE app_config_fragments f
+            SET rank = al.rank
+            FROM app_config_allow_list al
+            WHERE al.config_name = f.config_name
+              AND al.scope_type = f.scope_type
+            """
+        )
+    )
+    op.execute(
+        sa.text(
+            """
+            UPDATE app_config_fragments
+            SET rank = CASE scope_type
+                WHEN 'public' THEN 100
+                WHEN 'domain' THEN 200
+                WHEN 'user' THEN 300
+            END
+            WHERE rank IS NULL
+            """
+        )
+    )
+    op.alter_column("app_config_fragments", "rank", nullable=False)
+    op.drop_column("app_config_allow_list", "rank")

src/ai/backend/manager/models/app_config_allow_list/orders.py

@@ -27,6 +27,12 @@ def scope_type(ascending: bool = True) -> QueryOrder:
             return AppConfigAllowListRow.scope_type.asc()
         return AppConfigAllowListRow.scope_type.desc()
 
+    @staticmethod
+    def rank(ascending: bool = True) -> QueryOrder:
+        if ascending:
+            return AppConfigAllowListRow.rank.asc()
+        return AppConfigAllowListRow.rank.desc()
+
     @staticmethod
     def created_at(ascending: bool = True) -> QueryOrder:
         if ascending:

src/ai/backend/manager/models/app_config_allow_list/row.py

@@ -19,9 +19,10 @@ class AppConfigAllowListRow(Base):  # type: ignore[misc]
     """Permission to write config fragments for one config name at one scope type.
 
     A config fragment may be created, updated, or purged only when a matching row
-    exists here; without one, the write is rejected. Reads never consult this table.
-    There is at most one row per ``(config_name, scope_type)``, and only admins
-    create or remove these rows.
+    exists here; without one, the write is rejected. ``rank`` is the merge priority
+    the entry's fragments carry (low → high; higher wins) — admin-owned here so
+    fragment owners cannot re-order the merge. There is at most one row per
+    ``(config_name, scope_type)``, and only admins create or remove these rows.
     """
 
     __tablename__ = "app_config_allow_list"
@@ -48,6 +49,11 @@ class AppConfigAllowListRow(Base):  # type: ignore[misc]
         StrEnumType(AppConfigScopeType),
         nullable=False,
     )
+    rank: Mapped[int] = mapped_column(
+        "rank",
+        sa.Integer,
+        nullable=False,
+    )
     created_at: Mapped[datetime] = mapped_column(
         "created_at",
         sa.DateTime(timezone=True),
@@ -67,6 +73,7 @@ def to_data(self) -> AppConfigAllowListData:
             id=self.id,
             config_name=self.config_name,
             scope_type=self.scope_type,
+            rank=self.rank,
             created_at=self.created_at,
             updated_at=self.updated_at,
         )

src/ai/backend/manager/models/app_config_fragment/orders.py

@@ -33,12 +33,6 @@ def scope_id(ascending: bool = True) -> QueryOrder:
             return AppConfigFragmentRow.scope_id.asc()
         return AppConfigFragmentRow.scope_id.desc()
 
-    @staticmethod
-    def rank(ascending: bool = True) -> QueryOrder:
-        if ascending:
-            return AppConfigFragmentRow.rank.asc()
-        return AppConfigFragmentRow.rank.desc()
-
     @staticmethod
     def created_at(ascending: bool = True) -> QueryOrder:
         if ascending:

src/ai/backend/manager/models/app_config_fragment/row.py

@@ -18,7 +18,11 @@
 
 
 class AppConfigFragmentRow(Base):  # type: ignore[misc]
-    """One scoped app config fragment — a single JSON document at ``(config_name, scope_type, scope_id)``."""
+    """One scoped app config fragment — a single JSON document at ``(config_name, scope_type, scope_id)``.
+
+    The fragment carries no merge priority of its own — its rank is the ``rank`` of
+    the allow-list entry for its ``(config_name, scope_type)``.
+    """
 
     __tablename__ = "app_config_fragments"
     __table_args__ = (
@@ -52,11 +56,6 @@ class AppConfigFragmentRow(Base):  # type: ignore[misc]
         sa.String(length=255),
         nullable=False,
     )
-    rank: Mapped[int] = mapped_column(
-        "rank",
-        sa.Integer,
-        nullable=False,
-    )
     config: Mapped[dict[str, Any]] = mapped_column(
         "config",
         JSONB,
@@ -82,7 +81,6 @@ def to_data(self) -> AppConfigFragmentData:
             config_name=self.config_name,
             scope_type=self.scope_type,
             scope_id=self.scope_id,
-            rank=self.rank,
             config=self.config,
             created_at=self.created_at,
             updated_at=self.updated_at,