Skip to content

Commit 6bcbc90

Browse files
authored
docs(experiments): note plan follow-ups (#3494)
1 parent a819168 commit 6bcbc90

1 file changed

Lines changed: 13 additions & 0 deletions

File tree

.plans/experimental-models-1.md

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,8 @@ Existing targeted coverage includes:
8888

8989
Experiment routing rewrites outbound requests to the selected variant `upstream.internal_id`, but direct partner responses can still echo that id. Implement response rewriting so clients receive the requested public model id in every response shape.
9090

91+
Optional consideration: evaluate whether experiment responses should also be retained for analysis or partner evaluation. This is not required by the current spec and should be designed separately from response rewriting, including explicit policy decisions for sensitive data handling, retention, wipe behavior, storage location, truncation, and admin/provider access.
92+
9193
Implementation target:
9294

9395
- Reuse or generalize existing free-model response rewriters in `apps/web/src/lib/ai-gateway/providers/openrouter/responses.ts` and sibling response helpers.
@@ -108,6 +110,17 @@ Only add these when a concrete consumer exists:
108110
- `model_experiment_request_stats` reporting view.
109111
- Analytics Engine dimensions for experiment, variant, or variant version.
110112

113+
### Experiment Model Properties
114+
115+
Add admin support for defining model properties for the experimented public model id, such as context window, supported request/response capabilities, pricing/display metadata, and any other fields needed by clients or routing logic. These properties are not currently configurable through the experiment workflow, so preview ids cannot fully describe model behavior independently of their upstream variants.
116+
117+
Implementation target:
118+
119+
- Decide whether model properties belong on the experiment, the public model id, or a separate model-metadata record referenced by the experiment.
120+
- Ensure clients can discover the experimented model's effective context window and capabilities before sending requests.
121+
- Keep variant upstream configuration separate from client-facing model properties unless a field is intentionally variant-specific.
122+
- Validate that configured properties match the request kinds and provider APIs supported by all active variants.
123+
111124
### Prompt Retention Operations
112125

113126
Only add these when retention automation is prioritized:

0 commit comments

Comments
 (0)