Skip to content

docs(experiments): note plan follow-ups #3494

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
markijbema merged 1 commit into from
May 27, 2026
Merged

docs(experiments): note plan follow-ups #3494

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions .plans/experimental-models-1.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,8 @@ Existing targeted coverage includes:

Experiment routing rewrites outbound requests to the selected variant `upstream.internal_id`, but direct partner responses can still echo that id. Implement response rewriting so clients receive the requested public model id in every response shape.

Optional consideration: evaluate whether experiment responses should also be retained for analysis or partner evaluation. This is not required by the current spec and should be designed separately from response rewriting, including explicit policy decisions for sensitive data handling, retention, wipe behavior, storage location, truncation, and admin/provider access.

Implementation target:

- Reuse or generalize existing free-model response rewriters in `apps/web/src/lib/ai-gateway/providers/openrouter/responses.ts` and sibling response helpers.
Expand All @@ -108,6 +110,17 @@ Only add these when a concrete consumer exists:
- `model_experiment_request_stats` reporting view.
- Analytics Engine dimensions for experiment, variant, or variant version.

### Experiment Model Properties

Add admin support for defining model properties for the experimented public model id, such as context window, supported request/response capabilities, pricing/display metadata, and any other fields needed by clients or routing logic. These properties are not currently configurable through the experiment workflow, so preview ids cannot fully describe model behavior independently of their upstream variants.

Implementation target:

- Decide whether model properties belong on the experiment, the public model id, or a separate model-metadata record referenced by the experiment.
- Ensure clients can discover the experimented model's effective context window and capabilities before sending requests.
- Keep variant upstream configuration separate from client-facing model properties unless a field is intentionally variant-specific.
- Validate that configured properties match the request kinds and provider APIs supported by all active variants.

### Prompt Retention Operations

Only add these when retention automation is prioritized:
Expand Down
Loading