You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: rfcs/0007-scorer-presets/0007-scorer-presets.md
+2-32Lines changed: 2 additions & 32 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -382,13 +382,7 @@ MLflow ships three built-in preset subclasses as starting points. Each call crea
382
382
383
383
#### Design Rationale
384
384
385
-
- **Safety is in `Rag` and `Agent`** because these presets aim to be complete starting points. Most users want safety checks without composing two presets.
386
-
- **Fluency is excluded from `Agent`** because agent evaluation emphasizes tool usage and task completion. Users who need it can compose: `Agent() | [Fluency()]`.
387
-
- **`ConversationalAgent` excludes `ConversationalRoleAdherence`** because it requires a defined persona in the system prompt, which not all agents have.
388
-
- **`RetrievalSufficiency` is excluded from `Rag`** because it requires `expected_response` or `expected_facts` (ground truth). Users who have expectations data can add it manually: `Rag() | [RetrievalSufficiency()]`.
389
-
- **`Correctness` is excluded from all presets** because it requires `expectations` (ground truth) data.
390
-
- **`Guidelines` and `ConversationalGuidelines` are excluded from all presets** because both require a `guidelines` constructor argument.
391
-
- **Only three built-in presets** (Rag, Agent, ConversationalAgent) — these represent clear, distinct evaluation patterns. Other groupings (e.g., safety, quality) are too vague or too small to justify a built-in preset. Users can create and persist their own groupings for their specific needs.
385
+
- **Only three built-in presets** (Rag, Agent, ConversationalAgent) — these represent clear, distinct evaluation patterns. Users can create and persist their own groupings for specific needs.
392
386
393
387
## Drawbacks
394
388
@@ -400,31 +394,7 @@ MLflow ships three built-in preset subclasses as starting points. Each call crea
400
394
401
395
### 1. `get_preset()` function (no class)
402
396
403
-
Instead of a `Preset` class, provide a simple function that returns a plain list. This approach is simpler and also supports persistence and customization.
404
-
405
-
Usage:
406
-
407
-
```python
408
-
from mlflow.genai.scorers import get_preset
409
-
410
-
# Simple usage
411
-
result = mlflow.genai.evaluate(scorers=get_preset("agent"))
**Pros:** Simpler. No validation changes needed. Returns fresh instances each call. `Literal` type gives IDE autocompletion. Going from function to class later is non-breaking. Can also support persistence by registering and loading presets by name.
426
-
427
-
**Cons:** No user-defined preset objects. Composition requires `+` with list concatenation. The preset concept disappears immediately -- it's just a list. No deduplication when combining presets. If this approach is preferred, the RFC can be updated to use it. This is a viable alternative if the class approach is deemed too heavy.
397
+
Instead of a `Preset` class, provide a simple function that returns a plain list. Simpler to implement and can also support persistence via `register_preset()` / `get_preset()`.
0 commit comments