Skip to content

Commit 6ab572e

Browse files
Nehanthclaude
andcommitted
Trim RFC: simplify design rationale and alternatives
Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Nehanth <nehanthnarendrula@gmail.com>
1 parent 4c6388c commit 6ab572e

1 file changed

Lines changed: 2 additions & 32 deletions

File tree

rfcs/0007-scorer-presets/0007-scorer-presets.md

Lines changed: 2 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -382,13 +382,7 @@ MLflow ships three built-in preset subclasses as starting points. Each call crea
382382

383383
#### Design Rationale
384384

385-
- **Safety is in `Rag` and `Agent`** because these presets aim to be complete starting points. Most users want safety checks without composing two presets.
386-
- **Fluency is excluded from `Agent`** because agent evaluation emphasizes tool usage and task completion. Users who need it can compose: `Agent() | [Fluency()]`.
387-
- **`ConversationalAgent` excludes `ConversationalRoleAdherence`** because it requires a defined persona in the system prompt, which not all agents have.
388-
- **`RetrievalSufficiency` is excluded from `Rag`** because it requires `expected_response` or `expected_facts` (ground truth). Users who have expectations data can add it manually: `Rag() | [RetrievalSufficiency()]`.
389-
- **`Correctness` is excluded from all presets** because it requires `expectations` (ground truth) data.
390-
- **`Guidelines` and `ConversationalGuidelines` are excluded from all presets** because both require a `guidelines` constructor argument.
391-
- **Only three built-in presets** (Rag, Agent, ConversationalAgent) — these represent clear, distinct evaluation patterns. Other groupings (e.g., safety, quality) are too vague or too small to justify a built-in preset. Users can create and persist their own groupings for their specific needs.
385+
- **Only three built-in presets** (Rag, Agent, ConversationalAgent) — these represent clear, distinct evaluation patterns. Users can create and persist their own groupings for specific needs.
392386

393387
## Drawbacks
394388

@@ -400,31 +394,7 @@ MLflow ships three built-in preset subclasses as starting points. Each call crea
400394

401395
### 1. `get_preset()` function (no class)
402396

403-
Instead of a `Preset` class, provide a simple function that returns a plain list. This approach is simpler and also supports persistence and customization.
404-
405-
Usage:
406-
407-
```python
408-
from mlflow.genai.scorers import get_preset
409-
410-
# Simple usage
411-
result = mlflow.genai.evaluate(scorers=get_preset("agent"))
412-
413-
# Extending a preset
414-
scorers = get_preset("agent") + [Guidelines(name="tone", guidelines=["Be professional"])]
415-
result = mlflow.genai.evaluate(scorers=scorers)
416-
417-
# Create a custom preset and persist it
418-
register_preset("my_team_agent", scorers=[Safety(), Fluency(), my_custom_scorer])
419-
420-
# Load it later
421-
scorers = get_preset("my_team_agent")
422-
result = mlflow.genai.evaluate(scorers=scorers)
423-
```
424-
425-
**Pros:** Simpler. No validation changes needed. Returns fresh instances each call. `Literal` type gives IDE autocompletion. Going from function to class later is non-breaking. Can also support persistence by registering and loading presets by name.
426-
427-
**Cons:** No user-defined preset objects. Composition requires `+` with list concatenation. The preset concept disappears immediately -- it's just a list. No deduplication when combining presets. If this approach is preferred, the RFC can be updated to use it. This is a viable alternative if the class approach is deemed too heavy.
397+
Instead of a `Preset` class, provide a simple function that returns a plain list. Simpler to implement and can also support persistence via `register_preset()` / `get_preset()`.
428398

429399
### 2. Tag-based filtering
430400

0 commit comments

Comments
 (0)