Add RFC 0007: Scorer Presets for Common Evaluation Patterns by Nehanth · Pull Request #13 · mlflow/rfcs

Nehanth · 2026-04-28T14:50:23Z

Summary

This RFC proposes a Preset class for MLflow that packages a named, immutable collection of scorers for common evaluation patterns. MLflow ships five built-in presets (RAG, AGENT, CONVERSATIONAL_AGENT, SAFETY, QUALITY), and users can define their own.

Presets can be passed directly in the scorers list alongside individual scorers:

from mlflow.genai.scorers import AGENT, Guidelines

mlflow.genai.evaluate(
    data=eval_dataset,
    scorers=[AGENT, Guidelines(name="tone", guidelines=["Be professional"])],
)

Duplicate scorers across presets are automatically deduplicated during validation.

Based on mlflow/mlflow#21445.

Also introduces a new scorer proposal: TaskSuccess — a ground-truth-free scorer for evaluating agent result quality.

Co-Authored-By: Claude noreply@anthropic.com

Proposes a Preset class that packages a named collection of scorers for common evaluation patterns (RAG, agent, conversational-agent, safety, quality). Presets can be passed directly in the scorers list alongside individual scorers, with automatic deduplication. Based on mlflow/mlflow#21445. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Nehanth <nehanthnarendrula@gmail.com>

Each built-in preset is now a subclass (Agent, Rag, ConversationalAgent, SafetyPreset, Quality) that creates fresh scorer instances on each call. Eliminates shared mutable state and enables future preset-specific configuration. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Nehanth <nehanthnarendrula@gmail.com>

Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Nehanth <nehanthnarendrula@gmail.com>

mprahl · 2026-05-05T21:16:20Z

+        return len(self._scorers)
+
+    def __add__(self, other):
+        if isinstance(other, (Preset, list)):


Does it make sense to deduplicate scorers when combining presets especially since some have overlap? Edit: I see this is addressed below. It might be good to mention it here too.

Yes, I've added deduplication in add so duplicates are removed when presets are combined.

mprahl · 2026-05-05T21:19:31Z

+`validate_scorers()` deduplicates by scorer type after flattening:
+
+```python
+def _deduplicate_scorers(scorers: list[Scorer]) -> list[Scorer]:


We should keep Preset deduplicated whenever scorers are added instead of relying on validate_scorers to do it IMO.

I think we should have both, add (I added the deduplication there as well) handles deduplication when a user combines presets or scorers using the + operator, and validate_scorers() handles it when multiple presets are passed directly in the scorers list.

mprahl · 2026-05-05T21:22:12Z

@@ -0,0 +1,682 @@
+---


This RFC has too many implementation details. The RFC should be scoped to requirements, APIs, database schema changes, etc. This is delving into PR level implementation.

Can we keep this scoped to the desired UX, the proposed groupings, and the class definition?

Trimmed it down!

mprahl · 2026-05-05T21:26:39Z

+- **Type checking.** `isinstance(preset, Agent)` works — code can distinguish which preset is being used.
+- **Custom control flow.** Each preset can override methods for preset-specific validation or behavior.
+
+### Deduplication


How does deduplication account for scorers with different configurations (e.g. ConversationalGuidelines with separate configs)?

Deduplication works by (type(scorer), scorer.name), so if you use different names for your
ConversationalGuidelines scorers, they'll be kept. If two scorers have the same type and name, one gets removed.

mprahl · 2026-05-05T21:28:33Z

+
+MLflow ships five built-in preset subclasses. Each call creates fresh scorer instances.
+
+> **Note:** **`TaskSuccess`** is a new scorer proposed in [mlflow/mlflow#22972](https://github.com/mlflow/mlflow/issues/22972). It evaluates whether an agent successfully accomplished the user's task without requiring ground truth data — unlike `Correctness`, which requires an `expectations` column. This scorer would be added to `Agent`, `ConversationalAgent`, and `Quality`. This work can be part of this RFC or be a future addition after this RFC is completed.


Is this relevant for this RFC? It seems out of scope.

You're right, it's out of scope for this RFC. I'll remove it. I did create a separate issue for it mlflow#22972, once it's implemented it would be added to the Agent, ConversationalAgent, and Quality presets.comm

mprahl · 2026-05-05T21:30:08Z

+
+### Built-in Presets as Subclasses
+
+Each built-in preset is a subclass of `Preset` that hardcodes its scorer list. This means each call creates **fresh scorer instances** (no shared mutable singletons) and opens the door for preset-specific configuration and control flow in the future.


I don't really see a benefit to these presets if you can't configure the scorers. If it's merely to categorize different scorers, there are better ways to do this such as documentation or a function like get_agent_scorers() which returns the classes for discoverability.

The reason I’m going in this direction is because, after this RFC is approved, subclassing opens up more than just categorization. For example, each preset could accept different judge models (e.g, Agent(model="openai:/gpt-4o")), different inference parameters, or other preset-specific configurations.

If this RFC gets approved, I can also update it to use the function-based approach I proposed in the alternatives section, if that’s preferred.

I agree with Matt about customization capability. The class-based approach is more ergonomic to me but I think it should come with a good customizability and persistence.

mprahl · 2026-05-05T21:32:56Z

+    def __init__(self):
+        super().__init__("rag", [
+            RetrievalRelevance(),
+            RetrievalSufficiency(),


Doesn't this require expected_response or expected_facts?

Yes, RetrievalSufficiency does require ground truth, so I'm removing it from this preset. We want presets to work out of the box without requiring ground truth for easy evaluation.

- Add deduplication to Preset __init__ and __add__ - Remove TaskSuccess from presets (out of scope) - Remove RetrievalSufficiency from Rag preset (requires ground truth) - Trim implementation details per reviewer feedback Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Nehanth <nehanthnarendrula@gmail.com>

Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Nehanth <nehanthnarendrula@gmail.com>

Nehanth · 2026-05-06T20:54:14Z

Hey @mprahl, thanks for taking the time to review my RFC proposal. I've updated the RFC per you feedback commented back one some of the comments you made.

mprahl · 2026-05-18T20:13:43Z

+    def __len__(self):
+        return len(self._scorers)
+
+    def __add__(self, other):


Do you think it'd be less surprising to implement __or__ for "set" union behavior since you are deduplicating?

mprahl

I’m not convinced this should be a feature in its current form. Without a way to configure or cleanly override scorers inside a preset, the value seems limited to boilerplate reduction. At the same time, if built-in preset membership changes across releases, users will get silent behavior and cost changes on upgrade. That combination makes presets feel more like unstable convenience sugar than a strong API surface.

B-Step62

Thank you for the RFC submission @Nehanth!

The idea of bundling multiple scorers into sharable preset sounds very useful to me. My main feedback is to support persistence of presets in MLflow server and provide great customization UX. I can see built-in presets are useful for new users to getting started and solves "which scorer should I start with?" question, but many teams need customization for adopting them for the real use cases.

B-Step62 · 2026-05-18T20:27:24Z

+)
+```
+
+```python


I believe we need to persist presets more than a single python session. Otherwise L65 is essentially same as defining a normal python list of scorers. Team sharing benefit mentioned below only holds with persistence in a shared server.

B-Step62 · 2026-05-18T20:28:51Z

+        self._scorers = tuple(self._deduplicate(scorers))
+
+    @staticmethod
+    def _deduplicate(scorers):


nit: Instead of deduplication, I would block adding an existing scorer in the preset.

B-Step62 · 2026-05-18T20:30:58Z

+
+### Built-in Presets as Subclasses
+
+Each built-in preset is a subclass of `Preset` that hardcodes its scorer list. This means each call creates **fresh scorer instances** (no shared mutable singletons) and opens the door for preset-specific configuration and control flow in the future.


I agree with Matt about customization capability. The class-based approach is more ergonomic to me but I think it should come with a good customizability and persistence.

B-Step62 · 2026-05-18T20:32:36Z

+class SafetyPreset(Preset):
+    def __init__(self):
+        super().__init__("safety", [
+            Safety(),
+            ConversationalSafety(),
+        ])
+
+class Quality(Preset):
+    def __init__(self):
+        super().__init__("quality", [
+            RelevanceToQuery(),
+            Fluency(),
+            Completeness(),
+        ])


RAG/Agent/Conversation presets look great, the other two look less useful or intuitive to me. Safety one just bundles two scorers with 'safety' in its name. "Quality" sounds a bit too vague imo, technically most of scorers aims to evaluate some aspect of the agent quality. Shall we start with the first three + great customization story?

etirelli · 2026-05-19T19:42:10Z

I fully agree with @mprahl on this:

if built-in preset membership changes across releases, users will get silent behavior and cost changes on upgrade. That combination makes presets feel more like unstable convenience sugar than a strong API surface.

If this is not handled properly, this feature would end up being a burden for us to maintain, and create friction for users on upgrades, rather than being a benefit.
One way to solve this issue is to implement persistence as suggested by @B-Step62 , as that would empower users to control how they use and manage presets across versions. A function based implementation would in such cases be more flexible and efficient than a class hierarchy based one.

subclassing opens up more than just categorization. For example, each preset could accept different judge models (e.g, Agent(model="openai:/gpt-4o")), different inference parameters, or other preset-specific configurations.

A function based implementation does not prevent such customizations.

Nehanth mentioned this pull request Apr 28, 2026

[FR] Scorer presets for common evaluation patterns (agent, RAG, safety) mlflow/mlflow#21445

Open

2 tasks

Nehanth and others added 3 commits April 28, 2026 11:04

Fix naming consistency throughout RFC

00db80f

Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Nehanth <nehanthnarendrula@gmail.com>

Remove _BUILTIN_PRESETS and list_presets from main proposal

f42a950

Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Nehanth <nehanthnarendrula@gmail.com>

mprahl reviewed May 5, 2026

View reviewed changes

Nehanth and others added 2 commits May 6, 2026 16:48

Add back validate_scorers code block for review context

22f64f8

Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Nehanth <nehanthnarendrula@gmail.com>

Nehanth requested a review from mprahl May 8, 2026 15:37

mprahl reviewed May 18, 2026

View reviewed changes

mprahl requested changes May 18, 2026

View reviewed changes

B-Step62 reviewed May 18, 2026

View reviewed changes


		MLflow ships five built-in preset subclasses. Each call creates fresh scorer instances.

		> Note: `TaskSuccess` is a new scorer proposed in [mlflow/mlflow#22972](https://github.com/mlflow/mlflow/issues/22972). It evaluates whether an agent successfully accomplished the user's task without requiring ground truth data — unlike `Correctness`, which requires an `expectations` column. This scorer would be added to `Agent`, `ConversationalAgent`, and `Quality`. This work can be part of this RFC or be a future addition after this RFC is completed.


		### Built-in Presets as Subclasses

		Each built-in preset is a subclass of `Preset` that hardcodes its scorer list. This means each call creates fresh scorer instances (no shared mutable singletons) and opens the door for preset-specific configuration and control flow in the future.

Conversation

Nehanth commented Apr 28, 2026

Summary

Uh oh!

mprahl May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Nehanth commented May 6, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mprahl left a comment

Choose a reason for hiding this comment

Uh oh!

B-Step62 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

etirelli commented May 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mprahl May 5, 2026 •

edited

Loading