Skip to content

Commit 7d53e0a

Browse files
Qardclaude
andauthored
feat(eval): add classifier support (#143)
* feat(eval): add classifier support Adds a Classifier interface mirroring Scorer that returns categorical labels (id + optional label + optional metadata) instead of numeric scores, with multi-label support. Implements the cross-SDK classifiers spec: spans use type=classifier/purpose=scorer, errors are non-fatal and aggregated into classifier_errors on the eval span's metadata, and classifiers run concurrently with each other and with scorers within each case. Adds a required-validation that at least one of Scorers or Classifiers must be provided. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(eval): skip scorer goroutine when no scorers configured Mirror the existing classifier gating so runs that only configure classifiers don't pay for an empty scorer goroutine. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(eval): remove Classifications C() helper Inline the one-item helper at the 5 callsites — the explicit Classifications{{ID: ...}} literal is clear enough on its own. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 8b5b0bd commit 7d53e0a

7 files changed

Lines changed: 881 additions & 5 deletions

File tree

eval/classifiers.go

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
package eval
2+
3+
import (
4+
"context"
5+
)
6+
7+
// Classifier is an interface for categorizing the output of a task.
8+
//
9+
// Unlike [Scorer], which returns numeric scores, a Classifier returns one or
10+
// more [Classification] items with structured id/label/metadata.
11+
type Classifier[I, R any] interface {
12+
// Name returns the name of this classifier.
13+
Name() string
14+
// Run categorizes the task result.
15+
// It returns zero or more Classification results.
16+
Run(context.Context, TaskResult[I, R]) (Classifications, error)
17+
}
18+
19+
// Classification represents a single category result returned by a Classifier.
20+
type Classification struct {
21+
// Name is the grouping key for this classification.
22+
// If empty, it defaults to the name of the Classifier that produced it.
23+
Name string
24+
25+
// ID is the stable identifier for the category (required).
26+
ID string
27+
28+
// Label is an optional human-readable label.
29+
Label string
30+
31+
// Metadata is optional additional metadata for this classification.
32+
Metadata map[string]interface{}
33+
}
34+
35+
// Classifications is a collection of Classification results returned by classifiers.
36+
type Classifications = []Classification
37+
38+
// ClassifyFunc is a function that classifies a task result and returns a list of Classifications.
39+
type ClassifyFunc[I, R any] func(ctx context.Context, result TaskResult[I, R]) (Classifications, error)
40+
41+
// NewClassifier creates a new classifier with the given name and classify function.
42+
func NewClassifier[I, R any](name string, fn ClassifyFunc[I, R]) Classifier[I, R] {
43+
return &classifierImpl[I, R]{
44+
name: name,
45+
fn: fn,
46+
}
47+
}
48+
49+
type classifierImpl[I, R any] struct {
50+
name string
51+
fn ClassifyFunc[I, R]
52+
}
53+
54+
func (c *classifierImpl[I, R]) Name() string {
55+
return c.name
56+
}
57+
58+
func (c *classifierImpl[I, R]) Run(ctx context.Context, result TaskResult[I, R]) (Classifications, error) {
59+
return c.fn(ctx, result)
60+
}

0 commit comments

Comments
 (0)