Skip to content

feat(eval): add classifier support#143

Merged
Stephen Belanger (Qard) merged 3 commits into
mainfrom
classifiers
Jun 3, 2026
Merged

feat(eval): add classifier support#143
Stephen Belanger (Qard) merged 3 commits into
mainfrom
classifiers

Conversation

@Qard

Copy link
Copy Markdown
Contributor

Summary

Ports the classifiers feature from the Ruby SDK (braintrustdata/braintrust-sdk-ruby#154) into the Go SDK, implementing the cross-SDK classifiers spec.

  • New `eval.Classifier[I, R]` interface and `Classification` struct mirroring `Scorer`/`Score`. Returns categorical labels (`id` + optional `label` + optional `metadata`) rather than a numeric score; multi-label classifiers may return several `Classification` items.
  • Each classifier runs in its own span with `type: "classifier"` / `purpose: "scorer"` and input/output JSON matching the spec wire format. Per-case results are aggregated onto the eval span as `braintrust.classifications`.
  • Classifier failures are non-fatal: per-classifier errors are aggregated and merged into the eval span's `braintrust.metadata.classifier_errors` map. Failing classifiers do not abort other classifiers or scorers.
  • Classifiers run concurrently with each other (one goroutine per classifier) and concurrently with the scorer pass within each case.
  • Adds a runtime validation that at least one of `Scorers` or `Classifiers` must be provided (spec MUST).
  • Adds a customer-facing example at `examples/classifiers/` and a kitchen-sink internal example at `examples/internal/classifiers/`.

Test plan

  • `make lint` — 0 issues
  • `go test ./eval/... -count=1` — all tests pass, including 9 new classifier tests
  • `go test ./eval/ -race` — clean under the race detector
  • `go build ./...` in the examples module — both new examples build
  • End-to-end smoke against a real workspace: `go run examples/classifiers/classifiers.go` and `go run examples/internal/classifiers/main.go` — verify classifier spans appear with `type=classifier` and aggregated classifications render in the UI

🤖 Generated with Claude Code

Comment thread eval/eval.go Outdated
Comment thread eval/classifiers.go Outdated
Stephen Belanger (Qard) and others added 3 commits June 4, 2026 01:11
Adds a Classifier interface mirroring Scorer that returns categorical
labels (id + optional label + optional metadata) instead of numeric
scores, with multi-label support. Implements the cross-SDK classifiers
spec: spans use type=classifier/purpose=scorer, errors are non-fatal
and aggregated into classifier_errors on the eval span's metadata, and
classifiers run concurrently with each other and with scorers within
each case. Adds a required-validation that at least one of Scorers or
Classifiers must be provided.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirror the existing classifier gating so runs that only configure
classifiers don't pay for an empty scorer goroutine.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Inline the one-item helper at the 5 callsites — the explicit
Classifications{{ID: ...}} literal is clear enough on its own.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@Qard
Stephen Belanger (Qard) enabled auto-merge (squash) June 3, 2026 17:12
@Qard
Stephen Belanger (Qard) merged commit 7d53e0a into main Jun 3, 2026
6 checks passed
@Qard
Stephen Belanger (Qard) deleted the classifiers branch June 3, 2026 17:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants