Skip to content

Commit 1144d58

Browse files
sebbycorpclaude
authored and
sebbycorp
committed
Add community evaluators registry page
- Add /evaluators/ page with card grid parsed from data/evaluators.yaml - Each card shows evaluator name, description, language, tags, author, and link to source on GitHub - Include search/filter functionality - Add "How to Use" section with eval_config.yaml + CLI examples - Add "Build Your Own Evaluator" CTA with scaffold command and links to contribution guide and fork - Add Evaluators nav link across all page layouts (home, docs, single) - Remove remaining emojis from docs nav theme toggles - Data file mirrors index.yaml from agentevals-dev/evaluators repo Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent ce29cf2 commit 1144d58

7 files changed

Lines changed: 594 additions & 4 deletions

File tree

‎content/evaluators/_index.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
---
2+
title: "Community Evaluators"
3+
description: "Browse community-contributed evaluators for AgentEvals."
4+
---

‎data/evaluators.yaml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
generatedAt: "2026-03-20T00:02:55Z"
2+
3+
evaluators:
4+
- name: peters_evaluator
5+
description: "sample evaluator that returns a 0.123 score"
6+
language: python
7+
path: evaluators/peters_evaluator/peters_evaluator.py
8+
tags: ["test","example"]
9+
author: peterj
10+
lastUpdated: "2026-03-20T00:02:55Z"
11+
- name: response_quality
12+
description: "Checks that responses are non-empty, meet a minimum length, and don't just echo back the user input"
13+
language: python
14+
path: evaluators/response_quality/response_quality.py
15+
tags: ["quality","response","length"]
16+
author: agentevals-dev
17+
lastUpdated: "2026-03-20T00:02:55Z"
18+
- name: tool_coverage
19+
description: "Verifies that each invocation made at least a minimum number of tool calls"
20+
language: python
21+
path: evaluators/tool_coverage/tool_coverage.py
22+
tags: ["tools","coverage","validation"]
23+
author: agentevals-dev
24+
lastUpdated: "2026-03-20T00:02:55Z"

‎layouts/docs/list.html‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,11 +10,12 @@
1010
<div class="nav-links">
1111
<a href="{{ "/" | relURL }}">Home</a>
1212
<a href="{{ "docs/" | relURL }}">Docs</a>
13+
<a href="{{ "evaluators/" | relURL }}">Evaluators</a>
1314
<a href="{{ .Site.Params.discord }}" target="_blank">Discord</a>
1415
<a href="{{ .Site.Params.github }}" target="_blank" class="btn-sm">GitHub</a>
1516
<button class="theme-toggle" onclick="toggleTheme()" aria-label="Toggle theme">
16-
<span class="icon-sun">☀️</span>
17-
<span class="icon-moon">🌙</span>
17+
<span class="icon-sun">&#9728;</span>
18+
<span class="icon-moon">&#9790;</span>
1819
</button>
1920
</div>
2021
</nav>

‎layouts/docs/single.html‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,11 +10,12 @@
1010
<div class="nav-links">
1111
<a href="{{ "/" | relURL }}">Home</a>
1212
<a href="{{ "docs/" | relURL }}">Docs</a>
13+
<a href="{{ "evaluators/" | relURL }}">Evaluators</a>
1314
<a href="{{ .Site.Params.discord }}" target="_blank">Discord</a>
1415
<a href="{{ .Site.Params.github }}" target="_blank" class="btn-sm">GitHub</a>
1516
<button class="theme-toggle" onclick="toggleTheme()" aria-label="Toggle theme">
16-
<span class="icon-sun">☀️</span>
17-
<span class="icon-moon">🌙</span>
17+
<span class="icon-sun">&#9728;</span>
18+
<span class="icon-moon">&#9790;</span>
1819
</button>
1920
</div>
2021
</nav>

‎layouts/evaluators/list.html‎

Lines changed: 176 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,176 @@
1+
{{ define "main" }}
2+
3+
<!-- Navigation -->
4+
<nav class="nav">
5+
<a href="{{ "/" | relURL }}" class="nav-logo">
6+
<img src="{{ "images/logo-color.png" | relURL }}" alt="AgentEvals" class="logo-dark">
7+
<img src="{{ "images/logo-color-transparent.png" | relURL }}" alt="AgentEvals" class="logo-light">
8+
</a>
9+
<button class="nav-toggle" onclick="document.querySelector('.nav-links').classList.toggle('active')" aria-label="Menu">&#9776;</button>
10+
<div class="nav-links">
11+
<a href="{{ "/" | relURL }}">Home</a>
12+
<a href="{{ "docs/" | relURL }}">Docs</a>
13+
<a href="{{ "evaluators/" | relURL }}" class="nav-active">Evaluators</a>
14+
<a href="{{ .Site.Params.discord }}" target="_blank">Discord</a>
15+
<a href="{{ .Site.Params.github }}" target="_blank" class="btn-sm">GitHub</a>
16+
<button class="theme-toggle" onclick="toggleTheme()" aria-label="Toggle theme">
17+
<span class="icon-sun">&#9728;</span>
18+
<span class="icon-moon">&#9790;</span>
19+
</button>
20+
</div>
21+
</nav>
22+
23+
<div class="eval-registry">
24+
<div class="container">
25+
<!-- Header -->
26+
<div class="eval-header">
27+
<h1>Community Evaluators</h1>
28+
<p>Browse and use community-contributed evaluators, or submit your own.</p>
29+
<div class="eval-header-actions">
30+
<div class="eval-search-wrapper">
31+
<input type="text" id="evalSearch" class="eval-search" placeholder="Search evaluators..." oninput="filterEvaluators()">
32+
</div>
33+
<a href="https://github.com/agentevals-dev/evaluators#contributing-an-evaluator" target="_blank" class="btn btn-primary">
34+
+ Submit an Evaluator
35+
</a>
36+
</div>
37+
</div>
38+
39+
<!-- Stats bar -->
40+
<div class="eval-stats">
41+
<span class="eval-stat"><strong>{{ len .Site.Data.evaluators.evaluators }}</strong> evaluators</span>
42+
<span class="eval-stat-sep">&#183;</span>
43+
<span class="eval-stat">Last updated: {{ .Site.Data.evaluators.generatedAt | time.Format "Jan 2, 2006" }}</span>
44+
<span class="eval-stat-sep">&#183;</span>
45+
<a href="https://github.com/agentevals-dev/evaluators" target="_blank" class="eval-stat-link">View on GitHub &#8599;</a>
46+
</div>
47+
48+
<!-- Evaluator Grid -->
49+
<div class="eval-grid" id="evalGrid">
50+
{{ range .Site.Data.evaluators.evaluators }}
51+
<div class="eval-card" data-name="{{ .name }}" data-desc="{{ .description }}" data-tags="{{ delimit .tags " " }}" data-author="{{ .author }}">
52+
<div class="eval-card-header">
53+
<div class="eval-card-icon">
54+
{{ if eq .language "python" }}
55+
<span class="lang-icon lang-python">py</span>
56+
{{ else if eq .language "javascript" }}
57+
<span class="lang-icon lang-js">js</span>
58+
{{ else if eq .language "typescript" }}
59+
<span class="lang-icon lang-ts">ts</span>
60+
{{ else }}
61+
<span class="lang-icon">&#x2699;</span>
62+
{{ end }}
63+
</div>
64+
<div class="eval-card-meta">
65+
<h3>{{ .name }}</h3>
66+
<span class="eval-card-author">by {{ .author }}</span>
67+
</div>
68+
</div>
69+
<p class="eval-card-desc">{{ .description }}</p>
70+
<div class="eval-card-tags">
71+
{{ range .tags }}
72+
<span class="eval-tag">{{ . }}</span>
73+
{{ end }}
74+
</div>
75+
<div class="eval-card-footer">
76+
<a href="https://github.com/agentevals-dev/evaluators/tree/main/{{ .path | path.Dir }}" target="_blank" class="eval-card-link">
77+
View source &#8599;
78+
</a>
79+
<span class="eval-card-lang">{{ .language }}</span>
80+
</div>
81+
</div>
82+
{{ end }}
83+
</div>
84+
85+
<!-- Empty state (hidden by default, shown by JS when search has no results) -->
86+
<div class="eval-empty" id="evalEmpty" style="display:none;">
87+
<p>No evaluators match your search.</p>
88+
</div>
89+
90+
<!-- How to Use -->
91+
<div class="eval-howto">
92+
<h2>Using Community Evaluators</h2>
93+
<div class="eval-howto-grid">
94+
<div class="eval-howto-step">
95+
<div class="step-number">1</div>
96+
<h3>Reference in eval config</h3>
97+
<div class="eval-howto-code">
98+
<pre>metrics:
99+
- name: response_quality
100+
type: remote
101+
source: github
102+
ref: evaluators/response_quality/response_quality.py
103+
threshold: 0.7
104+
config:
105+
min_response_length: 20</pre>
106+
</div>
107+
</div>
108+
<div class="eval-howto-step">
109+
<div class="step-number">2</div>
110+
<h3>Run as usual</h3>
111+
<div class="eval-howto-code">
112+
<pre><span class="cmd">agentevals</span> run traces/my_trace.json \
113+
<span class="flag">--config</span> <span class="string">eval_config.yaml</span> \
114+
<span class="flag">--eval-set</span> <span class="string">eval_set.json</span></pre>
115+
</div>
116+
<p class="eval-howto-note">Evaluators are downloaded automatically and cached in <code>~/.cache/agentevals/evaluators/</code>.</p>
117+
</div>
118+
</div>
119+
</div>
120+
121+
<!-- Submit CTA -->
122+
<div class="eval-cta">
123+
<h2>Build Your Own Evaluator</h2>
124+
<p>Evaluators are standalone scoring programs that read EvalInput JSON from stdin and write EvalResult JSON to stdout. Scaffold one in seconds:</p>
125+
<div class="eval-howto-code" style="max-width:500px;margin:1.5rem auto;">
126+
<pre><span class="cmd">pip</span> install agentevals
127+
<span class="cmd">agentevals</span> evaluator init my_evaluator</pre>
128+
</div>
129+
<div class="eval-cta-buttons">
130+
<a href="https://github.com/agentevals-dev/evaluators#contributing-an-evaluator" target="_blank" class="btn btn-primary">
131+
Contribution Guide
132+
</a>
133+
<a href="https://github.com/agentevals-dev/evaluators/fork" target="_blank" class="btn btn-secondary">
134+
Fork the Repo
135+
</a>
136+
</div>
137+
</div>
138+
</div>
139+
</div>
140+
141+
<!-- Footer -->
142+
<footer class="footer">
143+
<div class="footer-content">
144+
<a href="{{ "/" | relURL }}" class="footer-logo">
145+
<img src="{{ "images/logo-color.png" | relURL }}" alt="AgentEvals" class="logo-dark">
146+
<img src="{{ "images/logo-color-transparent.png" | relURL }}" alt="AgentEvals" class="logo-light">
147+
</a>
148+
<div class="footer-links">
149+
<a href="{{ "docs/" | relURL }}">Docs</a>
150+
<a href="{{ "evaluators/" | relURL }}">Evaluators</a>
151+
<a href="{{ .Site.Params.github }}" target="_blank">GitHub</a>
152+
<a href="{{ .Site.Params.discord }}" target="_blank">Discord</a>
153+
</div>
154+
<span class="footer-copy">&copy; {{ now.Year }} AgentEvals. Open source under Apache 2.0.</span>
155+
</div>
156+
</footer>
157+
158+
<script>
159+
function filterEvaluators() {
160+
var query = document.getElementById('evalSearch').value.toLowerCase();
161+
var cards = document.querySelectorAll('.eval-card');
162+
var visible = 0;
163+
cards.forEach(function(card) {
164+
var name = card.getAttribute('data-name').toLowerCase();
165+
var desc = card.getAttribute('data-desc').toLowerCase();
166+
var tags = card.getAttribute('data-tags').toLowerCase();
167+
var author = card.getAttribute('data-author').toLowerCase();
168+
var match = name.indexOf(query) !== -1 || desc.indexOf(query) !== -1 || tags.indexOf(query) !== -1 || author.indexOf(query) !== -1;
169+
card.style.display = match ? '' : 'none';
170+
if (match) visible++;
171+
});
172+
document.getElementById('evalEmpty').style.display = visible === 0 ? '' : 'none';
173+
}
174+
</script>
175+
176+
{{ end }}

‎layouts/index.html‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313
<a href="#interfaces">Interfaces</a>
1414
<a href="#get-started">Get Started</a>
1515
<a href="{{ "docs/" | relURL }}">Docs</a>
16+
<a href="{{ "evaluators/" | relURL }}">Evaluators</a>
1617
<a href="{{ .Site.Params.discord }}" target="_blank">Discord</a>
1718
<a href="{{ .Site.Params.github }}" target="_blank" class="btn-sm">GitHub</a>
1819
<button class="theme-toggle" onclick="toggleTheme()" aria-label="Toggle theme">

0 commit comments

Comments
 (0)