Skip to content

skill-evaluation-research.md

Latest commit

 

History

History
378 lines (268 loc) · 18.8 KB

skill-evaluation-research.md

File metadata and controls

378 lines (268 loc) · 18.8 KB
Raw

Skill Evaluation Research Report

Date: 2026-02-18 Status: Complete Sources: 10+ industry sources analyzed

Executive Summary

We analyzed how leading platforms evaluate AI agent skills and compared them against our AQE skill evaluation infrastructure. Our system is already more sophisticated than most in areas like trust tiers, multi-model testing, and learning integration. However, we have clear gaps in skill quality scoring (LLM-as-a-judge for how well skills are written), activation conflict detection, adaptive rubric-based grading, and eval-driven development workflow. This report synthesizes findings from Tessl, Anthropic's official guidance, OpenAI's eval methodology, Google Vertex AI's rubric systems, DeepEval, and Anthropic's engineering blog.

1. Tessl Framework Analysis

Source: docs.tessl.io/evaluate/evaluating-skills, docs.tessl.io/evaluate/evaluate-skill-quality-using-scenarios

Two-Layer Evaluation System

Layer 1: tessl skill review — Static Quality Assessment (LLM-as-a-Judge)

  • Validation Checks: Deterministic grading on structure (frontmatter, line count, licensing, metadata) — ~12-15 structural criteria
  • Implementation Score: LLM judges conciseness, actionability, workflow clarity, progressive disclosure — 1-4 point subscores per metric
  • Activation Score: LLM judges description specificity, trigger term quality, distinctiveness/conflict risk
  • Scoring benchmarks: 90%+ = best practice, 70-89% = good, <70% = needs revision

Layer 2: tessl eval run — Scenario-Based End-to-End Evaluation

  • Each scenario has 3 files:
    • capability.text — which skill capability is targeted
    • TASK.md — a real task for the agent to solve
    • criteria.json — grading rubric
  • Scenarios can be auto-generated from skill instructions, then manually reviewed
  • Agent actually executes the task, and output is graded against criteria
  • CLI: tessl eval run, tessl eval view, tessl eval list, tessl eval retry

Key Tessl Insight

Tessl separates skill quality (how well is the SKILL.md written?) from skill effectiveness (does the agent produce correct output?). We only do the latter.

2. Anthropic's Official Skill Authoring Guidance

Source: Skill authoring best practices

Evaluation-Driven Development (Most Important Takeaway)

Anthropic recommends building evals BEFORE writing extensive skill documentation:

  1. Identify gaps: Run Claude on representative tasks without a skill. Document failures.
  2. Create evaluations: Build 3+ scenarios that test those gaps
  3. Establish baseline: Measure Claude's performance without the skill
  4. Write minimal instructions: Just enough to pass evaluations
  5. Iterate: Execute evaluations, compare against baseline, refine

"This approach ensures you're solving actual problems rather than anticipating requirements that may never materialize."

Skill Quality Checklist (from Anthropic)

Core Quality:

  • Description is specific, includes trigger terms, written in third person
  • SKILL.md body under 500 lines
  • Progressive disclosure with one-level-deep references
  • No time-sensitive information
  • Consistent terminology throughout

Key Principles:

  • Conciseness is critical — context window is a shared resource. Challenge every paragraph: "Does Claude really need this?"
  • Degrees of freedom — match specificity to task fragility (high freedom for flexible tasks, low freedom for fragile operations)
  • Test with ALL models — what works for Opus may need more detail for Haiku

Naming Conventions:

  • Use gerund form: processing-pdfs, analyzing-spreadsheets (not pdf-helper, utils)
  • Lowercase, hyphens only, max 64 characters

Iterative Development with Claude A/B Pattern

  • Claude A = skill author (helps design/refine the skill)
  • Claude B = skill user (tests the skill on real tasks in a fresh context)
  • Observe Claude B's behavior → bring findings back to Claude A → refine → repeat

Relevance to AQE

Many of our 97 skills likely violate these principles. An automated quality scorer could flag:

  • Skills over 500 lines
  • Inconsistent terminology
  • Vague descriptions
  • Time-sensitive content
  • Deeply nested references

3. Anthropic's Agent Eval Engineering Guide

Source: Demystifying evals for AI agents

Three Grader Types

Type Speed Cost Best For
Code-based Fast Free Exact match, regex, schema validation, outcome verification
Model-based Medium $$ Subjective quality, reasoning assessment, rubric grading
Human Slow $$$$ Gold-standard calibration, complex judgment

Key Metrics

  • pass@k: Probability of ≥1 success across k attempts (when one working solution suffices)
  • pass^k: Probability ALL k trials succeed (critical for consistency-required agents)

Critical Best Practices

  1. Start with 20-50 tasks from REAL user failures — outperforms hundreds of synthetic ones
  2. Grade outputs, not paths — agents often discover valid approaches evaluators didn't anticipate
  3. Build in partial credit — recognize continua of success, not just binary pass/fail
  4. Read transcripts regularly — distinguish genuine failures from grading bugs
  5. Monitor saturation — when evals hit 100%, graduate them to regression suites and create harder ones

Relevance to AQE

Our must_contain keyword checks are code-based graders only. We're missing:

  • Model-based grading for subjective quality dimensions
  • Partial credit scoring (currently binary pass/fail per keyword)
  • pass@k / pass^k metrics for measuring consistency across trials
  • Saturation monitoring for knowing when to create harder evals

4. OpenAI's Eval Skills Methodology

Source: Testing Agent Skills Systematically with Evals

Four Success Dimensions

Every skill eval should measure across these dimensions:

  1. Outcome goals: Did the task complete correctly?
  2. Process goals: Did the agent follow intended steps?
  3. Style goals: Does output adhere to conventions?
  4. Efficiency goals: Were unnecessary operations minimized?

Evaluation Dataset Structure

Start with 10-20 test cases covering:

  • Explicit invocation: Direct skill name reference
  • Implicit invocation: Skill triggers via description match
  • Contextual invocation: Real-world scenarios
  • Negative controls: Cases where skill should NOT activate

Advanced Checks to Layer In

Beyond basic output validation:

  • Command counting: Detect looping or inefficient workflows
  • Token tracking: Monitor input/output usage for prompt bloat
  • Build verification: Run actual builds as end-to-end validation
  • Repository cleanliness: Ensure no unwanted files generated
  • Permission validation: Verify least-privilege constraints

Key Insight

"The fundamental shift is from subjective assessment ('this feels better') to measurable validation."

Relevance to AQE

We're missing:

  • Negative control test cases — verifying skills DON'T activate incorrectly
  • Efficiency metrics — token usage and command counting per skill invocation
  • Process goals — did the agent follow the right steps, not just get the right answer?

5. Google Vertex AI: Adaptive vs Static Rubrics

Source: Vertex AI Rubric-Based Metrics

Two Rubric Approaches

Approach Description Best For
Adaptive Rubrics Auto-generate unique pass/fail checks per prompt Varied prompts, no reference answers
Static Rubrics Fixed scoring guidelines across all prompts Consistent benchmarks, standardized metrics

Adaptive Rubrics (Recommended Default)

For each input, the system dynamically generates prompt-specific checks:

  • "Is the response under 100 words?"
  • "Does it mention the Sept 15 deadline?"
  • "Does it return valid JSON with keys risk_level and next_steps?"

These catch highly specific contracts that static rubrics miss.

Relevance to AQE

Our current eval system uses entirely static rubrics (must_contain patterns). Adopting adaptive rubrics would:

  • Auto-generate context-specific validation per test case
  • Catch skill-specific output requirements we currently miss
  • Reduce manual effort writing must_contain lists

6. DeepEval Agent Metrics

Source: DeepEval AI Agent Evaluation, Tool Correctness Metric

Agent-Specific Metrics

Metric What It Measures
Tool Correctness Did the agent select the right tools and call them with correct arguments?
Task Completion Did the agent accomplish what the user asked?
Reasoning Quality Was the agent's decision-making process sound?

Key Insight

"AI agents fail in fundamentally different ways than simple LLM applications—an agent might select the right tool but pass wrong arguments, create a brilliant plan but fail to follow it, or complete the task but waste resources on redundant steps."

Relevance to AQE

For our MCP-integrated skills, we should evaluate:

  • Did the skill invoke the correct MCP tools?
  • Were tool arguments correct?
  • Was the overall task completed?
  • Were any redundant tool calls made?

7. Anthropic's Empirical Evaluation Guide

Source: Create strong empirical evaluations

Grading Method Hierarchy

Choose the fastest, most reliable, most scalable method:

  1. Code-based: Exact match, string match — fastest, most reliable
  2. LLM-based: Rubric-driven assessment — fast, flexible, requires calibration
  3. Human: Most flexible — slow, expensive, avoid if possible

Eval Methods by Quality Dimension

Dimension Method Example
Task fidelity Exact match Sentiment == "positive"
Consistency Cosine similarity Paraphrased inputs → similar outputs
Relevance ROUGE-L Summary captures key information
Tone/Style LLM Likert scale Rate empathy 1-5
Safety LLM binary classification Contains PHI? yes/no
Context use LLM ordinal scale References conversation history? 1-5

Tips for LLM-Based Grading

  • Detailed, clear rubrics with specific criteria and examples
  • Empirical output — "Output only 'correct' or 'incorrect'" or 1-5 scale
  • Encourage reasoning — ask LLM to think first, then score (chain-of-thought improves grading accuracy)
  • Use a different model to grade than the model that generated the output

Relevance to AQE

We should adopt:

  • Cosine similarity for consistency testing across similar inputs
  • LLM Likert scales for subjective quality dimensions
  • Chain-of-thought grading — have the judge reason before scoring
  • Cross-model grading — don't use the same model to both generate and evaluate

8. Comparison: AQE vs Industry

Capability AQE Tessl Anthropic OpenAI Vertex AI DeepEval
Trust tiers T0-T4 None None None None None
Multi-model testing Built-in None Recommended None None None
Learning integration MCP + ReasoningBank None None None None None
Schema validation 4-layer Basic None None None None
Skill quality scoring Missing LLM judge Checklist None None None
Adaptive rubrics Missing None None None Built-in None
Negative controls Missing None Recommended Required None None
Tool correctness Missing None None None None Built-in
Partial credit Missing None Recommended None None None
pass@k / pass^k Missing None Built-in None None None
Eval-driven development Missing None Core workflow Core workflow None None
Activation conflict detection Missing Built-in None None None None
Auto-generated evals Missing Built-in None None Adaptive None
Efficiency metrics Missing None None Command/token counting None None
Cosine similarity grading Missing None Documented None None None

Legend: Bold = strong implementation

9. Prioritized Recommendations

P0 — High Impact, Addresses Core Gaps

1. Eval-Driven Development Workflow

Why: Anthropic and OpenAI both make this their #1 recommendation. Write evals before skill content. Implementation: Add aqe skill eval-init <skill-name> that creates baseline eval from task failures, then aqe skill eval-compare to measure improvement. Effort: Medium

2. LLM-as-a-Judge Skill Quality Scorer

Why: Tessl has this, Anthropic recommends it. We have 39 Tier-0 skills with no quality signal. Implementation: Score each SKILL.md on: conciseness (under 500 lines), actionability, description specificity, trigger term quality, terminology consistency, progressive disclosure usage. Output a 0-100 score with specific improvement suggestions. Effort: Medium

3. Activation Conflict Detector

Why: With 97 skills, overlapping descriptions cause mis-routing. Tessl scores this as "distinctiveness." Implementation: Embed all skill descriptions → compute pairwise cosine similarity → flag pairs above 0.7 threshold → suggest description refinements. Effort: Low (we already have HNSW/vector infrastructure)

P1 — Medium Impact, Deepens Eval Quality

4. Adaptive Rubric Grading

Why: Our must_contain keyword matching misses context-specific requirements. Vertex AI's adaptive rubrics auto-generate checks. Implementation: For each eval test case, have an LLM generate prompt-specific pass/fail checks, then grade the output against those checks. Effort: Medium

5. Multi-Dimensional Grading (Beyond Keywords)

Why: Anthropic documents 6+ grading methods (exact match, cosine similarity, ROUGE-L, Likert, binary classification, ordinal). We only use keyword match. Implementation: Add grading modes to eval YAML: grading_method: [keyword_match | cosine_similarity | llm_likert | llm_rubric | schema_check] Effort: Medium-High

6. Negative Control Test Cases

Why: OpenAI requires testing when skills should NOT activate. We don't test this. Implementation: Add negative_test_cases section to eval YAML — inputs where the skill should decline or redirect. Effort: Low

P2 — Refinements and Efficiency

7. pass@k / pass^k Consistency Metrics

Why: Anthropic's engineering guide highlights these as critical for measuring agent reliability across multiple trials. Implementation: Run each eval test case k times, compute both metrics. Surface pass^k for skills used in CI/production. Effort: Low

8. Efficiency Metrics (Token/Command Tracking)

Why: OpenAI recommends tracking token usage and command counts per skill invocation to detect bloat. Implementation: Add max_tokens, max_tool_calls fields to eval success criteria. Effort: Low

9. Auto-Generate Eval YAML from SKILL.md

Why: 39 skills at Tier 0 need evals. Tessl auto-generates scenarios from skill instructions. Implementation: LLM reads SKILL.md → generates eval YAML with test cases, expected outputs, and success criteria → human reviews and refines. Effort: Medium

10. Tool Correctness Metrics (for MCP Skills)

Why: DeepEval shows agent-specific metrics are different. Our MCP-integrated skills should validate tool selection and arguments. Implementation: For MCP-calling skills, verify: correct tools selected, correct arguments passed, no redundant calls. Effort: Medium

10. What We Already Do Better Than Everyone

These are genuine competitive advantages we should maintain and publicize:

  1. Trust Tier System (T0-T4) — No other platform has a graduated trust system with clear upgrade paths
  2. Multi-Model Consistency Testing — Built-in variance tracking across Sonnet/Haiku is unique
  3. MCP + ReasoningBank Learning — Eval results feed into shared learning, no competitor does this
  4. Structured YAML Eval Format — More organized than Tessl's 3-file-per-scenario approach
  5. 4-Layer Validation Architecture — Schema → Validator → Eval Suite → CI is more rigorous than industry standard
  6. Fleet-Wide Pattern Sharing — Successful patterns propagate to other agents automatically

Sources