fix: retries are never happening in eval runner

[leonardmq]

What

Extracted root-cause fix from #1547 (as discussed there), targeting main directly since evals on main are affected today.

EvalRunner.run_job classifies transient failures with _is_retryable_error and re-raises them as RetryableError so AsyncJobRunner retries them (max_retries=2). But the classifier checks isinstance(e, litellm.RateLimitError) etc., while the model adapter wraps every provider exception in KilnRunError (base_adapter.py) before it reaches the retry loop. The wrapper never matched, so no transient eval error was ever retried on a real run — rate-limited items failed on their first attempt. Tests didn't catch it because they raise bare litellm errors instead of wrapped ones.

Fix

_is_retryable_error now unwraps KilnRunError via a small _unwrap_kiln_run_error helper and classifies the underlying error. The helper walks nested wrappers (not currently possible, defense-in-depth) and guards a contract-violating None original.

That's the whole change — no new retry knobs or backoff behavior; the runner's existing max_retries=2 / 1s delay simply works now. The fuller retry work (configurable schedule for background jobs, jittered exponential backoff) stays in #1547.

Tests

• Wrapped rate limit → retryable; wrapped non-transient → not; nested wrap → retryable.
• Full suite green (6219 passed); ruff / format / ty clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JNMsZVT2b9PuANGAHkPyb1

[coderabbitai]

Walkthrough

Adds KilnRunError unwrapping in eval retry classification, uses the underlying exception text for RetryableError messages, and expands tests for wrapped, nested, and non-retryable failures.

Changes

Retry error unwrapping

Layer / File(s)	Summary
Unwrap KilnRunError before retry classification libs/core/kiln_ai/adapters/eval/eval_runner.py	Adds KilnRunError handling and unwraps nested .original exceptions before retry checks run.
Use unwrapped message for RetryableError libs/core/kiln_ai/adapters/eval/eval_runner.py	Raises RetryableError with the innermost exception text instead of the wrapper text.
Tests for wrapped retry failures libs/core/kiln_ai/adapters/eval/test_eval_runner.py	Covers wrapped and nested KilnRunError retry classification, RetryableError messaging, and the no-persisted-run case for wrapped rate-limit failures.

Estimated code review effort: 2 (Simple) | ~12 minutes

Possibly related PRs

• Kiln-AI/Kiln#1200: Both PRs modify eval_runner.py retry decision logic around RetryableError.
• Kiln-AI/Kiln#1531: Both PRs unwrap KilnRunError.original to drive eval error handling and reporting.
• Kiln-AI/Kiln#1542: Both PRs change how wrapped eval errors are unwrapped before downstream handling.

Suggested reviewers: scosman

Poem

A bunny peeks behind the veil,
Finds the real error in the trail.
Hop, unwrap, and retry anew,
With clearer words and tests thatూ?
Thump thump — the wrapper’s through 🐰

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly states the main fix: eval runner retries were not happening.
Description check	✅ Passed	The description covers the bug, root cause, fix, and tests, but omits the issue link, CLA confirmation, and checklist items.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch leonard/fix-transient-retry-never-fires

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

📊 Coverage Report

Overall Coverage: 92%

Diff: origin/main...HEAD

• libs/core/kiln_ai/adapters/eval/eval_runner.py (100%)

Summary

• Total: 7 lines
• Missing: 0 lines
• Coverage: 100%

---

• 📊 HTML Coverage Report - Interactive coverage report
• 📈 Diff Coverage Report - Detailed diff analysis
• Github Actions Run - View the full coverage report

[gemini-code-assist]

Code Review

This pull request introduces a helper function _unwrap_kiln_run_error to extract the underlying exception from a wrapped KilnRunError and integrates it into _is_retryable_error to ensure transient errors (like rate limits) are correctly classified for retries. Unit tests are also added to verify this behavior. Feedback suggests unwrapping the exception in run_job as well, to ensure that logged messages and raised RetryableErrors contain the detailed underlying error message rather than the generic KilnRunError text, aligning with the docstring's intent.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.