Judge feedback batch: job-backed run of a pre-existing batch (eval parity) + deep-review fixes

[leonardmq]

Brings the judge_feedback_batch background-job worker to parity with the eval job worker, reshapes it to run a pre-existing batch (eval parity), and applies fixes from a deep multi-phase code review of the whole judge_feedback_batch feature. Targets integration/auto-mode-e2e (the auto-mode e2e integration branch). Includes the work merged from #1543.

Job shape: run a pre-existing batch (eval parity)

The job was reshaped from "create-and-run" to "run an existing batch", mirroring EvalJobWorker:

• The batch config is created first via the synchronous POST .../judge_feedback_batches (persisted under the git-sync write lock); the job then just runs it by id.
• JudgeFeedbackBatchJobParams is now {project_id, task_id, judge_feedback_batch_id} (was a full CreateJudgeFeedbackBatchRequest). run()/describe() load the batch by id and read its config off disk.
• Because the id is caller-supplied and lives in job.params, it's retrievable via GET /api/jobs/{id} even if the run fails (the old create-and-run job only surfaced the minted id in its success result). The job result now echoes judge_feedback_batch_id from params.
• The worker no longer writes the config, so the earlier "wrap the batch save in the git-sync context" worker workaround is gone — the create endpoint owns that write.

Worker parity

• Progress: the worker previously reported progress once at the end and used train_set_size as the denominator, so the bar stalled below 100% when the matching set exceeded max_samples. The runner now takes an optional progress_callback and streams (num_judged, error_count, planned_total) per judged chunk; live + final progress use the capped denominator min(train_set_size, max_samples), reaching 100% on full coverage (mirrors the eval worker). success excludes errors (num_judged - error_count) so processed = success + error holds and the bar can't overshoot.
• Metadata / jobs UI: added JudgeFeedbackBatchJobProperties + describe() (mirrors EvalJobProperties) — judge/eval names, algorithm, model, run config (only when generating outputs), tags, max_samples, stop_after_failures — and render them in the jobs table (was a raw Judge_feedback_batch label).
• Errors: per-item errors reach the View Errors UI, and are now logged live via an error_callback the moment each is collected — previously the button appeared mid-run (gated by the streamed error count) but the messages only landed in a post-run loop, so the log read "No error messages recorded" until completion. Message quality improved by unwrapping KilnRunError.original like the eval worker.
• Field descriptions: added to JudgeFeedbackBatchJobResult, JudgeFeedbackBatchJobParams, and the project_id/task_id params.

Deep code review fixes

Multi-phase review (runner / worker / data model / REST API / UI / tests). 0 critical, 6 moderate (all fixed), 24 mild (12 fixed, rest documented). Highlights:

• Progress double-counting: num_judged already includes errored items, so reporting success = num_judged alongside a separate error count breached the processed = success + error contract — now success = num_judged - error_count in both the streamed callback and the final snapshot.
• Gate-mode determinism: gate mode now samples deterministically (sorted by id) before capping, so two runs cover the same task_run_ids — the pairing the gate exists for (a random shuffle previously picked disjoint subsets past max_samples). Train-signal mode keeps the random minibatch.
• Data model: added a JudgeFeedbackBatch model_validator (generate_outputs ⇒ run_config_id, non-empty target_tags) as defense-in-depth (mirrors EvalRun).
• Runner tests for the empty candidate set and gate-mode hit_cap=True (incl. proof that two gate runs cover the same ids), plus a test asserting errors are delivered live (interleaved with progress).
• Polish: _error_detail on save errors, concurrency = max(1, concurrency) guard, 404 message/wording (Judge feedback batch not found), judge_feedback_batch_from_id accepts a preloaded task (removes the run endpoint's double task load), docstrings covering generate-outputs mode.

Agent policy

The synchronous run / create-and-run endpoints are superseded by the create + job flow, so the agent goes through POST /api/jobs/judge_feedback_batch/run for dashboard visibility:

• create → allow (cheap, no model calls — the agent creates the batch, then runs it as a job)
• run (existing, sync) → deny
• create-and-run (sync) → deny
• run as job (/api/jobs/...) → allow + approval
• list / get / runs (GET) → allow

Regenerated the agent-check annotation JSONs so the policy is actually enforced (the chat backend reads the dumped JSONs, not the live spec) — fixes the check_api_bindings CI job. The synchronous create-and-run / run endpoints are left in place for callers not yet migrated.

UI hardening

• Render judge-batch properties in the jobs table (eval_name, stop_after_failures, tags, etc.).
• Guard the target_tags render with optional chaining so a job record with incomplete properties can't throw and crash the whole jobs table (per gemini-code-assist review).

Verification

Full uv run ./checks.sh --agent-mode green (lint, format, typecheck, py + web tests, schema, build). api_schema.d.ts regenerated.

🤖 Generated with Claude Code

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

Walkthrough

Updates judge feedback batch jobs to stream progress and errors, expose job properties for describe/UI use, enforce batch configuration invariants, and align backend API and web UI rendering with the new job shape.

Changes

Judge Feedback Batch Progress, Describe, and Validation

Layer / File(s)	Summary
Runner progress callback and error detail extraction libs/core/kiln_ai/adapters/eval/judge_feedback_batch_runner.py, libs/core/kiln_ai/adapters/eval/test_judge_feedback_batch_runner.py	Runner adds async progress and error callbacks, deterministic gate-mode candidate ordering, concurrency clamping, and unified per-item error formatting; tests cover streaming, cap handling, empty candidates, and live error delivery.
Worker describe, run progress, and API reuse app/desktop/studio_server/jobs/workers/judge_feedback_batch.py, app/desktop/studio_server/jobs/workers/test_judge_feedback_batch.py, app/desktop/studio_server/judge_feedback_batch_api.py	Worker adds async describe()/_describe_sync() for properties, run() forwards progress and error callbacks while reporting final capped totals, and the API reuses a preloaded task while updating its not-found text and git-sync save handling.
JudgeFeedbackBatch model validation libs/core/kiln_ai/datamodel/judge_feedback_batch.py, libs/core/kiln_ai/datamodel/test_judge_feedback_batch.py	Adds a model validator requiring run_config_id when generate_outputs is true and at least one target_tags entry, with tests for both cases.
Frontend job properties and table rendering app/web_ui/src/lib/stores/jobs_api.ts, app/web_ui/src/lib/components/jobs_table.svelte, app/web_ui/src/lib/components/jobs_table.test.ts, app/web_ui/src/lib/api_schema.d.ts	The web UI adds typed judge feedback batch job properties, a dedicated jobs-table rendering branch, matching test coverage, and schema/docs updates for the endpoint and params.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related PRs

• Kiln-AI/Kiln#1530: Both PRs touch the worker/job-properties contract used by the jobs table UI.

Suggested reviewers: scosman, chiang-daniel

Poem

A rabbit hopped through batch and log,
With progress ticks and tidy fog 🐇
New fields bloom, and tables sing,
Judge the batch — let metrics ping!
Hop, hop, hooray, the bits align 🥕

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 17.39% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check	⚠️ Warning	The description covers the PR goal, but it omits required template sections for Related Issues, CLA, and checklists.	Add the missing template sections: Related Issues, CLA confirmation, and the test/checklist items, even if briefly filled out.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title matches the main change: judge feedback batch jobs now run an existing batch with parity and review fixes.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch leonard/judge-feedback-batch-job-polish

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

📊 Coverage Report

Overall Coverage: 92%

Diff: origin/integration/auto-mode-e2e...HEAD

• app/desktop/studio_server/jobs/workers/judge_feedback_batch.py (100%)
• app/desktop/studio_server/judge_feedback_batch_api.py (100%)
• libs/core/kiln_ai/adapters/eval/judge_feedback_batch_runner.py (86.4%): Missing lines 61,235,380
• libs/core/kiln_ai/datamodel/judge_feedback_batch.py (100%)

Summary

• Total: 100 lines
• Missing: 3 lines
• Coverage: 97%

Line-by-line

View line-by-line diff coverage

libs/core/kiln_ai/adapters/eval/judge_feedback_batch_runner.py

Lines 57-65

  57     (often generic) `format_error_message` text; the underlying cause survives on `.original`, so
  58     surface that for the developer-facing error log. Mirrors `EvalJobWorker._error_detail`.
  59     """
  60     if isinstance(error, KilnRunError) and error.original is not None:
! 61         return str(error.original)
  62     return str(error)
  63 
  64 
  65 def score_passes(

---

Lines 231-239

  231         retry_delay: float = 2.0,
  232     ):
  233         task = judge_feedback_batch.parent_task()
  234         if task is None:
! 235             raise ValueError("Judge feedback batch must have a parent task")
  236         eval = eval_config.parent_eval()
  237         if eval is None:
  238             raise ValueError("Eval config must have a parent eval")

---

Lines 376-384

  376                         error=f"Unexpected error judging item: {_error_detail(scored)}",
  377                     )
  378                     errors.append(unexpected_error)
  379                     if error_callback is not None:
! 380                         await error_callback(unexpected_error)
  381                     continue
  382                 if scored.error is not None:
  383                     errors.append(scored.error)
  384                     if error_callback is not None:

---

---

• 📊 HTML Coverage Report - Interactive coverage report
• 📈 Diff Coverage Report - Detailed diff analysis
• Github Actions Run - View the full coverage report

[gemini-code-assist]

Code Review

This pull request introduces live progress streaming and detailed job properties for judge feedback batch jobs, enhancing both the backend runner and the frontend jobs UI. It also adds deterministic candidate selection in gate mode to ensure stable subsets across runs, along with model validation for batch configurations. The review feedback highlights a potential UI crash in the jobs table if target_tags is undefined for older or malformed jobs, suggesting optional chaining as a safeguard.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@app/desktop/studio_server/jobs/workers/judge_feedback_batch.py`:
- Around line 216-251: The judge/save errors are only being forwarded to
ctx.report_error after runner.run completes, so the error count can update
before any messages exist in the View Errors UI. Update judge_feedback_batch.py
so errors are reported incrementally during execution by wiring an error
callback or observer through the runner path used by report_progress and
result.errors, similar to EvalJobWorker, and keep the final post-run snapshot
only as a catch-up for any remaining errors.

In `@libs/core/kiln_ai/datamodel/judge_feedback_batch.py`:
- Around line 113-124: The after-model validation in
JudgeFeedbackBatch.validate_config is now rejecting older persisted batches when
load_from_file() calls model_validate() directly. Update the loading path or add
a backward-compatibility migration so legacy files can still be read, and keep
the stricter checks only for newly created/edited batches; use the
JudgeFeedbackBatch.validate_config and load_from_file entry points to locate the
fix.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 34d8f9c9-c528-4eb2-8880-e3c2ac794eb5

📥 Commits

Reviewing files that changed from the base of the PR and between 16f3933 and 938fdda.

📒 Files selected for processing (11)

• app/desktop/studio_server/jobs/workers/judge_feedback_batch.py
• app/desktop/studio_server/jobs/workers/test_judge_feedback_batch.py
• app/desktop/studio_server/judge_feedback_batch_api.py
• app/web_ui/src/lib/api_schema.d.ts
• app/web_ui/src/lib/components/jobs_table.svelte
• app/web_ui/src/lib/components/jobs_table.test.ts
• app/web_ui/src/lib/stores/jobs_api.ts
• libs/core/kiln_ai/adapters/eval/judge_feedback_batch_runner.py
• libs/core/kiln_ai/adapters/eval/test_judge_feedback_batch_runner.py
• libs/core/kiln_ai/datamodel/judge_feedback_batch.py
• libs/core/kiln_ai/datamodel/test_judge_feedback_batch.py

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

libs/core/kiln_ai/adapters/eval/judge_feedback_batch_runner.py (1)

374-393: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Unguarded callback exceptions can abort the whole run and discard already-judged results.

error_callback and progress_callback are awaited directly with no try/except. If the caller's callback raises (e.g. the worker's report_progress/report_item_error in judge_feedback_batch.py doing I/O against the job store), the exception propagates out of run(), and none of the already-collected judged_runs/errors/failing_runs from prior chunks get returned — a purely cosmetic reporting failure would fail the entire batch job. The worker layer doesn't wrap await runner.run(...) in a try/except either, so nothing catches this downstream.

Since the whole point of these hooks is to report progress/errors without changing outcome semantics, consider isolating callback failures so a broken reporter can't sink the run.

🛡️ Proposed fix: isolate callback exceptions

                     errors.append(unexpected_error)
                     if error_callback is not None:
-                        await error_callback(unexpected_error)
+                        await self._safe_callback(error_callback, unexpected_error)
                     continue
                 if scored.error is not None:
                     errors.append(scored.error)
                     if error_callback is not None:
-                        await error_callback(scored.error)
+                        await self._safe_callback(error_callback, scored.error)
                 if scored.run is not None:
                     judged_runs.append(scored.run)
                     if not scored.passed:
                         failing_runs.append(scored.run)
             # Stream live progress against the planned (capped) count so a background job's bar
             # advances per chunk and reaches 100% on full coverage.
             if progress_callback is not None:
-                await progress_callback(num_judged, len(errors), len(candidates))
+                await self._safe_callback(
+                    progress_callback, num_judged, len(errors), len(candidates)
+                )

async def _safe_callback(self, callback, *args) -> None:
    try:
        await callback(*args)
    except Exception:
        logger.exception(
            "Callback failed for judge feedback batch %s; continuing run",
            self.judge_feedback_batch.id,
        )

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@libs/core/kiln_ai/adapters/eval/judge_feedback_batch_runner.py` around lines
374 - 393, The callback hooks in judge feedback batch execution can throw and
abort the whole run, so isolate failures in the `JudgeFeedbackBatchRunner.run`
flow instead of letting them propagate. Wrap both `error_callback` and
`progress_callback` awaits in a small safe helper or local try/except, log the
callback failure, and continue so already collected `judged_runs`, `errors`, and
`failing_runs` are still returned. Use the existing `error_callback`,
`progress_callback`, and `run()` symbols to locate the change.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@libs/core/kiln_ai/adapters/eval/judge_feedback_batch_runner.py`:
- Around line 374-393: The callback hooks in judge feedback batch execution can
throw and abort the whole run, so isolate failures in the
`JudgeFeedbackBatchRunner.run` flow instead of letting them propagate. Wrap both
`error_callback` and `progress_callback` awaits in a small safe helper or local
try/except, log the callback failure, and continue so already collected
`judged_runs`, `errors`, and `failing_runs` are still returned. Use the existing
`error_callback`, `progress_callback`, and `run()` symbols to locate the change.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: e33f0b02-50d5-486d-b245-00c77c3e6739

📥 Commits

Reviewing files that changed from the base of the PR and between d495639 and 5f60d43.

📒 Files selected for processing (4)

• app/desktop/studio_server/jobs/workers/judge_feedback_batch.py
• app/desktop/studio_server/jobs/workers/test_judge_feedback_batch.py
• libs/core/kiln_ai/adapters/eval/judge_feedback_batch_runner.py
• libs/core/kiln_ai/adapters/eval/test_judge_feedback_batch_runner.py

🚧 Files skipped from review as they are similar to previous changes (3)

• libs/core/kiln_ai/adapters/eval/test_judge_feedback_batch_runner.py
• app/desktop/studio_server/jobs/workers/test_judge_feedback_batch.py
• app/desktop/studio_server/jobs/workers/judge_feedback_batch.py