feat(judge-feedback-batch): expose concurrency in job params by leonardmq · Pull Request #1544 · Kiln-AI/Kiln

leonardmq · 2026-07-02T14:14:57Z

What

Add an optional concurrency field to JudgeFeedbackBatchJobParams and forward it to JudgeFeedbackBatchRunner.run() inside the job worker.

Why

The runner already accepts a concurrency argument (bounding how many items are judged in parallel per chunk), but the background job worker called runner.run(...) without it — so callers of the job API could never tune parallelism and were always stuck on the runner's default.

Details

concurrency: int | None = None on JudgeFeedbackBatchJobParams, with a field description. Null keeps the runner's mode-aware default (5 when generating fresh outputs, 25 when judging existing ones); values below 1 are clamped to 1 by the runner.
Worker now passes concurrency=params.concurrency to runner.run(...).
Regenerated api_schema.d.ts (the param is part of the jobs API request body).
Added a parametrized test asserting the param (including None) is forwarded to the runner.

Checks

pytest on the worker tests: 9 passed
ruff check/format, ty check, and check_schema.sh: all pass

🤖 Generated with Claude Code

Add an optional `concurrency` field to JudgeFeedbackBatchJobParams and forward it to JudgeFeedbackBatchRunner.run. Null keeps the runner's mode-aware default (5 when generating outputs, 25 when judging existing ones); values below 1 are clamped to 1 by the runner. Regenerated api_schema.d.ts and added a param-forwarding test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019w3CFeeewrMyMRdTv3SgQm

coderabbitai · 2026-07-02T14:15:18Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 35942280-2c40-4d34-bc5e-ad1735209ae6

📥 Commits

Reviewing files that changed from the base of the PR and between 1f8e9bf and c900ec2.

📒 Files selected for processing (3)

app/desktop/studio_server/jobs/workers/judge_feedback_batch.py
app/desktop/studio_server/jobs/workers/test_judge_feedback_batch.py
app/web_ui/src/lib/api_schema.d.ts

✅ Files skipped from review due to trivial changes (1)

app/web_ui/src/lib/api_schema.d.ts

🚧 Files skipped from review as they are similar to previous changes (2)

app/desktop/studio_server/jobs/workers/judge_feedback_batch.py
app/desktop/studio_server/jobs/workers/test_judge_feedback_batch.py

Walkthrough

Adds an optional concurrency field to judge feedback batch job params, forwards it to the batch runner, adds validation and forwarding tests, and updates the generated API schema.

Changes

Judge Feedback Batch Concurrency

Layer / File(s)	Summary
Concurrency param, runner wiring, and test `app/desktop/studio_server/jobs/workers/judge_feedback_batch.py`, `app/desktop/studio_server/jobs/workers/test_judge_feedback_batch.py`	Adds optional `concurrency: int \| None` to `JudgeFeedbackBatchJobParams`, forwards it to `JudgeFeedbackBatchRunner.run()`, and adds tests for forwarding and minimum-value validation.
Generated API schema update `app/web_ui/src/lib/api_schema.d.ts`	Adds optional nullable `concurrency?: number \| null` to the `JudgeFeedbackBatchJobParams` schema type.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

Kiln-AI/Kiln#1542: Also changes the judge feedback batch runner call chain and concurrency handling around JudgeFeedbackBatchRunner.run(...).
Kiln-AI/Kiln#1543: Touches the same JudgeFeedbackBatchJobParams and worker path in judge_feedback_batch.py.

Poem

A rabbit hopped with bounded glee,
“Run in parallel, but safely!”
One tiny field joined the batch,
Tests checked the forward-and-catch.
🐇

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: exposing concurrency in judge feedback batch job params.
Description check	✅ Passed	The description covers what, why, details, and checks, but it omits the Related Issues, CLA, and checklist sections from the template.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch leonard/judge-feedback-batch-concurrency-param

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

gemini-code-assist

Code Review

This pull request introduces an optional concurrency parameter to the JudgeFeedbackBatchJobParams model, allowing users to control the maximum number of items judged in parallel, and forwards this parameter to the batch runner. It also adds a corresponding unit test and updates the TypeScript API schema definitions. The reviewer suggests enforcing validation at the Pydantic level using ge=1 instead of silently clamping values below 1 in the runner.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

coderabbitai

🧹 Nitpick comments (1)

app/desktop/studio_server/jobs/workers/judge_feedback_batch.py (1)
39-44: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low value

Consider validating concurrency at the schema level.

The field accepts any int, including 0 or negative values, which the runner silently clamps to 1 (per judge_feedback_batch_runner.py's concurrency = max(1, concurrency)). Adding a gt=0 constraint would surface invalid input explicitly via Pydantic validation instead of silent clamping downstream.
♻️ Optional validation tightening
     concurrency: int | None = Field(
         default=None,
         description="Max items judged in parallel by the runner. Leave null to use the runner's "
         "default (5 when generating fresh outputs, 25 when judging existing ones). Values below 1 "
         "are clamped to 1 by the runner.",
+        gt=0,
     )
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@app/desktop/studio_server/jobs/workers/judge_feedback_batch.py` around lines
39 - 44, The JudgeFeedbackBatch schema currently allows non-positive concurrency
values and relies on downstream clamping in judge_feedback_batch_runner, so
tighten validation at the model level. Update the concurrency Field in
JudgeFeedbackBatch to require a value greater than 0, so invalid inputs are
rejected by Pydantic before reaching the runner instead of being silently
corrected. Use the existing concurrency attribute in JudgeFeedbackBatch as the
place to apply the constraint.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@app/desktop/studio_server/jobs/workers/judge_feedback_batch.py`:
- Around line 39-44: The JudgeFeedbackBatch schema currently allows non-positive
concurrency values and relies on downstream clamping in
judge_feedback_batch_runner, so tighten validation at the model level. Update
the concurrency Field in JudgeFeedbackBatch to require a value greater than 0,
so invalid inputs are rejected by Pydantic before reaching the runner instead of
being silently corrected. Use the existing concurrency attribute in
JudgeFeedbackBatch as the place to apply the constraint.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 4016004b-0836-4b9c-a206-7e27d6206427

📥 Commits

Reviewing files that changed from the base of the PR and between cc5347f and 1f8e9bf.

📒 Files selected for processing (3)

app/desktop/studio_server/jobs/workers/judge_feedback_batch.py
app/desktop/studio_server/jobs/workers/test_judge_feedback_batch.py
app/web_ui/src/lib/api_schema.d.ts

github-actions · 2026-07-02T14:18:19Z

📊 Coverage Report

Overall Coverage: 92%

Diff: origin/integration/auto-mode-e2e...HEAD

app/desktop/studio_server/jobs/workers/judge_feedback_batch.py (100%)

Summary

Total: 1 line
Missing: 0 lines
Coverage: 100%

📊 HTML Coverage Report - Interactive coverage report
📈 Diff Coverage Report - Detailed diff analysis
Github Actions Run - View the full coverage report

Add ge=1 to the concurrency param so invalid input returns a 422 up front (consistent with max_samples / stop_after_failures) instead of being silently clamped by the runner. Update the description, regenerate api_schema.d.ts, and add a validation test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019w3CFeeewrMyMRdTv3SgQm

gemini-code-assist Bot reviewed Jul 2, 2026

View reviewed changes

Comment thread app/desktop/studio_server/jobs/workers/judge_feedback_batch.py

coderabbitai Bot reviewed Jul 2, 2026

View reviewed changes

tawnymanticore self-requested a review July 2, 2026 14:18

tawnymanticore approved these changes Jul 2, 2026

View reviewed changes

leonardmq mentioned this pull request Jul 2, 2026

feat(eval-job): expose concurrency in job params #1545

Merged

leonardmq merged commit 40c6ead into integration/auto-mode-e2e Jul 2, 2026
12 checks passed

leonardmq deleted the leonard/judge-feedback-batch-concurrency-param branch July 2, 2026 15:28

leonardmq mentioned this pull request Jul 2, 2026

feat: assistant auto-mode + the eval/judge job backend (context gauge, chat resilience, queued messages) #1517

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(judge-feedback-batch): expose concurrency in job params#1544

feat(judge-feedback-batch): expose concurrency in job params#1544
leonardmq merged 2 commits into
integration/auto-mode-e2efrom
leonard/judge-feedback-batch-concurrency-param

leonardmq commented Jul 2, 2026

Uh oh!

coderabbitai Bot commented Jul 2, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Uh oh!

github-actions Bot commented Jul 2, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

leonardmq commented Jul 2, 2026

What

Why

Details

Checks

Uh oh!

coderabbitai Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📊 Coverage Report

Diff: origin/integration/auto-mode-e2e...HEAD

Summary

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

coderabbitai Bot commented Jul 2, 2026 •

edited

Loading

github-actions Bot commented Jul 2, 2026 •

edited

Loading