Multi-turn eval megabranch: eval builder + claim/evidence pipeline + upstream pre-integration

[chiang-daniel]

One production branch for the multi-turn eval work, per the agreed topology: forked from feat/multiturn-megabranch, containing all eval-builder / claim-evidence integration work (previously on dchiang/claim-evidence-review via PR #1441's chain) plus pre-integrated upstream tips: scosman/evals_v2, scosman/code_tools, and sfierro/sdg-batch-plan (which brings data-guide-v2-jobs and the eval-runner transient-retry fix).

Replaces PR #1441 and the dchiang/multiturn-synthetic-user middle hop - this branch supersedes that chain. When the pre-integrated upstream branches land on main and main flows into the megabranch, git dedupes the shared commits.

Notable resolutions in the sdg-batch-plan merge: /generate's prompts table keeps Sam's simplified version and the eval builder forks its full-featured variant locally; eval_steps_utils resurrected at its new $lib/utils location; the retry fix now also covers the v2 multi-turn re-drive path (new test).

Full checks green per commit; paid e2e harness run against the matching kiln_server megabranch.

🤖 Generated with Claude Code

[coderabbitai]

Actionable comments posted: 8

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

app/web_ui/static/fonts/OFL.txt (1)

1-93: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win

License file addition requires human sign-off, not agent action.

Adding a font license file is a legal/licensing action. As per coding guidelines, "Agents must never make legal decisions, including completing CLA attestations, setting license metadata tags, or adding license files; humans must handle these actions." Please confirm a human has explicitly reviewed and approved bundling this license file (and the underlying Inter font) before merge.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@app/web_ui/static/fonts/OFL.txt` around lines 1 - 93, Human review and
explicit approval are required before adding the OFL.txt license and bundling
the underlying Inter font. Do not modify or finalize this licensing addition
until a human confirms approval for both the license file and font distribution.

Source: Coding guidelines

🧹 Nitpick comments (9)

AGENTS.md (1)

63-71: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Fix minor grammatical errors in the new instructions.

There are slight grammatical issues with "a CLA attestations" and "in metadata file".

📝 Proposed grammatical fixes

 ### Never Make Legal Decisions as an Agent
 
 Agents are not allowed to make any legal decisions, including:
- - Filling out a CLA attestations in a PR template
- - Setting a license tag in metadata file (OSS/MIT/etc)
+ - Filling out a CLA attestation in a PR template
+ - Setting a license tag in a metadata file (OSS/MIT/etc)
  - Adding license files
 
 These all must be done by humans.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@AGENTS.md` around lines 63 - 71, Correct the grammar in the “Never Make Legal
Decisions as an Agent” instructions: change “a CLA attestations” to “a CLA
attestation” and add the missing article in “in metadata file” so it reads “in a
metadata file.”

app/web_ui/src/routes/(app)/specs/[project_id]/[task_id]/builder/claim_card.svelte (1)

20-46: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Consider moving tokenize_evidence into claim_evidence.ts.

It's a pure string-parsing function with no Svelte dependency, matching the shape of resolve_citation_span (which already lives in claim_evidence.ts and is unit-tested in claim_evidence.test.ts). Every other non-trivial parsing helper in this feature has dedicated tests; this one currently doesn't, and can't easily get them while it's embedded in the component.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@app/web_ui/src/routes/`(app)/specs/[project_id]/[task_id]/builder/claim_card.svelte
around lines 20 - 46, Move the pure tokenize_evidence helper from the Svelte
component into claim_evidence.ts alongside resolve_citation_span, export it for
reuse, and import it in claim_card.svelte without changing its behavior. Add
unit tests in claim_evidence.test.ts covering citation tokenization, unmatched
citations, plain text, and mixed evidence.

libs/core/kiln_ai/synthetic_user/parser.py (1)

7-8: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Docstring says "Greedy" but the regex is non-greedy. Line 8 describes "Greedy first match" while _extract (Line 27) uses .*? (non-greedy) and its own docstring (Line 26) says "non-greedy content." The non-greedy form is what makes first-match-wins and duplicate-tag handling correct; the "Greedy" wording could mislead a future edit toward .* and break that. Suggest rewording to "First match, non-greedy content per known tag."

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@libs/core/kiln_ai/synthetic_user/parser.py` around lines 7 - 8, Update the
tag-rules documentation near _extract to replace “Greedy first match per known
tag” with wording that accurately describes first-match behavior using
non-greedy content. Keep the existing nested-tag limitation and regex
implementation unchanged.

app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/data_guide_setup/guide_refine_view.svelte (1)

99-103: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Drop the "replaces the old standalone button" reference.

This comment explains what changed relative to the prior UI rather than documenting current behavior.

As per path instructions, "Do not add comments in code explaining changes; explain changes in chat instead."

✏️ Proposed fix

   // --- Edit chooser dialog ---
   // The header "Edit" action opens this first, letting the user pick between
-  // editing the guide text by hand or reviewing generated examples (and
-  // refining). Replaces the old standalone "Refine Data Guide" button.
+  // editing the guide text by hand or reviewing generated examples (and
+  // refining).

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@app/web_ui/src/routes/`(app)/generate/[project_id]/[task_id]/data_guide_setup/guide_refine_view.svelte
around lines 99 - 103, Remove the historical “Replaces the old standalone
‘Refine Data Guide’ button” wording from the comment above edit_chooser_dialog,
leaving only documentation of the dialog’s current behavior.

Source: Path instructions

app/web_ui/src/lib/stores/kiln_pro_batch_store.ts (1)

38-60: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low value

Optional: chain polls with setTimeout to avoid overlapping in-flight requests.

setInterval fires every POLL_INTERVAL_MS regardless of whether the previous async tick() has resolved. If a status GET ever exceeds the interval, requests overlap and a slower/older response can land after a newer one. Self-scheduling the next poll after each completion sidesteps this.

♻️ Suggested pattern

-  timer = setInterval(tick, POLL_INTERVAL_MS)
-  // Kick off an immediate poll so the UI updates without waiting a full tick.
-  tick()
+  const loop = async () => {
+    await tick()
+    if (timer !== null || is_cancelled()) return
+    timer = setTimeout(loop, POLL_INTERVAL_MS)
+  }
+  // Kick off an immediate poll so the UI updates without waiting a full tick.
+  loop()

Note this requires reworking stop()/timer to a setTimeout handle. Given it polls the local server every 1s, the risk is low.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@app/web_ui/src/lib/stores/kiln_pro_batch_store.ts` around lines 38 - 60,
Optionally replace the setInterval scheduling around the async tick function
with a self-scheduling setTimeout loop that schedules the next poll only after
each tick completes, preventing overlapping requests and stale responses. Update
timer and stop to manage the timeout handle while preserving immediate polling,
cancellation, error handling, and stopping when the status is no longer running.

app/desktop/studio_server/jobs/test_api.py (1)

105-114: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Several tests leave long-sleeping jobs uncancelled, unlike the rest of the file.

test_list_filter_by_type (project_scoped sleeps 5s), test_list_filter_by_project_id (two 5s project_scoped jobs), test_list_limit, and test_list_since_excludes_older (multiple ~2.5s noop jobs) never cancel or await-terminal the jobs they create, unlike every other job-creating test in this file. This leaves dangling asyncio.sleep() tasks that get destroyed (not cancelled) when the test's event loop tears down, typically producing "Task was destroyed but it is pending!" warnings.

Fixing this once in the registry fixture teardown is more robust than patching each test individually.

♻️ Proposed fixture-level cleanup

-@pytest.fixture
-def registry(monkeypatch):
+@pytest_asyncio.fixture
+async def registry(monkeypatch):
     """Patch a fresh registry in for isolation, then register the test workers."""
     reg = JobRegistry(max_concurrent=10)
     monkeypatch.setattr(jobs_api, "job_registry", reg)
     reg.register_type(NoopJobWorker)
     reg.register_type(ProjectScopedWorker)
     reg.register_type(ReconcileCompleteWorker)
     reg.register_type(NonPausableWorker)
-    return reg
+    yield reg
+    # Guard against tests forgetting to clean up their jobs: cancel anything
+    # still non-terminal so no sleeping task outlives the test's event loop.
+    for job_id, record in list(reg._jobs.items()):
+        if not record.status.is_terminal:
+            await _safe_cancel(reg, job_id)

Also applies to: 253-262, 274-286, 288-293, 296-304

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@app/desktop/studio_server/jobs/test_api.py` around lines 105 - 114, Update
the registry fixture to perform teardown cleanup after each test: cancel all
jobs still tracked by the JobRegistry and await their terminal completion before
the event loop is closed. Keep the existing isolated registry setup and worker
registrations unchanged, and use the registry’s existing job-management APIs
rather than modifying the individual list/filter tests.

app/web_ui/src/lib/stores/jobs_store.ts (1)

207-224: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

synced/connection exports bypass the subscriber-count-driven connection lifecycle.

The exported synced/connection readables wrap the raw internal writable .subscribe directly, unlike jobs/active_jobs_count which go through the custom subscribe that increments subscriber_count to open/close the EventSource. Today every consumer also subscribes to jobs/active_jobs_count alongside them, so the connection still opens — but any future component that subscribes only to synced/connection (without the job list) would silently never trigger connect(), freezing at idle/false forever.

♻️ Proposed fix: route synced/connection through the same subscriber-count tracking

+  function wrap_readable<T>(store: { subscribe: Readable<T>["subscribe"] }): Readable<T>["subscribe"] {
+    return (run, invalidate) => {
+      if (subscriber_count === 0) {
+        connect()
+      }
+      subscriber_count += 1
+      const unsub = store.subscribe(run, invalidate)
+      return () => {
+        unsub()
+        subscriber_count -= 1
+        if (subscriber_count <= 0) {
+          subscriber_count = 0
+          disconnect()
+        }
+      }
+    }
+  }
+
   return {
     subscribe,
-    synced: { subscribe: synced.subscribe } as Readable<boolean>,
-    connection: {
-      subscribe: connection.subscribe,
-    } as Readable<JobsConnection>,
+    synced: { subscribe: wrap_readable(synced) } as Readable<boolean>,
+    connection: { subscribe: wrap_readable(connection) } as Readable<JobsConnection>,
     set_project,
     _disconnect: disconnect,
   }

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@app/web_ui/src/lib/stores/jobs_store.ts` around lines 207 - 224, Update the
returned `synced` and `connection` readables in `createJobsStore` to subscribe
through the store’s custom subscriber-counting `subscribe` lifecycle, rather
than directly wrapping the internal writable subscriptions. Ensure subscribers
to either export independently trigger `connect()` and teardown via
`disconnect()` consistently with `jobs` and `active_jobs_count`.

app/web_ui/src/routes/(app)/jobs/+page.svelte (1)

53-57: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Mark this placeholder page with a TODO per repo convention.

The temporary nature is only conveyed via UI text; a TODO comment would make it discoverable by TODO scans/repo tooling ahead of the "final PR" cleanup.

As per coding guidelines: "Use TODO comments for temporary code, placeholders, or work that must be addressed before merging, and remove all TODO comments before the final PR."

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@app/web_ui/src/routes/`(app)/jobs/+page.svelte around lines 53 - 57, Add a
TODO comment adjacent to the AppPage declaration identifying this route as a
temporary placeholder that must be removed before merging. Keep the existing
title, subtitle, and action_buttons behavior unchanged.

Source: Coding guidelines

libs/server/kiln_server/test_document_api.py (1)

4781-4821: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Test coverage gap vs. sibling tests.

Unlike test_run_extractor_config_no_tags/test_run_extractor_config_with_tags, this test doesn't assert mock_run_extractor.assert_called_once() or verify extractor_runner.documents, leaving the end-to-end wiring (filtered docs actually passed to the runner) unverified for the new document_ids path.

✅ Suggested additional assertions

         mock_get_documents_filtered.assert_called_once_with(
             mock_project,
             exclude_extracted_by_extractor_config_id=mock_extractor_config.id,
             target_tags=None,
             target_document_ids=["doc-a", "doc-b", "doc-c"],
         )
+        mock_run_extractor.assert_called_once()
+        call_args = mock_run_extractor.call_args[0]
+        extractor_runner = call_args[0]
+        assert len(extractor_runner.documents) == 1
+        assert extractor_runner.documents[0].id == document.id

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@libs/server/kiln_server/test_document_api.py` around lines 4781 - 4821,
Extend test_run_extractor_config_with_document_ids to assert mock_run_extractor
is called exactly once and verify its extractor_runner argument contains the
filtered document in extractor_runner.documents. Preserve the existing request,
filtering-arguments, and successful-response assertions while covering that
document_ids-selected documents are passed to the runner.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@app/desktop/studio_server/jobs/api.py`:
- Around line 276-289: Change the get_job_errors endpoint from async def to a
standard def function so FastAPI executes the synchronous
job_registry.run_id_for and error_log.read_errors calls in its threadpool, while
preserving the existing run ID resolution and empty-list behavior.

In `@app/web_ui/src/app.css`:
- Around line 2-15: Remove the quotes around InterVariable in both `@font-face`
font-family declarations and update every downstream font-family reference to
use the unquoted name, ensuring the web UI lint and format checks pass.

In `@app/web_ui/src/routes/`(app)/+layout.svelte:
- Around line 454-457: Remove the existing ProgressWidget invocation from the
sidebar footer, leaving DataGuideProgressWidget as the sole progress widget
inside the footer li.

In
`@app/web_ui/src/routes/`(app)/generate/[project_id]/[task_id]/data_guide_setup/guide_refine_view.svelte:
- Around line 81-83: Update the template around the settings widget and Verify
Changes controls so the settings widget always renders, removing the
editing_has_changes conditional that hides it when there are no edits. Keep it
disabled using the existing verify_disabled state, and ensure the related UI
behavior matches the comment’s “always shown but disabled” intent.

In
`@app/web_ui/src/routes/`(app)/generate/[project_id]/[task_id]/data_guide/refine/+page.svelte:
- Around line 116-122: Update both preview-sample initialization paths,
including run_initial_preview and handle_refine, to pass the API response
through dedupe_by_input before assigning preview_samples or mapping into
reviewed_samples. Match the existing behavior in data_guide_setup/+page.svelte
while preserving the current reviewed-samples fields.

In
`@app/web_ui/src/routes/`(app)/specs/[project_id]/[task_id]/builder/claim_evidence.ts:
- Around line 156-186: Guard the final judgement grading in
build_claim_review_payload so a verdict with agrees === null cannot be passed to
graded_claim and persisted as "disagree". Validate final_judgement_verdict
before constructing the payload, using the existing reviewed-state behavior and
failing or otherwise rejecting incomplete reviews consistently with the
surrounding flow.

In
`@app/web_ui/src/routes/`(app)/specs/[project_id]/[task_id]/builder/plan_prompts_table.svelte:
- Around line 45-66: Replace the hard-coded edit_dialog_id and
document.getElementById lookups in open_edit and save_edit with a local dialog
element reference using Svelte bind:this. Use that reference to call showModal
and close, remove both `@ts-expect-error` directives, and preserve the existing
edit and save behavior.

In
`@app/web_ui/src/routes/`(app)/specs/[project_id]/[task_id]/compare/+page.svelte:
- Around line 1090-1106: Update the conditional rendering block around
eval_data_cache[section.eval_id]?.eval_input_filter_id to first ensure
eval_data_cache[section.eval_id] is loaded, matching the guard used by the
nearby has_default_eval_config === false branch. Keep showing “Add Eval Data”
only for loaded evals without eval_input_filter_id, and show the eval-builder
message when the loaded metadata identifies an EvalInput-typed slice.

---

Outside diff comments:
In `@app/web_ui/static/fonts/OFL.txt`:
- Around line 1-93: Human review and explicit approval are required before
adding the OFL.txt license and bundling the underlying Inter font. Do not modify
or finalize this licensing addition until a human confirms approval for both the
license file and font distribution.

---

Nitpick comments:
In `@AGENTS.md`:
- Around line 63-71: Correct the grammar in the “Never Make Legal Decisions as
an Agent” instructions: change “a CLA attestations” to “a CLA attestation” and
add the missing article in “in metadata file” so it reads “in a metadata file.”

In `@app/desktop/studio_server/jobs/test_api.py`:
- Around line 105-114: Update the registry fixture to perform teardown cleanup
after each test: cancel all jobs still tracked by the JobRegistry and await
their terminal completion before the event loop is closed. Keep the existing
isolated registry setup and worker registrations unchanged, and use the
registry’s existing job-management APIs rather than modifying the individual
list/filter tests.

In `@app/web_ui/src/lib/stores/jobs_store.ts`:
- Around line 207-224: Update the returned `synced` and `connection` readables
in `createJobsStore` to subscribe through the store’s custom subscriber-counting
`subscribe` lifecycle, rather than directly wrapping the internal writable
subscriptions. Ensure subscribers to either export independently trigger
`connect()` and teardown via `disconnect()` consistently with `jobs` and
`active_jobs_count`.

In `@app/web_ui/src/lib/stores/kiln_pro_batch_store.ts`:
- Around line 38-60: Optionally replace the setInterval scheduling around the
async tick function with a self-scheduling setTimeout loop that schedules the
next poll only after each tick completes, preventing overlapping requests and
stale responses. Update timer and stop to manage the timeout handle while
preserving immediate polling, cancellation, error handling, and stopping when
the status is no longer running.

In
`@app/web_ui/src/routes/`(app)/generate/[project_id]/[task_id]/data_guide_setup/guide_refine_view.svelte:
- Around line 99-103: Remove the historical “Replaces the old standalone ‘Refine
Data Guide’ button” wording from the comment above edit_chooser_dialog, leaving
only documentation of the dialog’s current behavior.

In `@app/web_ui/src/routes/`(app)/jobs/+page.svelte:
- Around line 53-57: Add a TODO comment adjacent to the AppPage declaration
identifying this route as a temporary placeholder that must be removed before
merging. Keep the existing title, subtitle, and action_buttons behavior
unchanged.

In
`@app/web_ui/src/routes/`(app)/specs/[project_id]/[task_id]/builder/claim_card.svelte:
- Around line 20-46: Move the pure tokenize_evidence helper from the Svelte
component into claim_evidence.ts alongside resolve_citation_span, export it for
reuse, and import it in claim_card.svelte without changing its behavior. Add
unit tests in claim_evidence.test.ts covering citation tokenization, unmatched
citations, plain text, and mixed evidence.

In `@libs/core/kiln_ai/synthetic_user/parser.py`:
- Around line 7-8: Update the tag-rules documentation near _extract to replace
“Greedy first match per known tag” with wording that accurately describes
first-match behavior using non-greedy content. Keep the existing nested-tag
limitation and regex implementation unchanged.

In `@libs/server/kiln_server/test_document_api.py`:
- Around line 4781-4821: Extend test_run_extractor_config_with_document_ids to
assert mock_run_extractor is called exactly once and verify its extractor_runner
argument contains the filtered document in extractor_runner.documents. Preserve
the existing request, filtering-arguments, and successful-response assertions
while covering that document_ids-selected documents are passed to the runner.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: fca809ee-655c-4a03-ac83-12a36a813d43

📥 Commits

Reviewing files that changed from the base of the PR and between f428ef9 and daf73bf.

⛔ Files ignored due to path filters (47)

• app/desktop/studio_server/api_client/kiln_ai_server_client/api/chat/get_client_version_policy_v1_chat_version_policy_get.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/api/copilot/batch_plan_v1_copilot_batch_plan_post.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/api/copilot/build_claim_evidence_v1_copilot_build_claim_evidence_post.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/api/copilot/draft_input_data_guide_v1_copilot_draft_input_data_guide_post.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/api/copilot/refine_judge_prompt_v1_copilot_refine_judge_prompt_post.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/api/jobs/get_data_guide_job_result_v1_jobs_data_guide_job_job_id_result_get.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/api/jobs/start_data_guide_job_v1_jobs_data_guide_job_start_post.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/api/synthetic_user/__init__.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/api/synthetic_user/generate_v1_synthetic_user_generate_post.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/__init__.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/batch_plan_input.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/batch_plan_output.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/build_claim_evidence_input.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/build_claim_evidence_output.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/change.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/citation.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/claim.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/client_version_policy.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/data_guide_job_output.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/data_guide_job_result_response.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/draft_input_data_guide_input.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/draft_input_data_guide_output.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/expected_result.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/final_judgement.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/generate_synthetic_users_request.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/generate_synthetic_users_response.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/generate_v1_synthetic_user_generate_post_response_401.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/generate_v1_synthetic_user_generate_post_response_500.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/generate_v1_synthetic_user_generate_post_response_502.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/generate_v1_synthetic_user_generate_post_response_502_code.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/graded_claim.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/graded_final_judgement.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/graded_trace.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/human_grade.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/jinja_input_transform.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/job_type.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/judge_score.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/kiln_agent_run_config_properties.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/refine_judge_prompt_input.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/refine_judge_prompt_output.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/source.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/synthetic_user_case.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/validation_error.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/desktop/studio_server/api_client/kiln_ai_server_client/models/validation_error_context.py is excluded by !app/desktop/studio_server/api_client/kiln_ai_server_client/**
• app/web_ui/package-lock.json is excluded by !**/package-lock.json
• app/web_ui/static/fonts/InterVariable-Italic.woff2 is excluded by !**/*.woff2
• app/web_ui/static/fonts/InterVariable.woff2 is excluded by !**/*.woff2

📒 Files selected for processing (256)

• .agents/skills/claude-maintain-models/SKILL.md
• .agents/skills/release-digest/SKILL.md
• .agents/skills/release-digest/scripts/gather_changes.py
• AGENTS.md
• app/desktop/desktop_server.py
• app/desktop/dev_server.py
• app/desktop/git_sync/middleware.py
• app/desktop/git_sync/save_context.py
• app/desktop/git_sync/test_save_context.py
• app/desktop/studio_server/api_models/copilot_models.py
• app/desktop/studio_server/api_models/eval_builder_models.py
• app/desktop/studio_server/batch_plan_api.py
• app/desktop/studio_server/copilot_api.py
• app/desktop/studio_server/data_gen_api.py
• app/desktop/studio_server/eval_api.py
• app/desktop/studio_server/eval_builder_api.py
• app/desktop/studio_server/jobs/__init__.py
• app/desktop/studio_server/jobs/api.py
• app/desktop/studio_server/jobs/error_log.py
• app/desktop/studio_server/jobs/events.py
• app/desktop/studio_server/jobs/models.py
• app/desktop/studio_server/jobs/registry.py
• app/desktop/studio_server/jobs/test_api.py
• app/desktop/studio_server/jobs/test_error_log.py
• app/desktop/studio_server/jobs/test_events.py
• app/desktop/studio_server/jobs/test_registry.py
• app/desktop/studio_server/jobs/workers/__init__.py
• app/desktop/studio_server/jobs/workers/noop.py
• app/desktop/studio_server/multiturn_sdg_api.py
• app/desktop/studio_server/provider_api.py
• app/desktop/studio_server/synthetic_user/__init__.py
• app/desktop/studio_server/synthetic_user/client.py
• app/desktop/studio_server/synthetic_user/test_client.py
• app/desktop/studio_server/test_batch_plan_api.py
• app/desktop/studio_server/test_copilot_api.py
• app/desktop/studio_server/test_data_gen_api.py
• app/desktop/studio_server/test_eval_api.py
• app/desktop/studio_server/test_eval_builder_api.py
• app/desktop/studio_server/test_eval_builder_e2e_paid.py
• app/desktop/studio_server/test_multiturn_sdg_api.py
• app/desktop/studio_server/utils/copilot_utils.py
• app/desktop/studio_server/utils/eval_builder_utils.py
• app/desktop/studio_server/utils/response_utils.py
• app/desktop/studio_server/utils/test_copilot_utils.py
• app/web_ui/.env.example
• app/web_ui/src/app.css
• app/web_ui/src/lib/api_schema.d.ts
• app/web_ui/src/lib/chat/streaming_chat.ts
• app/web_ui/src/lib/components/SidebarJobsIndicator.svelte
• app/web_ui/src/lib/components/SidebarJobsIndicator.test.ts
• app/web_ui/src/lib/components/add_example_dialog.svelte
• app/web_ui/src/lib/components/eval_types/code_eval_helpers.test.ts
• app/web_ui/src/lib/components/eval_types/code_eval_helpers.ts
• app/web_ui/src/lib/components/extraction_dialog.svelte
• app/web_ui/src/lib/components/extractor_picker.svelte
• app/web_ui/src/lib/components/jobs_dialog.component.test.ts
• app/web_ui/src/lib/components/jobs_dialog.svelte
• app/web_ui/src/lib/components/jobs_table.svelte
• app/web_ui/src/lib/components/jobs_table.test.ts
• app/web_ui/src/lib/eval/default_judge.ts
• app/web_ui/src/lib/stores/data_guide_job_store.ts
• app/web_ui/src/lib/stores/job_status.test.ts
• app/web_ui/src/lib/stores/job_status.ts
• app/web_ui/src/lib/stores/jobs_api.test.ts
• app/web_ui/src/lib/stores/jobs_api.ts
• app/web_ui/src/lib/stores/jobs_dialog.test.ts
• app/web_ui/src/lib/stores/jobs_dialog.ts
• app/web_ui/src/lib/stores/jobs_store.test.ts
• app/web_ui/src/lib/stores/jobs_store.ts
• app/web_ui/src/lib/stores/kiln_pro_batch_store.ts
• app/web_ui/src/lib/ui/animations/refining_animation.svelte
• app/web_ui/src/lib/ui/data_guide_progress_widget.svelte
• app/web_ui/src/lib/ui/dialog.svelte
• app/web_ui/src/lib/ui/documents_table.svelte
• app/web_ui/src/lib/ui/floating_menu.svelte
• app/web_ui/src/lib/ui/floating_menu_types.ts
• app/web_ui/src/lib/ui/icons/copy_documents_icon.svelte
• app/web_ui/src/lib/ui/icons/csv_icon.svelte
• app/web_ui/src/lib/ui/icons/document_icon.svelte
• app/web_ui/src/lib/ui/icons/jobs_icon.svelte
• app/web_ui/src/lib/ui/icons/settings_gear_icon.svelte
• app/web_ui/src/lib/ui/icons/stars_icon.svelte
• app/web_ui/src/lib/ui/increment_ui.svelte
• app/web_ui/src/lib/ui/increment_ui.test.ts
• app/web_ui/src/lib/ui/kiln_copilot/copilot_auth_page.svelte
• app/web_ui/src/lib/ui/run_config_component/run_config_component.svelte
• app/web_ui/src/lib/ui/see_all_dialog.svelte
• app/web_ui/src/lib/utils/copilot_utils.ts
• app/web_ui/src/lib/utils/dedupe_by_input.ts
• app/web_ui/src/lib/utils/eval_steps_utils.ts
• app/web_ui/src/lib/utils/formatters.ts
• app/web_ui/src/lib/utils/get_eval_steps.test.ts
• app/web_ui/src/lib/utils/sse.test.ts
• app/web_ui/src/lib/utils/sse.ts
• app/web_ui/src/lib/utils/task_run_picker.svelte
• app/web_ui/src/routes/(app)/+layout.svelte
• app/web_ui/src/routes/(app)/docs/library/[project_id]/+page.svelte
• app/web_ui/src/routes/(app)/docs/library/[project_id]/upload_file_dialog.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/data_gen_intro.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/data_guide/+page.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/data_guide/refine/+page.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/data_guide/refine_handoff_store.ts
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/data_guide_chooser/+page.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/data_guide_chooser/+page.ts
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/data_guide_setup/+page.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/data_guide_setup/generation_settings_trigger.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/data_guide_setup/guide_preview.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/data_guide_setup/guide_refine_view.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/data_guide_setup/guide_setup_form.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/data_guide_setup/run_options_tiles.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/data_guide_setup_copilot/+page.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/data_guide_setup_copilot/+page.ts
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/data_guide_setup_copilot/[job_id]/+page.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/data_guide_setup_copilot/[job_id]/+page.ts
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/data_guide_setup_copilot/add_manual_structured_dialog.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/data_guide_setup_copilot/add_samples_picker_dialog.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/data_guide_setup_copilot/all_samples_dialog.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/data_guide_setup_copilot/existing_run_picker_dialog.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/data_guide_setup_copilot/import_csv_dialog.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/data_guide_setup_copilot/input_examples_uploader.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/data_guide_setup_copilot/pending_draft_store.ts
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/data_guide_setup_copilot/select_from_library_dialog.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/data_guide_setup_copilot/tag_first_selector.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/generate_samples_modal.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/generated_data_node.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/kiln_pro_batch_plan.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/kiln_pro_inputs.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/kiln_pro_plan_summary.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/kiln_pro_plans_table.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/kiln_pro_prompts_table.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/qna/+page.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/qna/extraction_dialog.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/synth/+page.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/synth_batch_chooser.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/synth_data_guidance_datamodel.test.ts
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/synth_data_guidance_datamodel.ts
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/synth_data_guide.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/synth_kiln_pro.svelte
• app/web_ui/src/routes/(app)/generate/data_guide_pro_auth/+page.svelte
• app/web_ui/src/routes/(app)/generate/data_guide_pro_auth/+page.ts
• app/web_ui/src/routes/(app)/jobs/+page.svelte
• app/web_ui/src/routes/(app)/sidebar_rail.svelte
• app/web_ui/src/routes/(app)/sidebar_rail_item.svelte
• app/web_ui/src/routes/(app)/sidebar_rail_item.test.ts
• app/web_ui/src/routes/(app)/specs/[project_id]/[task_id]/+page.svelte
• app/web_ui/src/routes/(app)/specs/[project_id]/[task_id]/[spec_id]/[eval_id]/+page.svelte
• app/web_ui/src/routes/(app)/specs/[project_id]/[task_id]/[spec_id]/[eval_id]/compare_run_configs/+page.svelte
• app/web_ui/src/routes/(app)/specs/[project_id]/[task_id]/[spec_id]/[eval_id]/eval_configs/+page.svelte
• app/web_ui/src/routes/(app)/specs/[project_id]/[task_id]/builder/+page.svelte
• app/web_ui/src/routes/(app)/specs/[project_id]/[task_id]/builder/+page.ts
• app/web_ui/src/routes/(app)/specs/[project_id]/[task_id]/builder/batch_plan_approval.svelte
• app/web_ui/src/routes/(app)/specs/[project_id]/[task_id]/builder/batch_plan_guidance.ts
• app/web_ui/src/routes/(app)/specs/[project_id]/[task_id]/builder/claim_card.svelte
• app/web_ui/src/routes/(app)/specs/[project_id]/[task_id]/builder/claim_evidence.test.ts
• app/web_ui/src/routes/(app)/specs/[project_id]/[task_id]/builder/claim_evidence.ts
• app/web_ui/src/routes/(app)/specs/[project_id]/[task_id]/builder/claim_evidence_review.svelte
• app/web_ui/src/routes/(app)/specs/[project_id]/[task_id]/builder/claim_trace_modal.svelte
• app/web_ui/src/routes/(app)/specs/[project_id]/[task_id]/builder/plan_prompts_table.svelte
• app/web_ui/src/routes/(app)/specs/[project_id]/[task_id]/compare/+page.svelte
• app/web_ui/src/routes/(app)/specs/[project_id]/[task_id]/spec_builder/+page.svelte
• app/web_ui/src/routes/(app)/specs/[project_id]/[task_id]/spec_builder/create_spec_form.svelte
• app/web_ui/src/routes/(fullscreen)/setup/(setup)/create_task/edit_task.test.ts
• app/web_ui/static/fonts/OFL.txt
• app/web_ui/tailwind.config.js
• app/web_ui/tests/e2e/act/discover/docs-library.spec.ts
• app/web_ui/tests/e2e/act_data_guide_actions.spec.ts
• app/web_ui/tests/e2e/act_data_guide_routing.spec.ts
• app/web_ui/tests/e2e/act_kiln_pro_batch_generate.spec.ts
• app/web_ui/tests/e2e/act_kiln_pro_batch_plan.spec.ts
• app/web_ui/tests/e2e/mock_kiln_server/server.ts
• libs/core/kiln_ai/adapters/data_gen/data_gen_prompts.py
• libs/core/kiln_ai/adapters/data_gen/data_gen_task.py
• libs/core/kiln_ai/adapters/data_gen/test_data_gen_task.py
• libs/core/kiln_ai/adapters/eval/base_eval.py
• libs/core/kiln_ai/adapters/eval/drive_fingerprint.py
• libs/core/kiln_ai/adapters/eval/eval_helpers.py
• libs/core/kiln_ai/adapters/eval/eval_runner.py
• libs/core/kiln_ai/adapters/eval/registry.py
• libs/core/kiln_ai/adapters/eval/test_base_eval.py
• libs/core/kiln_ai/adapters/eval/test_drive_fingerprint.py
• libs/core/kiln_ai/adapters/eval/test_eval_helpers.py
• libs/core/kiln_ai/adapters/eval/test_eval_runner.py
• libs/core/kiln_ai/adapters/eval/test_registry.py
• libs/core/kiln_ai/adapters/eval/test_v2_eval_code_eval.py
• libs/core/kiln_ai/adapters/eval/v2_eval_code_eval.py
• libs/core/kiln_ai/adapters/ml_model_list.py
• libs/core/kiln_ai/adapters/model_adapters/adapter_stream.py
• libs/core/kiln_ai/adapters/model_adapters/litellm_adapter.py
• libs/core/kiln_ai/adapters/model_adapters/test_adapter_stream.py
• libs/core/kiln_ai/adapters/model_adapters/test_litellm_adapter.py
• libs/core/kiln_ai/adapters/model_adapters/test_thinking_level_paid.py
• libs/core/kiln_ai/datamodel/__init__.py
• libs/core/kiln_ai/datamodel/claim_review.py
• libs/core/kiln_ai/datamodel/copilot_models/input_data_guide.py
• libs/core/kiln_ai/datamodel/data_guide.py
• libs/core/kiln_ai/datamodel/eval.py
• libs/core/kiln_ai/datamodel/json_schema.py
• libs/core/kiln_ai/datamodel/task_run.py
• libs/core/kiln_ai/datamodel/test_claim_review.py
• libs/core/kiln_ai/datamodel/test_eval_model.py
• libs/core/kiln_ai/datamodel/test_json_schema.py
• libs/core/kiln_ai/synthetic_user/__init__.py
• libs/core/kiln_ai/synthetic_user/case.py
• libs/core/kiln_ai/synthetic_user/drive_loop.py
• libs/core/kiln_ai/synthetic_user/driver.py
• libs/core/kiln_ai/synthetic_user/eval_drive.py
• libs/core/kiln_ai/synthetic_user/models.py
• libs/core/kiln_ai/synthetic_user/parser.py
• libs/core/kiln_ai/synthetic_user/prompt.py
• libs/core/kiln_ai/synthetic_user/role_swap.py
• libs/core/kiln_ai/synthetic_user/runner.py
• libs/core/kiln_ai/synthetic_user/test_case.py
• libs/core/kiln_ai/synthetic_user/test_drive_loop.py
• libs/core/kiln_ai/synthetic_user/test_driver.py
• libs/core/kiln_ai/synthetic_user/test_eval_drive.py
• libs/core/kiln_ai/synthetic_user/test_models.py
• libs/core/kiln_ai/synthetic_user/test_parser.py
• libs/core/kiln_ai/synthetic_user/test_prompt.py
• libs/core/kiln_ai/synthetic_user/test_role_swap.py
• libs/core/kiln_ai/synthetic_user/test_runner.py
• libs/core/kiln_ai/tools/mcp_session_manager.py
• libs/core/kiln_ai/utils/jinja_engine.py
• libs/core/kiln_ai/utils/test_jinja_engine.py
• libs/server/kiln_server/document_api.py
• libs/server/kiln_server/server.py
• libs/server/kiln_server/spec_api.py
• libs/server/kiln_server/test_document_api.py
• libs/server/kiln_server/utils/agent_checks/annotations/delete_api_jobs_id.json
• libs/server/kiln_server/utils/agent_checks/annotations/get_api_jobs.json
• libs/server/kiln_server/utils/agent_checks/annotations/get_api_jobs_events.json
• libs/server/kiln_server/utils/agent_checks/annotations/get_api_jobs_id.json
• libs/server/kiln_server/utils/agent_checks/annotations/get_api_jobs_id_errors.json
• libs/server/kiln_server/utils/agent_checks/annotations/get_api_jobs_id_result.json
• libs/server/kiln_server/utils/agent_checks/annotations/get_api_jobs_id_wait.json
• libs/server/kiln_server/utils/agent_checks/annotations/get_api_projects_project_id_tasks_task_id_copilot_data_guide_job_job_id_result.json
• libs/server/kiln_server/utils/agent_checks/annotations/get_api_projects_project_id_tasks_task_id_copilot_data_guide_job_job_id_status.json
• libs/server/kiln_server/utils/agent_checks/annotations/get_api_projects_project_id_tasks_task_id_generate_inputs_batch_job_id.json
• libs/server/kiln_server/utils/agent_checks/annotations/get_api_projects_project_id_tasks_task_id_generate_outputs_batch_job_id.json
• libs/server/kiln_server/utils/agent_checks/annotations/get_api_provider_verify_kiln_copilot_api_key.json
• libs/server/kiln_server/utils/agent_checks/annotations/post_api_copilot_classify_spec_description.json
• libs/server/kiln_server/utils/agent_checks/annotations/post_api_jobs_id_cancel.json
• libs/server/kiln_server/utils/agent_checks/annotations/post_api_jobs_id_pause.json
• libs/server/kiln_server/utils/agent_checks/annotations/post_api_jobs_id_resume.json
• libs/server/kiln_server/utils/agent_checks/annotations/post_api_jobs_type.json
• libs/server/kiln_server/utils/agent_checks/annotations/post_api_projects_project_id_tasks_task_id_copilot_batch_plan.json
• libs/server/kiln_server/utils/agent_checks/annotations/post_api_projects_project_id_tasks_task_id_copilot_data_guide_job_start.json
• libs/server/kiln_server/utils/agent_checks/annotations/post_api_projects_project_id_tasks_task_id_copilot_parse_import_file.json
• libs/server/kiln_server/utils/agent_checks/annotations/post_api_projects_project_id_tasks_task_id_eval_builder_build_claims.json
• libs/server/kiln_server/utils/agent_checks/annotations/post_api_projects_project_id_tasks_task_id_eval_builder_refine_judge.json
• libs/server/kiln_server/utils/agent_checks/annotations/post_api_projects_project_id_tasks_task_id_eval_builder_review_pipeline.json
• libs/server/kiln_server/utils/agent_checks/annotations/post_api_projects_project_id_tasks_task_id_eval_builder_review_traces.json
• libs/server/kiln_server/utils/agent_checks/annotations/post_api_projects_project_id_tasks_task_id_generate_inputs_batch.json
• libs/server/kiln_server/utils/agent_checks/annotations/post_api_projects_project_id_tasks_task_id_generate_outputs_batch.json
• libs/server/kiln_server/utils/agent_checks/annotations/post_api_projects_project_id_tasks_task_id_multiturn_sdg_generate_cases.json
• libs/server/kiln_server/utils/agent_checks/annotations/post_api_projects_project_id_tasks_task_id_multiturn_sdg_run_cases_batch.json
• specs/projects/evals_v2/spec_fidelity_review/unit_27-code-eval.md

💤 Files with no reviewable changes (2)

• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/qna/extraction_dialog.svelte
• app/web_ui/src/routes/(app)/generate/[project_id]/[task_id]/generated_data_node.svelte

[coderabbitai]

🚀 Performance & Scalability | 🟡 Minor | ⚡ Quick win

Avoid blocking the event loop with synchronous file I/O.

error_log.read_errors performs synchronous file I/O. Since this endpoint is defined as async def, the synchronous file read will block the asyncio event loop.

To offload this blocking call to FastAPI's threadpool, you can define the endpoint as a standard def function instead of async def (since job_registry.run_id_for is also synchronous).

⚡ Proposed fix

-    async def get_job_errors(
+    def get_job_errors(
         id: Annotated[str, Path(description="The job id.")],
         run_id: Annotated[
             str | None,

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	async def get_job_errors(
	id: Annotated[str, Path(description="The job id.")],
	run_id: Annotated[
	str | None,
	Query(description="Read the error log for a specific past run id."),
	] = None,
	) -> list[dict[str, Any]]:
	# Always 200, never errors (functional_spec §5). A plain non-reconciling
	# lookup of the current run_id — we don't recompute state for a
	# best-effort diagnostic read.
	resolved_run_id = run_id or job_registry.run_id_for(id)
	if resolved_run_id is None:
	return []
	return error_log.read_errors(resolved_run_id)
	def get_job_errors(
	id: Annotated[str, Path(description="The job id.")],
	run_id: Annotated[
	str | None,
	Query(description="Read the error log for a specific past run id."),
	] = None,
	) -> list[dict[str, Any]]:
	# Always 200, never errors (functional_spec §5). A plain non-reconciling
	# lookup of the current run_id — we don't recompute state for a
	# best-effort diagnostic read.
	resolved_run_id = run_id or job_registry.run_id_for(id)
	if resolved_run_id is None:
	return []
	return error_log.read_errors(resolved_run_id)

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@app/desktop/studio_server/jobs/api.py` around lines 276 - 289, Change the
get_job_errors endpoint from async def to a standard def function so FastAPI
executes the synchronous job_registry.run_id_for and error_log.read_errors calls
in its threadpool, while preserving the existing run ID resolution and
empty-list behavior.

[coderabbitai]

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Drop the quotes around InterVariable to satisfy stylelint.

Stylelint (font-family-name-quotes) flags the quoted name on lines 3 and 10, which will fail the web lint gate. Ensure every downstream font-family reference uses the unquoted name too. As per coding guidelines, web UI lint/format checks must pass before merging.

🔧 Proposed fix

 `@font-face` {
-  font-family: "InterVariable";
+  font-family: InterVariable;
   font-style: normal;
@@
 `@font-face` {
-  font-family: "InterVariable";
+  font-family: InterVariable;
   font-style: italic;

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	@font-face {
	font-family: "InterVariable";
	font-style: normal;
	font-weight: 100 900;
	font-display: swap;
	src: url("/fonts/InterVariable.woff2") format("woff2");
	}
	@font-face {
	font-family: "InterVariable";
	font-style: italic;
	font-weight: 100 900;
	font-display: swap;
	src: url("/fonts/InterVariable-Italic.woff2") format("woff2");
	}
	`@font-face` {
	font-family: InterVariable;
	font-style: normal;
	font-weight: 100 900;
	font-display: swap;
	src: url("/fonts/InterVariable.woff2") format("woff2");
	}
	`@font-face` {
	font-family: InterVariable;
	font-style: italic;
	font-weight: 100 900;
	font-display: swap;
	src: url("/fonts/InterVariable-Italic.woff2") format("woff2");
	}

🧰 Tools

🪛 Stylelint (17.14.0)

[error] 3-3: Expected no quotes around "InterVariable" (font-family-name-quotes)

(font-family-name-quotes)

---

[error] 10-10: Expected no quotes around "InterVariable" (font-family-name-quotes)

(font-family-name-quotes)

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@app/web_ui/src/app.css` around lines 2 - 15, Remove the quotes around
InterVariable in both `@font-face` font-family declarations and update every
downstream font-family reference to use the unquoted name, ensuring the web UI
lint and format checks pass.

Sources: Coding guidelines, Linters/SAST tools

[coderabbitai]

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Two progress widgets now render side-by-side instead of one.

The line-range summary describes this as swapping ProgressWidget for DataGuideProgressWidget, but the code adds DataGuideProgressWidget while leaving the existing <ProgressWidget /> in place — both now render in the sidebar footer <li>. If this wasn't intentional (i.e. ProgressWidget should have been removed), it duplicates progress UI for every user.

🔧 If the old widget should be removed

         <li class="mt-auto pt-2 bg-transparent">
           <DataGuideProgressWidget />
-          <ProgressWidget />
         </li>

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	<li class="mt-auto pt-2 bg-transparent">
	<DataGuideProgressWidget />
	<ProgressWidget />
	</li>
	<li class="mt-auto pt-2 bg-transparent">
	<DataGuideProgressWidget />
	</li>

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@app/web_ui/src/routes/`(app)/+layout.svelte around lines 454 - 457, Remove
the existing ProgressWidget invocation from the sidebar footer, leaving
DataGuideProgressWidget as the sole progress widget inside the footer li.

[coderabbitai]

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Settings widget is hidden, not just disabled, when there are no edits.

Comment says the widget is "always shown but disabled," but the template gates it entirely behind {#if editing_has_changes}. Either the conditional should be dropped so the widget always renders (matching the comment/intended UX), or the comment should be corrected to describe the actual hide/show behavior.

Also applies to: 370-376

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@app/web_ui/src/routes/`(app)/generate/[project_id]/[task_id]/data_guide_setup/guide_refine_view.svelte
around lines 81 - 83, Update the template around the settings widget and Verify
Changes controls so the settings widget always renders, removing the
editing_has_changes conditional that hides it when there are no edits. Keep it
disabled using the existing verify_disabled state, and ensure the related UI
behavior matches the comment’s “always shown but disabled” intent.

[coderabbitai]

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Missing dedupe_by_input here, unlike the sibling setup flow.

Both run_initial_preview and handle_refine map preview samples straight into reviewed_samples without deduping. The parallel flow in data_guide_setup/+page.svelte wraps the identical API response in dedupe_by_input(data as PreviewSample[]) before mapping. Since both hit the same data_gen_guide_preview endpoint, this page can surface duplicate rows that the setup flow already guards against.

🔧 Proposed fix

+import { dedupe_by_input } from "$lib/utils/dedupe_by_input"
...
       if (!data) throw new KilnError("No preview inputs returned", null)

-      preview_samples = data as PreviewSample[]
+      preview_samples = dedupe_by_input(data as PreviewSample[])
...
       if (!preview_data) throw new KilnError("No preview inputs returned", null)

-      guide = refined_guide
-      preview_samples = preview_data as PreviewSample[]
+      guide = refined_guide
+      preview_samples = dedupe_by_input(preview_data as PreviewSample[])

Also applies to: 190-194

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@app/web_ui/src/routes/`(app)/generate/[project_id]/[task_id]/data_guide/refine/+page.svelte
around lines 116 - 122, Update both preview-sample initialization paths,
including run_initial_preview and handle_refine, to pass the API response
through dedupe_by_input before assigning preview_samples or mapping into
reviewed_samples. Match the existing behavior in data_guide_setup/+page.svelte
while preserving the current reviewed-samples fields.

[coderabbitai]

🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

build_claim_review_payload can silently mislabel an ungraded final judgement as "disagree".

graded_claim maps verdict.agrees truthy→"agree", falsy→"disagree", so agrees === null (not yet reviewed) collapses into "disagree". build_claim_review_payload filters ungraded sub-claims before grading them, but calls graded_claim on final_judgement_verdict unconditionally. The only current call site (build_graded_traces) is safe because it pre-filters via is_trace_reviewed, but the function itself has no guard — a future direct call (bypassing that filter) would persist a wrong, silent grade rather than failing.

🛡️ Proposed guard

 export function build_claim_review_payload(
   trace: TraceClaims,
   review: TraceReview,
 ): ClaimReviewPayload {
+  if (review.final_judgement_verdict.agrees === null) {
+    throw new Error(
+      "build_claim_review_payload called before the final judgement was graded",
+    )
+  }
   return {
     judge_score: trace.judge_score,

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	function graded_claim(claim: Claim, verdict: ClaimVerdict): GradedClaim {
	return {
	claim: claim.claim,
	evidence: claim.evidence,
	expected_result: claim.expected_result,
	human_grade: verdict.agrees ? "agree" : "disagree",
	human_feedback: verdict.why.trim() || null,
	}
	}
	// Build the persisted per-claim grades for one reviewed trace. Only claims
	// the reviewer actually graded are included (sub-claim verdicts are
	// optional); the final judgement is always graded by the time save is
	// reachable (is_trace_reviewed gates it).
	export function build_claim_review_payload(
	trace: TraceClaims,
	review: TraceReview,
	): ClaimReviewPayload {
	return {
	judge_score: trace.judge_score,
	judge_reasoning: trace.judge_reasoning,
	claims: trace.claims
	.map((claim, i) => ({ claim, verdict: review.claim_verdicts[i] }))
	.filter(({ verdict }) => verdict && verdict.agrees !== null)
	.map(({ claim, verdict }) => graded_claim(claim, verdict)),
	final_judgement: graded_claim(
	trace.final_judgement,
	review.final_judgement_verdict,
	),
	}
	}
	function graded_claim(claim: Claim, verdict: ClaimVerdict): GradedClaim {
	return {
	claim: claim.claim,
	evidence: claim.evidence,
	expected_result: claim.expected_result,
	human_grade: verdict.agrees ? "agree" : "disagree",
	human_feedback: verdict.why.trim() || null,
	}
	}
	// Build the persisted per-claim grades for one reviewed trace. Only claims
	// the reviewer actually graded are included (sub-claim verdicts are
	// optional); the final judgement is always graded by the time save is
	// reachable (is_trace_reviewed gates it).
	export function build_claim_review_payload(
	trace: TraceClaims,
	review: TraceReview,
	): ClaimReviewPayload {
	if (review.final_judgement_verdict.agrees === null) {
	throw new Error(
	"build_claim_review_payload called before the final judgement was graded",
	)
	}
	return {
	judge_score: trace.judge_score,
	judge_reasoning: trace.judge_reasoning,
	claims: trace.claims
	.map((claim, i) => ({ claim, verdict: review.claim_verdicts[i] }))
	.filter(({ verdict }) => verdict && verdict.agrees !== null)
	.map(({ claim, verdict }) => graded_claim(claim, verdict)),
	final_judgement: graded_claim(
	trace.final_judgement,
	review.final_judgement_verdict,
	),
	}
	}

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@app/web_ui/src/routes/`(app)/specs/[project_id]/[task_id]/builder/claim_evidence.ts
around lines 156 - 186, Guard the final judgement grading in
build_claim_review_payload so a verdict with agrees === null cannot be passed to
graded_claim and persisted as "disagree". Validate final_judgement_verdict
before constructing the payload, using the existing reviewed-state behavior and
failing or otherwise rejecting incomplete reviews consistently with the
surrounding flow.

[coderabbitai]

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Avoid global getElementById lookup for the edit dialog — use bind:this instead.

The dialog is opened/closed via a hard-coded DOM id (document.getElementById(edit_dialog_id)). If this component is ever mounted twice on the same page (plausible given it's a general-purpose "builder-local plan table"), both instances would share the same DOM id, and one instance's edit action could open/mutate the wrong instance's dialog. A local element reference sidesteps this and also removes the two @ts-expect-error casts.

🛡️ Proposed fix

-  const edit_dialog_id = "builder_plan_prompt_edit_dialog"
   let editing_index: number | null = null
   let editing_value = ""
+  let edit_dialog_el: HTMLDialogElement | null = null

   function open_edit(index: number) {
     editing_index = index
     editing_value = prompts[index]
-    // `@ts-expect-error` showModal is not typed on HTMLElement
-    document.getElementById(edit_dialog_id)?.showModal()
+    edit_dialog_el?.showModal()
   }

   function save_edit() {
     if (editing_index !== null) {
       const value = editing_value.trim()
       if (value && value !== prompts[editing_index]) {
         on_edit?.(editing_index, value)
       }
     }
     editing_index = null
-    // `@ts-expect-error` close is not typed on HTMLElement
-    document.getElementById(edit_dialog_id)?.close()
+    edit_dialog_el?.close()
   }

-<dialog id={edit_dialog_id} class="modal">
+<dialog bind:this={edit_dialog_el} class="modal">

Also applies to: 218-218

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@app/web_ui/src/routes/`(app)/specs/[project_id]/[task_id]/builder/plan_prompts_table.svelte
around lines 45 - 66, Replace the hard-coded edit_dialog_id and
document.getElementById lookups in open_edit and save_edit with a local dialog
element reference using Svelte bind:this. Use that reference to call showModal
and close, remove both `@ts-expect-error` directives, and preserve the existing
edit and save behavior.

[coderabbitai]

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Gate on eval_data_cache loading state before choosing "Add Eval Data" vs. "No eval data".

eval_data_cache[section.eval_id]?.eval_input_filter_id is falsy while the eval metadata is still loading (before fetch_eval_data resolves), so this briefly renders — and lets users click — "Add Eval Data" for evals whose data is actually minted by the eval builder, then flips to the correct message once the fetch completes. The sibling branch just above (has_default_eval_config === false, lines ~983-1004) already guards this correctly with eval_data_cache[section.eval_id] !== undefined; this new block should follow the same pattern.

🛡️ Proposed fix

                                 <div class="text-left">
                                   {`#if` getEvalDatasetSize(section.eval_id) === 0}
-                                    {`#if` eval_data_cache[section.eval_id]?.eval_input_filter_id}
+                                    {`#if` eval_data_cache[section.eval_id] === undefined}
+                                      <div class="mt-1">
+                                        <div class="loading loading-spinner loading-xs"></div>
+                                      </div>
+                                    {:else if eval_data_cache[section.eval_id]?.eval_input_filter_id}
                                       <!-- EvalInput-typed slice: data is minted by the
                                         eval builder; the add-data flow tags TaskRuns,
                                         which doesn't apply. -->

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	{#if eval_data_cache[section.eval_id]?.eval_input_filter_id}
	<!-- EvalInput-typed slice: data is minted by the
	eval builder; the add-data flow tags TaskRuns,
	which doesn't apply. -->
	<div class="text-xs text-gray-500">
	No eval data. This eval's data is
	created by the eval builder.
	</div>
	{:else}
	<button
	class="btn btn-xs mt-1"
	on:click={() =>
	navigateToAddData(section.eval_id)}
	>
	Add Eval Data
	</button>
	{/if}
	{`#if` eval_data_cache[section.eval_id] === undefined}
	<div class="mt-1">
	<div class="loading loading-spinner loading-xs"></div>
	</div>
	{:else if eval_data_cache[section.eval_id]?.eval_input_filter_id}
	<!-- EvalInput-typed slice: data is minted by the
	eval builder; the add-data flow tags TaskRuns,
	which doesn't apply. -->
	<div class="text-xs text-gray-500">
	No eval data. This eval's data is
	created by the eval builder.
	</div>
	{:else}
	<button
	class="btn btn-xs mt-1"
	on:click={() =>
	navigateToAddData(section.eval_id)}
	>
	Add Eval Data
	</button>
	{/if}

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@app/web_ui/src/routes/`(app)/specs/[project_id]/[task_id]/compare/+page.svelte
around lines 1090 - 1106, Update the conditional rendering block around
eval_data_cache[section.eval_id]?.eval_input_filter_id to first ensure
eval_data_cache[section.eval_id] is loaded, matching the guard used by the
nearby has_default_eval_config === false branch. Keep showing “Add Eval Data”
only for loaded evals without eval_input_filter_id, and show the eval-builder
message when the loaded metadata identifies an EvalInput-typed slice.