feat: Multiturn eval builder #1457
Merged
chiang-daniel merged 13 commits intoJun 24, 2026
Merged
Conversation
Extends spec_with_copilot to handle multi-turn synthetic-user batches. When the request carries a `multi_turn.batch_tag`, the endpoint: - finds existing chain leaves tagged synthetic_user_batch:<batch_tag> - applies the spec's eval + golden filter tags to them - creates Eval with evaluation_data_type=full_trace and train_set_filter_id=None - skips example synthesis and TaskRun creation A request-shape validator enforces mutual exclusion with sdg_session_config and requires evaluate_full_trace=True for the multi-turn path. Also adds classify_spec_description as a stub endpoint that returns 501 until the kiln_server classifier ships. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the v2 eval builder at /specs_v2/{project_id}/{task_id} — a single-
page wizard that simplifies v1's 9-template carousel + multi-field form
into one description box → Q&A → editable refine → generate → review →
save flow. Sidebar entry under "Evals V2 (Beta)".
Reuses v1 spec_builder components (Questions, RefineSpec, ReviewExamples)
on the shared screens so the look-and-feel stays in sync with v1. The
multi-turn branch at Step 4 + 5 + 6 has its own custom UI for chat-trace
review and ties into the new spec_with_copilot multi-turn save path.
Step 1 calls classify_spec_description (currently 501) with a graceful
fallback to "issue" defaults. Multi-turn generation (Step 4) is a stub —
real run_cases_batch SSE wiring is still pending; surface a clear error
until that lands.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Step 4 multi-turn now drives the real run_cases_batch endpoint instead
of simulating with hardcoded chains:
1. Resolve the task's default run config → target_run_config (the
drive loop invokes the agent the same way a normal task run would).
2. POST /multiturn_sdg/generate_cases → N=10 synthetic-user cases.
3. POST /multiturn_sdg/run_cases_batch as SSE; consume the stream via
fetch + ReadableStream (same pattern as streaming_chat.ts since the
endpoint is POST and EventSource is GET-only).
4. Dispatch BatchEvents into component state: batch_started seeds the
batch_tag used by the multi-turn save, turn_completed updates the
cumulative trace per case, case_completed appends a Chain to the
review cards, batch_completed advances to Step 5.
SU driver model is hardcoded for MVP (claude_4_5_haiku via openrouter)
per design.md; surfacing the choice in the UI is deferred.
With this in place the multi-turn save path uses a real batch_tag and
the previous "no real chains, route through single-turn pipeline"
stopgap can be removed. The save step is now a single-path call to
spec_with_copilot with multi_turn={batch_tag}.
Sidebar entry now carries a comment noting that the v1 Evals tab is
intended to be removed once v2 ships GA.
Also adds get_task_composite_id to the edit_task test's $lib/stores
mock so the new transitive import doesn't trip an incomplete-mock
warning during test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…cription Two related bugs surfaced when walking the v2 builder without a real classifier (which currently 501s): 1. Step 1's free-text `description` never flowed into `property_values.issue_description`. Step 3's Refine screen rendered the "Original" column from `property_values`, so issue_description showed up empty even though the user had typed it in Step 1. The missing required field then silently blocked the form's Continue button (RefineSpec validator). 2. `refine_submitting` was set true by FormContainer on submit but never reset by the v2 handler. After a successful click the flag stayed true; if the user navigated back to Step 3 the button would be permanently disabled. For (1): seed `property_values.issue_description = description` at the start of classify_then_continue, before the classifier call. Real classifier response (when it ships) still overwrites; on 501 fallback we keep the user's input as the issue description. For (2): clear `refine_submitting` before advancing — the handler doesn't await any network call so there's no submitting state to preserve. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tasks created via the new-task wizard don't auto-set default_run_config_id, so first-time multi-turn users were blocked at Step 4 with "Task has no default run config — set one in task settings" even though the task had a perfectly usable run config available. Prefer the default when set; otherwise pick the first available config. Only error when the task has zero configs (genuinely unrunnable). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
v1's RefineSpec component (which v2 was reusing for both modes) is overkill for multi-turn: examples fields don't apply (the synthetic- user chains from Step 4 are the real "examples"), and the Original / Refined two-column diff plus duplicate name inputs add UX friction when only the description is meaningfully editable. For multi-turn tasks, render a stripped-down variant: one editable name input, one editable description textarea (with the refinement reason shown inline when present), and unincorporated-feedback note if any. Single-turn keeps using v1's RefineSpec. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Step 4 multi-turn does two sequential operations: first a single LLM
call to /multiturn_sdg/generate_cases (can take 5-15s for 10 cases),
then the longer streaming run_cases_batch SSE batch (minutes). The UI
was showing the same "0 of 10 ready" message during the case-gen call,
which is misleading — no chains are running yet, the copilot is still
authoring the personas.
Add a multi_turn_phase state ("idle" | "generating_cases" |
"running_batch") and branch the Step 4 copy on it. Title + subtitle +
status line all reflect what's actually happening behind the spinner.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Step 5 multi-turn was letting users hit Save with chain_verdicts still null — the golden ratings ended up empty and the underlying eval got created with no validated examples. Disable the button until every chain has been marked pass or fail; add a tooltip on the disabled state so the user understands what's blocking. Single-turn already enforces this via v1's ReviewExamples component (its submit_disabled = !all_feedback_aligned && !all_examples_reviewed) so no change needed there. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Multi-turn save was calling /api/copilot/clarify_spec just to harvest the judge_result field — clarify_spec also runs topic + input + output example generation that we throw away, costing 5-10 minutes per save. Beyond perf, server-side judge generation has a deeper problem (per Steve): the judge model has to come from the server's model registry, which can't include the user's locally-hosted models, custom fine-tunes, or models behind keys the server doesn't have. So judge model choice belongs on the client. Synthesize the judge config client-side: default model gpt_4o via openrouter, with a generic conversation-trace-evaluation prompt that inlines the spec definition. Extracted into app/web_ui/src/lib/eval/default_judge.ts so studio_server, future CLI tooling, or a UI judge-model picker can share the same defaults. Trade-off acknowledged inline: the templated prompt is weaker than clarify_spec's LLM-authored spec-specific rubric (which cited concrete red flags by name). The path forward is a UI picker that lets users override both model and prompt; this commit just stops the unbounded wait. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Several UI issues called out from internal review:
- v2 was constrained to max-w-3xl (768px) across every step; v1 widens
to 900/1400px per step. Mirror v1's getPageClass-style logic via
page_max_w_for(step): 1400px for review and refine-with-suggestions
(single-turn), 900px otherwise. Fixes the "v2 not using full
real-estate" feel and lets v1's reused Questions/RefineSpec/
ReviewExamples components render at their intended widths.
- Move per-step title + subtitle into AppPage props (title /
subtitle), matching v1's page header pattern. Removes the duplicate
in-body h1 + paragraph that lived in each step's markup. Titles
audited to match v1's tone ("Create Eval", "Clarify Eval", "Refine
Eval", "Review Conversations" / "Review Examples", "Creating Eval").
- Drop the redundant "Case N" persona_summary line on multi-turn
review cards — "Conversation N of N" already conveys that.
- Add filename_string_short_validator to the multi-turn Step 3 Eval
Name input so the user gets immediate red-text feedback if the name
exceeds 32 chars (the FilenameStringShort limit on EvalOutputScore.name)
or violates the other filename rules. Previously a too-long name
would only fail server-side with a 422 at save time.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (14)
WalkthroughAdds a multi-turn Copilot spec builder across backend and web UI. The API now accepts single-turn or multi-turn save inputs, tags persisted chain leaves for evals, and the web app adds the v2 builder, multi-turn review UI, and related navigation and support routes. ChangesMulti-turn Copilot Builder
Sequence Diagram(s)sequenceDiagram
participant User
participant builder_page as builder/+page.svelte
participant copilot_api as copilot_api.py
participant copilot_utils as copilot_utils.py
User->>builder_page: start a multi-turn spec save
builder_page->>copilot_api: POST /spec_with_copilot { multi_turn.batch_tag }
copilot_api->>copilot_utils: find_multi_turn_chain_leaves(task, batch_tag)
copilot_api->>copilot_utils: tag_multi_turn_chains_for_eval(eval_tag, golden_tag)
copilot_api->>builder_page: return spec_id and eval_id
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Suggested reviewers
Poem
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces Evals V2 (Beta), a new single-page wizard for spec authoring that supports both single-turn and multi-turn synthesis paths, including SSE streaming for multi-turn conversation generation. The code review feedback identifies several critical issues in the Svelte frontend and Python backend: potential runtime crashes in +page.svelte due to unsafe access to chain_verdicts, a SvelteKit routing issue where onMount fails to re-run on task navigation (causing state leaks), and a lack of cleanup for active generation streams which could cause resource leaks. Additionally, a rollback mechanism is recommended in copilot_utils.py to prevent orphaned tags on TaskRuns if file saving fails mid-transaction.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| {/each} | ||
| </div> | ||
|
|
||
| {#if chain_verdicts[ci]?.verdict !== null} |
Contributor
There was a problem hiding this comment.
If chain_verdicts[ci] is undefined (which can happen if chain_verdicts is empty or not yet updated during a reactive tick), chain_verdicts[ci]?.verdict !== null will evaluate to true (since undefined !== null is true). The template will then attempt to bind to chain_verdicts[ci].feedback, throwing a TypeError and crashing the entire application. Changing the condition to {#if chain_verdicts[ci]?.verdict} safely guards against this crash.
{#if chain_verdicts[ci]?.verdict}
Comment on lines
+80
to
+88
| onMount(async () => { | ||
| try { | ||
| task = await load_task(project_id, task_id) | ||
| } catch (e) { | ||
| task_error = e instanceof Error ? e.message : "Failed to load task." | ||
| } finally { | ||
| task_loading = false | ||
| } | ||
| }) |
Contributor
There was a problem hiding this comment.
In SvelteKit, when navigating between pages that share the same route template (e.g., switching tasks), the component instance is reused and onMount does not run again. This means the new task's data will not be loaded. Additionally, any state from the previous task's wizard will leak into the new task. Replacing onMount with a reactive statement that resets the wizard state and loads the task whenever project_id or task_id changes resolves both issues.
function reset_wizard() {
current_step = "describe"
description = ""
spec_type = "issue"
name = ""
property_values = {
issue_description: "",
issue_examples: "",
non_issue_examples: "",
}
classifying = false
classify_error = null
question_set = null
selections = []
other_texts = []
refined_property_values = {}
suggested_edits = {}
not_incorporated_feedback = ""
single_turn_examples = []
sdg_session_config = null
judge_info = null
multi_turn_chains = []
multi_turn_progress = 0
real_multi_turn_batch_tag = null
chain_verdicts = []
review_rows = []
}
$: if (project_id && task_id) {
reset_wizard()
task_loading = true
task_error = null
load_task(project_id, task_id)
.then((loaded_task) => {
task = loaded_task
})
.catch((e) => {
task_error = e instanceof Error ? e.message : "Failed to load task."
})
.finally(() => {
task_loading = false
})
}
Comment on lines
+278
to
+279
| let generation_loading = false | ||
| let generation_error: string | null = null |
Contributor
There was a problem hiding this comment.
Declare a component-level AbortController and abort any active multi-turn generation stream when the component is destroyed to prevent background resource leaks.
let generation_loading = false
let generation_error: string | null = null
let generation_abort_controller: AbortController | null = null
onDestroy(() => {
if (generation_abort_controller) {
generation_abort_controller.abort()
}
})
Comment on lines
+398
to
+404
| async function on_generate_multi_turn() { | ||
| generation_loading = true | ||
| generation_error = null | ||
| multi_turn_progress = 0 | ||
| multi_turn_chains = [] | ||
| real_multi_turn_batch_tag = null | ||
| multi_turn_phase = "idle" |
Contributor
There was a problem hiding this comment.
Abort any active generation stream and instantiate a new AbortController before starting a new multi-turn generation batch.
async function on_generate_multi_turn() {
if (generation_abort_controller) {
generation_abort_controller.abort()
}
generation_abort_controller = new AbortController()
generation_loading = true
generation_error = null
multi_turn_progress = 0
multi_turn_chains = []
real_multi_turn_batch_tag = null
multi_turn_phase = "idle"
Comment on lines
+465
to
+477
| const response = await fetch(url, { | ||
| method: "POST", | ||
| headers: { | ||
| "Content-Type": "application/json", | ||
| Accept: "text/event-stream", | ||
| }, | ||
| body: JSON.stringify({ | ||
| cases: cases_resp.data.cases, | ||
| turns: TURNS_PER_CASE, | ||
| target_run_config, | ||
| su_driver: SU_DRIVER_DEFAULT, | ||
| }), | ||
| }) |
Contributor
There was a problem hiding this comment.
Pass the abort signal to the fetch call so that the active multi-turn generation stream can be cancelled.
const response = await fetch(url, {
method: "POST",
headers: {
"Content-Type": "application/json",
Accept: "text/event-stream",
},
signal: generation_abort_controller.signal,
body: JSON.stringify({
cases: cases_resp.data.cases,
turns: TURNS_PER_CASE,
target_run_config,
su_driver: SU_DRIVER_DEFAULT,
}),
})
Comment on lines
+337
to
+342
| for leaf in leaves: | ||
| tags = set(leaf.tags or []) | ||
| tags.add(eval_tag) | ||
| tags.add(golden_tag) | ||
| leaf.tags = sorted(tags) | ||
| leaf.save_to_file() |
Contributor
There was a problem hiding this comment.
If save_to_file fails for any of the leaves during the tagging process, some leaves will have been successfully mutated and saved with the new tags, while others will not. Since the overall transaction fails and the spec/eval are deleted, this leaves orphaned tags on those TaskRuns. Implementing a rollback mechanism to revert the tags of successfully modified leaves on failure ensures data consistency.
Suggested change
| for leaf in leaves: | |
| tags = set(leaf.tags or []) | |
| tags.add(eval_tag) | |
| tags.add(golden_tag) | |
| leaf.tags = sorted(tags) | |
| leaf.save_to_file() | |
| tagged_leaves = [] | |
| try: | |
| for leaf in leaves: | |
| original_tags = list(leaf.tags or []) | |
| tags = set(original_tags) | |
| tags.add(eval_tag) | |
| tags.add(golden_tag) | |
| leaf.tags = sorted(tags) | |
| leaf.save_to_file() | |
| tagged_leaves.append((leaf, original_tags)) | |
| except Exception: | |
| for leaf, original_tags in tagged_leaves: | |
| try: | |
| leaf.tags = original_tags | |
| leaf.save_to_file() | |
| except Exception: | |
| pass | |
| raise |
| <script lang="ts"> | ||
| import AppPage from "../../../app_page.svelte" | ||
| import { page } from "$app/stores" | ||
| import { onMount } from "svelte" |
Contributor
There was a problem hiding this comment.
Import onDestroy from svelte to support cleanup of the active multi-turn generation stream when the component is destroyed.
import { onMount, onDestroy } from "svelte"
Comment on lines
+1036
to
+1039
| on:click={() => { | ||
| chain_verdicts[ci].verdict = "fail" | ||
| chain_verdicts = [...chain_verdicts] | ||
| }} |
Contributor
There was a problem hiding this comment.
Guard the click handler to ensure chain_verdicts[ci] is defined before setting its properties, preventing potential runtime crashes.
on:click={() => {
if (chain_verdicts[ci]) {
chain_verdicts[ci].verdict = "fail"
chain_verdicts = [...chain_verdicts]
}
}}
Comment on lines
+1047
to
+1050
| on:click={() => { | ||
| chain_verdicts[ci].verdict = "pass" | ||
| chain_verdicts = [...chain_verdicts] | ||
| }} |
Contributor
There was a problem hiding this comment.
Guard the click handler to ensure chain_verdicts[ci] is defined before setting its properties, preventing potential runtime crashes.
on:click={() => {
if (chain_verdicts[ci]) {
chain_verdicts[ci].verdict = "pass"
chain_verdicts = [...chain_verdicts]
}
}}
📊 Coverage ReportOverall Coverage: 92% Diff: origin/dchiang/multiturn-synthetic-user...HEAD
Summary
Line-by-lineView line-by-line diff coverageapp/desktop/studio_server/copilot_api.pyLines 499-507 499 spec_name=request.name,
500 )
501 task_runs = dataset_runs.task_runs
502 for run in task_runs:
! 503 run.parent = task
504 models_to_save.extend(task_runs)
505
506 # Snapshot the generation config on the Spec (single-turn only).
507 topic_cfg = request.sdg_session_config.topic_generation_configLines 554-563 554
555 for run in task_runs:
556 run.save_to_file()
557 saved_models.append(run)
! 558 if dataset_runs is not None:
! 559 dataset_runs.save_pending_feedback(run)
560
561 spec.save_to_file()
562 saved_models.append(spec)Lines 576-585 576 except Exception:
577 # Reverse any leaf tags we added in this run before deleting the
578 # saved models, so a failed multi-turn save doesn't leave orphan
579 # tags pointing at a now-deleted eval.
! 580 if tagged_leaves:
! 581 untag_multi_turn_chains_for_eval(tagged_leaves)
582 for model in reversed(saved_models):
583 try:
584 model.delete()
585 except Exception:app/desktop/studio_server/utils/copilot_utils.pyLines 345-353 345 for leaf in leaves:
346 current = set(leaf.tags or [])
347 added = {eval_tag, golden_tag} - current
348 if not added:
! 349 continue
350 leaf.tags = sorted(current | added)
351 leaf.save_to_file()
352 if tagged_out is not None:
353 tagged_out.append((leaf, added))Lines 362-372 362 so pre-existing tags on the leaf are preserved. Best-effort: a per-leaf
363 save failure is logged and the loop continues — the original save error
364 that triggered cleanup is the one the user needs to see.
365 """
! 366 for leaf, added_tags in tagged_leaves:
! 367 try:
! 368 leaf.tags = sorted(set(leaf.tags or []) - added_tags)
! 369 leaf.save_to_file()
! 370 except Exception:
! 371 logger.exception(f"Failed to untag leaf {leaf.id} during cleanup")
|
… rollback fix - Move v2 builder to /specs/[project_id]/[task_id]/builder (drop /specs_v2/ URL). - Pro-gate the new builder route via CopilotRequiredCard. - Flip v1 listing's "Create Eval" CTA: Pro users land on the v2 builder; non-Pro keeps the existing select_workflow flow. - Replace "Evals V2 Beta" sidebar entry with a temporary "Evals Legacy" entry (TODO-marked for removal post-GA) for side-by-side comparison. - Fix rollback gap: multi-turn save failures now untag any leaves we added tags to in this run, preserving pre-existing tags. - Surface guardrails: multi-turn comparison notice on the eval detail page, Step 1 beta hint, default-run-config fallback notice. - Posthog events on the v2 builder flow (open, step entered, save success/error, CTA branch). - Apply CR audit fixes (critical + moderate) on stale or false comments. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Match v1 spec_builder's width pattern: wrap the entire AppPage in the page_max_w div so title and body share the same constraint. - Replace raw alert/info divs with the shared Warning component (6 sites in builder + 1 in compare_run_configs). - Standardize Back/Cancel buttons to btn-ghost btn-sm. - Wire classify_error through FormElement's error_message prop. - Bump step indicator from text-xs to text-sm to match v1. - Drop the "Beta: every spec treated as Issue" hint — coordinate with Mike via the bug bash announcement instead. - Add v1's AbortController plumbing (abort_copilot_request, new_copilot_abort_signal, is_abort_error) and onDestroy cleanup. Long-running Copilot calls pass the signal, Back buttons cancel in-flight requests, catch blocks silently ignore AbortError. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace Step 5's vertical card stack with a focused paginator: one conversation at a time, dot-row navigator at the top, persona + verdict buttons inside the card, single bottom action bar. - Pass auto-advances to the next un-reviewed conversation; Fail keeps the user on the current case and focuses the feedback input, since the judge prompt needs the reason to learn from. - Save is gated until every chain has a verdict AND every failed chain has a non-empty reason. Tooltip explains what's missing. - Single bottom action bar owns Back/Prev/Next/Save so the user sees one set of buttons instead of two stacked rows. - Dots are small solid markers with numbered labels below, connected by solid lines (matches DaisyUI .steps line convention). - Feedback wording mirrors v1 review_examples: "Describe why this fails" / "Describe why this passes (optional)". Adds a DEV-ONLY mock route at /dev_mock_review for iterating on the paginator without running through the full eval-builder flow. TODO-marked for removal post-bash. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
chiang-daniel
merged commit 2ee9cdd
into
dchiang/multiturn-synthetic-user
10 of 12 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1441 — review only the delta against
dchiang/multiturn-synthetic-user. Do not merge until #1441 lands.Adds the multi-turn eval builder UI and the
multi_turnbranch onspec_with_copilot. Tagged the existing chain leaves produced by #1441's synthetic-user runner as the eval dataset rather than re-driving synthesis. V1-shapeEvalConfig(config_type=llm_as_judge)is created so V2's legacy dispatch path (preserved in #1454) continues to score these evals.The v2 builder lives at
/specs/[project_id]/[task_id]/builder(NOT/specs_v2/...). Pro-gated at the route + at the v1 listing's "Create Eval" CTA. A temporary "Evals Legacy" sidebar entry routes to v1's manual workflow for side-by-side comparison during the bug bash; marked with aTODO(eval-v2)for removal once v2 ships GA.Deferred until #1454 (Steve's Evals V2) lands:
EvalInput[MultiTurnSyntheticEvalInputData](V2 datamodel slot) so the same cases can be re-driven against different run configsLlmJudgePropertiesshapeincompatible_input_shape— we route through legacyGEvalinstead)Known limitations bug-bash testers should know about:
classify_spec_descriptionendpoint is stubbed (returns 501) — every spec is treated as an Issue eval. Beta hint shown on Step 1.full_traceevals.Test plan
/specs/{p}/{t}/builderas non-Pro: confirm upgrade prompt renders instead of builderselect_workflow🤖 Generated with Claude Code