Skip to content

Commit 99c30f8

Browse files
committed
feat(agent): programmatic auto-continue on promissory stops
The system prompt forbids 'let me try another approach' followed by silence, but the LLM regularly violates that rule — leaving the user staring at a stderr with no resolution. Move enforcement from prompt to code: after every streamText call, regex-match the final assistant text against a promissory-phrase pattern. If matched, inject an [auto-continue] user message and re-stream (up to 3 times per send). Why code beats prompt: - Deterministic. Regex either matches or doesn't, no model variance. - Survives prompt drift. Future system-prompt rewrites can't silently remove the guard. - Cheap. Capped at MAX_AUTOCONTINUE=3 — at most 3 extra round-trips. - Visible. Emits a [agent: auto-continuing …] hint into the stream so the user sees why the agent kept going. Pattern catches: 'let'?s try', 'let me (try|investigate|check|explore| re-examine|verify|look)', 'i'?ll (try|explore|investigate|check|retry| rerun)', 'another approach', 'trying (a different|another)', 'continue investigating', 'i need to (check|run|investigate)'. Belt + suspenders: also tightened the prompt rule with explicit forbidden phrases and a final-narration requirement, so a model that ignores the regex still gets nagged via prompt.
1 parent 033e0a5 commit 99c30f8

1 file changed

Lines changed: 115 additions & 39 deletions

File tree

src/agent/llm/session.ts

Lines changed: 115 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,22 @@ import { runScript } from '../shell/runtime'
77
import type { ExecContext } from '../shell/runtime'
88
import { collect } from '../shell/types'
99

10+
/**
11+
* Phrases that signal the model promised follow-up action without
12+
* executing it. Triggered programmatically — far more reliable than
13+
* adding more "do not say this" rules to the system prompt, which
14+
* the model regularly violates.
15+
*/
16+
const PROMISSORY_PATTERN =
17+
/\b(let'?s try|let me (try|investigate|check|explore|re-?examine|verify|look)|i'?ll (try|explore|investigate|check|re-?try|re-?run)|another approach|trying (a different|another)|let me approach|could you|continue investigating|let me run|i need to (check|run|investigate))\b/i
18+
19+
/**
20+
* Up to this many extra streamText calls per send() to recover from
21+
* promissory-stop. Caps the cost: the agent can't burn unlimited
22+
* tokens looping on itself.
23+
*/
24+
const MAX_AUTOCONTINUE = 3
25+
1026
const SYSTEM_PROMPT = `You operate ComfyUI through a POSIX-like shell running in the user's browser.
1127
1228
THE ONLY EXISTING MOUNTS ARE:
@@ -260,6 +276,26 @@ Iteration / self-correction rules (CRITICAL):
260276
after the first attempt. The step budget is effectively unlimited.
261277
- If a run-js call throws, READ the error message and immediately retry
262278
with a corrected snippet in the SAME turn — no asking permission.
279+
Phrases that MUST be followed by another tool call in the same turn,
280+
not by a stop:
281+
"Let's try another approach"
282+
"Let me try a different way"
283+
"I'll explore this further"
284+
"Let me investigate"
285+
"Let me check"
286+
"I'll re-examine"
287+
Any sentence ending with ":" or "..." that promises future action
288+
If you write any of these, the very next tokens MUST be a run_shell
289+
tool call. Stopping after such a sentence is forbidden — it leaves
290+
the user staring at an error with no resolution.
291+
- Final answer narration rule: every turn that runs tools MUST end with
292+
a 1-3 sentence plain-language summary of what you found, separate
293+
from the tool output. Tool output alone is not an answer. Examples:
294+
GOOD: "Found 2 orphan nodes: id 15 (MarkdownNote — looks
295+
intentional, just documentation) and id 19 (Preview3D — has
296+
model_file widget set but no upstream link)."
297+
BAD: just leaving the JSON [15, 19] visible in the tool output and
298+
going silent. The user shouldn't have to decode raw output.
263299
- "X is not defined" → the local name might be wrong; the injected locals
264300
are exactly: app, api, document, window, useCanvasStore, useCommandStore,
265301
useWorkflowStore, useMissingModelStore, useExecutionErrorStore,
@@ -505,44 +541,84 @@ export async function streamSession(
505541
? { openai: { reasoningEffort: opts.reasoningEffort } }
506542
: undefined
507543

508-
const { textStream, text } = streamText({
509-
model: openai(opts.model),
510-
system,
511-
messages: opts.messages,
512-
abortSignal: opts.signal,
513-
// Effectively unlimited — rely on the model to decide completion.
514-
// A safety valve + repeat-script detection could be added later.
515-
stopWhen: stepCountIs(opts.maxSteps ?? 1000),
516-
providerOptions,
517-
tools: {
518-
run_shell: tool({
519-
description:
520-
'Run one or more shell commands. Supports pipes (|), sequencing (&&, ||, ;), and redirection (>, >>). Returns stdout, stderr, and exit code.',
521-
inputSchema: z.object({
522-
script: z.string().describe('The shell script to execute')
523-
}),
524-
execute: async ({ script }) => {
525-
const res = await runScript(script, opts.execContext)
526-
const stdout = await collect(res.stdout)
527-
const inv: ToolInvocation = {
528-
script,
529-
stdout,
530-
stderr: res.stderr,
531-
exitCode: res.exitCode
532-
}
533-
toolCalls.push(inv)
534-
onTool(inv)
535-
return {
536-
stdout,
537-
stderr: res.stderr ?? '',
538-
exitCode: res.exitCode
539-
}
544+
const tools = {
545+
run_shell: tool({
546+
description:
547+
'Run one or more shell commands. Supports pipes (|), sequencing (&&, ||, ;), and redirection (>, >>). Returns stdout, stderr, and exit code.',
548+
inputSchema: z.object({
549+
script: z.string().describe('The shell script to execute')
550+
}),
551+
execute: async ({ script }: { script: string }) => {
552+
const res = await runScript(script, opts.execContext)
553+
const stdout = await collect(res.stdout)
554+
const inv: ToolInvocation = {
555+
script,
556+
stdout,
557+
stderr: res.stderr,
558+
exitCode: res.exitCode
540559
}
541-
})
542-
}
543-
})
544-
545-
for await (const delta of textStream) onDelta(delta)
546-
const finalText = await text
547-
return { text: finalText, toolCalls }
560+
toolCalls.push(inv)
561+
onTool(inv)
562+
return {
563+
stdout,
564+
stderr: res.stderr ?? '',
565+
exitCode: res.exitCode
566+
}
567+
}
568+
})
569+
}
570+
571+
let messages: ModelMessage[] = [...opts.messages]
572+
let combinedText = ''
573+
let autoContinues = 0
574+
575+
while (true) {
576+
if (opts.signal?.aborted) break
577+
const before = toolCalls.length
578+
const { textStream, text } = streamText({
579+
model: openai(opts.model),
580+
system,
581+
messages,
582+
abortSignal: opts.signal,
583+
stopWhen: stepCountIs(opts.maxSteps ?? 1000),
584+
providerOptions,
585+
tools
586+
})
587+
for await (const delta of textStream) onDelta(delta)
588+
const finalText = await text
589+
combinedText += finalText
590+
const toolsDuringTurn = toolCalls.length - before
591+
592+
// Auto-continue heuristic: the model wrote a promissory phrase
593+
// ("let me try another approach", "I'll investigate further", …)
594+
// but stopped emitting tool calls. Inject a continue prompt and
595+
// re-stream — up to MAX_AUTOCONTINUE times.
596+
const stalled =
597+
finalText.length > 0 &&
598+
PROMISSORY_PATTERN.test(finalText) &&
599+
// If the model just made a tool call this turn AND ended with the
600+
// promise, it's mid-iteration — give it one more push. If it
601+
// didn't make any tool call this turn, it's definitely stuck.
602+
autoContinues < MAX_AUTOCONTINUE
603+
604+
if (!stalled) break
605+
606+
autoContinues++
607+
messages = [
608+
...messages,
609+
{ role: 'assistant', content: finalText },
610+
{
611+
role: 'user',
612+
content:
613+
toolsDuringTurn === 0
614+
? '[auto-continue] You promised follow-up action but stopped without executing it. Run the next tool call now — do not narrate, act.'
615+
: '[auto-continue] Your last turn surfaced something that needs another step. Take that step now via run_shell — no preamble.'
616+
}
617+
]
618+
onDelta(
619+
`\n\n[agent: auto-continuing — promissory stop detected, attempt ${autoContinues}/${MAX_AUTOCONTINUE}]\n\n`
620+
)
621+
}
622+
623+
return { text: combinedText, toolCalls }
548624
}

0 commit comments

Comments
 (0)