fix(agent): corrected read-before-write prompt, install_skill capture-and-continue#816
Open
gewenyu99 wants to merge 6 commits into
Open
fix(agent): corrected read-before-write prompt, install_skill capture-and-continue#816gewenyu99 wants to merge 6 commits into
gewenyu99 wants to merge 6 commits into
Conversation
…_skill halt
Three field fixes from the 2026-07-07 triage, all aimed at agents spiraling instead of adapting:
1. YARA repeat-block guard (code, not prompt): the runtime notes already say "retrying the same edit will just block again", but post-2.38.1 data shows ~4.2 blocks per affected run — the prompt alone doesn't hold. Both fences now track the exact payloads they block (tool + content hash). An identical retry gets an escalated reason ("this exact content was ALREADY blocked — change the code, not the retry"), and from the third attempt on the agent is told to note it in the setup report and move on. Wired into the pi security extension's pre-execution scan (bash/write/edit) and the anthropic PreToolUse Bash hook; the tracker lives in yara-hooks so both share one implementation.
2. PI_RUNTIME_NOTES: tell the agent not to delete .posthog-events.json (the host removes it after the run; `rm` is the #1 bash denial at ~31/week) and to `write` new files directly instead of `read`ing a missing path first (the fence errors on reads of nonexistent files and the read doesn't count).
3. posthog-integration prompt STEP 2: if install_skill fails, emit [ERROR-RESOURCE-MISSING] and halt — same pattern STEP 1 already has for load_skill_menu — instead of freelancing the integration without the skill.
Generated-By: PostHog Code
Task-Id: bcb96bcb-e37f-421d-841d-738647c73b8e
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Results will be posted here when complete. |
… capture-and-continue on skill install failure Reworks three pieces of this PR per review: 1. rm .posthog-events.json shouldn't be fence-blocked (it isn't on anthropic): Bash sits in BASE_ALLOWED_TOOLS, so the claude-agent-sdk auto-approves it and canUseTool is never consulted — the install/build allowlist only ever bites on pi, where this rm was the #1 denial. wizardCanUseTool now allows exactly `rm [-f] [./].posthog-events.json` (after the operator checks, so chaining still dies), and the "don't delete it" prompt note is gone. The pi host-side cleanup stays as a safety net. 2. The read-before-write pi note treated a self-inflicted problem as a harness fence: pi's write/edit have NO read requirement (verified in pi 0.79.8 source — write is mkdir+writeFile, edit fuzzy-matches oldText; no read tracking exists). The wizard's own shared prompt line ("You must read a file immediately before attempting to write it, even if you have previously read it") is what sent agents to read nonexistent paths. That line now states the accurate rule — read existing files before editing/overwriting; never read a path that doesn't exist yet — and the pi note is deleted. 3. install_skill failure no longer halts. A freestyled integration beats an outright failure, but the run must be measurable: new [SKILL-INSTALL-FAILED] signal, STEP 2 tells the agent to emit it and continue from its own knowledge (noting the missing skill in the setup report), and both harnesses capture a `wizard: agent continued without skill` event with the detail — complementing the tool-level `wizard: skill install failed` event from #807. Generated-By: PostHog Code Task-Id: bcb96bcb-e37f-421d-841d-738647c73b8e
The repeat-block escalation touched the same pi/security.ts + yara-hooks.ts scanning paths that #804 (pi scans through warlock, legacy scanner deleted) rewrites — it belongs on top of that PR, not racing it from main. Reverted the tracker, its fence wiring, and its tests to main here; this PR keeps the non-YARA fixes (plan-file rm allowance, corrected read-before-write prompt, install_skill capture-and-continue). The guard follows as a stacked PR on fix/pi-harness-use-warlock. Generated-By: PostHog Code Task-Id: bcb96bcb-e37f-421d-841d-738647c73b8e
gewenyu99
changed the title
gewenyu99
added a commit
that referenced
this pull request
Jul 7, 2026
…(stacked on #804) The YARA guard split out of #816, rebuilt on top of #804's warlock-based pi fence, plus the one surviving piece of the closed #818: 1. Repeat-block escalation: agents retry identically-blocked payloads (~4.2 blocks per affected run) despite the prompt saying not to. Both fences (pi pre-execution scan, anthropic PreToolUse Bash hook) now hash the payloads they block; an identical retry gets an escalated reason ("this exact content was ALREADY blocked — change the code, not the retry"), and from the third attempt the agent is told to note it in the setup report and move on. The escalation decorates #804's remediation-bearing block message, never replaces it. Tracker lives in yara-hooks, shared by both fences; policy denies are unaffected. 2. Edit scanning scoped to replacement text: #804 still scans JSON.stringify(input.edits), which includes each edit's oldText — pre-existing file content. That blocked edits whose surroundings contained a violation and even edits REMOVING one (field FPs: capture edits adjacent to existing identify() PII; pre-existing violations blocking their own replacement). Now scans only the joined newText, matching the anthropic path which scans new_string only. Generated-By: PostHog Code Task-Id: bcb96bcb-e37f-421d-841d-738647c73b8e
#834 allows scoped `rm [-f] <relative-file>` in the pi fence (where the allowlist actually bites), which covers the plan file and more, without touching the shared policy. This PR keeps the read-before-write prompt correction and the install_skill capture-and-continue. Generated-By: PostHog Code Task-Id: 9b33e996-4d8c-4906-8405-63fb4bcd7d07
gewenyu99
changed the title
edwinyjlim
approved these changes
Jul 8, 2026
Co-authored-by: Edwin Lim <edwin@posthog.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Item C from the 2026-07-07 triage (report), reworked per review. The YARA repeat-block guard was split out of this PR — it touches the scanning paths that #804 rewrites, so it now stacks on
fix/pi-harness-use-warlockas its own PR. The plan-filermallowance was also dropped — superseded by #834, which allows scopedrm [-f] <relative-file>in the pi fence (where the allowlist actually bites) without touching the shared policy. This PR keeps the two remaining fixes:1. Accurate read-before-write guidance (the old line caused the failures)
pi's write/edit have no read requirement at all (verified in pi 0.79.8 source —
writeis mkdir+writeFile,editfuzzy-matchesoldText; no read tracking exists). The wizard's own shared prompt line — "You must read a file immediately before attempting to write it, even if you have previously read it" — is what marched agents into reading nonexistent paths before creating new files. The line now states the real rule: read existing files immediately before editing/overwriting; never read a path that doesn't exist yet — just write it.2.
install_skillfailure: capture-and-continue, not haltA freestyled integration beats an outright failure — but we need to know it happened. New
[SKILL-INSTALL-FAILED]signal; STEP 2 tells the agent to emit it and continue from its own knowledge of the framework and PostHog docs (noting the missing skill in the setup report). Both harnesses parse it and capture awizard: agent continued without skillevent with the detail, complementing the tool-levelwizard: skill install failedevent from #807. A bare marker still reports as''so the failure is never lost.Tests
pnpm build && pnpm testgreen (1214 tests): 3 signal-parsing tests for[SKILL-INSTALL-FAILED].🤖 Generated with Claude Code