Skip to content

Commit e7b5dfc

Browse files
committed
fix(tests): narrow recall assertion to exiftool only
The banned list previously included PIL/piexif/exifread, which are all pip-installable. A valid recall guideline can legitimately recommend "pip install <pkg>" for any of them, causing the assertion to fail on correct behavior. Only exiftool is definitively absent in the sandbox (not installed, not pip-installable), so it's the only reliable marker that recall did NOT steer Claude away from the unavailable tool.
1 parent 8ed92eb commit e7b5dfc

1 file changed

Lines changed: 4 additions & 7 deletions

File tree

tests/e2e/test_sandbox_learn_recall.py

Lines changed: 4 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -159,11 +159,8 @@ def test_learn_then_recall_flow(sandbox_ready, sandbox_workspace):
159159
log.info(f"session 2: checking {len(commands)} bash commands for forbidden tools")
160160
joined = "\n".join(commands).lower()
161161

162-
# Recall should steer Claude away from the tools that failed in session 1.
163-
# The guideline text itself may name these tools, but we're checking actual
164-
# bash invocations, not string mentions — so a command like
165-
# `python3 -c "import PIL"` would fail this check, while the guideline's
166-
# prose mentioning PIL as unavailable does not.
162+
# Recall should steer Claude away from tools guaranteed-unavailable in the
163+
# sandbox. Only `exiftool` is definitively absent (not installed, can't be
164+
# pip-installed). Other libraries (PIL, piexif, exifread) may appear in a
165+
# valid guideline as "install via pip and use", so we don't ban them.
167166
assert not re.search(r"\bexiftool\b", joined), "session 2 invoked exiftool despite recall guideline:\n" + "\n".join(commands)
168-
for banned in ("from pil", "import pil", "import piexif", "import exifread"):
169-
assert banned not in joined, f"session 2 tried {banned!r} despite recall guideline:\n" + "\n".join(commands)

0 commit comments

Comments
 (0)