cookbook: sharpen Pattern-1 prompts in 08-llamaindex + 10-openai-agents#38
Closed
DK09876 wants to merge 2 commits into
Closed
cookbook: sharpen Pattern-1 prompts in 08-llamaindex + 10-openai-agents#38DK09876 wants to merge 2 commits into
DK09876 wants to merge 2 commits into
Conversation
…gent-safety) Pairs with the hindsight-superagent integration (hindsight-core PR #1128). 22 cells demonstrating: - SafeHindsight drop-in wrapper for the Hindsight client - PII redact-on-write (default on) - Guard blocking a blatant prompt injection - retain_batch with safety pipeline per-item - on_guard observability callback for verdict logging - async-with lifecycle / aclose connection cleanup Verified end-to-end against live Hindsight + Superagent + OpenAI on Python 3.12. All cells run cleanly with no errors; Guard correctly identifies "IGNORE ALL PREVIOUS INSTRUCTIONS" as a prompt_injection / system_prompt_extraction with the matched CWE code, and Redact strips the seeded email address before storage. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
V2 audit found gpt-4o-mini repeatedly answering 'I don't have any information' on cells 7 and 8/9 of notebooks 08 and 10 — despite the prior cell having stored exactly the asked-about info. Pure LLM tool-choice variance, not an integration bug; the same notebooks' Pattern-2 cells (memory_instructions auto-injection / HindsightMemory adapter) work perfectly. Same fix template the Claude SDK cookbook (commit d92ccab) and the LangGraph + Haystack cookbook PRs (#36, #31) just adopted: prepend an explicit recall directive so the model has to ground its answer in memory. Two cells per notebook. Integration code in #1866 (openai-agents) and #1867 (llamaindex) is unchanged. This is a cookbook-prose polish PR; opening separately so the integration PRs land independently. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Closing per vectorize-io/hindsight#866 (comment) (review-log 2026-06-02). Empirical retest of the four cookbook polish commits showed the Claude-SDK prompt-sharpening template does not transfer to gpt-4o-mini-driven notebooks. LangGraph regressed (empty-query recall error); Haystack, LlamaIndex, and OpenAI Agents notebooks produced the same "I don't have any information" responses as before. Reverting the polish commits. The Pattern-1 cells remain a known LLM tool-call variance with gpt-4o-mini, not an integration bug. Pattern-2 cells (memory_instructions / HindsightMemory adapter / hooks) in the same notebooks work perfectly. Real fix is choosing Pattern-2 by default in cookbook prose, which is a separate scope. |
This was referenced Jun 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Sharpens the Pattern-1 recall cells in notebooks
08-llamaindex-react-agent.ipynband10-openai-agents-memory.ipynbso gpt-4o-mini reliably invokes recall before answering. Same pattern the Claude SDK cookbook (d92ccab) and cookbook PRs #36 (LangGraph) + #31 (Haystack) just adopted.Why
2026-06-02 V2 audit found cells 7 and 8/9 in both notebooks producing "I don't have any information…" responses even though the prior cell stored exactly the asked-about content. Pure LLM tool-choice variance, not an integration bug — the same notebooks' Pattern-2 cells (
memory_instructions/HindsightMemory) work perfectly.What changed
notebooks/08-llamaindex-react-agent.ipynbcells 7 + 9 prepend an explicit recall directivenotebooks/10-openai-agents-memory.ipynbcells 7 + 8 prepend an explicit recall directiveIntegration code is unchanged.
Test plan
🤖 Generated with Claude Code