feat(integrations): add Haystack integration (#1256)

* feat(integrations): add Haystack integration for persistent agent memory

Add hindsight-haystack package providing Haystack Tool instances backed
by Hindsight's retain/recall/reflect APIs. Uses async client methods with
event-loop-safe sync wrapper to work correctly inside Haystack's agent
runtime.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(haystack): use persistent event loop for async client calls

aiohttp binds its session to the creating event loop, so asyncio.run()
(which creates/destroys a loop per call) breaks on sequential calls.
Switch to a persistent daemon-thread event loop with
run_coroutine_threadsafe. Also removes unused per-operation timeout
constants and adds _run_sync tests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(haystack): add HindsightToolset with auto-recall/retain, fix review issues

- Add HindsightToolset(Toolset) with auto_recall and auto_retain flags
  that automatically inject recalled memories into the system prompt
  before each turn and retain user/assistant messages after each turn
- Fix _ensure_bank to retry on transient errors instead of permanently
  disabling bank creation
- Fix reflect_on_memory to return structured_output JSON when
  response_schema is set
- Truncate error messages to avoid dumping raw HTTP responses to agents
- Extract _build_backend_kwargs() and _build_tools() as shared helpers
- Add 20 new tests (60 -> 80 total) covering toolset, auto-recall,
  auto-retain, structured output, and bank creation retry

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(haystack): address review round 2 — max_recall_results, role metadata, _run_sync cleanup

- Add max_recall_results param to HindsightToolset (default 10) to cap
  auto-recall prompt injection size, matching Pydantic AI pattern
- Auto-retain now includes role + source metadata on messages, matching
  LlamaIndex's metadata pattern for distinguishable conversation turns
- _recall_for_prompt now calls the API directly with result cap instead
  of going through the formatted string from recall_memory
- Serialize/deserialize max_recall_results in to_dict/from_dict
- Add tests for max_recall_results and role metadata (82 total)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(haystack): default to Cloud without configure(); add gated E2E + bucketing

- resolve_client() now falls back to DEFAULT_HINDSIGHT_API_URL and the
  HINDSIGHT_API_KEY env var when configure() was never called (it previously
  raised "No Hindsight API URL configured"). Updated the unit test to assert
  the cloud-default + env-key behavior. Satisfies the "default to Cloud" goal.
- Add a gated tests/test_e2e.py (retain/recall/reflect tools against a live
  Hindsight server), marked requires_real_llm; register the marker in
  pyproject; the test-haystack-integration CI job now runs the deterministic
  bucket (-m "not requires_real_llm").

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(haystack): close owned clients at exit; run E2E client I/O on the bridge loop

The tools run async client calls on a persistent background event loop. aiohttp
sessions bound to that loop were never closed, surfacing as "Unclosed client
session/connector" warnings. Track module-owned Hindsight clients (those created
when the caller didn't pass client=) and close them on the loop via an atexit
hook, then stop the loop. The live E2E now performs all client I/O through that
same loop (acreate_bank/adelete_bank/aclose via _run_sync) and logs cleanup
failures instead of swallowing them — zero unclosed-connector warnings.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(haystack): sync openapi.json with main

Branch carried an older snapshot of hindsight-docs/static/openapi.json
that pre-dated main's addition of OperationProgress. Re-checkout main's
openapi.json so check-openapi-compatibility passes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(haystack): strip api_key from to_dict() so it doesn't leak to YAML

_build_backend_kwargs was emitting the api_key in the serializable dict
that to_dict() returns. Haystack pipelines get dumped to YAML for
inspection, checkpointing, and sharing — a serialized key leaks into
every dump. Reviewer (benfrank241) flagged this on #1256.

Drop the api_key from the serialized backend_kwargs. resolve_client()
already reads HINDSIGHT_API_KEY from the env var as a final fallback,
so a redeployed pipeline picks the key back up from the host's
environment rather than from the YAML.

The test_tools_round_trip_serialization_with_client test previously
asserted the leak — flipped it to assert the key is NOT present
and added a json.dumps probe asserting the literal key value also
doesn't appear under any other field name. Pre-fix, the test fails:
  AssertionError: api_key must not appear in serialized backend_kwargs
   — would leak to YAML pipeline dumps
  assert 'api_key' not in {'api_key': 'client-key', ...}

86/86 tests pass post-fix.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* ci: re-trigger CI

A previous push to this branch silently did not trigger a pull_request
event in GitHub Actions, leaving the PR without a CI run for the latest
HEAD. Push an empty commit to force a new event.

* ci: empty commit to attach pull_request CI check to the PR head

(Previous pushes did not auto-trigger pull_request workflow events for
reasons internal to GitHub Actions; manual workflow_dispatch runs passed
green but their checks don't roll up onto the PR. Re-poking the head
to surface the green state on the PR.)

* ci: trailing newline to force CI retrigger

* fix(haystack): register in changelog/gallery + docs page + tidy tools

Review follow-ups for the Haystack integration:

1. Add haystack to the INTEGRATIONS map in generate_changelog.py so the
   release script's changelog step resolves the slug (was missing, which
   would fail the release).
2. Add the integrations.json gallery entry, a doc page at
   docs-integrations/haystack.md, and an icon — required by
   check-integrations.mjs (forward: entry needs a doc page; reverse: a
   released integration must appear in the gallery).
3. Drop the inaccurate 'Raises: HindsightError' clause from
   create_hindsight_tools — resolution always succeeds (URL defaults to
   Cloud) so it never raises; the error type stays exported as the
   conventional public catch type.
4. Replace the _TOOL_DEFS dict-of-3-tuples with a frozen _ToolDef dataclass
   and drop the redundant method-name field (it equalled the dict key).

* fix(haystack): use official Haystack logo for gallery icon

Replace the placeholder glyph with the real deepset Haystack mark (teal
#0EAF9C rounded square + white symbol), extracted as vector from deepset's
own website source (deepset-ai/haystack-home site-logo partial).

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: DK09876 <dk09876@DK09876s-MacBook-Pro.local>

Author: 3 people

.github/workflows/test.yml

@@ -59,6 +59,7 @@ jobs:
       integrations-vapi: ${{ steps.filter.outputs.integrations-vapi }}
       integrations-flowise: ${{ steps.filter.outputs.integrations-flowise }}
       integrations-google-adk: ${{ steps.filter.outputs.integrations-google-adk }}
+      integrations-haystack: ${{ steps.filter.outputs.integrations-haystack }}
       tools-agent-sdk: ${{ steps.filter.outputs.tools-agent-sdk }}
       integrations-roo-code: ${{ steps.filter.outputs.integrations-roo-code }}
       dev: ${{ steps.filter.outputs.dev }}
@@ -141,6 +142,8 @@ jobs:
             - 'hindsight-integrations/langgraph/**'
           integrations-llamaindex:
             - 'hindsight-integrations/llamaindex/**'
+          integrations-haystack:
+            - 'hindsight-integrations/haystack/**'
           integrations-paperclip:
             - 'hindsight-integrations/paperclip/**'
           integrations-opencode:
@@ -3273,6 +3276,45 @@ jobs:
       # (requires_real_llm) needs a live Hindsight server and runs separately.
       run: uv run pytest tests -v -m "not requires_real_llm"
 
+  test-haystack-integration:
+    needs: [detect-changes]
+    if: >-
+      github.event_name != 'pull_request_review' &&
+      (github.event_name == 'workflow_dispatch' ||
+      needs.detect-changes.outputs.integrations-haystack == 'true' ||
+      needs.detect-changes.outputs.ci == 'true')
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v6
+      with:
+        ref: ${{ github.event.pull_request.head.sha || '' }}
+
+    - name: Install uv
+      uses: astral-sh/setup-uv@v7
+      with:
+        enable-cache: true
+        prune-cache: false
+
+    - name: Set up Python
+      uses: actions/setup-python@v6
+      with:
+        python-version-file: ".python-version"
+
+    - name: Build haystack integration
+      working-directory: ./hindsight-integrations/haystack
+      run: uv build
+
+    - name: Install dependencies
+      working-directory: ./hindsight-integrations/haystack
+      run: uv sync --frozen
+
+    - name: Run tests
+      working-directory: ./hindsight-integrations/haystack
+      # PR CI runs only the deterministic bucket; the real-LLM E2E bucket
+      # (requires_real_llm) needs a live Hindsight server and runs separately.
+      run: uv run pytest tests -v -m "not requires_real_llm"
+
   test-openai-agents-integration:
     needs: [detect-changes]
     if: >-
@@ -4257,6 +4299,7 @@ jobs:
       - test-llamaindex-integration
       - test-openai-agents-integration
       - test-agentcore-integration
+      - test-haystack-integration
       - test-pip-slim
       - test-embed
       - test-embed-windows

hindsight-dev/hindsight_dev/generate_changelog.py

@@ -68,6 +68,7 @@ class IntegrationMeta:
     "flowise": IntegrationMeta("@vectorize-io/flowise-nodes-hindsight", "Flowise"),
     "google-adk": IntegrationMeta("hindsight-google-adk", "Google ADK"),
     "superagent": IntegrationMeta("hindsight-superagent", "Superagent"),
+    "haystack": IntegrationMeta("hindsight-haystack", "Haystack"),
     "roo-code": IntegrationMeta("hindsight-roo-code", "Roo Code"),
 }
 

hindsight-docs/docs-integrations/haystack.md

@@ -0,0 +1,126 @@
+---
+sidebar_position: 34
+title: "Haystack Persistent Memory with Hindsight | Integration"
+description: "Add persistent long-term memory to Haystack agents with Hindsight. Provides retain/recall/reflect Tools plus a HindsightToolset with optional auto-recall and auto-retain."
+---
+
+# Haystack
+
+Persistent long-term memory for [Haystack](https://haystack.deepset.ai/) agents via Hindsight. The `hindsight-haystack` package gives you two complementary patterns:
+
+- **`create_hindsight_tools(...)`** — Returns a list of Haystack `Tool`s (`retain_memory`, `recall_memory`, `reflect_on_memory`) the model can call directly inside a turn.
+- **`HindsightToolset`** — A Haystack `Toolset` that bundles the same tools and adds optional **auto-recall** (inject relevant memories into the system prompt before each turn) and **auto-retain** (store user + assistant messages after each turn).
+
+## Installation
+
+```bash
+pip install hindsight-haystack
+```
+
+## Quick Start
+
+```python
+from hindsight_client import Hindsight
+from hindsight_haystack import create_hindsight_tools
+from haystack.components.agents import Agent
+from haystack.components.generators.chat import OpenAIChatGenerator
+from haystack.dataclasses import ChatMessage
+
+client = Hindsight(base_url="http://localhost:8888")
+
+tools = create_hindsight_tools(
+    client=client,
+    bank_id="user-123",
+    mission="Track user preferences",
+)
+
+agent = Agent(
+    chat_generator=OpenAIChatGenerator(model="gpt-4o-mini"),
+    tools=tools,
+    system_prompt=(
+        "You are a helpful assistant with long-term memory. "
+        "Use retain_memory to store important facts. "
+        "Use recall_memory to search memory before answering."
+    ),
+)
+
+result = agent.run(messages=[ChatMessage.from_user("Remember that I prefer dark mode")])
+print(result["messages"][-1].text)
+```
+
+## Automatic Memory with HindsightToolset
+
+For automatic recall and retain without relying on the agent to call tools:
+
+```python
+from hindsight_haystack import HindsightToolset
+
+toolset = HindsightToolset(
+    client=client,
+    bank_id="user-123",
+    mission="Track user preferences",
+    auto_recall=True,   # Inject memories into the system prompt before each turn
+    auto_retain=True,   # Store user + assistant messages after each turn
+)
+
+agent = Agent(
+    chat_generator=OpenAIChatGenerator(model="gpt-4o-mini"),
+    tools=toolset,
+    system_prompt="You are a helpful assistant with long-term memory.",
+)
+
+# Use toolset.run() for automatic memory behavior
+result = toolset.run(agent, messages=[ChatMessage.from_user("I prefer dark mode")])
+```
+
+## Selective Tools
+
+```python
+# Only retain + recall (no reflect)
+tools = create_hindsight_tools(
+    client=client,
+    bank_id="user-123",
+    include_reflect=False,
+)
+```
+
+## Configuration
+
+Call `configure()` once to set connection defaults so you can omit `client=`/`hindsight_api_url=` on every call:
+
+```python
+from hindsight_haystack import configure
+
+configure(
+    hindsight_api_url="http://localhost:8888",
+    api_key="your-api-key",
+    budget="mid",
+    tags=["source:haystack"],
+    context="my-app",
+    mission="Track user preferences",
+)
+
+tools = create_hindsight_tools(bank_id="user-123")
+```
+
+The API URL defaults to Hindsight Cloud (`https://api.hindsight.vectorize.io`), and the API key falls back to the `HINDSIGHT_API_KEY` environment variable.
+
+## Requirements
+
+- Python 3.10+
+- `haystack-ai >= 2.12.0`
+- `hindsight-client >= 0.4.0`
+
+## Prerequisites
+
+A running Hindsight instance:
+
+**Hindsight Cloud (recommended):** [Sign up](https://ui.hindsight.vectorize.io/signup) — no self-hosting required.
+
+**Self-hosted:**
+
+```bash
+pip install hindsight-all
+export HINDSIGHT_API_LLM_API_KEY=your-api-key
+hindsight-api  # starts on http://localhost:8888
+```

hindsight-docs/src/data/integrations.json

@@ -200,6 +200,16 @@
       "link": "/sdks/integrations/google-adk",
       "icon": "/img/icons/google-adk.png"
     },
+    {
+      "id": "haystack",
+      "name": "Haystack",
+      "description": "Persistent memory for Haystack agents. Provides retain/recall/reflect Tools plus a HindsightToolset with optional auto-recall and auto-retain.",
+      "type": "official",
+      "by": "hindsight",
+      "category": "framework",
+      "link": "/sdks/integrations/haystack",
+      "icon": "/img/icons/haystack.svg"
+    },
     {
       "id": "nemoclaw",
       "name": "NemoClaw",

hindsight-docs/static/img/icons/haystack.svg

@@ -0,0 +1,108 @@
+# hindsight-haystack
+
+Haystack integration for [Hindsight](https://github.com/vectorize-io/hindsight) — persistent long-term memory for AI agents.
+
+Provides Haystack `Tool` instances that give any Haystack `Agent` persistent memory via Hindsight's retain/recall/reflect APIs.
+
+## Installation
+
+```bash
+pip install hindsight-haystack
+```
+
+## Quick Start
+
+```python
+from hindsight_client import Hindsight
+from hindsight_haystack import create_hindsight_tools
+from haystack.components.agents import Agent
+from haystack.components.generators.chat import OpenAIChatGenerator
+from haystack.dataclasses import ChatMessage
+
+client = Hindsight(base_url="http://localhost:8888")
+
+tools = create_hindsight_tools(
+    client=client,
+    bank_id="user-123",
+    mission="Track user preferences",
+)
+
+agent = Agent(
+    chat_generator=OpenAIChatGenerator(model="gpt-4o-mini"),
+    tools=tools,
+    system_prompt=(
+        "You are a helpful assistant with long-term memory. "
+        "Use retain_memory to store important facts. "
+        "Use recall_memory to search memory before answering."
+    ),
+)
+
+result = agent.run(messages=[ChatMessage.from_user("Remember that I prefer dark mode")])
+print(result["messages"][-1].text)
+```
+
+## Automatic Memory with HindsightToolset
+
+For automatic recall and retain without relying on the agent to call tools:
+
+```python
+from hindsight_haystack import HindsightToolset
+
+toolset = HindsightToolset(
+    client=client,
+    bank_id="user-123",
+    mission="Track user preferences",
+    auto_recall=True,   # Inject memories into system prompt before each turn
+    auto_retain=True,    # Store user + assistant messages after each turn
+)
+
+agent = Agent(
+    chat_generator=OpenAIChatGenerator(model="gpt-4o-mini"),
+    tools=toolset,
+    system_prompt="You are a helpful assistant with long-term memory.",
+)
+
+# Use toolset.run() for automatic memory behavior
+result = toolset.run(agent, messages=[ChatMessage.from_user("I prefer dark mode")])
+```
+
+## Selective Tools
+
+```python
+# Only retain + recall (no reflect)
+tools = create_hindsight_tools(
+    client=client,
+    bank_id="user-123",
+    include_reflect=False,
+)
+```
+
+## Configuration
+
+```python
+from hindsight_haystack import configure
+
+configure(
+    hindsight_api_url="http://localhost:8888",
+    api_key="your-api-key",
+    budget="mid",
+    tags=["source:haystack"],
+    context="my-app",
+    mission="Track user preferences",
+)
+
+# Now you can skip client= and url= arguments
+tools = create_hindsight_tools(bank_id="user-123")
+```
+
+## Requirements
+
+- Python 3.10+
+- `haystack-ai >= 2.12.0`
+- `hindsight-client >= 0.4.0`
+
+## Documentation
+
+- [Integration docs](https://docs.hindsight.vectorize.io/docs/sdks/integrations/haystack)
+- [Hindsight API docs](https://docs.hindsight.vectorize.io)
+

hindsight-integrations/haystack/README.md

@@ -0,0 +1,36 @@
+"""Hindsight memory integration for Haystack agents.
+
+Provides Haystack-compatible ``Tool`` instances backed by Hindsight's
+retain/recall/reflect APIs. Use ``create_hindsight_tools()`` to create
+tools for any Haystack ``Agent``.
+
+Usage::
+
+    from hindsight_haystack import create_hindsight_tools
+    from haystack.components.agents import Agent
+    from haystack.components.generators.chat import OpenAIChatGenerator
+
+    tools = create_hindsight_tools(bank_id="user-123", client=client)
+    agent = Agent(chat_generator=OpenAIChatGenerator(), tools=tools)
+"""
+
+from .config import (
+    HindsightHaystackConfig,
+    configure,
+    get_config,
+    reset_config,
+)
+from .errors import HindsightError
+from .tools import HindsightToolset, create_hindsight_tools
+
+__version__ = "0.1.0"
+
+__all__ = [
+    "configure",
+    "get_config",
+    "reset_config",
+    "HindsightHaystackConfig",
+    "HindsightError",
+    "create_hindsight_tools",
+    "HindsightToolset",
+]