fix(openclaw): strip runtime metadata from memory content (#1968)

* feat(openclaw): pass retain context guidance to prevent routing metadata misattribution

Hindsight's fact extraction LLM was misinterpreting routing identifiers
(sender open_id, bank ID, channel, provider) as semantic actors, project
names, or organizations. After many conversation turns, the bank name
(e.g. saber-prod) would override the actual project being discussed
(e.g. x-power-cli).

This adds interpretation guidance via the retain API 'context' field:
- New DEFAULT_RETAIN_CONTEXT constant explains that [context] block
  sender/channel/provider are routing identifiers, not human names
- Bank IDs, session keys, agent IDs, thread IDs, and tags are also
  marked as operational routing identifiers, not project names
- Assistant-role first-person statements are attributed to the AI
- Context is passed through the full chain: buildRetainRequest →
  scopeClient.retain → Hindsight SDK API
- RetainQueue persists and flushes context correctly
- Backfill CLI also passes context
- New 'retainContext' config option allows customization

includeSenderContext behavior is unchanged; the [context] block remains
in transcript content, but extraction LLM now knows how to interpret it.

7 files changed, 97 insertions(+).

* fix(openclaw): remove platform-specific examples from DEFAULT_RETAIN_CONTEXT

* test(openclaw): harden retain context handling

* fix(openclaw): strip runtime metadata from memory content

* refactor(openclaw): remove dead session-context surface

Following the removal of transcript context-prepending, drop the now-unused
formatRetentionSessionContext / RetentionSessionContext and the ignored
prepareRetentionTranscript session-context parameter (and the discarded
object built at the live call site). Remove the inert includeSenderContext
config option (no longer read) from the type, manifest schema, and UI label.
Collapse the session-context tests to two regression guards asserting that
retained JSON/text content carries no context header.

---------

Co-authored-by: Nicolò Boschi <boschi1997@gmail.com>

Author: de1tydev

hindsight-integrations/openclaw/README.md

@@ -95,6 +95,7 @@ Optional settings in `~/.openclaw/openclaw.json` under `plugins.entries.hindsigh
 | `bankIdPrefix`             | —                              | Prefix for bank IDs (e.g. `"prod"`)                                                                                                                                                                                                                                                                         |
 | `retainTags`               | `[]`                           | Tags applied to every retained document, useful for cross-agent/source labeling (e.g. `source_system:openclaw`, `agent:agentname`). Auto-retain also merges inline per-message tags from `<retain_tags>...</retain_tags>` or `<hindsight_retain_tags>...</hindsight_retain_tags>` blocks in user messages.  |
 | `retainSource`             | `"openclaw"`                   | `source` value written into retained document metadata                                                                                                                                                                                                                                                      |
+| `retainContext`            | built-in OpenClaw guidance     | Interpretation guidance sent through the Hindsight retain API `context` field. The default tells the extraction LLM that sender/channel/provider metadata, bank IDs, session keys, source systems, and tags are operational routing metadata, not human names, project names, or organizations.             |
 | `dynamicBankGranularity`   | `["agent", "channel", "user"]` | Fields used to derive bank ID. Options: `agent`, `channel`, `user`, `provider`                                                                                                                                                                                                                              |
 | `excludeProviders`         | `["heartbeat"]`                | Message providers to skip for recall/retain (e.g. `heartbeat`, `slack`, `telegram`, `discord`)                                                                                                                                                                                                              |
 | `autoRecall`               | `true`                         | Auto-inject memories before each turn. Set to `false` when the agent has its own recall tool.                                                                                                                                                                                                               |
@@ -162,6 +163,8 @@ Glob syntax:
 
 Retained documents use stable session-scoped IDs derived from the OpenClaw `sessionKey`. Every retain in a session shares one document id like `openclaw:agent:agentname:discord:channel:123`, so all turns of the conversation accumulate under a single Hindsight document (on legacy APIs without `update_mode: 'append'` support, the integration falls back to per-retain ids — `...:turn:000001`, `...:window:000002` for chunked retention — so prior turns aren't overwritten). Retained documents include richer metadata such as `session_key`, `agent_id`, `provider`, `channel_id`, `thread_id`, `sender_id`, `turn_index`, and `retention_scope`. Each message in the retained JSON also carries a structured `timestamp` field (ISO 8601) lifted from OpenClaw's per-message time, so facts are not polluted by inline weekday/date prefixes.
 
+`retainContext` is sent separately from the transcript content and gives Hindsight's extraction LLM interpretation guidance for the retained document. The default is designed for OpenClaw transcripts: it explains that sender/channel/provider metadata is operational routing data, that assistant-role first-person statements belong to the AI assistant, and that bank IDs or tags should not be treated as the discussed project. Sender/channel/provider stay in retain request metadata/context and are not prepended into retained transcript content.
+
 ## Documentation
 
 For full documentation, configuration options, troubleshooting, and development guide, see:

hindsight-integrations/openclaw/openclaw.plugin.json

@@ -5,8 +5,14 @@
   "configContracts": {
     "secretInputs": {
       "paths": [
-        { "path": "llmApiKey", "expected": "string" },
-        { "path": "hindsightApiToken", "expected": "string" }
+        {
+          "path": "llmApiKey",
+          "expected": "string"
+        },
+        {
+          "path": "hindsightApiToken",
+          "expected": "string"
+        }
       ]
     }
   },
@@ -112,6 +118,11 @@
         "description": "Source value written into retained document metadata. Defaults to 'openclaw'.",
         "default": "openclaw"
       },
+      "retainContext": {
+        "type": "string",
+        "description": "Interpretation guidance sent via the Hindsight retain API context field. It tells the extraction LLM that OpenClaw sender/channel/provider values, bank IDs, session keys, source systems, and tags are operational routing metadata, not human names, project names, or organizations.",
+        "default": "This content is an AI-assistant conversation transcript from OpenClaw. Retain request metadata may include routing identifiers such as 'sender_id' (an opaque user ID, not a human name), 'channel_id' (a chat identifier), and 'provider' (the messaging platform name). These are operational routing metadata, not semantic actors or people. Messages with role 'assistant' are from the AI assistant; first-person statements in assistant messages refer to the AI, not the human user. Messages with role 'user' are from the human user. Bank IDs, session keys, agent IDs, thread IDs, source systems, and tags in metadata are also operational routing identifiers, not human names, project names, or organizations."
+      },
       "autoRecall": {
         "type": "boolean",
         "description": "Automatically recall memories on every prompt and inject them as context. Set to false when agent has its own recall tool.",
@@ -159,11 +170,6 @@
         "description": "When true (default) and retainFormat is 'json', each message's content is an Anthropic-shaped array of typed blocks including tool_use (agent tool calls) and tool_result (truncated at 2000 chars). Operational MCP tools — Hindsight's own recall/retain/search — are filtered out to avoid feedback loops. Set to false to retain text-only content, one string per message.",
         "default": true
       },
-      "includeSenderContext": {
-        "type": "boolean",
-        "description": "When true (default), prepend a session context block (sender, channel, provider) to retained transcripts so similarity search can distinguish memories by speaker even when dynamicBankGranularity does not include 'user'. Set false to omit.",
-        "default": true
-      },
       "retainEveryNTurns": {
         "type": "integer",
         "description": "Retain every Nth turn instead of every turn. 1 = every turn (default). Values > 1 enable chunked retention with a sliding window.",
@@ -277,12 +283,16 @@
       },
       "ignoreSessionPatterns": {
         "type": "array",
-        "items": { "type": "string" },
+        "items": {
+          "type": "string"
+        },
         "description": "Session key glob patterns to skip entirely (no recall, no retain). E.g. [\"agent:main:**\", \"agent:*:cron:**\"]. * matches non-colon chars, ** matches anything."
       },
       "statelessSessionPatterns": {
         "type": "array",
-        "items": { "type": "string" },
+        "items": {
+          "type": "string"
+        },
         "description": "Session key glob patterns for read-only sessions: retain is always skipped, recall is skipped when skipStatelessSessions is true. E.g. [\"agent:*:subagent:**\", \"agent:*:heartbeat:**\"]."
       },
       "skipStatelessSessions": {
@@ -377,6 +387,10 @@
       "label": "Retain Source",
       "placeholder": "openclaw"
     },
+    "retainContext": {
+      "label": "Retain Context",
+      "placeholder": "Guidance for interpreting retained OpenClaw transcripts"
+    },
     "autoRecall": {
       "label": "Auto-Recall",
       "placeholder": "true (inject memories on every prompt)"
@@ -405,10 +419,6 @@
       "label": "Retain Tool Calls",
       "placeholder": "true (default) — include tool_use / tool_result blocks in retained JSON"
     },
-    "includeSenderContext": {
-      "label": "Include Sender Context",
-      "placeholder": "true (default) — prepend sender/channel/provider context to retained content"
-    },
     "retainEveryNTurns": {
       "label": "Retain Every N Turns",
       "placeholder": "1 (every turn, default)"

hindsight-integrations/openclaw/src/backfill.test.ts

@@ -197,8 +197,9 @@ describe("backfill helpers", () => {
   it("treats a symlinked bin path as direct execution", async () => {
     const { isDirectExecution } = await import("./backfill.js");
     const dir = mkdtempSync(join(tmpdir(), "hindsight-backfill-bin-"));
-    const modulePath = join(process.cwd(), "dist", "backfill.js");
+    const modulePath = join(dir, "backfill.js");
     const symlinkPath = join(dir, "hindsight-openclaw-backfill");
+    writeFileSync(modulePath, "#!/usr/bin/env node\n", "utf8");
     symlinkSync(modulePath, symlinkPath);
 
     const moduleUrl = pathToFileURL(modulePath).href;

hindsight-integrations/openclaw/src/backfill.ts

@@ -17,7 +17,7 @@ function loadPackageVersion(): string {
 }
 
 const USER_AGENT = `hindsight-openclaw/${loadPackageVersion()}`;
-import { detectExternalApi, detectLLMConfig } from "./index.js";
+import { DEFAULT_RETAIN_CONTEXT, detectExternalApi, detectLLMConfig } from "./index.js";
 import type { BankStats, PluginConfig } from "./types.js";
 import {
   buildBackfillPlan,
@@ -547,6 +547,7 @@ export async function runCli(argv: string[] = process.argv.slice(2)): Promise<vo
         }
         await client.retain(entry.bankId, entry.transcript, {
           documentId: entry.documentId,
+          context: pluginConfig.retainContext || DEFAULT_RETAIN_CONTEXT,
           metadata,
           async: true,
         });