Skip to content

Commit 74cbedf

Browse files
Skiipy11claude
andcommitted
v1.4.0: compact responses, security hardening, async consolidation, memory deletion
Token optimization: - Briefing and search default to compact format (~70-80% token reduction) - Three format tiers: summary (counts only), compact (200-char truncation), full - Importance-ranked sorting (critical/high first) - Low-importance events filtered in compact mode Security: - Full XML entity escaping in consolidation (& < > " ') - JSON code-fence stripping for LLM output - Top-level structure validation on LLM responses Performance: - O(1) fact/status supersedes via targeted Qdrant key/subject queries - Async consolidation with job ID polling (POST /consolidate → 202) - Briefing pagination (limit param, max 500) - New Qdrant payload indexes: key, subject New features: - DELETE /memory/:id — soft-delete with agent-scoped permissions + audit trail - brain_delete MCP tool - Request correlation IDs (X-Request-ID header) - Configurable MCP timeouts (BRAIN_MCP_TIMEOUT, BRAIN_MCP_CONSOLIDATION_TIMEOUT) Reliability: - Graceful shutdown (SIGTERM/SIGINT with 10s drain) - Alias cache pre-seeds 67 KNOWN_TECH entries on startup - Entity name normalization prevents case-variant duplicates - SQLite silent catches replaced with proper WARN logging Testing: - 41 new tests (validation + entity extraction), 81 total, all passing Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 1bb6dcb commit 74cbedf

16 files changed

Lines changed: 936 additions & 137 deletions

File tree

.env.example

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,3 +57,10 @@ CONSOLIDATION_MODEL=gpt-4o-mini
5757
# --- Memory Decay ---
5858
# Decay factor per day without access (0.98 = 2% decay/day). Only affects facts and statuses.
5959
DECAY_FACTOR=0.98
60+
61+
# --- Event TTL ---
62+
# EVENT_TTL_DAYS=30 # Auto-expire low-importance, never-accessed events after N days (default: 30)
63+
64+
# --- MCP Server Timeouts ---
65+
# BRAIN_MCP_TIMEOUT=15000 # Default timeout for MCP→API calls in ms (default: 15000)
66+
# BRAIN_MCP_CONSOLIDATION_TIMEOUT=120000 # Timeout for sync consolidation calls in ms (default: 120000)

README.md

Lines changed: 32 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -292,6 +292,7 @@ curl "http://localhost:8084/memory/search?q=deployment&entity=Docker&limit=5" \
292292
| `category` | Filter by category |
293293
| `entity` | Filter to memories linked to this entity (by name or alias) |
294294
| `limit` | Max results (default 10) |
295+
| `format` | `compact` (default) truncates to 200 chars; `full` returns complete content + all metadata |
295296
| `include_superseded` | Set to `true` to include superseded memories |
296297

297298
### `GET /briefing` — Session briefing
@@ -301,7 +302,15 @@ curl "http://localhost:8084/briefing?since=2025-01-15T00:00:00Z&agent=claude-cod
301302
-H "X-Api-Key: YOUR_KEY"
302303
```
303304

304-
Returns categorized updates (events, facts, statuses, decisions) from all agents since the given timestamp. Excludes entries from the requesting agent by default. The summary includes `entities_mentioned` — a ranked list of entities that appeared in the briefing period.
305+
Returns categorized updates (events, facts, statuses, decisions) from all agents since the given timestamp. Excludes entries from the requesting agent by default. Results are sorted by importance (critical/high first), then recency.
306+
307+
| Param | Description |
308+
|-------|-------------|
309+
| `since` | ISO 8601 timestamp (required) |
310+
| `agent` | Requesting agent — its entries are excluded |
311+
| `format` | `compact` (default): 200-char truncation, skips low-importance events. `summary`: counts + headlines only. `full`: complete content. |
312+
| `limit` | Max memories to retrieve (default 100, max 500) |
313+
| `include` | Set to `all` to include own entries |
305314

306315
### `GET /memory/query` — Structured query
307316

@@ -343,10 +352,29 @@ Requires a structured storage backend (SQLite, Postgres, or Baserow). Returns a
343352
### `POST /consolidate` — Trigger LLM consolidation
344353

345354
```bash
355+
# Async (default) — returns job ID immediately
346356
curl -X POST http://localhost:8084/consolidate -H "X-Api-Key: YOUR_KEY"
357+
# {"status":"started","job_id":"a1b2c3d4-..."}
358+
359+
# Poll job status
360+
curl http://localhost:8084/consolidate/job/a1b2c3d4-... -H "X-Api-Key: YOUR_KEY"
361+
362+
# Sync (blocking) — waits for completion
363+
curl -X POST "http://localhost:8084/consolidate?sync=true" -H "X-Api-Key: YOUR_KEY"
364+
```
365+
366+
Runs the consolidation engine on demand. The engine finds duplicates to merge, contradictions to flag, connections between memories, cross-memory insights, and named entities to extract/normalize. The alias cache is refreshed after each run. Also runs automatically on a schedule when `CONSOLIDATION_ENABLED=true`. Jobs auto-expire after 1 hour.
367+
368+
### `DELETE /memory/:id` — Delete a memory
369+
370+
```bash
371+
curl -X DELETE http://localhost:8084/memory/a1b2c3d4-... \
372+
-H "X-Api-Key: YOUR_KEY" \
373+
-H "Content-Type: application/json" \
374+
-d '{"reason": "Contains incorrect information"}'
347375
```
348376

349-
Runs the consolidation engine on demand. The engine finds duplicates to merge, contradictions to flag, connections between memories, cross-memory insights, and named entities to extract/normalize. The alias cache is refreshed after each run. Also runs automatically on a schedule when `CONSOLIDATION_ENABLED=true`.
377+
Soft-deletes a memory (marks it inactive). Agent-scoped API keys can only delete their own memories. The `reason` field is optional but logged for audit purposes.
350378

351379
### `POST /webhook/n8n` — n8n workflow logging
352380

@@ -398,7 +426,7 @@ Uses fast-path regex extraction only (no LLM calls, no cost). Processes all acti
398426

399427
### MCP Server (Claude Code, Cursor, Windsurf)
400428

401-
The MCP server exposes 7 tools: `brain_store`, `brain_search`, `brain_briefing`, `brain_query`, `brain_stats`, `brain_consolidate`, `brain_entities`.
429+
The MCP server exposes 8 tools: `brain_store`, `brain_search`, `brain_briefing`, `brain_query`, `brain_stats`, `brain_consolidate`, `brain_entities`, `brain_delete`.
402430

403431
**Claude Code (`~/.claude.json`):**
404432
```json
@@ -621,7 +649,7 @@ multi-agent-memory/
621649
│ ├── Dockerfile
622650
│ └── package.json
623651
├── mcp-server/ # MCP server for Claude/Cursor
624-
│ ├── src/index.js # 7 tools: store, search, briefing, query, stats, consolidate, entities
652+
│ ├── src/index.js # 8 tools: store, search, briefing, query, stats, consolidate, entities, delete
625653
│ └── package.json
626654
├── adapters/
627655
│ ├── bash/ # CLI adapter (curl + jq)

adapters/bash/brain.sh

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ shift
3232

3333
# === Parse args ===
3434
TYPE="" CONTENT="" CLIENT_ID="global" CATEGORY="" IMPORTANCE=""
35-
QUERY="" LIMIT="" SINCE="" INCLUDE="" SOURCE="" KEY="" SUBJECT="" STATUS_VALUE=""
35+
QUERY="" LIMIT="" SINCE="" INCLUDE="" SOURCE="" KEY="" SUBJECT="" STATUS_VALUE="" FORMAT=""
3636

3737
while [[ $# -gt 0 ]]; do
3838
case $1 in
@@ -49,6 +49,7 @@ while [[ $# -gt 0 ]]; do
4949
--key) KEY="$2"; shift 2 ;;
5050
--subject) SUBJECT="$2"; shift 2 ;;
5151
--status_value) STATUS_VALUE="$2"; shift 2 ;;
52+
--format) FORMAT="$2"; shift 2 ;;
5253
*) echo "Unknown arg: $1" >&2; exit 1 ;;
5354
esac
5455
done
@@ -108,7 +109,7 @@ case "$CMD" in
108109
exit 1
109110
fi
110111

111-
QS="q=$(jq -rn --arg q "$QUERY" '$q | @uri')"
112+
QS="q=$(jq -rn --arg q "$QUERY" '$q | @uri')&format=${FORMAT:-compact}"
112113
[ -n "$TYPE" ] && QS="${QS}&type=$(echo -n "$TYPE" | jq -sRr @uri)"
113114
[ -n "$SOURCE" ] && QS="${QS}&source_agent=$(echo -n "$SOURCE" | jq -sRr @uri)"
114115
[ -n "$CLIENT_ID" ] && [ "$CLIENT_ID" != "global" ] && QS="${QS}&client_id=$(echo -n "$CLIENT_ID" | jq -sRr @uri)"
@@ -136,7 +137,7 @@ case "$CMD" in
136137
exit 1
137138
fi
138139

139-
QS="since=$(jq -rn --arg s "$SINCE" '$s | @uri')&agent=${SOURCE_AGENT}"
140+
QS="since=$(jq -rn --arg s "$SINCE" '$s | @uri')&agent=${SOURCE_AGENT}&format=${FORMAT:-compact}"
140141
[ -n "$INCLUDE" ] && QS="${QS}&include=${INCLUDE}"
141142

142143
RESPONSE=$(curl -s --max-time 15 -H "${AUTH_HEADER}" "${API_URL}/briefing?${QS}")

api/src/index.js

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
import express from 'express';
2+
import crypto from 'crypto';
23
import { authMiddleware } from './middleware/auth.js';
34
import { rateLimitMiddleware } from './middleware/ratelimit.js';
45
import { memoryRouter } from './routes/memory.js';
@@ -26,6 +27,13 @@ const HOST = process.env.HOST || '127.0.0.1';
2627

2728
app.use(express.json({ limit: '1mb' }));
2829

30+
// Request correlation ID
31+
app.use((req, res, next) => {
32+
req.requestId = req.headers['x-request-id'] || crypto.randomUUID();
33+
res.setHeader('x-request-id', req.requestId);
34+
next();
35+
});
36+
2937
// Health check (no auth)
3038
app.get('/health', (req, res) => {
3139
res.json({ status: 'ok', service: 'shared-brain', timestamp: new Date().toISOString() });
@@ -90,9 +98,26 @@ async function start() {
9098
console.log('[shared-brain] Consolidation disabled (CONSOLIDATION_ENABLED=false)');
9199
}
92100

93-
app.listen(PORT, HOST, () => {
101+
const server = app.listen(PORT, HOST, () => {
94102
console.log(`[shared-brain] Memory API running on ${HOST}:${PORT}`);
95103
});
104+
105+
// Graceful shutdown
106+
const shutdown = (signal) => {
107+
console.log(`[shared-brain] ${signal} received — shutting down gracefully...`);
108+
server.close(() => {
109+
console.log('[shared-brain] HTTP server closed');
110+
process.exit(0);
111+
});
112+
// Force exit after 10s if connections don't drain
113+
setTimeout(() => {
114+
console.error('[shared-brain] Forced exit after timeout');
115+
process.exit(1);
116+
}, 10_000).unref();
117+
};
118+
119+
process.on('SIGTERM', () => shutdown('SIGTERM'));
120+
process.on('SIGINT', () => shutdown('SIGINT'));
96121
} catch (err) {
97122
console.error('[shared-brain] Failed to start:', err.message);
98123
process.exit(1);

api/src/routes/briefing.js

Lines changed: 115 additions & 70 deletions
Original file line numberDiff line numberDiff line change
@@ -1,103 +1,148 @@
11
import { Router } from 'express';
22
import { scrollPoints, getCollectionInfo, computeEffectiveConfidence } from '../services/qdrant.js';
3-
// Briefing primarily uses Qdrant — structured store imports kept for potential future use
43

54
export const briefingRouter = Router();
65

6+
const COMPACT_MAX_CONTENT = 200;
7+
const IMPORTANCE_RANK = { critical: 4, high: 3, medium: 2, low: 1 };
8+
79
// GET /briefing — Session briefing: what happened since timestamp
10+
// format=compact (default): truncated content, skip low-importance events — lean for agents
11+
// format=summary: counts + headlines only — minimal tokens
12+
// format=full: complete content — original behavior
813
briefingRouter.get('/', async (req, res) => {
914
try {
10-
const { agent, since, include } = req.query;
15+
const { agent, since, include, limit: limitParam, format: formatParam } = req.query;
16+
const format = ['compact', 'summary', 'full'].includes(formatParam) ? formatParam : 'compact';
1117

1218
if (!since) {
1319
return res.status(400).json({
1420
error: 'Missing required parameter: since (ISO 8601 timestamp)',
15-
example: '/briefing?since=2026-03-09T00:00:00Z&agent=claude-code',
21+
example: '/briefing?since=2026-03-09T00:00:00Z&agent=claude-code&format=compact',
1622
});
1723
}
1824

19-
const briefing = {
20-
since,
21-
requesting_agent: agent || 'unknown',
22-
generated_at: new Date().toISOString(),
23-
events: [],
24-
facts_updated: [],
25-
status_changes: [],
26-
decisions: [],
27-
summary: {},
28-
};
29-
30-
// Get recent events from Qdrant (semantic store has everything)
25+
// Get recent events from Qdrant
26+
const scrollLimit = Math.min(Math.max(parseInt(limitParam) || 100, 1), 500);
3127
const filter = { created_after: since };
32-
const recent = await scrollPoints(filter, 100);
28+
const recent = await scrollPoints(filter, scrollLimit);
3329
const points = recent.points || [];
3430

35-
// Categorize results
36-
for (const point of points) {
31+
// Filter points: exclude requesting agent's own entries (unless include=all)
32+
const filteredPoints = points.filter(point => {
3733
const p = point.payload;
38-
// Skip entries from the requesting agent (they already know what they did)
39-
// Unless include=all is set
40-
if (agent && p.source_agent === agent && include !== 'all') continue;
41-
42-
const entry = {
43-
id: point.id,
44-
content: p.text,
45-
source_agent: p.source_agent,
46-
client_id: p.client_id,
47-
category: p.category,
48-
importance: p.importance,
49-
confidence: computeEffectiveConfidence(p),
50-
created_at: p.created_at,
51-
};
34+
if (agent && p.source_agent === agent && include !== 'all') return false;
35+
return true;
36+
});
5237

53-
switch (p.type) {
54-
case 'event': briefing.events.push(entry); break;
55-
case 'fact': briefing.facts_updated.push(entry); break;
56-
case 'status': briefing.status_changes.push(entry); break;
57-
case 'decision': briefing.decisions.push(entry); break;
58-
}
59-
}
60-
61-
// Sort each by created_at descending
62-
const sortDesc = (a, b) => new Date(b.created_at) - new Date(a.created_at);
63-
briefing.events.sort(sortDesc);
64-
briefing.facts_updated.sort(sortDesc);
65-
briefing.status_changes.sort(sortDesc);
66-
briefing.decisions.sort(sortDesc);
38+
// In compact mode, skip low-importance events (keep facts/statuses/decisions regardless)
39+
const relevantPoints = format === 'compact'
40+
? filteredPoints.filter(p => p.payload.type !== 'event' || p.payload.importance !== 'low')
41+
: filteredPoints;
6742

68-
// Collect entities mentioned across all briefing entries
43+
// Collect entity counts (before any truncation)
6944
const entityCounts = {};
70-
for (const point of points) {
71-
const entities = point.payload.entities || [];
72-
for (const ent of entities) {
73-
const key = ent.name;
74-
if (!entityCounts[key]) entityCounts[key] = { name: ent.name, type: ent.type, count: 0 };
75-
entityCounts[key].count++;
45+
for (const point of relevantPoints) {
46+
for (const ent of (point.payload.entities || [])) {
47+
if (!entityCounts[ent.name]) entityCounts[ent.name] = { name: ent.name, type: ent.type, count: 0 };
48+
entityCounts[ent.name].count++;
7649
}
7750
}
7851
const entitiesMentioned = Object.values(entityCounts).sort((a, b) => b.count - a.count);
7952

80-
// Summary stats
81-
briefing.summary = {
82-
total_entries: points.length,
83-
events: briefing.events.length,
84-
facts_updated: briefing.facts_updated.length,
85-
status_changes: briefing.status_changes.length,
86-
decisions: briefing.decisions.length,
53+
// Build summary (always included)
54+
const summary = {
55+
total_entries: relevantPoints.length,
56+
total_in_period: points.length,
57+
events: 0, facts_updated: 0, status_changes: 0, decisions: 0,
8758
agents_active: [...new Set(points.flatMap(p => p.payload.observed_by || [p.payload.source_agent]))],
8859
clients_mentioned: [...new Set(points.map(p => p.payload.client_id).filter(c => c !== 'global'))],
89-
entities_mentioned: entitiesMentioned.slice(0, 20),
60+
entities_mentioned: entitiesMentioned.slice(0, 15),
61+
};
62+
63+
// Categorize
64+
const buckets = { events: [], facts_updated: [], status_changes: [], decisions: [] };
65+
for (const point of relevantPoints) {
66+
const p = point.payload;
67+
const confidence = computeEffectiveConfidence(p);
68+
69+
// Build entry based on format
70+
let entry;
71+
if (format === 'summary') {
72+
// Minimal: first line of content only (headline)
73+
const firstLine = (p.text || '').split('\n')[0].slice(0, 120);
74+
entry = {
75+
id: point.id,
76+
headline: firstLine,
77+
source_agent: p.source_agent,
78+
client_id: p.client_id,
79+
importance: p.importance,
80+
created_at: p.created_at,
81+
};
82+
} else if (format === 'compact') {
83+
// Truncated: first 200 chars + flag if truncated
84+
const text = p.text || '';
85+
const truncated = text.length > COMPACT_MAX_CONTENT;
86+
entry = {
87+
id: point.id,
88+
content: truncated ? text.slice(0, COMPACT_MAX_CONTENT) + '...' : text,
89+
truncated,
90+
source_agent: p.source_agent,
91+
client_id: p.client_id,
92+
importance: p.importance,
93+
confidence: +confidence.toFixed(3),
94+
created_at: p.created_at,
95+
};
96+
} else {
97+
// Full: original behavior
98+
entry = {
99+
id: point.id,
100+
content: p.text,
101+
source_agent: p.source_agent,
102+
client_id: p.client_id,
103+
category: p.category,
104+
importance: p.importance,
105+
confidence: +confidence.toFixed(4),
106+
created_at: p.created_at,
107+
};
108+
}
109+
110+
switch (p.type) {
111+
case 'event': buckets.events.push(entry); summary.events++; break;
112+
case 'fact': buckets.facts_updated.push(entry); summary.facts_updated++; break;
113+
case 'status': buckets.status_changes.push(entry); summary.status_changes++; break;
114+
case 'decision': buckets.decisions.push(entry); summary.decisions++; break;
115+
}
116+
}
117+
118+
// Sort by importance then recency (critical/high first, then by date)
119+
const sortByImportanceAndDate = (a, b) => {
120+
const impDiff = (IMPORTANCE_RANK[b.importance] || 0) - (IMPORTANCE_RANK[a.importance] || 0);
121+
if (impDiff !== 0) return impDiff;
122+
return new Date(b.created_at) - new Date(a.created_at);
123+
};
124+
for (const arr of Object.values(buckets)) {
125+
arr.sort(sortByImportanceAndDate);
126+
}
127+
128+
const briefing = {
129+
since,
130+
format,
131+
requesting_agent: agent || 'unknown',
132+
generated_at: new Date().toISOString(),
133+
summary,
134+
...buckets,
90135
};
91136

92-
// Get collection stats
93-
try {
94-
const info = await getCollectionInfo();
95-
briefing.brain_stats = {
96-
total_memories: info.points_count,
97-
vectors_count: info.vectors_count,
98-
};
99-
} catch (e) {
100-
// Non-critical
137+
// Collection stats (compact/full only — skip for summary to save tokens)
138+
if (format !== 'summary') {
139+
try {
140+
const info = await getCollectionInfo();
141+
briefing.brain_stats = {
142+
total_memories: info.points_count,
143+
vectors_count: info.vectors_count,
144+
};
145+
} catch (e) { /* non-critical */ }
101146
}
102147

103148
res.json(briefing);

0 commit comments

Comments
 (0)