You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* fix(memory): phase 1 tails - tool honesty, session save lock, combined extraction, curator provenance
- memory add_atom now reports 'quarantined for human review (reason: ...)'
instead of 'added atom' when the guard scan routed the atom to
quarantine, so the agent (and user) are not misled about what landed
in the live store
- session Store.Save no longer holds the store mutex during embedding
(a potential 10s HTTP call); vector-index add happens after the lock
is released, persistence semantics unchanged
- session-end extraction uses ONE combined LLM call (summary + facts in
a single JSON response) when both episode and fact extraction are
enabled, with automatic fallback to the two single-purpose calls on
parse failure; all existing safety filters preserved
- curator MergeSkills keeps the WORSE provenance of the two inputs
(Untrusted/NeedsReview = OR, Sources = deduped union) instead of
inheriting the keeper's trust while concatenating a tainted body
* feat(memory): phase 2 smartness - semantic dedup, intent confidence, consolidation, index fingerprint, stats API
- semantic dedup on add: after the exact-match tier, atoms whose top-1
cosine similarity to a live atom exceeds semantic_dedup_threshold
(default 0.92, 0 disables) refresh the existing atom instead of
duplicating; in-batch duplicates handled via direct embedding
- predicted-intent confidence now weights recall scores; intents below
0.3 confidence are skipped entirely
- extractor confidence default is now 0.7 (was 1.0) so unconfident
extractions no longer get maximal retention and eviction resistance
- new ConsolidateAtoms: greedy similarity clustering (default 0.9) +
one LLM merge call per group; originals kept on any failure; merged
atoms keep highest confidence and union of associations; live store
only, quarantine untouched
- vector index persists a corpus fingerprint (count + FNV-1a of sorted
atom IDs) in vectors_meta.json; a mismatch on load forces a rebuild
instead of silently serving a stale corpus (legacy files rebuild once)
- new Stats() API: live/quarantined atoms, quarantine reason breakdown,
index vectors/dirty, store size, and recall timeout/failure counters
for degradation observability
* test: phase 3 proof - PiGuard CI E2E + fuzz soaks (and a fuzz-found panic fix)
- new guard E2E test (env-gated ODEK_E2E_GUARD=1) exercises the real
sidecar through guard.New over HTTP and socket_path: confident-BENIGN
regression, injection detection, batch classification
- new piguard-e2e CI workflow: builds the vendored daemon + gateway,
caches the ONNX model by pinned revision, runs the E2E on
ubuntu-latest (socket mode included), always tears down
- fuzz suites for extractJSON, SKILL.md parsing, and session loading
- fix a REAL panic found by the fuzzer: parseYAMLValue sliced
s[1:len(s)-1] on a 1-char quoted string (' or "), crashing on any
SKILL.md frontmatter line like 'key: "'
* feat(memory): phase 4 operability - 'odek memory extended stats' and 'consolidate'
- stats: store/index sizes, quarantine reason breakdown, recall
timeout/failure counters with a degradation warning - makes guard
false-positive rates and recall health visible to operators
- consolidate: merges near-duplicate live atoms via the operator's
configured LLM backend (10-minute bounded run)
* Potential fix for pull request finding 'CodeQL / Workflow does not contain permissions'
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* docs: sync CONFIG/MEMORY/EXTENDED_MEMORY/AGENTS with roadmap changes
- CONFIG.md: add semantic_dedup_threshold and consolidate_similarity_threshold
to the extended config example and field reference
- MEMORY.md: combined session-end extraction call, bounded drain at close
(episode extraction no longer fire-and-forget), quarantine-not-reject
guard behavior with the honest add_atom report, stats in the CLI surface
and observability sections, complete memory tool action enum
- EXTENDED_MEMORY.md: consolidate/stats now have CLI commands
- AGENTS.md: PIGuard E2E and fuzz soak commands in Testing
* test: move guard E2E from CI to a local runner script
The GitHub Actions job was too slow to keep (a ~735 MB ONNX model
download plus two image builds per run). docker/piguard-e2e.sh now
provisions the same stack locally (build-if-missing, health wait,
env-gated test, always-tear-down) with an optional --linux flag that
runs the test binary inside a container for full socket-mode coverage
(the host socket subtest skips on macOS because unix sockets do not
cross the Docker Desktop VM boundary). Verified locally: host run
passes, container run passes all 6 subtests including socket mode.
---------
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
go test -fuzz=FuzzExtractJSON -fuzztime=30s ./internal/memory/extended/
246
+
go test -fuzz=FuzzParseSkillContent -fuzztime=30s ./internal/skills/
247
+
go test -fuzz=FuzzSessionLoad -fuzztime=30s ./internal/session/
238
248
```
239
249
240
250
Note: MCP client E2E tests build the fakeserver from `internal/mcpclient/testdata/main.go` at test time (no pre-compiled binary). macOS temp dirs are classified as `LocalWrite` (not `SystemWrite`), and the Docker availability check verifies daemon reachability before running sandbox tests.
Copy file name to clipboardExpand all lines: docs/CONFIG.md
+4Lines changed: 4 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -375,6 +375,8 @@ The `memory` section controls the persistent memory system (see [docs/MEMORY.md]
375
375
"semantic_search_overfetch": 4,
376
376
"semantic_search_min_score": 0.55,
377
377
"semantic_search_rerank": true,
378
+
"semantic_dedup_threshold": 0.92,
379
+
"consolidate_similarity_threshold": 0.9,
378
380
"atom_max_chars": 300,
379
381
"memory_budget_chars": 2000,
380
382
"decay_half_life_days": 30,
@@ -409,6 +411,8 @@ The `memory` section controls the persistent memory system (see [docs/MEMORY.md]
409
411
|`semantic_search_overfetch`|`4`| — | — | Candidate multiplier before filtering and reranking. |
410
412
|`semantic_search_min_score`|`0.55`| — | — | Minimum cosine similarity for a candidate to be considered. |
411
413
|`semantic_search_rerank`|`true`| — | — | Use the memory LLM to rerank candidates. |
414
+
|`semantic_dedup_threshold`|`0.92`| — | — | Cosine similarity at or above which an incoming atom is treated as a paraphrase of an existing live atom and refreshes it instead of appending. `0` disables the semantic tier (exact-match dedup always runs). |
415
+
|`consolidate_similarity_threshold`|`0.9`| — | — | Pairwise cosine similarity at or above which live atoms are grouped for LLM merging by `odek memory extended consolidate` / `ConsolidateAtoms`. |
412
416
|`atom_max_chars`|`300`|`ODEK_MEMORY_EXTENDED_ATOM_MAX_CHARS`|`--memory-extended-atom-max-chars`| Maximum stored text length per atom. |
413
417
|`memory_budget_chars`|`2000`|`ODEK_MEMORY_EXTENDED_MEMORY_BUDGET_CHARS`|`--memory-extended-memory-budget-chars`| Maximum injected Extended Memory context per turn. |
414
418
|`decay_half_life_days`|`30`| — | — | Days until an atom's recall/eviction weight halves. |
├── user_model.json # persisted inferred user state + pending review queue
105
105
└── associations.json # bidirectional atom association links
106
106
```
107
107
108
-
All files are written atomically and use `0600` permissions. The vector store and metadata are rebuilt from the chunk files if they are ever corrupted. The user model and associations are loaded at startup and persisted on every change.
108
+
All files are written atomically and use `0600` permissions. The vector store and metadata are rebuilt from the chunk files if they are ever corrupted. `vectors_meta.json` records both the embedding-space fingerprint (which embedding backend produced the vectors) and a corpus fingerprint (atom count plus a hash of the sorted atom IDs); a mismatch on either — including a missing corpus fingerprint in files written before it was tracked — marks the persisted index stale so it is rebuilt once instead of silently serving an outdated corpus. The user model and associations are loaded at startup and persisted on every change.
109
109
110
110
## Dedicated Memory LLM
111
111
@@ -163,6 +163,23 @@ Extracted atoms are immediately:
163
163
4. Embedded and written to the atom store.
164
164
5. Checked against the 100 MB size cap; low-retention atoms are evicted if needed.
165
165
166
+
Explicit LLM-provided `confidence` values in (0, 1] are kept. A missing or out-of-range confidence defaults to **0.7** for extraction-origin atoms, so unqualified extracted memories do not inflate their retention score.
167
+
168
+
### Write-Path Deduplication
169
+
170
+
Atoms are deduplicated in two tiers at persistence time:
171
+
172
+
1.**Exact match**: an atom whose normalized text (lowercased, whitespace-collapsed) equals an existing live atom refreshes it — new `CreatedAt`, the higher confidence wins, the original ID is kept — instead of appending a duplicate.
173
+
2.**Semantic match**: if no exact match exists and `semantic_dedup_threshold` is non-zero, the incoming atom is compared against the live corpus via the vector index (top-1 similarity); atoms stored earlier in the same batch, which the index does not cover yet, are compared by embedding both texts directly. A match at or above the threshold is refreshed the same way. The dedup search uses the pre-batch index state and never triggers extra index rebuilds: the single index invalidation after the batch is preserved.
174
+
175
+
### Atom Consolidation
176
+
177
+
`ExtendedMemory.ConsolidateAtoms(ctx)` — exposed as `odek memory extended consolidate` — merges groups of live atoms that are near-duplicates (pairwise cosine similarity at or above `consolidate_similarity_threshold`). For each group it asks the memory LLM — one call per group — to merge the texts into a single concise atom, stores the merged atom through the normal add path (scan and size cap apply) keeping the group's highest confidence, a refreshed `CreatedAt`, and the union of the group's outward associations, then removes the originals. Quarantined atoms are never consolidated. On any failure for a group (LLM error, empty response, scan rejection) the originals are kept untouched.
178
+
179
+
### Observability
180
+
181
+
`ExtendedMemory.Stats()` returns a `Stats` snapshot — surfaced as `odek memory extended stats` — for operators and monitoring: live/quarantined atom counts, quarantine entries grouped by reason prefix (`tainted`, `scan_rejected`), index vector count and dirty flag, on-disk store size, and recall error counters (`RecallTimeouts` for context-deadline-exceeded errors, `RecallFailures` for all other recall errors). The CLI prints a degradation warning when either counter is non-zero.
182
+
166
183
## Semantic Search
167
184
168
185
Extended Memory replaces episode-based recall with semantic search over atom vectors.
@@ -200,6 +217,8 @@ The final recall result is also bounded by `memory_budget_chars`. Tainted atoms
200
217
201
218
When `follow_up_anticipation_enabled` is true (default), the memory LLM receives the current user message, the last several user messages, and the current user-state model. It returns a JSON array of up to `predictive_intents` likely follow-up intents. Each intent is embedded and searched, and the union of literal-query matches and predicted-intent matches is injected into the main agent's context. For each predicted intent, the system also performs a type-targeted recall for `convention`, `file`, and `error` atoms so the agent can pre-load relevant conventions, references, and known failure modes.
202
219
220
+
Intents carry a `confidence` (0.0-1.0). Intents below 0.3 are skipped entirely; otherwise the composite score of every atom found via a predicted intent is multiplied by that intent's confidence, so a speculative intent cannot outrank literal-query matches.
221
+
203
222
Example:
204
223
205
224
- User: "Refactor the auth package to remove JWT."
@@ -355,6 +374,8 @@ Extended Memory is configured under the `memory.extended` section.
355
374
"user_state_max_pending": 20,
356
375
"associations_enabled": true,
357
376
"association_semantic_top_k": 3,
377
+
"semantic_dedup_threshold": 0.92,
378
+
"consolidate_similarity_threshold": 0.9,
358
379
"proactive_return_after_break": true,
359
380
"style_mirroring_enabled": true,
360
381
"anaphora_resolution_enabled": true,
@@ -401,6 +422,8 @@ Extended Memory is configured under the `memory.extended` section.
401
422
|`user_state_max_pending`|`20`| Maximum pending-review entries kept in the user model. |
402
423
|`associations_enabled`|`true`| Enable bidirectional atom associations (temporal, task, semantic). |
403
424
|`association_semantic_top_k`|`3`| Number of semantic neighbours linked when building associations. |
425
+
|`semantic_dedup_threshold`|`0.92`| Cosine similarity at or above which an incoming atom is treated as a paraphrase of an existing live atom and refreshes it instead of appending. `0` disables the semantic tier (exact-match dedup always runs). |
426
+
|`consolidate_similarity_threshold`|`0.9`| Cosine similarity at or above which live atoms are grouped as near-duplicates for `ConsolidateAtoms` merging. |
404
427
|`proactive_return_after_break`|`true`| On session resume, inject a "where you left off" summary. |
405
428
|`style_mirroring_enabled`|`true`| Inject a style-guidance directive based on the inferred user model. |
406
429
|`anaphora_resolution_enabled`|`true`| Resolve the first pronoun in a user message against recent trusted atoms when the top atom's score is high enough. |
0 commit comments