Importance-weighted governance decision storage for AI agent compliance

[nagasatish007]

Use Case

Building multi-agent systems with TealTiger governance — every tool call, LLM request, and agent action produces a deterministic governance decision (allow/deny/monitor) with a risk score. These decisions need persistent memory that:

1. Retains high-risk security denials (secret detected, PII blocked) for months
2. Naturally forgets routine low-risk approvals after days/weeks
3. Allows contextual recall: "what governance decisions were made for this agent in similar situations?"

Goal: governance memory that behaves like human memory — critical security events are memorable, routine approvals fade.

Problem Statement

Governance decisions currently live in flat storage (in-memory or database rows). Every decision gets the same retention regardless of importance. This creates two problems:

1. Storage bloat — millions of routine ALLOW decisions accumulate with no decay
2. No contextual recall — when making a new governance decision, there's no way to ask "has this agent been denied for similar actions before?" without scanning the entire history
3. No importance hierarchy — a critical DENY (risk_score=95, secret detected) has the same storage lifetime as a trivial ALLOW (risk_score=0, routine tool call)

We need memory that understands importance and context, not just timestamps.

How This Feature Would Help

Hindsight's importance-based retention and contextual recall would solve all three problems:

• Natural decay: Routine ALLOWs (importance=0.60) fade within days. Critical DENYs (importance=0.95) persist for compliance audits.
• Contextual governance: Before evaluating a new tool call, recall similar past decisions. If an agent was denied 5 times for PII in web_search results, that context informs the next evaluation.
• Anomaly detection: Query "show me agents whose denial rate spiked vs. their historical baseline" — impossible with flat storage, natural with Hindsight's memory model.

This makes governance adaptive without making it non-deterministic — the policy engine is still rule-based, but it has memory of past outcomes.

Proposed Solution

A HindsightGovernanceMemory adapter that maps TealTiger decisions to Hindsight memories:

from hindsight import HindsightClient
from tealtiger.integrations.hindsight import HindsightGovernanceMemory

hindsight = HindsightClient(api_key="your-key")

memory = HindsightGovernanceMemory(
    client=hindsight,
    importance_map={
        "deny": 0.95,    # Critical — retain for compliance
        "monitor": 0.70, # Notable — retain for pattern detection
        "allow": 0.60,   # Routine — natural decay
    },
)

# Store every governance decision with importance-weighted retention
memory.store(decision)

# Contextual recall before new decisions
past_decisions = memory.recall(
    agent_id="research-agent",
    context="tool_call:web_search",
    limit=5,
)


### Alternatives Considered

1. **Flat database (PostgreSQL)** — stores everything equally, no importance weighting, no contextual recall without custom vector search
2. **Dakera integration** — we built this (decay-weighted retention), but it lacks Hindsight's contextual recall and semantic similarity
3. **Redis with TTL** — simple expiry, but no importance hierarchy (everything expires at the same rate regardless of severity)

### Priority

Important - affects my workflow

### Additional Context

We built a similar pattern with Dakera (another agent memory server): https://github.com/agentguard-ai/tealtiger/blob/main/docs/integrations/dakera.md

Key design principle from that integration: **Storage = evidence/continuity, NOT authority.** A stored ALLOW from yesterday cannot authorize today's action. Every new request gets a fresh deterministic evaluation. Storage informs; it doesn't permit.

Happy to contribute a cookbook example or PR.

- TealTiger: https://github.com/agentguard-ai/tealtiger
- PyPI: `pip install tealtiger`
- Apache 2.0 license


### Checklist

- [x] I would be willing to contribute this feature

[Sanderhoff-alt]

Interesting use case. One question about the "naturally forgets" part: do you imagine this as soft forgetting through importance-weighted decay during recall, or as hard deletion/archival through retention policies?

For governance decisions, especially DENY events involving PII/secrets, the distinction seems important: "not recalled by default" is very different from "physically deleted".

Would the desired Hindsight adapter need both mechanisms, or is the first version mainly about importance-weighted contextual recall?

[Sanderhoff-alt]

Another related question: how should we evaluate whether importance-weighted decay actually improves governance memory, beyond being an intuitive human-memory analogy?

A concrete benchmark could look like this:

Create a synthetic governance event stream for several agents over 30-90 days, including:

• thousands of routine ALLOW events
• recurring DENY events for similar PII, secret, or prompt-injection cases
• a few one-off critical DENY events
• agents whose DENY rate spikes after a certain point

Then compare different retrieval and retention strategies on fixed tasks:

1. Before a new tool call, retrieve the 5 most useful prior governance decisions for the same agent/context.
2. Detect agents with a meaningful denial-rate spike.
3. Suppress or remove routine ALLOW noise while keeping retrieval/storage cost low.
4. Preserve audit-relevant DENY events.

Strong baselines would be useful here:

• semantic top-k only
• semantic top-k + metadata filters
• fixed TTL by decision type
• recency-weighted retrieval
• fixed retention windows for ALLOW/MONITOR/DENY

The key question is whether importance-weighted decay beats these baselines on measurable metrics, for example:

• precision@5 for recalled governance decisions
• recall of relevant prior DENY/MONITOR events
• ALLOW noise rate in retrieved context
• anomaly detection precision/recall for denial spikes
• storage or retrieval cost reduction without losing audit-relevant events

That would make the decay mechanism easier to justify as an engineering feature, rather than mainly as an analogy to human memory curves.

[nagasatish007]

Great questions @Sanderhoff-alt — both are things we've worked through in the Dakera integration.

On soft vs. hard forgetting:

For v1, we only need soft forgetting through importance-weighted decay during recall. Here's why:

• Governance DENY events (PII, secrets) are compliance artifacts — physically deleting them would violate audit requirements (SOC 2, EU AI Act Article 14). They need to exist but don't need to surface in every recall.
• The adapter would use a two-tier approach:
  • Recall tier: importance-weighted decay — routine ALLOWs fade from context windows, critical DENYs stay easily recallable
  • Retention tier: hard retention policy (configurable) — DENYs retained for 90/180/365 days minimum regardless of recall weight, ALLOWs archived or purged after 30 days

So "naturally forgets" means "doesn't pollute retrieval context" — not "ceases to exist." The distinction between "not recalled by default" and "physically deleted" maps to recall-tier vs. retention-tier.

On benchmarking:

The benchmark you propose is exactly what we'd want to build. Concretely:

Metric	What it measures
precision@5 for contextual recall	Does retrieved context help the next governance decision?
ALLOW noise rate	What % of retrieved context is low-value routine approvals?
Denial spike detection F1	Can we catch agents whose behavior changed?
Audit-relevant DENY preservation	Do we ever lose a compliance-critical event?

The baselines you listed are the right ones. Our hypothesis is that importance-weighted decay will beat:

• Semantic top-k alone (which has no recency or severity signal)
• Fixed TTL (which can't differentiate a critical DENY from a routine ALLOW with the same age)
• Recency-only weighting (which surfaces recent ALLOWs over older critical DENYs)

We'd propose building this as a cookbook example with the synthetic event stream you described. Happy to contribute the governance event generator (we already have one for PBT testing in TealTiger) and the evaluation harness. The Hindsight team would own the retrieval strategy implementations.

Practical next step: I can put together a synthetic governance event dataset (30 days, 5 agents, ~10k events with the distribution you described) and a simple eval script. Would that be useful as a starting point?

[Sanderhoff-alt]

One more question on the risk_score -> memory importance mapping:

Do you expect risk_score to directly drive decay/retention, or should it just be one signal used by Hindsight when deciding memory importance?

I ask because high-risk events are not always useful future context if they are false positives or one-off incidents, while low-risk ALLOW events may still be useful in aggregate as a baseline for anomaly detection. Also, risk scores can change meaning as policy/scorer versions evolve.

So I am curious whether v1 should be a simple direct mapping from governance risk to memory importance, or whether the adapter should leave room for a richer importance model later.

[Sanderhoff-alt]

@nicoloboschi Stepping back from TealTiger specifically: is Hindsight considering memory lifecycle governance as a core concept?

For example, should Hindsight eventually provide primitives for layered memory governance, such as importance-based recall decay, retention/archive policies, and configurable lifecycle policies for different classes of memories?

Or would the preferred direction be to expose general governance extension points, so users can plug in their own scorer/policy engine and define decay, retention, and archival behavior themselves?

In other words, independent of any particular governance engine, is memory governance/forgetting something you see as part of Hindsight’s own roadmap, or mainly as an integration-level concern?

[nagasatish007]

@Sanderhoff-alt — good question. The answer is: leave room for a richer model, but ship v1 with a sensible default mapping that users can override.

Concretely, the adapter should take an importance_fn parameter:

memory = HindsightGovernanceMemory(
    client=hindsight,
    # Default: simple decision-type mapping
    importance_fn=default_importance,
)

def default_importance(decision) -> float:
    """Default v1 mapping: decision action drives base importance."""
    base = {"deny": 0.90, "require_approval": 0.85, "monitor": 0.70, "allow": 0.55}
    return base.get(decision.action, 0.60)

Users who want richer logic can override it:

def custom_importance(decision) -> float:
    """Use risk_score as one signal among several."""
    base = 0.50
    # Risk score contributes, but doesn't dominate
    risk_contribution = (decision.risk_score / 100) * 0.3
    # Repeated denials for same agent+tool are more important
    recurrence_boost = 0.15 if decision.is_recurring else 0.0
    # False-positive-flagged events are deprioritized
    fp_penalty = -0.20 if decision.metadata.get("false_positive") else 0.0
    return min(base + risk_contribution + recurrence_boost + fp_penalty, 1.0)

memory = HindsightGovernanceMemory(client=hindsight, importance_fn=custom_importance)

Why not map risk_score directly to importance:

You're right on both points:

1. High-risk false positives shouldn't persist forever — a risk_score: 95 that turned out to be a regex false positive shouldn't crowd out real denials in future recall
2. Low-risk ALLOWs in aggregate ARE useful for baseline construction — they should decay slowly from recall context but persist long enough to compute statistical baselines

The v1 design principle: importance_fn is a pluggable function. The default is simple (decision type only, risk_score ignored). The interface leaves room for richer models (recurrence, false-positive feedback, policy version drift) without baking assumptions into the adapter.

This also addresses the policy/scorer version evolution point: if risk_score semantics change between policy versions, the importance_fn can incorporate policy_digest to normalize scores across versions.

For the benchmark dataset you proposed earlier — I'd suggest testing both the simple default mapping and 2-3 richer variants to see which actually produces better precision@5 for contextual recall. That would tell us whether the complexity is worth it for v1, or whether the simple mapping is good enough until real usage data says otherwise.