feat: implemented memory_isolation assertion

[Tech-Psycho95]

Summary

Implemented the memory_isolation assertion feature from #27.

Now, the harness fails if there is data from another user, session, tenant or memory scope found in a trace, by scanning the full serialised trace for forbidden_markers as defined in the scenario.

Changes

src/agent_harness/assertions.py

• Added evaluate_memory_isolation(scenario, trace) — serialises the entire trace to JSON and scans for each forbidden marker as a substring; reports all leaked markers in evidence
• Registered memory_isolation in the evaluate_assertions() dispatcher

src/agent_harness/scenario.py

• Added validation for memory_isolation assertions: requires expected.memory_isolation.forbidden_markers to be a non-empty list of non-empty strings

scenarios/memory_isolation/cross_session_leak_001.yaml

• Example scenario matching the shape suggested in the issue

examples/traces/

• memory_isolation_leak.json — failing trace (foreign markers present in messages and events)
• memory_isolation_clean.json — passing trace (no foreign markers)

tests/test_assertions.py

• 10 new unit tests: pass, fail in messages/tool_calls/events, multi-marker reporting, not_run on missing/empty config, scope in pass evidence, graceful skip of non-string markers, dispatcher routing

Closes #27

Test results

New tests (memory_isolation):

tests/test_assertions.py::test_memory_isolation_passes_with_clean_trace PASSED
tests/test_assertions.py::test_memory_isolation_fails_with_marker_in_messages PASSED
tests/test_assertions.py::test_memory_isolation_fails_with_marker_in_tool_calls PASSED
tests/test_assertions.py::test_memory_isolation_fails_with_marker_in_events PASSED
tests/test_assertions.py::test_memory_isolation_reports_multiple_leaked_markers PASSED
tests/test_assertions.py::test_memory_isolation_not_run_when_config_missing PASSED
tests/test_assertions.py::test_memory_isolation_not_run_when_list_empty PASSED
tests/test_assertions.py::test_memory_isolation_includes_scope_in_pass_evidence PASSED
tests/test_assertions.py::test_memory_isolation_skips_non_string_markers PASSED
tests/test_assertions.py::test_dispatcher_routes_memory_isolation PASSED

10 passed in 0.09s

Full suite: 136 passed in 3.48s — no regressions.

CLI:
```bash
# Should fail — trace contains foreign markers
agent-harness run scenarios/memory_isolation/cross_session_leak_001.yaml \
  --trace-file examples/traces/memory_isolation_leak.json

# Should pass — trace is clean
agent-harness run scenarios/memory_isolation/cross_session_leak_001.yaml \
  --trace-file examples/traces/memory_isolation_clean.json

[mertsatilmaz]

Thanks for the implementation. This is the right overall shape for #27: scenario validation, dispatcher wiring, evaluator logic, fixtures, and tests are all included.

Before merge, I want three cleanup items:

1. Please document the new memory_isolation assertion in the assertion/scenario docs.

Include the expected YAML shape:

expected:
memory_isolation:
forbidden_markers:
- "alice@example.com"
- "Project Falcon API key"
scope:
user_id: "bob"
session_id: "session_b"
tenant_id: "tenant_2"

and explain that the assertion fails if any forbidden marker appears in the returned trace.

2. Please change the trace serialization in evaluate_memory_isolation to preserve Unicode:

trace_text = json.dumps(trace.to_dict(), ensure_ascii=False)

Without this, non-ASCII forbidden markers may be escaped in the JSON string and missed by substring matching.

3. Please document that this MVP scans the entire serialized trace, including messages, tool calls, events, and nested data. That behavior is acceptable for now, but it needs to be explicit because any occurrence of a forbidden marker anywhere in the trace will fail the assertion.

After those changes and CI passing, this looks mergeable.

[Tech-Psycho95]

Thanks for the implementation. This is the right overall shape for #27: scenario validation, dispatcher wiring, evaluator logic, fixtures, and tests are all included.

Before merge, I want three cleanup items:

1. Please document the new memory_isolation assertion in the assertion/scenario docs.

Include the expected YAML shape:

expected: memory_isolation: forbidden_markers: - "alice@example.com" - "Project Falcon API key" scope: user_id: "bob" session_id: "session_b" tenant_id: "tenant_2"

and explain that the assertion fails if any forbidden marker appears in the returned trace.

2. Please change the trace serialization in evaluate_memory_isolation to preserve Unicode:

trace_text = json.dumps(trace.to_dict(), ensure_ascii=False)

Without this, non-ASCII forbidden markers may be escaped in the JSON string and missed by substring matching.

3. Please document that this MVP scans the entire serialized trace, including messages, tool calls, events, and nested data. That behavior is acceptable for now, but it needs to be explicit because any occurrence of a forbidden marker anywhere in the trace will fail the assertion.

After those changes and CI passing, this looks mergeable.

Hi @mertsatilmaz, I've addressed all three review points:

• Added docs/assertions/memory-isolation.md with the YAML shape, detection mechanism, and a note that scope is audit-only metadata
• Fixed json.dumps to use ensure_ascii=False so non-ASCII forbidden markers are matched correctly
• Added a comment in evaluate_memory_isolation making the full-trace scan behaviour explicit

The PR now has 2 clean commits and all 136 tests pass. Could you take another look when you get a chance? Happy to make any further changes.

[mertsatilmaz]

LGTM, welcome to the team.