Skip to content

Commit c20c9c6

Browse files
johnteeejohnteee
committed
Add run export/undo, audit chain, config loader, plugins, and integration tests
1 parent c4f09f3 commit c20c9c6

25 files changed

Lines changed: 2382 additions & 53 deletions

‎docs/acceptance.md‎

Lines changed: 193 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -324,21 +324,203 @@ shareable.*
324324

325325
---
326326

327+
---
328+
329+
#### Interactive Diff Preview + y/n Approval (new)
330+
331+
**File:** `test_approval_ui.py` (integration)
332+
333+
**Story (AC-NEW-4 P0):** *As a developer, I want to preview a destructive edit
334+
as a unified diff and approve or reject it inline.*
335+
336+
| Assertion | Details |
337+
|---|---|
338+
| `y` returns `True` | Case-insensitive |
339+
| `n` returns `False` | Case-insensitive |
340+
| Unified diff shown for write tool | `-old` / `+new` lines present |
341+
| New-file creation diff shown | All lines prefixed `+` |
342+
| Non-write tool shows arg summary | Tool name always visible |
343+
| Invalid input re-prompts | Up to `max_prompts` then auto-deny |
344+
| `e` (explain) shows details then re-prompts | Full metadata printed |
345+
| Path traversal in args handled | No crash, returns False |
346+
347+
---
348+
349+
#### A2A Traceparent Propagation (new)
350+
351+
**File:** `test_a2a_traceparent.py` (integration)
352+
353+
**Story (AC-NEW-11 P1):** *As an SRE, I want every A2A delegation to carry
354+
traceparent so I can see the full multi-agent call tree in Tempo/Jaeger.*
355+
356+
| Assertion | Details |
357+
|---|---|
358+
| `generate_traceparent()` produces valid W3C format | version-traceId-parentId-flags |
359+
| `parse_traceparent()` extracts all four fields | Validated lengths and hex |
360+
| Malformed header raises `TraceparentError` | Bad format detected |
361+
| `A2AClient.delegate(traceparent=tp)` injects header | HTTP request carries it |
362+
| Without `traceparent`, header absent | No leakage |
363+
| Result carries outbound traceparent | For span continuation |
364+
| Every generated traceparent is unique | 20 consecutive calls all different |
365+
366+
---
367+
368+
#### Configurable PII Redaction (new)
369+
370+
**File:** `test_redaction_config.py` (integration)
371+
372+
**Story (AC-NEW-15 P2):** *As a regulated user, I want to toggle which PII
373+
patterns are redacted in audit logs per deployment.*
374+
375+
| Assertion | Details |
376+
|---|---|
377+
| Default redacts Bearer tokens | Audit log value absent |
378+
| `bearer_tokens=False` preserves token | Value survives in log |
379+
| `api_keys=False` preserves `sk-` token | |
380+
| `aws_keys=False` preserves AKIA key | |
381+
| `extra_patterns` adds custom regex | Custom replacement visible |
382+
| All patterns disabled preserves all tokens | |
383+
384+
---
385+
386+
#### Migration Dry-Run (new)
387+
388+
**File:** `test_migration_dry_run.py` (integration)
389+
390+
**Story (AC-NEW-17 P2):** *As a user upgrading TeaAgent, I want a dry-run
391+
preview that shows which migrations would run without applying them.*
392+
393+
| Assertion | Details |
394+
|---|---|
395+
| `apply_pending(dry_run=True)` does not create table | No SQL executed |
396+
| `result.dry_run_pending` lists pending versions | Correct version set |
397+
| Partial-applied state reflected in dry-run | Only unapplied shown |
398+
| Dry-run does not call `mark_applied` | `applied_versions()` still empty |
399+
| Normal run after dry-run applies all | Tables created |
400+
| Already-applied → `dry_run_pending=[]` | |
401+
402+
---
403+
404+
#### Disk-Full Graceful Degradation (new)
405+
406+
**File:** `test_disk_full_degradation.py` (integration)
407+
408+
**Story (AC-NEW-18 P2):** *As a user whose disk fills mid-run, I want
409+
graceful degradation: audit buffered to memory, error surfaced, no corruption.*
410+
411+
| Assertion | Details |
412+
|---|---|
413+
| `OSError(ENOSPC)` on write does not raise | Run continues |
414+
| In-memory events captured after disk full | Events list populated |
415+
| `_disk_write_error` synthetic event recorded | Error details in payload |
416+
| `audit.disk_error` property returns the `OSError` | |
417+
| `disk_error` is `None` on success | |
418+
| Further writes skipped (no retry) | `append_jsonl_line` called once |
419+
| Non-ENOSPC `OSError` also handled | `EACCES` treated the same |
420+
421+
---
422+
423+
#### Ultrawork Completion Notification (new)
424+
425+
**File:** `test_ultrawork_notify.py` (integration)
426+
427+
**Story (AC-NEW-8 P2):** *As a user, I want desktop/webhook notification when
428+
a long-running ultrawork task finishes.*
429+
430+
| Assertion | Details |
431+
|---|---|
432+
| Webhook POST delivers `worker_id` and `event` | Verified against live server |
433+
| Shell command executed on stop | `subprocess.run` called with command |
434+
| Webhook failure silenced | Unreachable URL does not raise |
435+
| Both webhook and shell fire together | |
436+
| `NotifyConfig` defaults are `None` | No accidental notifications |
437+
| `UltraworkStore(notify_config=cfg).stop()` fires notification | End-to-end |
438+
439+
---
440+
441+
#### Eval HTML Report (new)
442+
443+
**File:** `test_eval_report.py` (integration)
444+
445+
**Story (AC-NEW-21 P2):** *As a model evaluator, I want `render_html_report`
446+
to produce a self-contained HTML file comparing pass/fail, scores, and
447+
reasoning.*
448+
449+
| Assertion | Details |
450+
|---|---|
451+
| Returns a string | Non-empty HTML |
452+
| Case names present | All three test cases in output |
453+
| Pass/fail indicators shown | PASS ✓ / FAIL ✗ labels |
454+
| Pass rate percentage present | `n%` format |
455+
| Valid HTML structure | `<html>` and `</html>` tags |
456+
| Judge scores and reasoning rendered | `0.9` and reasoning text |
457+
| Empty report renders gracefully | No crash, valid HTML skeleton |
458+
459+
---
460+
461+
#### Benchmark Latency / Cost Tracking (new)
462+
463+
**File:** `test_benchmark.py` (integration)
464+
465+
**Story (AC-NEW-22 P2):** *As a CI maintainer, I want `run_benchmark` to
466+
track p50/p95 latency across eval suites and detect regressions.*
467+
468+
| Assertion | Details |
469+
|---|---|
470+
| `run_benchmark()` returns `BenchmarkResult` | Typed result |
471+
| `case_metrics` length equals suite case count | One entry per case |
472+
| Per-case `latency_ms >= 0` | Measured with `perf_counter` |
473+
| `p50_ms` and `p95_ms` computed | `p95 >= p50` |
474+
| `mean_ms > 0` for real runner | Non-zero average |
475+
| Output captured per case | `CaseMetrics.output` set |
476+
| `regression_detected(baseline)` returns bool | Compares against thresholds |
477+
| No regression when baseline is very high | |
478+
| Regression detected when runner is slow | 50 ms > 2 ms threshold |
479+
| `to_dict()` serialisable | Contains `suite_name`, `p50_ms`, `p95_ms` |
480+
481+
---
482+
483+
## Integration Tests (tests/integration/)
484+
485+
| File | Coverage |
486+
|---|---|
487+
| `test_runner_cost_tracking.py` | `RunResult` cost fields + audit event fields (IT-1) |
488+
| `test_cancel_token.py` | Cancel token — pre-cancel, mid-run, thread-safe (IT-2) |
489+
| `test_tool_rate_limit.py` | Sliding-window quota, concurrency safety, expiry (IT-3) |
490+
| `test_mcp_tool_adapter.py` | `register_mcp_tools` discovery, annotations, filter (IT-4) |
491+
| `test_skill_loader.py` | `load_skills` discovery, dedup, cap, prompt injection (IT-5) |
492+
| `test_audit_sink_isolation.py` | Crashing sink does not propagate or block other sinks (IT-6) |
493+
| `test_file_policy.py` | `DenyRule` matching, `FilePolicy.assert_allowed`, runner wiring (IT-7) |
494+
| `test_webhook_sink.py` | HTTP delivery, HMAC, filter, failure suppression (IT-8) |
495+
| `test_error_hints.py` | All error classes have default hints, `str()` rendering (IT-9) |
496+
| `test_subagent_budget_inheritance.py` | Subagent depth limit, error dict, registry guard (IT-10) |
497+
| `test_run_resume_checkpoint.py` | Checkpoint save on tool completion, pending-approval, SQLite round-trip, obs replay (IT-11) |
498+
| `test_destructive_approval_lifecycle.py` | Pause → approve → resume; deny path; auto-approve handler; read-only block (IT-12) |
499+
| `test_streaming_tool_calls.py` | `on_chunk` callbacks, audit events, token accumulation (IT-13) |
500+
| `test_schema_migration_live.py` | Migration ordering, idempotency, data survival, version tracking (IT-14) |
501+
| `test_dpop_replay_concurrency.py` | Concurrent JTI consume: exactly one success; expiry reset (IT-15) |
502+
| `test_audit_chain.py` | Hash-chain validity, tampering/insertion/deletion detection |
503+
| `test_plugins.py` | Plugin discovery, isolation, custom group |
504+
| `test_a2a_circuit_breaker.py` | Circuit open/close, skip, reset, backward compat |
505+
| `test_config_loader.py` | Layer precedence, env override, workspace profile applied |
506+
| `test_run_undo.py` | Pre-write capture, file deletion/restore, path traversal guard |
507+
| `test_run_export.py` | Export/import round-trip, hash-chain survival, error cases |
508+
| `test_approval_ui.py` | Diff preview, y/n/e flow, path traversal, max prompts |
509+
| `test_a2a_traceparent.py` | W3C format gen/parse, header injection, result field |
510+
| `test_redaction_config.py` | Toggle groups, custom pattern, default behaviour |
511+
| `test_migration_dry_run.py` | Dry-run preview, no SQL, pending version list |
512+
| `test_disk_full_degradation.py` | ENOSPC graceful degradation, in-memory fallback |
513+
| `test_ultrawork_notify.py` | Webhook delivery, shell command, failure silence |
514+
| `test_eval_report.py` | HTML rendering, case names, pass/fail, judge scores |
515+
| `test_benchmark.py` | p50/p95/mean latency, regression detection, serialisation |
516+
517+
---
518+
327519
## Next User Stories
328520

329-
The following stories are specified and prioritised but do not yet have
330-
acceptance tests. Each maps to a Review gap analysis recommendation.
521+
The following stories remain unimplemented.
331522

332523
| ID | Story | Priority |
333524
|---|---|---|
334-
| AC-NEW-1 | `teaagent init` first-time wizard | P2 |
335-
| AC-NEW-4 | Interactive diff preview + inline y/n approval | P0 |
336-
| AC-NEW-8 | Desktop/webhook notification on ultrawork completion | P2 |
337-
| AC-NEW-11 | A2A delegation carries `traceparent` header | P1 |
338-
| AC-NEW-15 | Configurable PII redaction categories | P2 |
339525
| AC-NEW-16 | HTTPS_PROXY / CA bundle / mTLS "just works" | P2 |
340-
| AC-NEW-17 | `teaagent upgrade` schema migration dry-run preview | P2 |
341-
| AC-NEW-18 | Graceful degradation when disk fills mid-run | P2 |
342526
| AC-NEW-20 | Local model (Ollama/vLLM) with full governance | P2 |
343-
| AC-NEW-21 | `teaagent eval run <suite>` with HTML report | P2 |
344-
| AC-NEW-22 | `teaagent benchmark` latency/cost regression tracking | P2 |

‎teaagent/__init__.py‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,14 +7,23 @@
77
except PackageNotFoundError:
88
__version__ = '0+local'
99

10+
from teaagent.a2a_trace import TraceparentError, generate_traceparent, parse_traceparent
1011
from teaagent.agentcard import CircuitBreakerConfig
1112
from teaagent.aibom import AIBOMManifest, build_aibom
13+
from teaagent.approval_ui import DiffApprovalHandler
1214
from teaagent.audit import AuditLogger
1315
from teaagent.audit_chain import (
1416
ChainVerificationResult,
1517
compute_event_hash,
1618
verify_audit_chain,
1719
)
20+
from teaagent.benchmark import (
21+
BenchmarkBaseline,
22+
BenchmarkResult,
23+
BenchmarkSuite,
24+
CaseMetrics,
25+
run_benchmark,
26+
)
1827
from teaagent.budget import RunBudget
1928
from teaagent.chat_agent import (
2029
ChatAgentConfig,
@@ -40,6 +49,7 @@
4049
from teaagent.context import ContextCompactor
4150
from teaagent.errors import RunCancelledError
4251
from teaagent.eval import EvalCase, EvalReport, run_eval
52+
from teaagent.eval_report import render_html_report
4353
from teaagent.file_policy import DenyRule, FilePolicy, load_file_policy
4454
from teaagent.graph_rag import GraphEdge, KnowledgeGraph, graph_retrieve
4555
from teaagent.graphqlite_production import (
@@ -86,6 +96,7 @@
8696
from teaagent.mcp_tool_adapter import register_mcp_tools
8797
from teaagent.memory import MemoryCatalog, MemoryEntry
8898
from teaagent.model_routing import ModelRoute, classify_task, route_model
99+
from teaagent.notify import NotifyConfig, fire_notification
89100
from teaagent.oauth21 import (
90101
HAS_CRYPTOGRAPHY,
91102
DPoPValidationResult,
@@ -120,6 +131,7 @@
120131
from teaagent.prompt import PromptBundle, assemble_agent_prompt, parse_model_decision
121132
from teaagent.rag import Document, InMemoryRetriever, agentic_retrieve
122133
from teaagent.readiness import ReadinessReport, assess_managed_agent_readiness
134+
from teaagent.redaction import RedactionConfig
123135
from teaagent.run_export import ExportManifest, export_run, import_run
124136
from teaagent.run_store import RunStore, RunSummary
125137
from teaagent.run_undo import UndoJournal, UndoResult
@@ -166,7 +178,17 @@
166178
'ApprovalPolicy',
167179
'ApprovalRequest',
168180
'AuditLogger',
181+
'BenchmarkBaseline',
182+
'BenchmarkResult',
183+
'BenchmarkSuite',
184+
'CaseMetrics',
185+
'run_benchmark',
186+
'render_html_report',
169187
'CircuitBreakerConfig',
188+
'DiffApprovalHandler',
189+
'TraceparentError',
190+
'generate_traceparent',
191+
'parse_traceparent',
170192
'CONFIG_KEYS',
171193
'ConfigLayer',
172194
'ConfigResolver',
@@ -237,7 +259,10 @@
237259
'OAuthStore',
238260
'PermissionMode',
239261
'ExportManifest',
262+
'NotifyConfig',
240263
'PluginLoadResult',
264+
'fire_notification',
265+
'RedactionConfig',
241266
'UndoJournal',
242267
'UndoResult',
243268
'export_run',

0 commit comments

Comments
 (0)