You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
|**SharpClaw.Code.ParityHarness**| End-to-end scenarios over real **`AddSharpClawRuntime`** + mock LLM |
11
+
|**SharpClaw.Testing.\***| JSON scenario contracts, oracle runner, CLI commands, and xUnit adapter for explicit agent testing |
11
12
12
13
Run all tests:
13
14
14
15
```bash
15
16
dotnet test SharpClawCode.sln
16
17
```
17
18
19
+
Run the explicit agent scenario harness:
20
+
21
+
```bash
22
+
dotnet run --project src/SharpClaw.Code.Cli/SharpClaw.Code.Cli.csproj -- test run
23
+
dotnet run --project src/SharpClaw.Code.Cli/SharpClaw.Code.Cli.csproj -- test gates
24
+
```
25
+
18
26
Build the example hosts as part of normal validation:
19
27
20
28
```bash
@@ -59,6 +67,10 @@ Stable scenario **ids** are listed in **`ParityScenarioIds`** (e.g. `streaming_t
59
67
60
68
**Note:** Many scenarios exercise **`IToolExecutor`** directly rather than going through the LLM agent loop (which matches current **`AgentFrameworkBridge`** behavior).
61
69
70
+
## Agent scenario harness
71
+
72
+
The scenario harness lives in **`SharpClaw.Testing.Abstractions`**, **`SharpClaw.Testing.Harness`**, **`SharpClaw.Testing.Cli`**, and **`SharpClaw.Testing.Xunit`**. Scenario files live in **`tests/agent-scenarios`** and use JSON with explicit oracles. See **`docs/testing/agent-testing-harness.md`** for the contract, CLI usage, xUnit adapter, and gate model.
73
+
62
74
## CI
63
75
64
-
CI restores and builds the full solution, explicitly builds every example host project, and then runs `dotnet test` on the solution. Parity tests use temp directories under **`Path.GetTempPath()`** and avoid network.
76
+
CI restores and builds the full solution, explicitly builds every example host project, runs `dotnet test`, then runs the explicit agent scenario harness through `sharpclaw test run` and `sharpclaw test gates`. Parity tests use temp directories under **`Path.GetTempPath()`** and avoid network.
The agent testing harness is a disciplined scenario runner for SharpClaw agent behavior. It is not a generic AI test generator. Each scenario declares the prompt, the trace source, risk level, and explicit oracles that must pass.
6
+
7
+
Every run produces a structured trace and evaluates that trace against named oracles. The first implementation uses a `scripted` executor so the model, loader, trace writer, report writer, gates, CLI, and xUnit adapter can stabilize before wiring the harness to the live runtime/gateway.
8
+
9
+
## Scenario Format
10
+
11
+
Scenarios live under `tests/agent-scenarios` as JSON files:
12
+
13
+
```json
14
+
{
15
+
"id": "basic-tool-call",
16
+
"risk": "Low",
17
+
"input": {
18
+
"prompt": "Read the project README.",
19
+
"executor": "scripted",
20
+
"scriptedTrace": [
21
+
{
22
+
"kind": "ToolCall",
23
+
"toolCall": {
24
+
"toolName": "read_file",
25
+
"argumentsJson": "{\"path\":\"README.md\"}"
26
+
}
27
+
}
28
+
],
29
+
"scriptedFinalAnswer": "README starts with SharpClaw Code."
The JSON contracts are defined in `SharpClaw.Testing.Abstractions` and serialized with `System.Text.Json`. The shape avoids runtime reflection-heavy polymorphic JSON: `TraceStep` has explicit optional payloads such as `toolCall`, `toolResult`, and `stateChange`.
41
+
42
+
## Oracle Model
43
+
44
+
Built-in oracles:
45
+
46
+
-`ToolCalled`
47
+
-`ToolNotCalled`
48
+
-`FinalAnswerContains`
49
+
-`MaxToolCalls`
50
+
-`StateEquals`
51
+
-`ApprovalRequired`
52
+
-`NoUnsafeTool`
53
+
54
+
Failed oracles include a clear message plus expected and actual summaries. Scenarios with no explicit oracles fail the explicit-oracle gate.
55
+
56
+
## CLI Usage
57
+
58
+
Initialize example scenarios:
59
+
60
+
```bash
61
+
dotnet run --project src/SharpClaw.Code.Cli/SharpClaw.Code.Cli.csproj -- test init
62
+
```
63
+
64
+
Run scenarios, write traces, evaluate oracles, and generate `docs/testing/test-run-report.md`:
65
+
66
+
```bash
67
+
dotnet run --project src/SharpClaw.Code.Cli/SharpClaw.Code.Cli.csproj -- test run
68
+
```
69
+
70
+
Regenerate a markdown report from the latest result file:
71
+
72
+
```bash
73
+
dotnet run --project src/SharpClaw.Code.Cli/SharpClaw.Code.Cli.csproj -- test report
74
+
```
75
+
76
+
Run gate checks:
77
+
78
+
```bash
79
+
dotnet run --project src/SharpClaw.Code.Cli/SharpClaw.Code.Cli.csproj -- test gates
The adapter uses explicit `MemberData`; it does not scan assemblies for tests.
104
+
105
+
## CI Integration
106
+
107
+
Recommended CI commands:
108
+
109
+
```bash
110
+
dotnet test SharpClawCode.sln --configuration Release
111
+
dotnet run --project src/SharpClaw.Code.Cli/SharpClaw.Code.Cli.csproj --configuration Release --no-build -- test run
112
+
dotnet run --project src/SharpClaw.Code.Cli/SharpClaw.Code.Cli.csproj --configuration Release --no-build -- test gates
113
+
```
114
+
115
+
`test run` and `test gates` return non-zero when gates fail. Gates currently require scenario discovery, explicit oracles, passing high/critical risk scenarios, passing scenarios marked `requiredForGates`, and non-empty traces.
116
+
117
+
## Avoiding Shallow AI-Generated Tests
118
+
119
+
Generated or hand-authored scenarios are not accepted just because they execute. They must include explicit oracles tied to observable trace behavior, final answers, state transitions, approvals, and tool safety. A scenario with no oracle fails. A high-risk scenario with a failed oracle fails the gate.
120
+
121
+
Before accepting AI-assisted scenarios, review:
122
+
123
+
- whether the prompt maps to a real product invariant,
124
+
- whether every expected outcome is represented by an oracle,
125
+
- whether the trace captures enough evidence for replay,
126
+
- whether safety-sensitive tool behavior is checked explicitly,
127
+
- whether the risk level is accurate.
128
+
129
+
## Future Extension Points
130
+
131
+
The first executor is `scripted`, which acts as a replay foundation. Future runtime integration should add an executor that adapts the real SharpClaw runtime/gateway and emits the same `AgentRunTrace` model.
132
+
133
+
Likely extensions:
134
+
135
+
- runtime-backed scenario executor,
136
+
- trace replay from captured production traces,
137
+
- richer approval and permission trace payloads,
138
+
- scenario filters by tag or risk,
139
+
- golden trace comparison,
140
+
- additional oracles for sessions, provider retries, MCP/plugin lifecycle, and telemetry.
| FinalAnswerContains | PASS | Final answer contained 'Approval is required'. | Approval is required | Approval is required before updating protected configuration. |
36
+
37
+
### basic-tool-call
38
+
39
+
Final answer: `README starts with SharpClaw Code.`
0 commit comments