Skip to content

Commit acaadc9

Browse files
westey-mwestey-m
authored
.NET: Add a verify-samples tool and skill (microsoft#5005)
* Add a verify-samples tool and skill * Address PR comments * Move verify-samples to eng folder and improve definitions
1 parent 3432984 commit acaadc9

16 files changed

Lines changed: 3287 additions & 0 deletions

‎dotnet/.github/skills/verify-samples-tool/SKILL.md‎

Lines changed: 213 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,213 @@
1+
---
2+
name: verify-samples-tool
3+
description: How to use the verify-samples tool to run, verify, and manage sample definitions in the Agent Framework repository. Use this when adding, updating, or running sample verification.
4+
---
5+
6+
# verify-samples Tool
7+
8+
The `verify-samples` project (`dotnet/eng/verify-samples/`) is an automated tool that runs sample projects and verifies their output using deterministic checks and AI-powered verification.
9+
10+
## Running verify-samples
11+
12+
```bash
13+
cd dotnet
14+
15+
# Run all samples across all categories
16+
dotnet run --project eng/verify-samples -- --log results.log --csv results.csv
17+
18+
# Run a specific category
19+
dotnet run --project eng/verify-samples -- --category 02-agents --log results.log
20+
21+
# Run specific samples by name
22+
dotnet run --project eng/verify-samples -- Agent_Step02_StructuredOutput Agent_Step09_AsFunctionTool
23+
24+
# Control parallelism (default 8)
25+
dotnet run --project eng/verify-samples -- --parallel 8 --log results.log
26+
27+
# Combine options
28+
dotnet run --project eng/verify-samples -- --category 03-workflows --parallel 4 --log results.log --csv results.csv
29+
```
30+
31+
### Required Environment Variables
32+
33+
The tool itself needs:
34+
- `AZURE_OPENAI_ENDPOINT` — for the AI verification agent
35+
- `AZURE_OPENAI_DEPLOYMENT_NAME` (optional, defaults to `gpt-5-mini`)
36+
37+
Individual samples require their own env vars (e.g., `AZURE_AI_PROJECT_ENDPOINT`). The tool automatically checks and skips samples with missing env vars.
38+
39+
### Output Files
40+
41+
- `--log results.log` — detailed per-sample log with stdout/stderr, AI reasoning, and a summary
42+
- `--csv results.csv` — tabular summary with Sample, ProjectPath, Status, FailedChecks, and Failures columns
43+
44+
## Sample Categories
45+
46+
Definitions are in the `dotnet/eng/verify-samples/` directory:
47+
48+
| Category | Config File | Registered Key |
49+
|----------|-------------|----------------|
50+
| 01-get-started | `GetStartedSamples.cs` | `01-get-started` |
51+
| 02-agents | `AgentsSamples.cs` | `02-agents` |
52+
| 03-workflows | `WorkflowSamples.cs` | `03-workflows` |
53+
54+
Categories are registered in `VerifyOptions.cs` in the `s_sampleSets` dictionary.
55+
56+
## SampleDefinition Properties
57+
58+
Each sample is defined as a `SampleDefinition` in the appropriate config file. Key properties:
59+
60+
```csharp
61+
new SampleDefinition
62+
{
63+
// Required: Display name for the sample
64+
Name = "Agent_Step02_StructuredOutput",
65+
66+
// Required: Relative path from dotnet/ to the sample project directory
67+
ProjectPath = "samples/02-agents/Agents/Agent_Step02_StructuredOutput",
68+
69+
// Environment variables the sample requires (throws if missing)
70+
RequiredEnvironmentVariables = ["AZURE_OPENAI_ENDPOINT"],
71+
72+
// Environment variables with defaults that would prompt on console if unset
73+
OptionalEnvironmentVariables = ["AZURE_OPENAI_DEPLOYMENT_NAME"],
74+
75+
// Skip this sample with a reason (for structural issues only)
76+
SkipReason = null, // or "Requires external service X."
77+
78+
// Deterministic checks: substrings that must appear in stdout
79+
MustContain = ["=== Section Header ==="],
80+
81+
// Substrings that must NOT appear in stdout
82+
MustNotContain = [],
83+
84+
// If true, only MustContain checks are used (no AI verification)
85+
IsDeterministic = false,
86+
87+
// AI verification: natural-language descriptions of expected output
88+
// Each entry describes one aspect to verify independently
89+
ExpectedOutputDescription =
90+
[
91+
"The output should show structured person information with Name, Age, and Occupation fields.",
92+
"The output should not contain error messages or stack traces.",
93+
],
94+
95+
// Stdin inputs to feed to the sample (for interactive samples)
96+
Inputs = ["Y", "Y", "Y"],
97+
98+
// Delay between stdin inputs in ms (default 2000, increase for LLM calls between inputs)
99+
InputDelayMs = 3000,
100+
}
101+
```
102+
103+
## How to Add a New Sample Definition
104+
105+
1. **Check the sample's Program.cs** to understand:
106+
- What environment variables it reads (look for `GetEnvironmentVariable`)
107+
- Whether it needs stdin input (look for `Console.ReadLine`, `Application.GetInput`)
108+
- Whether it has an external loop (look for `EXIT` patterns in YAML workflows)
109+
- What output it produces (section headers, markers, expected behavior)
110+
- Whether it exits on its own or runs as a server
111+
112+
2. **Choose the right verification strategy:**
113+
- **Deterministic** (`IsDeterministic = true`): Use `MustContain` for samples with fixed output strings. No AI verification.
114+
- **AI-verified** (default): Use `ExpectedOutputDescription` with semantic descriptions. Write expectations that are flexible enough for non-deterministic LLM output.
115+
- **Both**: Use `MustContain` for fixed markers AND `ExpectedOutputDescription` for LLM-generated content.
116+
117+
3. **Set `SkipReason` only for structural issues:**
118+
- Web servers that don't exit
119+
- Multi-process client/server architectures
120+
- Samples requiring external infrastructure (MCP servers you can't reach, Docker, etc.)
121+
- Do NOT skip for missing env vars — the tool checks those dynamically.
122+
123+
4. **For interactive samples, provide `Inputs`:**
124+
- Samples using `Application.GetInput(args)` need one initial input
125+
- Samples with `Console.ReadLine()` approval loops need `"Y"` inputs
126+
- YAML workflows with `externalLoop` need `"EXIT"` as the last input
127+
- Set `InputDelayMs` to 3000-8000ms for samples with LLM calls between inputs
128+
129+
5. **Add the definition** to the appropriate config file (e.g., `AgentsSamples.cs`) in the `All` list.
130+
131+
6. **Register new categories** (if needed) in `VerifyOptions.cs` `s_sampleSets` dictionary.
132+
133+
### Writing Good ExpectedOutputDescription
134+
135+
- Write descriptions that are **semantically flexible** — LLM output varies between runs
136+
- Each array entry should describe **one independent aspect** to verify
137+
- Always include `"The output should not contain error messages or stack traces."` as the last entry
138+
- Avoid exact wording expectations — use "should mention", "should contain information about", "should show"
139+
- Bad: `"The output should say 'The weather in Amsterdam is cloudy with a high of 15°C'"`
140+
- Good: `"The output should contain weather information about Amsterdam mentioning cloudy weather with a high of 15°C."`
141+
142+
### Example: Simple LLM Sample
143+
144+
```csharp
145+
new SampleDefinition
146+
{
147+
Name = "Agent_With_AzureOpenAIChatCompletion",
148+
ProjectPath = "samples/02-agents/AgentProviders/Agent_With_AzureOpenAIChatCompletion",
149+
RequiredEnvironmentVariables = ["AZURE_OPENAI_ENDPOINT"],
150+
OptionalEnvironmentVariables = ["AZURE_OPENAI_DEPLOYMENT_NAME"],
151+
ExpectedOutputDescription =
152+
[
153+
"The output should contain a joke about a pirate.",
154+
"The output should not contain error messages or stack traces.",
155+
],
156+
},
157+
```
158+
159+
### Example: Deterministic Sample
160+
161+
```csharp
162+
new SampleDefinition
163+
{
164+
Name = "Workflow_Declarative_GenerateCode",
165+
ProjectPath = "samples/03-workflows/Declarative/GenerateCode",
166+
IsDeterministic = true,
167+
MustContain = ["WORKFLOW: Parsing", "WORKFLOW: Defined"],
168+
ExpectedOutputDescription = ["The output should show a YAML workflow being parsed and C# code being generated from it."],
169+
},
170+
```
171+
172+
### Example: Interactive Sample with Approval Loop
173+
174+
```csharp
175+
new SampleDefinition
176+
{
177+
Name = "FoundryAgent_Hosted_MCP",
178+
ProjectPath = "samples/02-agents/ModelContextProtocol/FoundryAgent_Hosted_MCP",
179+
RequiredEnvironmentVariables = ["AZURE_AI_PROJECT_ENDPOINT"],
180+
OptionalEnvironmentVariables = ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
181+
Inputs = ["Y", "Y", "Y", "Y", "Y"],
182+
InputDelayMs = 5000,
183+
ExpectedOutputDescription = ["The output should show an agent using the Microsoft Learn MCP tool with approval prompts."],
184+
},
185+
```
186+
187+
### Example: Declarative Workflow with External Loop
188+
189+
```csharp
190+
new SampleDefinition
191+
{
192+
Name = "Workflow_Declarative_FunctionTools",
193+
ProjectPath = "samples/03-workflows/Declarative/FunctionTools",
194+
RequiredEnvironmentVariables = ["AZURE_AI_PROJECT_ENDPOINT"],
195+
OptionalEnvironmentVariables = ["AZURE_AI_MODEL_DEPLOYMENT_NAME"],
196+
Inputs = ["What are today's specials?", "EXIT"],
197+
InputDelayMs = 8000,
198+
ExpectedOutputDescription = ["The output should show a workflow calling function tools to answer a question about restaurant specials."],
199+
},
200+
```
201+
202+
### Example: Skipped Sample
203+
204+
```csharp
205+
new SampleDefinition
206+
{
207+
Name = "Agent_MCP_Server",
208+
ProjectPath = "samples/02-agents/ModelContextProtocol/Agent_MCP_Server",
209+
RequiredEnvironmentVariables = ["AZURE_OPENAI_ENDPOINT"],
210+
OptionalEnvironmentVariables = ["AZURE_OPENAI_DEPLOYMENT_NAME"],
211+
SkipReason = "Runs as an MCP stdio server that does not exit on its own.",
212+
},
213+
```

‎dotnet/agent-framework-dotnet.slnx‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
<Folder Name="/Samples/">
88
<File Path="samples/AGENTS.md" />
99
<File Path="samples/README.md" />
10+
<Project Path="eng/verify-samples/verify-samples.csproj" />
1011
</Folder>
1112
<Folder Name="/Samples/01-get-started/">
1213
<Project Path="samples/01-get-started/01_hello_agent/01_hello_agent.csproj" />

0 commit comments

Comments
 (0)