Skip to content

Commit e84017f

Browse files
NinjaRocksNinjaRocks
committed
- Add ch03-evaluation elaboration.
1 parent 35e6fd3 commit e84017f

3 files changed

Lines changed: 131 additions & 0 deletions

File tree

‎docs/verification-log.md‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
# Verification Log
2+
3+
Pre-publication verification record for *Generative AI in .NET*. One entry per pass. The most-recent fully-green entry is the print sign-off gate.
4+
5+
**Cadence:**
6+
7+
- **Pre-print:** weekly for the four weeks before print drop.
8+
- **Post-print:** monthly for six months, then quarterly (drives errata releases).
9+
10+
**What each entry covers:** package versions (Critical-5 list), code samples (build + smoke runs), URLs (chapter links + appendices), Anthropic API surface (model IDs in Appendix B + Chapter 4.2.4).
11+
12+
**Sign-off gate before print:**
13+
14+
- [ ] Most recent week is fully green.
15+
- [ ] No package on the watch list has a known breaking change pending.
16+
- [ ] Companion repo CI green on the latest commit.
17+
- [ ] Companion repo tagged `v1.0-print-ready` matching the manuscript version.
18+
19+
---
20+
21+
## Template
22+
23+
Copy this block to start a new entry. Date format `YYYY-MM-DD`.
24+
25+
```markdown
26+
## YYYY-MM-DD verification pass
27+
28+
### Packages (Critical-5 list)
29+
- [ ] Microsoft.Extensions.AI -- vX.Y.Z, no change / changelog reviewed
30+
- [ ] Microsoft.Extensions.AI.Abstractions -- ...
31+
- [ ] Microsoft.Extensions.AI.OpenAI -- ...
32+
- [ ] Microsoft.Extensions.AI.Ollama -- ...
33+
- [ ] Microsoft.Extensions.AI.AzureAIInference -- ...
34+
- [ ] Microsoft.Agents.AI -- ...
35+
- [ ] Microsoft.Agents.AI.OpenAI -- ...
36+
- [ ] Microsoft.Agents.AI.Workflows -- ...
37+
- [ ] Microsoft.Agents.AI.AzureAI -- ...
38+
- [ ] ModelContextProtocol -- ...
39+
- [ ] ModelContextProtocol.Core -- ...
40+
- [ ] ModelContextProtocol.AspNetCore -- ...
41+
- [ ] Microsoft.McpServer.ProjectTemplates -- ...
42+
- [ ] Microsoft.Azure.Functions.Worker.Extensions.Mcp -- ...
43+
- [ ] Azure.AI.OpenAI -- ...
44+
- [ ] OpenAI -- ...
45+
- [ ] OllamaSharp -- ...
46+
- [ ] Anthropic.SDK -- ...
47+
48+
### Code samples
49+
- [ ] CI matrix green on commit `<sha>`
50+
- [ ] Live-API smoke tests green (or skipped, with reason)
51+
52+
### URLs
53+
- [ ] Anthropic / Claude documentation links resolve
54+
- [ ] Microsoft Learn links resolve
55+
- [ ] Azure documentation links resolve
56+
- [ ] NuGet package pages resolve
57+
58+
### Anthropic API surface
59+
- [ ] Every model ID in `Appendix-B-Model-Quick-Reference.md` is callable
60+
- [ ] Every model ID in `Chapter-04.md` section 4.2.4 is callable
61+
- [ ] `Anthropic.SDK` API surface used in the chapter examples matches the latest stable
62+
63+
### Issues found / actions taken
64+
- (none) | <description + commit ref + manuscript section touched>
65+
```
66+
67+
---
68+
69+
## 2026-04-30 -- Initial sweep (kickoff)
70+
71+
**Status:** Partial -- snapshot only; subsequent weeks will be full passes.
72+
73+
### Packages
74+
- [x] Critical-5 list re-verified against live NuGet feed; companion repo pinned to `Microsoft.Agents.AI 1.3` and `ModelContextProtocol 1.2`.
75+
- [x] All 37 sample projects build clean on these versions (companion-repo commits `0047e61` + `35e6fd3`).
76+
- [x] CI matrix green (run 25136332826).
77+
78+
### Code samples
79+
- [x] All 37 samples build clean.
80+
- [ ] Live-API smoke tests -- not yet wired up (P2-1 in next-steps-plan).
81+
82+
### URLs
83+
- [ ] Not run yet -- queue for the first scheduled weekly pass.
84+
85+
### Anthropic API surface
86+
- [ ] Not run yet -- queue for the first scheduled weekly pass (P0-4 in next-steps-plan).
87+
88+
### Issues found / actions taken
89+
- 15 placeholder samples ported to 1.x stable APIs (book-repo commit `09bb7d9` cleared the API-update-pending punch list).
90+
91+
---
92+
93+
*New entries go below this line, most recent first.*

‎samples/ch03-rag/03.2.7-evaluation/Bootstrap.cs‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
using System.ComponentModel;
2+
using Microsoft.Extensions.AI;
3+
4+
namespace RagEvaluation;
5+
6+
internal sealed record QaPair(
7+
[property: Description("A question whose answer is contained in the source passage.")] string Question,
8+
[property: Description("The minimal correct answer drawn directly from the passage.")] string GroundTruth,
9+
[property: Description("Echo back the source chunk id so the pair stays traceable.")] string SourceChunkId);
10+
11+
internal static class Bootstrap
12+
{
13+
public static async Task<IReadOnlyList<QaPair>> GenerateAsync(
14+
IChatClient generator,
15+
IEnumerable<(string Id, string Text)> chunks,
16+
CancellationToken ct = default)
17+
{
18+
var pairs = new List<QaPair>();
19+
foreach (var (id, text) in chunks)
20+
{
21+
var resp = await generator.GetResponseAsync<QaPair>(
22+
$$"""
23+
Read the passage and propose ONE question whose answer is contained
24+
in the passage. Keep the question specific and the answer concise.
25+
Set sourceChunkId to "{{id}}".
26+
27+
Passage:
28+
{{text}}
29+
""", cancellationToken: ct);
30+
31+
if (resp.TryGetResult(out var pair))
32+
pairs.Add(pair);
33+
}
34+
return pairs;
35+
}
36+
}

‎samples/ch03-rag/03.2.7-evaluation/README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@ A minimal LLM-as-judge harness that scores candidate answers on **faithfulness**
66

77
The bundled `golden-dataset.json` is a 3-row demo. Replace it with your own questions/ground-truths and plug in your real RAG pipeline's answer in place of the `candidateAnswer` placeholder.
88

9+
`Bootstrap.cs` ships a small Q&A-pair generator that reads chunks of your indexed corpus and proposes one `QaPair` per chunk via the same `IChatClient` plumbing. Use it to seed a golden dataset from a corpus with no existing user logs, then human-review the result before checking it in.
10+
911
## Run it
1012

1113
```bash

0 commit comments

Comments
 (0)