Skip to content

Commit 4ecd0e6

Browse files
denisecasedenisecase
committed
update
1 parent 9cc5614 commit 4ecd0e6

42 files changed

Lines changed: 3365 additions & 12 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.claude/mcp_settings.json‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
{
2+
"mcpServers": {
3+
"filesystem": {
4+
"command": "npx",
5+
"args": [
6+
"@modelcontextprotocol/server-filesystem",
7+
"C:\\Repos"
8+
]
9+
}
10+
}
11+
}

‎.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,10 @@
1616
# committed or pushed. Clone, do not fork, to preserve upstream pull access.
1717
evaluation/local/
1818

19+
# WHY: Generated registry output from local test runs; not part of the specification.
20+
# Scenario-level copies in evaluation/scenarios/*/tools/ are committed as reference outputs.
21+
/tools/
22+
1923

2024
# === Universal (all projects, all languages) ===
2125

‎DECISIONS.md‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,40 @@ an adaptation of
104104
The adaptive-interfaces manifest borrows the pattern
105105
(explicit includes/excludes, declarative intent) without the schema binding.
106106

107+
## Decision 8. Registry structure: one file per tool surface
108+
109+
Use one TOML file per tool surface rather than one monolithic registry file,
110+
with a manifest.toml that lists all registry files.
111+
112+
### Rationale 8
113+
114+
Observed during scenario testing: re-verification of the GitHub API entries
115+
in a monolithic registry (560+ lines, 3 tool surfaces) took 12m 18s and
116+
required the agent to search and update entries across a large file.
117+
On large tool sets this is slow, fragile, and risks missed or duplicate updates.
118+
Per-surface files keep each file small and focused.
119+
Re-verification targets one file, not the entire registry.
120+
Agents load only the registries relevant to their current task.
121+
The manifest provides a stable index without requiring agents to
122+
scan the filesystem.
123+
124+
## Decision 9. Conclude step: inline scoring and artifact copy
125+
126+
The Conclude step requires the agent to score its own output
127+
against the rubric inline in the response,
128+
and to copy the registry file to the scenario folder if one exists.
129+
130+
### Rationale 9
131+
132+
Observed during scenario testing: the agent self-scored and copied
133+
the artifact unprompted in the github-api run 3.
134+
This behavior makes evaluation traceable without a separate manual step.
135+
The score is recorded alongside the evidence that justifies it,
136+
and the registry is preserved in the scenario folder for comparison
137+
across runs.
138+
Making these behaviors explicit in the spec ensures they happen
139+
consistently rather than opportunistically.
140+
107141
## See Also
108142

109143
- [Markdown Architectural Decision Records](https://adr.github.io/madr/)

‎MCP.md‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
# MCP
2+
3+
MCP stands for Model Context Protocol,
4+
an open standard Anthropic published in late 2024
5+
for connecting AI agents to external tools and data sources
6+
in a standardized way.
7+
8+
## Core Idea
9+
10+
Before MCP, every tool integration was bespoke.
11+
If you wanted Claude to read your filesystem,
12+
query a database, or call an API,
13+
you had to write custom code for each one.
14+
MCP standardizes the interface so any MCP-compatible agent
15+
can talk to any MCP-compatible server
16+
without custom integration code.
17+
Think of it like USB. Before USB,
18+
every peripheral had its own connector.
19+
MCP is the USB for AI tool connections.
20+
21+
## How It Works
22+
23+
An MCP server exposes a set of tools: functions the agent can call.
24+
The server declares:
25+
26+
- what tools exist
27+
- what inputs each tool takes
28+
- what each tool returns
29+
30+
The agent discovers these tools at runtime by reading the server's manifest,
31+
then invokes them by name with structured inputs.
32+
The server handles execution and returns structured outputs.
33+
34+
The three things MCP servers can expose
35+
36+
1. Tools: functions the agent can invoke (read a file, query a database, send a message)
37+
2. Resources: data the agent can read (file contents, database records)
38+
3. Prompts: pre-built prompt templates
39+
40+
ATD is doing MCP discovery manually for any tool surface.
41+
For MCP servers specifically,
42+
the discovery step is formalized.
43+
The server manifest tells exactly what tools exist and what their schemas are,
44+
rather than requiring docs-fetching or help-output parsing.
45+
46+
## Example
47+
48+
The MCP filesystem server exposes tools like:
49+
50+
```text
51+
read_file(path)
52+
write_file(path, contents)
53+
list_directory(path)
54+
create_directory(path)
55+
move_file(source, destination)
56+
search_files(path, pattern)
57+
get_file_info(path)
58+
```
59+
60+
Claude Code connects to the server,
61+
reads the manifest, and can now call any of these tools directly.
62+
No custom filesystem code needed.
63+
64+
## How Claude Code uses MCP
65+
66+
Claude Code has built-in MCP support.
67+
You configure servers in .claude/mcp_settings.json and
68+
Claude Code connects to them at session start.
69+
The agent can then invoke their tools as if they were native capabilities.
70+
71+
## The Ecosystem
72+
73+
There are already hundreds of MCP servers:
74+
for GitHub, Slack, Google Drive, databases, browsers, and more.
75+
The adaptive-interfaces org's ATD skill is particularly relevant
76+
here because mapping MCP server capabilities
77+
is exactly the kind of tool discovery ATD is designed to do.

‎README.md‎

Lines changed: 198 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,204 @@ npx markdownlint-cli2 "**/*.md"
8787
uvx skillcheck SKILL.md --min-desc-score 75
8888
```
8989

90+
## Get and Set GitHub Personal Access Token (Needed for Associated Scenario)
91+
92+
Before running the `github-api` scenario, create a GitHub personal access token
93+
with at least repo read scope for the verification calls.
94+
Hit ESC to escape the current Claude session and create a token.
95+
96+
1. Go to <https://github.com/settings/tokens>
97+
2. Click **Generate new token** / **Generate new token (classic)**
98+
3. Give it a name in the note field, e.g. `atd-scenario-test`
99+
4. Set expiration (e.g. 7-30 days for a test token)
100+
5. Select scopes:
101+
- `public_repo` - read access to public repositories (sufficient for read-only verification calls)
102+
- Add `repo` if you need to verify against private repositories
103+
6. Click **Generate token**
104+
7. Copy the token immediately; GitHub will not show it again
105+
106+
Set it in PowerShell before starting your Claude Code session:
107+
108+
```powershell
109+
$env:GITHUB_TOKEN = "ghp_yourtoken_here"
110+
```
111+
112+
Verify it is set:
113+
114+
```powershell
115+
$env:GITHUB_TOKEN
116+
```
117+
118+
## Set Up MCP Filesystem Server (Needed for Associated Scenario)
119+
120+
Before running the `mcp-filesystem-server` scenario,
121+
install and start the MCP reference filesystem server locally.
122+
Hit ESC to escape the current Claude session to complete setup.
123+
124+
### Install
125+
126+
Requires Node.js. Install the MCP filesystem server globally:
127+
128+
```shell
129+
npm install -g @modelcontextprotocol/server-filesystem
130+
```
131+
132+
### Start the server
133+
134+
The filesystem server requires at least one allowed directory.
135+
Pass the path you want it to expose, for example:
136+
137+
```powershell
138+
npx @modelcontextprotocol/server-filesystem C:\Repos
139+
```
140+
141+
The server starts on stdio by default.
142+
Note the server name and connection details for the scenario prompt.
143+
144+
### Verify it is running
145+
146+
In a separate terminal:
147+
148+
```powershell
149+
npx @modelcontextprotocol/server-filesystem C:\Repos
150+
```
151+
152+
### Configure Claude Code to connect
153+
154+
Add the server to your Claude Code MCP config
155+
(typically `.claude/mcp_settings.json` in your home directory):
156+
157+
```pwsh
158+
New-Item -ItemType Directory -Force -Path "$HOME\.claude"
159+
160+
Set-Content -Path "$HOME\.claude\mcp_settings.json" -Value '{
161+
"mcpServers": {
162+
"filesystem": {
163+
"command": "npx",
164+
"args": ["@modelcontextprotocol/server-filesystem", "C:\\Repos"]
165+
}
166+
}
167+
}'
168+
169+
Get-Content "$HOME\.claude\mcp_settings.json"
170+
```
171+
172+
The file created looks like:
173+
174+
```json
175+
{
176+
"mcpServers": {
177+
"filesystem": {
178+
"command": "npx",
179+
"args": ["@modelcontextprotocol/server-filesystem", "C:\\Repos"]
180+
}
181+
}
182+
}
183+
```
184+
185+
Restart Claude Code after editing the config.
186+
Verify the server is available in the session before running the scenario.
187+
188+
## Agent Options for Running Scenarios
189+
190+
**Claude Code** (recommended)
191+
Included with a Claude Pro subscription ($20/month).
192+
Terminal CLI that reads your repo, makes real HTTP calls, writes files,
193+
and executes commands.
194+
This is the right tool for running ATD scenarios end-to-end.
195+
196+
Install:
197+
198+
```shell
199+
npm install -g @anthropic-ai/claude-code
200+
```
201+
202+
**GitHub Copilot**
203+
Good for inline code suggestions while editing.
204+
Not suitable for running scenarios - lacks deep file system access
205+
and long context handling.
206+
207+
**ChatGPT / OpenAI**
208+
Separate product and ecosystem.
209+
Skills in this repo are specified for Claude's format and behavior.
210+
Cross-agent results are not comparable without adaptation.
211+
212+
## Run Scenarios (with Claude Code): General Instructions
213+
214+
Scenarios require an agent with file system access and the ability
215+
to make live HTTP calls.
216+
Claude.ai (chat) is useful for reviewing and refining skills
217+
but cannot execute scenarios - it has no file system access
218+
and cannot make real API calls.
219+
220+
From the repo root, start a Claude Code session:
221+
222+
```shell
223+
claude
224+
```
225+
226+
Select your authorization (e.g. Option 1) and it will open a browser to authenticate against your pro account.
227+
After auth is successful, follow the terminal prompts, e.g. "Login successful. Press Enter to continue…".
228+
Accept the notice (hit Enter).
229+
When asked " Use Claude Code's terminal setup?" chose your option (e.g., 1 or Enter to confirm).
230+
When asked "Is this a project you created or one you trust?", chose your option (e.g., 1 or Enter to confirm).
231+
232+
Then in the session, paste the commands provided below, e.g. something like this:
233+
234+
```shell
235+
─────────────────────────────────────────────────────────────────────────
236+
❯ Read SKILL.md then follow evaluation/scenarios/<scenario-name>/prompt.md
237+
─────────────────────────────────────────────────────────────────────────
238+
```
239+
240+
Read questions and answer or Hit Enter to accept default until done.
241+
After each scenario finishes:
242+
243+
1. Copy the `tools` folder to the scenario folder.
244+
2. Score performance against the rubric in `score.md`.
245+
246+
Note: Scenarios can be run in any order.
247+
The files in `tools/` reflect the order they were generated, e.g.,
248+
github-api, open-meteo-api, uv-cli, mcp-filesystem-server.
249+
250+
### Run Scenario 1: github-api (requires token)
251+
252+
```shell
253+
Run: if ($env:GITHUB_TOKEN) { "token is set" } else { "token is NOT set" }
254+
255+
Read SKILL.md then follow evaluation/scenarios/github-api/prompt.md
256+
```
257+
258+
### Run Scenario 2: mcp-filesystem-server
259+
260+
```shell
261+
Show me your MCP configuration
262+
263+
What MCP servers and tools are available?
264+
265+
Read SKILL.md then follow evaluation/scenarios/mcp-filesystem-server/prompt.md
266+
```
267+
268+
### Run Scenario 3: open-meteo-api (no auth)
269+
270+
```shell
271+
Read SKILL.md then follow evaluation/scenarios/open-meteo-api/prompt.md
272+
```
273+
274+
### Run Scenario 4: uv-cli (no auth)
275+
276+
```shell
277+
Read SKILL.md then follow evaluation/scenarios/uv-cli/prompt.md
278+
```
279+
280+
## Alternative: Explicitly Call ACS (not required)
281+
282+
```shell
283+
Read https://raw.githubusercontent.com/adaptive-interfaces/adaptive-conformance-specification/main/SKILL.md
284+
then read SKILL.md
285+
then follow evaluation/scenarios/<scenario-name>/prompt.md
286+
```
287+
90288
## License
91289
92290
MIT © 2026 [Adaptive Interfaces](https://github.com/adaptive-interfaces)

0 commit comments

Comments
 (0)