Skip to content

Commit d01d7e5

Browse files
aseembits93aseembits93
committed
Support launch benchmark prompt scenarios
1 parent 087220b commit d01d7e5

5 files changed

Lines changed: 384 additions & 95 deletions

File tree

‎experiment/locality-mcp-comparison/README.md‎

Lines changed: 41 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -23,8 +23,10 @@ The output parent page is:
2323
- `run-launch-readiness-benchmark.sh` - core benchmark runner.
2424
- `run-repeated.sh` - runs the benchmark multiple times.
2525
- `setup-codex-azure.sh` - writes Codex Azure config.
26-
- `prompts/locality-agent-prompt.md` - Locality-only agent prompt.
27-
- `prompts/notion-mcp-agent-prompt.md` - Notion-MCP-only agent prompt.
26+
- `prompts/Locality/*.md` - Locality-only scenario prompts.
27+
- `prompts/MCP/*.md` - Notion-MCP-only scenario prompts, paired by filename with `prompts/Locality/*.md`.
28+
- `prompts/locality-agent-prompt.md` - legacy Locality-only agent prompt used only when `prompts/Locality/` has no scenarios.
29+
- `prompts/notion-mcp-agent-prompt.md` - legacy Notion-MCP-only agent prompt used only when `prompts/Locality/` has no scenarios.
2830
- `scripts/timestamp-jsonl.py` - timestamps Codex JSON events.
2931
- `scripts/summarize-codex-events.py` - summarizes one Codex JSON trace.
3032
- `scripts/summarize-runs.py` - summarizes multiple run folders.
@@ -127,6 +129,22 @@ copies `AZURE_OPENAI_API_KEY` into sandbox-local secret storage when that
127129
environment variable is set locally; otherwise it uses the sandbox's existing
128130
Codex auth/config or `~/.config/locality-experiment/env`.
129131

132+
## Add Scenarios
133+
134+
The core runner discovers scenarios from `prompts/Locality/*.md`. To add a new
135+
benchmark scenario, add the same filename to both prompt directories:
136+
137+
```text
138+
prompts/Locality/scenario2.md
139+
prompts/MCP/scenario2.md
140+
```
141+
142+
When `--compare-mcp` is enabled, every Locality scenario must have a matching
143+
MCP prompt with the same basename. Each prompt should keep writing its report
144+
and trace to the standard paths under `OUT_DIR`, such as `OUT_DIR/report-body.md`
145+
for Locality and `OUT_DIR/notion-mcp-report-body.md` for MCP. The runner sets
146+
`OUT_DIR` separately for each scenario.
147+
130148
## Run Once
131149

132150
```bash
@@ -181,43 +199,44 @@ ssh -o StrictHostKeyChecking=accept-new "$SSH_TARGET" '
181199

182200
## Artifacts
183201

184-
Each run writes to:
202+
Each run writes shared setup artifacts to:
185203

186204
`experiment/runs/<run-id>/`
187205

188206
Important artifacts:
189207

190-
- `metrics.tsv` - phase wall-clock metrics.
208+
- `metrics.tsv` - phase wall-clock metrics with a `scenario` column.
191209
- `summary.json` - machine-readable run summary.
192-
- `report-body.md` - Locality report.
193-
- `notion-mcp-report-body.md` - MCP report.
194-
- `locality-codex-events.jsonl` - timestamped Codex JSON events.
195-
- `notion-mcp-codex-events.jsonl` - timestamped Codex JSON events.
196-
- `locality-prompt.md` and `notion-mcp-prompt.md` - exact prompts used for the run.
197-
- `locality-codex-command.txt` and `notion-mcp-codex-command.txt` - exact `codex exec` command and timeout wrapper.
198-
- `locality-codex-summary.json` - event counts, usage, errors.
199-
- `notion-mcp-codex-summary.json` - event counts, usage, errors.
200-
- `locality-speedscope.json` and `notion-mcp-speedscope.json` - Speedscope-compatible flame graph files generated from the JSON events.
210+
- `scenarios.tsv` - scenario manifest with prompt paths, output directories, and mounted report pages.
211+
- `scenarios/<scenario>/report-body.md` - Locality report for that scenario.
212+
- `scenarios/<scenario>/notion-mcp-report-body.md` - MCP report for that scenario.
213+
- `scenarios/<scenario>/locality-codex-events.jsonl` - timestamped Locality Codex JSON events.
214+
- `scenarios/<scenario>/notion-mcp-codex-events.jsonl` - timestamped MCP Codex JSON events.
215+
- `scenarios/<scenario>/locality-prompt.md` and `scenarios/<scenario>/notion-mcp-prompt.md` - exact prompts used for the scenario.
216+
- `scenarios/<scenario>/locality-codex-command.txt` and `scenarios/<scenario>/notion-mcp-codex-command.txt` - exact `codex exec` commands and timeout wrappers.
217+
- `scenarios/<scenario>/locality-codex-summary.json` - event counts, usage, errors.
218+
- `scenarios/<scenario>/notion-mcp-codex-summary.json` - event counts, usage, errors.
219+
- `scenarios/<scenario>/locality-speedscope.json` and `scenarios/<scenario>/notion-mcp-speedscope.json` - Speedscope-compatible flame graph files generated from the JSON events.
201220
- `locality-traces/*.jsonl` - raw Locality command and pull/hydration spans.
202221
- `locality-traces/*-summary.json` - top Locality spans by duration.
203222
- `locality-traces/*-spans.tsv` - tabular Locality span data.
204223
- `locality-traces/*-speedscope.json` - Speedscope-compatible Locality spans.
205-
- `locality-agent-locality-trace.jsonl` and `notion-mcp-agent-locality-trace.jsonl` - Locality spans emitted by any `loc` commands the agents run.
206-
- `locality-transcript.md` and `notion-mcp-transcript.md` - readable Codex event transcripts generated from the JSON events.
207-
- `locality-agent-trace.md` - agent-reported trace.
208-
- `notion-mcp-agent-trace.md` - agent-reported trace.
209-
- `loc-diff.out` - Locality push plan.
224+
- `scenarios/<scenario>/locality-agent-locality-trace.jsonl` and `scenarios/<scenario>/notion-mcp-agent-locality-trace.jsonl` - Locality spans emitted by any `loc` commands the agents run.
225+
- `scenarios/<scenario>/locality-transcript.md` and `scenarios/<scenario>/notion-mcp-transcript.md` - readable Codex event transcripts generated from the JSON events.
226+
- `scenarios/<scenario>/locality-agent-trace.md` - agent-reported Locality trace.
227+
- `scenarios/<scenario>/notion-mcp-agent-trace.md` - agent-reported MCP trace.
228+
- `scenarios/<scenario>/loc-diff.out` - Locality push plan.
210229

211230
Generate flame graph artifacts for a completed run with:
212231

213232
```bash
214233
python3 experiment/locality-mcp-comparison/scripts/codex-events-to-trace.py \
215-
experiment/runs/<run-id>/locality-codex-events.jsonl \
216-
experiment/runs/<run-id>/locality
234+
experiment/runs/<run-id>/scenarios/<scenario>/locality-codex-events.jsonl \
235+
experiment/runs/<run-id>/scenarios/<scenario>/locality
217236

218237
python3 experiment/locality-mcp-comparison/scripts/codex-events-to-trace.py \
219-
experiment/runs/<run-id>/notion-mcp-codex-events.jsonl \
220-
experiment/runs/<run-id>/notion-mcp
238+
experiment/runs/<run-id>/scenarios/<scenario>/notion-mcp-codex-events.jsonl \
239+
experiment/runs/<run-id>/scenarios/<scenario>/notion-mcp
221240
```
222241

223242
The generated Speedscope files use observed gaps between consecutive Codex JSON events. This makes the chart useful even when Codex flushes `item.started` and `item.completed` at the same timestamp. Treat these charts as agent-session timing, not exact internal shell, MCP, or model runtime profiling.

‎experiment/locality-mcp-comparison/prompts/Locality/scenario1.md‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
You are running the Locality-backed launch-readiness benchmark.
2+
3+
Goal: generate a concise, accurate launch-readiness report from the local git checkout plus hydrated Notion context exposed as local files by Locality.
4+
5+
Use only these context sources:
6+
- local git commands in `REPO_DIR`
7+
- mounted Locality files under the paths listed in `CONTEXT_PATHS_FILE`
8+
- `CONTEXT_INVENTORY`
9+
- `CONTEXT_SEARCH_RESULTS`
10+
- `OUT_DIR/git-data.json`
11+
12+
Do not use Notion MCP or direct Notion API tools in this run.
13+
Do not push to Notion.
14+
15+
Required work:
16+
1. Read `OUT_DIR/git-data.json`.
17+
2. Inspect relevant commits with `git show`, `git diff --stat`, and file reads as needed.
18+
3. Read the hydrated Notion context inventory and search hits.
19+
4. Open the most relevant mounted `page.md` files to connect git work to launch context.
20+
5. Read the benchmark case section in the Locality Launch Amika Environment page.
21+
6. Write the final Markdown report to `OUT_DIR/report-body.md`.
22+
7. Write a compact trace to `OUT_DIR/locality-agent-trace.md` listing:
23+
- git commands used
24+
- mounted Notion files read
25+
- key facts used from Notion
26+
- limitations
27+
28+
Report format:
29+
30+
# Locality Launch Readiness Report
31+
32+
## Executive Summary
33+
34+
## What Changed In Git
35+
36+
## How This Maps To Launch Readiness
37+
38+
## Cross-Page Context Used
39+
40+
## Risks And Blockers
41+
42+
## Recommended Next Actions
43+
44+
## Measurement Notes
45+
46+
The report should be human, specific, and grounded in evidence. Avoid generic filler. If a claim cannot be verified from git or Notion context, say so.

‎experiment/locality-mcp-comparison/prompts/MCP/scenario1.md‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
You are running the Notion-MCP launch-readiness comparison benchmark.
2+
3+
Goal: generate the same launch-readiness report using local git metadata plus Notion MCP for Notion context.
4+
5+
Use these context sources:
6+
- local git commands in `REPO_DIR`
7+
- `OUT_DIR/git-data.json`
8+
- Notion MCP tools for Notion search/read context
9+
10+
Do not read mounted Locality Notion files under `/home/amika/notion`.
11+
Do not use `loc` commands.
12+
Do not push or update Notion in this run.
13+
14+
Required work:
15+
1. Read `OUT_DIR/git-data.json`.
16+
2. Inspect relevant commits with local git commands as needed.
17+
3. Use Notion MCP to search/read context around:
18+
- Locality Launch Amika Environment
19+
- launch readiness
20+
- install/update/distribution
21+
- safe diff, push, review, and prompt-injection guidance
22+
- Live Mode, conflicts, File Provider, Windows Cloud Files
23+
- connector launch readiness
24+
- standups and internal daily-use workflow
25+
4. Read the benchmark case section in the Locality Launch Amika Environment page through Notion MCP.
26+
5. Write the final Markdown report to `OUT_DIR/notion-mcp-report-body.md`.
27+
6. Write a compact trace to `OUT_DIR/notion-mcp-agent-trace.md` listing:
28+
- git commands used
29+
- Notion MCP searches/calls attempted
30+
- Notion pages or excerpts used
31+
- limitations
32+
33+
Report format:
34+
35+
# Locality Launch Readiness Report
36+
37+
## Executive Summary
38+
39+
## What Changed In Git
40+
41+
## How This Maps To Launch Readiness
42+
43+
## Cross-Page Context Used
44+
45+
## Risks And Blockers
46+
47+
## Recommended Next Actions
48+
49+
## Measurement Notes
50+
51+
The report should be human, specific, and grounded in evidence. Avoid generic filler. If a claim cannot be verified from git or Notion context, say so.

0 commit comments

Comments
 (0)