Skip to content

Commit 54cd0e8

Browse files
Saga4Saga4
committed
experiment: add multi-source benchmark scenarios
1 parent 96293e1 commit 54cd0e8

8 files changed

Lines changed: 853 additions & 44 deletions

File tree

‎experiment/locality-mcp-comparison/README.md‎

Lines changed: 60 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,12 @@
22

33
This experiment compares two agent paths for a launch-readiness workflow:
44

5-
- **Locality path:** hydrate Notion through Locality, let the agent read mounted Markdown files, and write local Markdown artifacts under `OUT_DIR`.
6-
- **Notion MCP path:** let the agent use Notion MCP search/fetch for context, without reading mounted Locality files or using `loc`.
5+
- **Locality path:** hydrate or reuse prehydrated Locality files, let the agent
6+
read mounted Markdown/text/JSON files across connected sources, and write
7+
local Markdown artifacts under `OUT_DIR`.
8+
- **MCP path:** let the agent use MCP tools for Notion, Linear, and Slack when
9+
configured, plus local git/`gh` for repository evidence, without reading
10+
mounted Locality files or using `loc`.
711

812
The benchmark case lives in Notion at:
913

@@ -33,6 +37,8 @@ The output parent page is:
3337
to measure prompt handoff, tool calls, model thinking spans, and final output
3438
response spans while the session is running.
3539
- `scripts/summarize-codex-events.py` - summarizes one Codex JSON trace.
40+
- `scripts/deep-dive-report.py` - writes a per-run Markdown index of phase
41+
timings, tool buckets, timelines, and trace artifact paths.
3642
- `scripts/summarize-runs.py` - summarizes multiple run folders.
3743

3844
## Separation Rules
@@ -86,6 +92,34 @@ ssh -o StrictHostKeyChecking=accept-new "$ssh_target" '
8692
'
8793
```
8894

95+
For a sandbox that already has the desired Locality connections and hydrated
96+
files, use the existing state instead of creating an isolated temporary Notion
97+
mount:
98+
99+
```bash
100+
export SANDBOX=aseem-locality
101+
export SSH_TARGET="$(amika sandbox ssh --print "$SANDBOX")"
102+
export LOCALITY_SANDBOX="$SANDBOX"
103+
export LOCALITY_USE_EXISTING_STATE=1
104+
export LOCALITY_CONTEXT_HYDRATE=0
105+
```
106+
107+
If the multi-source roots are known, pass them explicitly. Use newline
108+
separation when paths contain spaces:
109+
110+
```bash
111+
export LOCALITY_CONTEXT_DIRS="$(cat <<'EOF'
112+
/home/amika/notion/Go To Market/Locality Launch - Amika Environment
113+
/home/amika/slack
114+
/home/amika/linear
115+
EOF
116+
)"
117+
```
118+
119+
The worker still locates and pulls the target Notion output page unless the run
120+
is MCP-only. `LOCALITY_CONTEXT_HYDRATE=0` applies only to the listed context
121+
directories and is intended for prehydrated sandboxes.
122+
89123
Verify Locality:
90124

91125
```bash
@@ -181,8 +215,11 @@ also retrieves compatibility outputs written under
181215
such as `/home/amika/report-body.md`.
182216

183217
Only `scenario1.md` receives precomputed git metadata at
184-
`GIT_DATA_FILE`. Other scenarios should not require that file; if they need
185-
repository context, they should inspect the repository directly with git.
218+
`GIT_DATA_FILE`. The multi-source scenarios `scenario7.md` and `scenario8.md`
219+
also receive this file so the agent can compare local git evidence with
220+
connected-source evidence. Other scenarios should not require that file; if
221+
they need repository context, they should inspect the repository directly with
222+
git.
186223

187224
## Run Once
188225

@@ -191,6 +228,21 @@ CODEX_MODEL=gpt-5.6-luna CODEX_REASONING_EFFORT=low \
191228
./experiment/locality-mcp-comparison/run-agent-comparison.sh
192229
```
193230

231+
Run only the two multi-source scenarios against prehydrated Locality state:
232+
233+
```bash
234+
export SANDBOX=aseem-locality
235+
export SSH_TARGET="$(amika sandbox ssh --print "$SANDBOX")"
236+
export LOCALITY_SANDBOX="$SANDBOX"
237+
export MCP_SANDBOX=aseem-mcp
238+
export LOCALITY_USE_EXISTING_STATE=1
239+
export LOCALITY_CONTEXT_HYDRATE=0
240+
241+
CODEX_MODEL=gpt-5.6-luna CODEX_REASONING_EFFORT=low \
242+
./experiment/locality-mcp-comparison/run-agent-comparison.sh \
243+
--scenario scenario7,scenario8
244+
```
245+
194246
By default this is artifact-only. It writes local Markdown reports under
195247
`target/launch-readiness-amika/<run-id>/artifacts/{locality,notion-mcp}` after
196248
syncing the remote sandbox `OUT_DIR`s back to the local machine. It does not
@@ -330,6 +382,10 @@ Important artifacts:
330382
scenarios and trials.
331383
- `token-usage/token-usage.tsv`, `token-usage/cost-usage.tsv`, and
332384
`token-usage/token-usage.json` - chart data, cost data, pricing, and manifest.
385+
- `deep-dive.md` - local wrapper report that indexes each scenario/strategy
386+
with phase timings, event counts, token totals, tool buckets, chronological
387+
tool calls, and links to the report, transcript, spans, Speedscope, Perfetto,
388+
SnakeViz, and Locality trace artifacts.
333389

334390
Cost charts default to the `gpt-5.6-luna` Standard short-context rates used by
335391
the benchmark harness. Override them for Azure/internal billing with

‎experiment/locality-mcp-comparison/prompts/Locality/scenario7.md‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
You are running the Locality-backed multi-source comparison benchmark.
2+
3+
User prompt:
4+
Have we seen this bug before? A user edited a mounted Notion page at
5+
`engineering-wiki/standups-with-locality/2026-07-02/page.md`, removed visible
6+
conflict markers, but Locality still could not push the file. The UI showed a
7+
pending problem with language like `daemon content cache path is missing`.
8+
Find out whether this is a known class of issue, whether it already appeared in
9+
Slack, Linear, GitHub work, or Notion docs, and what the safest next engineering
10+
action should be.
11+
12+
Use the Locality filesystem path for connected sources. Prefer the context
13+
directories listed in `CONTEXT_PATHS_FILE`, the inventory in
14+
`CONTEXT_INVENTORY`, and search hits in `CONTEXT_SEARCH_RESULTS`. Search mounted
15+
Notion, Slack, Linear, and GitHub-like Locality directories when they are
16+
available. Use `loc status`, `loc info`, `loc search`, `git`, and `gh` when they
17+
help verify evidence.
18+
19+
Do not use Notion MCP, Linear MCP, Slack MCP, direct Notion/Linear/Slack APIs,
20+
or browser automation in this run. Do not create docs, push changes, post
21+
messages, close issues, or update any remote source. Write the Markdown body to
22+
`REPORT_FILE`.
23+
24+
Required work:
25+
1. Identify the strongest matching prior incidents, issues, PRs, commits,
26+
Slack discussions, or Notion docs.
27+
2. Separate verified evidence from plausible interpretation.
28+
3. Explain whether this looks like a cache/shadow-state issue, a connector
29+
issue, a conflict-marker validation issue, a File Provider projection issue,
30+
or something else.
31+
4. Recommend the next safe action for engineering and the safest user recovery
32+
path.
33+
5. Write a compact trace to `TRACE_FILE` listing:
34+
- `loc`, `git`, `gh`, and shell searches used
35+
- mounted Locality files read
36+
- search terms used
37+
- source gaps or unavailable connectors
38+
39+
Report format:
40+
41+
# Prior Bug Evidence Report
42+
43+
## Answer
44+
45+
## Evidence Found
46+
47+
## Source-by-Source Findings
48+
49+
## Likely Root Cause Class
50+
51+
## Recommended Engineering Action
52+
53+
## Safe User Recovery Path
54+
55+
## Gaps And Confidence
56+
57+
The draft should be concise, specific, and grounded in source paths or command
58+
outputs. If a source is unavailable or no evidence is found, say that directly.

‎experiment/locality-mcp-comparison/prompts/Locality/scenario8.md‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
You are running the Locality-backed multi-source comparison benchmark.
2+
3+
User prompt:
4+
Can we make a new Locality release this week? Check the connected knowledge
5+
around launch/release readiness, current engineering risks, open issues,
6+
recent GitHub activity, and team discussions. Give a practical release call:
7+
go, no-go, or go with named conditions.
8+
9+
Use the Locality filesystem path for connected sources. Prefer the context
10+
directories listed in `CONTEXT_PATHS_FILE`, the inventory in
11+
`CONTEXT_INVENTORY`, and search hits in `CONTEXT_SEARCH_RESULTS`. Search mounted
12+
Notion, Slack, Linear, and GitHub-like Locality directories when they are
13+
available. Use `loc status`, `loc info`, `loc search`, `git`, and `gh` when they
14+
help verify evidence.
15+
16+
Do not use Notion MCP, Linear MCP, Slack MCP, direct Notion/Linear/Slack APIs,
17+
or browser automation in this run. Do not create a release, create docs, push
18+
changes, post messages, close issues, or update any remote source. Write the
19+
Markdown body to `REPORT_FILE`.
20+
21+
Required work:
22+
1. Find release-related context in Notion or other mounted docs.
23+
2. Check Linear for open blockers, recent fixes, and release-critical issues.
24+
3. Check Slack for launch/release discussions, unresolved decisions, or risk
25+
signals when Slack context is mounted.
26+
4. Check GitHub evidence through local git and `gh`: recent commits, open PRs,
27+
release tags, workflows, and notable failures when available.
28+
5. Produce a release recommendation with explicit conditions and owner-facing
29+
next actions.
30+
6. Write a compact trace to `TRACE_FILE` listing:
31+
- `loc`, `git`, `gh`, and shell searches used
32+
- mounted Locality files read
33+
- source gaps or unavailable connectors
34+
35+
Report format:
36+
37+
# Release Readiness Decision
38+
39+
## Recommendation
40+
41+
## Evidence Summary
42+
43+
## Source-by-Source Findings
44+
45+
## Blockers And Conditions
46+
47+
## Suggested Release Plan
48+
49+
## Gaps And Confidence
50+
51+
The draft should be concise, specific, and grounded in source paths or command
52+
outputs. If a source is unavailable or no evidence is found, say that directly.

‎experiment/locality-mcp-comparison/prompts/MCP/scenario7.md‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
You are running the MCP-backed multi-source comparison benchmark.
2+
3+
User prompt:
4+
Have we seen this bug before? A user edited a mounted Notion page at
5+
`engineering-wiki/standups-with-locality/2026-07-02/page.md`, removed visible
6+
conflict markers, but Locality still could not push the file. The UI showed a
7+
pending problem with language like `daemon content cache path is missing`.
8+
Find out whether this is a known class of issue, whether it already appeared in
9+
Slack, Linear, GitHub work, or Notion docs, and what the safest next engineering
10+
action should be.
11+
12+
Use MCP tools for Notion, Linear, and Slack when available. If a GitHub MCP
13+
server is available, use it for GitHub evidence; otherwise use local `git` and
14+
`gh` against the checked-out repository. Do not use `loc`, mounted Locality
15+
files, or Locality context inventory files in this run.
16+
17+
Do not create docs, push changes, post messages, close issues, or update any
18+
remote source. Write the Markdown body to `REPORT_FILE`.
19+
20+
Required work:
21+
1. Identify the strongest matching prior incidents, issues, PRs, commits,
22+
Slack discussions, or Notion docs.
23+
2. Separate verified evidence from plausible interpretation.
24+
3. Explain whether this looks like a cache/shadow-state issue, a connector
25+
issue, a conflict-marker validation issue, a File Provider projection issue,
26+
or something else.
27+
4. Recommend the next safe action for engineering and the safest user recovery
28+
path.
29+
5. Write a compact trace to `TRACE_FILE` listing:
30+
- MCP searches/calls attempted
31+
- `git` and `gh` commands used
32+
- source records or excerpts used
33+
- source gaps or unavailable MCP servers
34+
35+
Report format:
36+
37+
# Prior Bug Evidence Report
38+
39+
## Answer
40+
41+
## Evidence Found
42+
43+
## Source-by-Source Findings
44+
45+
## Likely Root Cause Class
46+
47+
## Recommended Engineering Action
48+
49+
## Safe User Recovery Path
50+
51+
## Gaps And Confidence
52+
53+
The draft should be concise, specific, and grounded in source records or command
54+
outputs. If a source is unavailable or no evidence is found, say that directly.

‎experiment/locality-mcp-comparison/prompts/MCP/scenario8.md‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
You are running the MCP-backed multi-source comparison benchmark.
2+
3+
User prompt:
4+
Can we make a new Locality release this week? Check the connected knowledge
5+
around launch/release readiness, current engineering risks, open issues,
6+
recent GitHub activity, and team discussions. Give a practical release call:
7+
go, no-go, or go with named conditions.
8+
9+
Use MCP tools for Notion, Linear, and Slack when available. If a GitHub MCP
10+
server is available, use it for GitHub evidence; otherwise use local `git` and
11+
`gh` against the checked-out repository. Do not use `loc`, mounted Locality
12+
files, or Locality context inventory files in this run.
13+
14+
Do not create a release, create docs, push changes, post messages, close
15+
issues, or update any remote source. Write the Markdown body to `REPORT_FILE`.
16+
17+
Required work:
18+
1. Find release-related context in Notion or other MCP-accessible docs.
19+
2. Check Linear for open blockers, recent fixes, and release-critical issues.
20+
3. Check Slack for launch/release discussions, unresolved decisions, or risk
21+
signals when Slack MCP is available.
22+
4. Check GitHub evidence through GitHub MCP when available, otherwise local git
23+
and `gh`: recent commits, open PRs, release tags, workflows, and notable
24+
failures when available.
25+
5. Produce a release recommendation with explicit conditions and owner-facing
26+
next actions.
27+
6. Write a compact trace to `TRACE_FILE` listing:
28+
- MCP searches/calls attempted
29+
- `git` and `gh` commands used
30+
- source records or excerpts used
31+
- source gaps or unavailable MCP servers
32+
33+
Report format:
34+
35+
# Release Readiness Decision
36+
37+
## Recommendation
38+
39+
## Evidence Summary
40+
41+
## Source-by-Source Findings
42+
43+
## Blockers And Conditions
44+
45+
## Suggested Release Plan
46+
47+
## Gaps And Confidence
48+
49+
The draft should be concise, specific, and grounded in source records or command
50+
outputs. If a source is unavailable or no evidence is found, say that directly.

0 commit comments

Comments
 (0)