Skip to content

Commit cf875f1

Browse files
aseembits93aseembits93
committed
Configure MCP auth for launch benchmark
1 parent 12e8767 commit cf875f1

4 files changed

Lines changed: 240 additions & 0 deletions

File tree

‎experiment/locality-mcp-comparison/README.md‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -129,6 +129,30 @@ copies `AZURE_OPENAI_API_KEY` into sandbox-local secret storage when that
129129
environment variable is set locally; otherwise it uses the sandbox's existing
130130
Codex auth/config or `~/.config/locality-experiment/env`.
131131

132+
## Launch Runner MCP Auth
133+
134+
When `run-launch-readiness-benchmark.sh` is run with `--compare-mcp`, it
135+
configures Codex MCP auth before running the MCP scenarios:
136+
137+
```bash
138+
export LINEAR_API_KEY=<linear-api-key>
139+
export NOTION_API_TOKEN=<notion-api-token>
140+
141+
# Optional Slack MCP support:
142+
export SLACK_BOT_TOKEN=<slack-bot-token>
143+
export SLACK_TEAM_ID=<slack-team-id>
144+
export SLACK_CHANNEL_IDS=<comma-delimited-channel-ids>
145+
```
146+
147+
`NOTION_TOKEN` and `NOTION_ACCESS_TOKEN` are accepted aliases for
148+
`NOTION_API_TOKEN`. Slack is configured only when `SLACK_BOT_TOKEN` is set; if
149+
any Slack MCP variable is set, both `SLACK_BOT_TOKEN` and `SLACK_TEAM_ID` are
150+
required.
151+
152+
The runner stores token-backed helper scripts and secret files under
153+
`~/.local/bin` and `~/.config/locality-launch-readiness/mcp` by default, then
154+
updates Codex MCP entries for `linear-server`, `notion`, and optionally `slack`.
155+
132156
## Add Scenarios
133157

134158
The core runner discovers scenarios from `prompts/Locality/*.md`. To add a new
@@ -208,6 +232,7 @@ Important artifacts:
208232
- `metrics.tsv` - phase wall-clock metrics with a `scenario` column.
209233
- `summary.json` - machine-readable run summary.
210234
- `scenarios.tsv` - scenario manifest with prompt paths, output directories, and mounted report pages.
235+
- `mcp-auth-setup.out` and `mcp-auth-setup.err` - Codex MCP setup logs when `--compare-mcp` is enabled.
211236
- `scenarios/<scenario>/report-body.md` - Locality report for that scenario.
212237
- `scenarios/<scenario>/notion-mcp-report-body.md` - MCP report for that scenario.
213238
- `scenarios/<scenario>/locality-codex-events.jsonl` - timestamped Locality Codex JSON events.

‎experiment/locality-mcp-comparison/prompts/Locality/scenario6.md‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
You are running the Locality-backed comparison benchmark.
2+
3+
User prompt:
4+
i want you to analyze the progress made by different team members from July 15 to July 21 in the year 2025 on the repo codeflash-ai/codeflash , read the linear issues for that time range and read the notion doc named 'Company' and create a notion doc in the page title `Locality Launch` followed by the current date and time in the title to distinguish it which summarizes your findings. use `loc` , `git` and `gh` to fulfil the tasks, do not rely on notion or linear mcp or api calls.
5+
6+
Use only these context sources:
7+
- `loc` commands
8+
- local git commands in `REPO_DIR`
9+
- GitHub context available through `gh`
10+
- mounted Locality files under the paths listed in `CONTEXT_PATHS_FILE`
11+
- `CONTEXT_INVENTORY`
12+
- `CONTEXT_SEARCH_RESULTS`
13+
- `OUT_DIR/git-data.json`
14+
15+
Do not use Notion MCP, Linear MCP, or direct Notion/Linear API tools in this run.
16+
Do not push to Notion or update any remote source. For this benchmark, satisfy the requested Notion doc by writing the Markdown body to `OUT_DIR/report-body.md`.
17+
18+
Required work:
19+
1. Read `OUT_DIR/git-data.json`.
20+
2. Inspect relevant codeflash-ai/codeflash repository progress for July 15 through July 21, 2025 using git and `gh` where available.
21+
3. Use `loc` and mounted Locality files to find and read relevant Notion context, including the doc named `Company` if available.
22+
4. Use Locality-accessible context for Linear issue information if available; if Linear issues are not accessible through allowed sources, state that limitation.
23+
5. Summarize progress by team member and connect it to the `Locality Launch` context where possible.
24+
6. Write the final Markdown draft to `OUT_DIR/report-body.md`.
25+
7. Write a compact trace to `OUT_DIR/locality-agent-trace.md` listing:
26+
- `loc`, git, and gh commands used
27+
- mounted Locality files read
28+
- key facts used from Locality context
29+
- unavailable or unverified Linear/Notion context
30+
31+
Report format:
32+
33+
# Locality Launch Team Progress Summary
34+
35+
## Summary
36+
37+
## Team Member Progress
38+
39+
## Linear Issue Context
40+
41+
## Company Context
42+
43+
## Launch Relevance
44+
45+
## Risks, Gaps, And Limitations
46+
47+
The draft should be concise, specific, and grounded in evidence. If a claim cannot be verified from loc, git, gh, or mounted Locality context, say so.

‎experiment/locality-mcp-comparison/prompts/MCP/scenario6.md‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
You are running the MCP comparison benchmark.
2+
3+
User prompt:
4+
i want you to analyze the progress made by different team members from July 15 to July 21 in the year 2025 on the repo codeflash-ai/codeflash , read the linear issues for that time range and read the notion doc named 'Company' and create a notion doc in the page title `Locality Launch` followed by the current date and time in the title to distinguish it which summarizes your findings.
5+
6+
Use these context sources:
7+
- local git commands in `REPO_DIR`
8+
- `OUT_DIR/git-data.json`
9+
- available MCP tools for Notion and Linear context
10+
11+
Do not read mounted Locality Notion files under `/home/amika/notion`.
12+
Do not use `loc` commands.
13+
Do not push or update Notion or any remote source in this run. For this benchmark, satisfy the requested Notion doc by writing the Markdown body to `OUT_DIR/notion-mcp-report-body.md`.
14+
15+
Required work:
16+
1. Read `OUT_DIR/git-data.json`.
17+
2. Inspect relevant codeflash-ai/codeflash repository progress for July 15 through July 21, 2025 using local git commands as needed.
18+
3. Use MCP tools to read Linear issues for that time range if available.
19+
4. Use MCP tools to find and read the Notion doc named `Company` if available.
20+
5. Summarize progress by team member and connect it to the `Locality Launch` context where possible.
21+
6. Write the final Markdown draft to `OUT_DIR/notion-mcp-report-body.md`.
22+
7. Write a compact trace to `OUT_DIR/notion-mcp-agent-trace.md` listing:
23+
- git commands used
24+
- MCP searches/calls attempted
25+
- Notion pages, Linear issues, or excerpts used
26+
- limitations
27+
28+
Report format:
29+
30+
# Locality Launch Team Progress Summary
31+
32+
## Summary
33+
34+
## Team Member Progress
35+
36+
## Linear Issue Context
37+
38+
## Company Context
39+
40+
## Launch Relevance
41+
42+
## Risks, Gaps, And Limitations
43+
44+
The draft should be concise, specific, and grounded in evidence. If a claim cannot be verified from git or MCP context, say so.

‎experiment/locality-mcp-comparison/run-launch-readiness-benchmark.sh‎

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,12 @@ Important environment:
2727
CODEX_REASONING_EFFORT Codex reasoning effort. Default: low
2828
CODEX_EXEC_TIMEOUT_SECONDS
2929
Per-strategy codex exec timeout. Default: 900. Use 0 to disable.
30+
LINEAR_API_KEY Required with --compare-mcp. Linear MCP bearer token.
31+
NOTION_API_TOKEN Required with --compare-mcp. NOTION_TOKEN and
32+
NOTION_ACCESS_TOKEN are accepted aliases.
33+
SLACK_BOT_TOKEN Optional with --compare-mcp. Slack bot token for Slack MCP.
34+
SLACK_TEAM_ID Required when SLACK_BOT_TOKEN is set.
35+
SLACK_CHANNEL_IDS Optional comma-delimited Slack channel allowlist.
3036
LOCALITY_EXPERIMENT_TRACE_FORCE_DIRECT
3137
Set to 1 to force CLI direct pull during traced Locality setup.
3238
SINCE Git window. Default: 24 hours ago
@@ -65,10 +71,17 @@ OUT_DIR="${OUT_DIR:-$REPO_DIR/experiment/runs/$RUN_ID}"
6571
CODEX_MODEL="${CODEX_MODEL:-gpt-5.6-luna}"
6672
CODEX_REASONING_EFFORT="${CODEX_REASONING_EFFORT:-low}"
6773
CODEX_EXEC_TIMEOUT_SECONDS="${CODEX_EXEC_TIMEOUT_SECONDS:-900}"
74+
LINEAR_API_KEY="${LINEAR_API_KEY:-}"
75+
NOTION_API_TOKEN="${NOTION_API_TOKEN:-${NOTION_TOKEN:-${NOTION_ACCESS_TOKEN:-}}}"
76+
SLACK_BOT_TOKEN="${SLACK_BOT_TOKEN:-}"
77+
SLACK_TEAM_ID="${SLACK_TEAM_ID:-}"
78+
SLACK_CHANNEL_IDS="${SLACK_CHANNEL_IDS:-}"
6879
LOCALITY_EXPERIMENT_TRACE_FORCE_DIRECT="${LOCALITY_EXPERIMENT_TRACE_FORCE_DIRECT:-0}"
6980
PROMPT_ROOT="${PROMPT_ROOT:-$SCRIPT_DIR/prompts}"
7081
LOCALITY_PROMPT_DIR="${LOCALITY_PROMPT_DIR:-$PROMPT_ROOT/Locality}"
7182
MCP_PROMPT_DIR="${MCP_PROMPT_DIR:-$PROMPT_ROOT/MCP}"
83+
MCP_SECRET_DIR="${MCP_SECRET_DIR:-$HOME/.config/locality-launch-readiness/mcp}"
84+
MCP_BIN_DIR="${MCP_BIN_DIR:-$HOME/.local/bin}"
7285
METRICS_TSV="$OUT_DIR/metrics.tsv"
7386
SUMMARY_JSON="$OUT_DIR/summary.json"
7487
CONTEXT_PATHS_FILE="$OUT_DIR/locality-context-paths.txt"
@@ -141,6 +154,99 @@ render_locality_traces() {
141154
done
142155
}
143156

157+
configure_codex_mcp_auth() {
158+
MCP_AUTH_DETAIL="linear=skipped; notion=skipped; slack=skipped"
159+
160+
command -v codex >/dev/null || return
161+
162+
if [ -z "$LINEAR_API_KEY" ]; then
163+
echo "LINEAR_API_KEY is required when --compare-mcp is enabled" >&2
164+
return 2
165+
fi
166+
if [ -z "$NOTION_API_TOKEN" ]; then
167+
echo "NOTION_API_TOKEN is required when --compare-mcp is enabled; NOTION_TOKEN and NOTION_ACCESS_TOKEN are accepted aliases" >&2
168+
return 2
169+
fi
170+
if { [ -n "$SLACK_BOT_TOKEN" ] || [ -n "$SLACK_TEAM_ID" ] || [ -n "$SLACK_CHANNEL_IDS" ]; } &&
171+
{ [ -z "$SLACK_BOT_TOKEN" ] || [ -z "$SLACK_TEAM_ID" ]; }; then
172+
echo "SLACK_BOT_TOKEN and SLACK_TEAM_ID are both required to configure Slack MCP" >&2
173+
return 2
174+
fi
175+
176+
mkdir -p "$MCP_SECRET_DIR" "$MCP_BIN_DIR" "$HOME/.codex" || return
177+
chmod 700 "$MCP_SECRET_DIR" "$HOME/.codex" || return
178+
179+
local notion_helper="$MCP_BIN_DIR/locality-launch-notion-mcp"
180+
(
181+
set -e
182+
umask 077
183+
printf '%s' "$LINEAR_API_KEY" > "$MCP_SECRET_DIR/linear-api-key"
184+
printf '%s' "$NOTION_API_TOKEN" > "$MCP_SECRET_DIR/notion-token"
185+
cat > "$notion_helper" <<SH
186+
#!/usr/bin/env bash
187+
set -euo pipefail
188+
189+
token_file="\${NOTION_API_TOKEN_FILE:-$MCP_SECRET_DIR/notion-token}"
190+
export OPENAPI_MCP_HEADERS="\$(
191+
python3 - "\$token_file" <<'PY'
192+
import json
193+
import pathlib
194+
import sys
195+
196+
token = pathlib.Path(sys.argv[1]).read_text(encoding="utf-8").strip()
197+
print(json.dumps({"Authorization": "Bearer " + token}, separators=(",", ":")))
198+
PY
199+
)"
200+
exec npx -y @notionhq/notion-mcp-server
201+
SH
202+
chmod 700 "$notion_helper"
203+
) || return
204+
205+
export LINEAR_API_KEY
206+
codex mcp remove linear-server >/dev/null 2>&1 || true
207+
codex mcp remove notion >/dev/null 2>&1 || true
208+
codex mcp remove slack >/dev/null 2>&1 || true
209+
codex mcp remove slack-server >/dev/null 2>&1 || true
210+
codex mcp add linear-server --url https://mcp.linear.app/mcp --bearer-token-env-var LINEAR_API_KEY || return
211+
codex mcp add notion -- "$notion_helper" || return
212+
213+
local slack_status="skipped"
214+
if [ -n "$SLACK_BOT_TOKEN" ]; then
215+
local slack_helper="$MCP_BIN_DIR/locality-launch-slack-mcp"
216+
(
217+
set -e
218+
umask 077
219+
printf '%s' "$SLACK_BOT_TOKEN" > "$MCP_SECRET_DIR/slack-bot-token"
220+
printf '%s' "$SLACK_TEAM_ID" > "$MCP_SECRET_DIR/slack-team-id"
221+
if [ -n "$SLACK_CHANNEL_IDS" ]; then
222+
printf '%s' "$SLACK_CHANNEL_IDS" > "$MCP_SECRET_DIR/slack-channel-ids"
223+
else
224+
rm -f "$MCP_SECRET_DIR/slack-channel-ids"
225+
fi
226+
cat > "$slack_helper" <<SH
227+
#!/usr/bin/env bash
228+
set -euo pipefail
229+
230+
secret_dir="$MCP_SECRET_DIR"
231+
export SLACK_BOT_TOKEN="\$(cat "\$secret_dir/slack-bot-token")"
232+
export SLACK_TEAM_ID="\$(cat "\$secret_dir/slack-team-id")"
233+
if [ -f "\$secret_dir/slack-channel-ids" ]; then
234+
export SLACK_CHANNEL_IDS="\$(cat "\$secret_dir/slack-channel-ids")"
235+
fi
236+
exec npx -y @modelcontextprotocol/server-slack
237+
SH
238+
chmod 700 "$slack_helper"
239+
) || return
240+
codex mcp add slack -- "$slack_helper" || return
241+
slack_status="configured"
242+
else
243+
rm -f "$MCP_SECRET_DIR/slack-bot-token" "$MCP_SECRET_DIR/slack-team-id" "$MCP_SECRET_DIR/slack-channel-ids"
244+
fi
245+
246+
MCP_AUTH_DETAIL="linear=configured; notion=configured; slack=$slack_status"
247+
echo "Configured Codex MCP auth: $MCP_AUTH_DETAIL"
248+
}
249+
144250
run_codex_agent() {
145251
local strategy="$1"
146252
local prompt_file="$2"
@@ -317,6 +423,24 @@ test -x "$LOC_BIN"
317423
git -C "$REPO_DIR" rev-parse --git-dir >/dev/null
318424
phase_end "locality" "validate_environment" "ok" "repo=$REPO_DIR; model=$CODEX_MODEL; effort=$CODEX_REASONING_EFFORT; scenarios=${#SCENARIO_FILES[@]}; prompts=$LOCALITY_PROMPT_DIR"
319425

426+
phase_start
427+
if [ "$COMPARE_MCP" -eq 1 ]; then
428+
set +e
429+
configure_codex_mcp_auth > "$OUT_DIR/mcp-auth-setup.out" 2> "$OUT_DIR/mcp-auth-setup.err"
430+
MCP_AUTH_RC=$?
431+
set -e
432+
if [ "$MCP_AUTH_RC" -eq 0 ]; then
433+
phase_end "notion_mcp" "mcp_auth_setup" "ok" "$MCP_AUTH_DETAIL; out=$OUT_DIR/mcp-auth-setup.out"
434+
else
435+
phase_end "notion_mcp" "mcp_auth_setup" "failed" "exit=$MCP_AUTH_RC; out=$OUT_DIR/mcp-auth-setup.out; err=$OUT_DIR/mcp-auth-setup.err"
436+
cat "$OUT_DIR/mcp-auth-setup.out" >&2 || true
437+
cat "$OUT_DIR/mcp-auth-setup.err" >&2 || true
438+
exit "$MCP_AUTH_RC"
439+
fi
440+
else
441+
phase_end "notion_mcp" "mcp_auth_setup" "skipped" "pass --compare-mcp to configure Codex MCP auth"
442+
fi
443+
320444
phase_start
321445
git -C "$REPO_DIR" fetch --quiet origin || true
322446
phase_end "locality" "git_fetch" "ok" "ref=$BASE_REF"

0 commit comments

Comments
 (0)