Skip to content

Commit 7877c26

Browse files
donutdanieldonutdaniel
committed
bump cargo version and cleanup benchmarks
1 parent 4180009 commit 7877c26

41 files changed

Lines changed: 3552 additions & 737 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎BENCHMARKS.md‎

Lines changed: 5 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -21,16 +21,16 @@ universal claim about every CLI vs MCP integration model.
2121
## Method
2222

2323
- Snapshot date: `2026-03-24 11:43:27 EDT`
24-
- Commit: `fc958aab7db662167c3adef4b016ceaf93155616`
24+
- Code state: local uncommitted worktree during the benchmark run
2525
- Machine: `Apple M4 Max`
2626
- OS: `macOS 26.2 (25C56)`
2727
- Runner: `hyperfine 1.20.0`
2828
- CLI binary: `./target/release/notioncli`
2929
- MCP server: `https://mcp.notion.com/mcp`
3030
- Orchestrator: `scripts/bench_strengthen.py`
31-
- Raw artifacts: `tmp/bench-publish-20260324/`
31+
- Raw artifacts: `benchmarks/publish-20260324/`
3232
- Reported metric: median of session medians from the exported `hyperfine` JSON
33-
files in `tmp/bench-publish-20260324/`
33+
files in `benchmarks/publish-20260324/`
3434
- Backend dependency: live Notion REST API and hosted MCP service
3535
- Transient retry policy: `bench_case_once.py --retries 2 --retry-delay-ms 250`
3636
- Comparison mode:
@@ -53,9 +53,8 @@ universal claim about every CLI vs MCP integration model.
5353
- Runs per case: `10`
5454
- Warmup per case: `2`
5555

56-
This snapshot is stricter than the older mixed comparable suite because it
57-
separates reads from writes, isolates mutating cases, and repeats the suite
58-
across multiple sessions.
56+
This snapshot separates reads from writes, isolates mutating cases, and repeats
57+
the suite across multiple sessions.
5958

6059
## Read Suite
6160

@@ -103,4 +102,3 @@ across multiple sessions.
103102
CLI mode would also change the comparison.
104103
- A different machine, network path, time of day, or Notion-side load can move
105104
these numbers around.
106-
- The older mixed comparable-suite snapshot is superseded by this one.

‎Cargo.lock‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[package]
22
name = "notioncli"
3-
version = "0.1.0"
3+
version = "0.1.1"
44
edition = "2024"
55
rust-version = "1.85"
66
description = "A Rust CLI for the Notion API"

‎README.md‎

Lines changed: 5 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -280,53 +280,32 @@ python3 scripts/notion_mcp_login.py
280280
python3 scripts/mcp_list_tools.py --schemas
281281
```
282282

283-
Built-in timer for quick spot checks:
284-
285-
```bash
286-
cp scripts/bench_cases.sample.json /tmp/bench_cases.json
287-
PAGE_ID="<page-id>" python3 scripts/bench_compare.py \
288-
--case-file /tmp/bench_cases.json --runs 20 --warmup 3
289-
```
290-
291283
Supported external benchmark:
292284

293285
```bash
294286
python3 scripts/bench_strengthen.py --root-parent-id "<page-id>"
295287
```
296288

297-
Legacy raw `hyperfine` runner for ad hoc suite development:
298-
299-
```bash
300-
eval "$(python3 scripts/bench_prepare_comparable.py)"
301-
CASE_FILE=scripts/bench_cases.comparable.json scripts/hyperfine_compare.sh
302-
python3 scripts/bench_cleanup_comparable.py
303-
```
304-
305289
Benchmark scripts:
306290

307291
| Script | Purpose |
308292
|---|---|
309293
| `notion_mcp_login.py` | MCP OAuth flow, stores local token |
310294
| `mcp_list_tools.py` | List MCP tools and schemas |
311295
| `bench_case_once.py` | Run one benchmark case |
312-
| `bench_compare.py` | Compare CLI vs MCP from a JSON case file |
313-
| `bench_prepare_comparable.py` | Create disposable fixtures (`eval` the output) |
314-
| `bench_cleanup_comparable.py` | Trash fixture container |
296+
| `bench_prepare_fixture.py` | Create disposable fixtures |
297+
| `bench_cleanup_fixture.py` | Trash fixture container |
315298
| `bench_strengthen.py` | Repeated suites with fixture isolation |
316-
| `hyperfine_compare.sh` | External `hyperfine`-based comparison |
299+
| `hyperfine_compare.sh` | `hyperfine` runner used by `bench_strengthen.py` |
317300

318-
Case files: `bench_cases.sample.json`, `bench_cases.comparable.json`,
319-
`bench_cases.readsuite.json`, `bench_cases.writesuite.json`,
320-
`bench_cases.readonly.json`.
301+
Case files: `bench_cases.readsuite.json`, `bench_cases.writesuite.json`.
321302

322303
Notes:
323304

324-
- `bench_compare.py` reports both `reused_session` and `fresh_session` MCP timings
325-
- `bench_compare.py` re-expands env placeholders per run, so `$BENCH_NONCE` works in write cases
326305
- `hyperfine_compare.sh` uses bounded transient retries by default (`BENCH_CASE_RETRIES=2`)
327306
- `NOTION_MCP_ACCESS_TOKEN` overrides the stored MCP token
328307
- Case files support `$ENV_VAR` placeholders
329-
- `bench_strengthen.py` is the supported path for checked-in results; the comparable suite is a legacy mixed-case harness
308+
- `bench_strengthen.py` is the supported path for checked-in results
330309

331310
## Notes
332311

‎benchmarks/publish-20260324/read/session-01/current-user.json‎

Lines changed: 156 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,156 @@
1+
{
2+
"results": [
3+
{
4+
"command": "cli",
5+
"mean": 0.41962979462000005,
6+
"stddev": 0.2028182550970922,
7+
"median": 0.32538593432000007,
8+
"user": 0.06990764,
9+
"system": 0.018722620000000002,
10+
"min": 0.28366195432,
11+
"max": 1.1213750383199999,
12+
"times": [
13+
0.32735320532000006,
14+
0.6027459973200001,
15+
0.31993708032000007,
16+
0.30523128832,
17+
0.30265962132,
18+
0.32148837132,
19+
0.30034783032000006,
20+
1.1213750383199999,
21+
0.28366195432,
22+
0.29606449732000006,
23+
0.43690803832,
24+
0.34339737132000003,
25+
0.34512203832000005,
26+
0.6393531223200001,
27+
0.56952516332,
28+
0.28845862132000005,
29+
0.5896412883200001,
30+
0.32341866332,
31+
0.30934591332000005,
32+
0.36656078832000005
33+
],
34+
"memory_usage_byte": [
35+
30457856,
36+
30457856,
37+
30457856,
38+
30457856,
39+
30474240,
40+
30474240,
41+
30474240,
42+
30474240,
43+
30474240,
44+
30474240,
45+
30474240,
46+
30474240,
47+
30474240,
48+
30474240,
49+
30556160,
50+
30556160,
51+
30556160,
52+
30556160,
53+
30556160,
54+
30556160
55+
],
56+
"exit_codes": [
57+
0,
58+
0,
59+
0,
60+
0,
61+
0,
62+
0,
63+
0,
64+
0,
65+
0,
66+
0,
67+
0,
68+
0,
69+
0,
70+
0,
71+
0,
72+
0,
73+
0,
74+
0,
75+
0,
76+
0
77+
]
78+
},
79+
{
80+
"command": "mcp",
81+
"mean": 0.8838547049700001,
82+
"stddev": 0.2360441574024102,
83+
"median": 0.8322424338200001,
84+
"user": 0.06260244000000001,
85+
"system": 0.01682382,
86+
"min": 0.74138441332,
87+
"max": 1.86565237132,
88+
"times": [
89+
0.76082749632,
90+
0.80306120532,
91+
0.84977495532,
92+
1.86565237132,
93+
0.82581820532,
94+
0.8468239133200001,
95+
0.74138441332,
96+
0.8566225383200001,
97+
0.7890333713200001,
98+
0.7783884553200001,
99+
0.86111991332,
100+
0.79286303832,
101+
0.8269409963200001,
102+
0.84751262132,
103+
0.8991457473200001,
104+
0.9469689133200001,
105+
0.8916624133200001,
106+
0.8290086633200001,
107+
0.8306034133200001,
108+
0.83388145432
109+
],
110+
"memory_usage_byte": [
111+
36585472,
112+
36585472,
113+
36585472,
114+
36585472,
115+
36585472,
116+
36585472,
117+
36585472,
118+
36585472,
119+
36585472,
120+
36585472,
121+
36585472,
122+
36585472,
123+
36585472,
124+
36585472,
125+
36585472,
126+
36585472,
127+
36585472,
128+
36585472,
129+
36585472,
130+
36585472
131+
],
132+
"exit_codes": [
133+
0,
134+
0,
135+
0,
136+
0,
137+
0,
138+
0,
139+
0,
140+
0,
141+
0,
142+
0,
143+
0,
144+
0,
145+
0,
146+
0,
147+
0,
148+
0,
149+
0,
150+
0,
151+
0,
152+
0
153+
]
154+
}
155+
]
156+
}

0 commit comments

Comments
 (0)