Commit bc4a2f4
docs: migrate README into Starlight docs site (website/) on GitHub Pages (#64)
* docs: migrate README into Starlight docs site (website/) on GitHub Pages
Split the 877-line README into a searchable Starlight (Astro) docs site
living in website/, deployed to GitHub Pages via a path-filtered workflow.
- Starlight 0.39 + Astro 6, base /nasde-toolkit/, Pagefind search (built-in),
llms.txt (starlight-llms-txt), mermaid (astro-mermaid), and
starlight-links-validator as a build-time broken-link gate.
- 25 content pages across Getting Started / Concepts / Reference / Guides,
migrated faithfully from README sections + docs/use-cases.md +
docs/benchmark-results.md. ADR/ARCHITECTURE/RELEASING stay internal in the
repo and are linked out to GitHub.
- Slimmed 877 → 108 lines: branding, four-steps, quick-start, skills table,
prominent link to the docs site. Everything else lives only on the site
(single source of truth). docs/use-cases.md + docs/benchmark-results.md
become pointer stubs. pyproject Documentation URL → live site.
- .github/workflows/docs-deploy.yml: path-filtered (website/**) push-to-main
Pages deploy, Node 20, npm ci. NOT a required check (avoids PR-wedge).
- .gitignore: website/node_modules, website/dist, website/.astro.
- Logo resized 4.99MB/2816px → 505KB/800px (sips), reused in site + README.
Validation: website build green (all internal links valid, 26 pages, mermaid
renders, llms.txt generated). Python CI intact — ruff/mypy/pytest (369) all
pass; tooling is scoped to src/+tests/ and never touches website/.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(website): brand theme, noesis logo, top-level nav, docs-first landing
UX/polish pass on the Starlight site after live review:
- **Logo**: replace the broken default favicon-as-logo with the Noesis logo
(src/assets/noesis-logo.png), shown alongside the "NASDE Toolkit" title
(no more replacesTitle, no "NASDE Toolkit | NASDE Toolkit" dup).
- **Brand theme**: src/styles/custom.css sets a teal/cyan accent from the
Noesis logo, consistent with noesis-docs, replacing Starlight's default
purple. Works in light + dark.
- **Top-level nav**: HeaderNav.astro overrides SocialIcons to add
"Documentation" + "Changelog" header links (the Docusaurus-style navbar
users expect), with active-state highlighting.
- **Landing reworked**: the splash now reads as documentation, not a product
page — hero tagline says "the official documentation", logo as hero image,
Get Started / What is NASDE? / GitHub actions, and pictogram LinkCards for
Getting Started / Concepts / Reference / Guides. The "what is NASDE / four
steps" product blurb lives in the docs (getting-started/overview), not on
the landing.
Search verified working — "calibration" returns "Calibrating the Rubric"
first; the earlier "call" result was a typo ("callibr"), not a config issue
(Pagefind has no fuzzy typo matching).
Verified live in Chrome (light + dark): logo, teal accent, nav links,
landing cards, sidebar active state, and the mermaid pipeline diagram all
render. Build green, all internal links valid, 26 pages.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(website): NASDE logo on hero, Noesis logo+favicon, fix light-mode hero contrast
Follow-up polish from live review:
- **Branding split**: NASDE product logo (nasde-toolkit-logo.png) as the hero
image on the landing; Noesis Vision logo in the navbar and as the favicon
(public/favicon.png). Product on the page, company in the chrome.
- **Title**: "Nasde Toolkit Docs" (normal case, not all-caps NASDE) in the
site title, hero, and "What is Nasde?" action.
- **Fix light-mode hero invisible title**: custom.css overrode
--sl-color-white globally but didn't set it for the light theme, so the
hero <h1> rendered white-on-white. Light theme now sets --sl-color-white
to the dark heading color (Starlight's inverted-scale convention).
Verified: hero title rgb(24,26,31) on white (light), rgb(255,255,255) on
dark; deep-page body text contrast intact in both.
- **Hero logo card**: rounded corners + soft card background so the
light-background NASDE artwork sits cleanly in dark mode.
Search note: Pagefind has no typo tolerance by design ("callibrate" won't
match "calibration") — accepted as the standard static-search trade-off;
correct spelling returns the right page first.
Verified live in Chrome, light + dark.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(website): consolidate 25 → 13 pages for balanced sidebar + filled TOC
Live review showed the imbalance vs noesis-docs: too many thin pages made the
left sidebar long (~23 items) while the right "On this page" TOC was often
empty (pages were continuous prose with no ## sections). noesis-docs does the
opposite — few dense pages, each with a rich TOC. This rebalances to match.
Approved "B-corrected" structure — 13 pages, each with a coherent flow:
- Getting Started (2): Overview · Quick Start (← Prerequisites + Installation
+ Quick Start)
- Concepts (4): How It Works (← Scoring + Pipeline, de-duplicated into one
staged narrative) · A Real Task · Token & Cost · Calibrating the Rubric
- Reference (3): CLI Reference (← Cheatsheet + Commands) · Configuration
(← Project Structure + variant.toml + task.toml) · Authentication & Opik
(← Authentication + Verifying Opik)
- Guides (4): Running & Configuring Runs (← Local Repo + Cloud + Reviewer +
Exporting) · Plugins & Skills (← Plugin + Skill-by-ref + Scoping) ·
Use Cases · Benchmark Results
Every page now has ## sections, so the right-hand TOC fills on all of them
(Token & Cost and Calibration also got headings added). All internal links
and anchors rewritten to the merged slugs.
Verified live in Chrome: short sidebar + populated TOC on calibration and
how-it-works; mermaid renders in How It Works. Build green, all internal
links valid, 13 doc pages.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(website): newcomer-first IA pass — 13→18 pages, agent/reviewer symmetry, glossary, boundaries
Information-architecture review (from a "never heard of NASDE" perspective,
one subagent pass + first-time-user walkthrough) surfaced gaps beyond the
earlier polish. This addresses them and finalizes the skeleton.
## Structural fixes
- **Agent ↔ reviewer symmetry (the big one)**: running-benchmarks.md now has a
"Configuring the agent under test" section (instructions / skills / MCP /
reasoning effort / scoping) mirroring the reviewer section, with a tip box
spelling out the parallel. Configuring the agent under test was previously
scattered across three pages.
- **General → specific**: Overview opens with "Why NASDE? — the problem" (the
hook) and adds "What NASDE is — and is not" (boundaries: not a CI
replacement, not production, needs Docker/subscription) before the mechanics.
## New pages (5)
- getting-started/reading-results — the run summary table, jobs/ layout,
interpreting assessment_summary.json, agent-noise vs judge-noise.
- concepts/key-terms — glossary (variant, trial, job, rubric, dimension,
reviewer, trajectory, Harbor, Opik, ...).
- creating-benchmarks/ (NEW GROUP): anatomy (what a benchmark is made of +
task-files mermaid) and assessment-criteria (how to write dimensions +
score ladders — the core value).
- guides/troubleshooting — Docker/auth/OOM(137)/flaky-eval/rate-limit +
what-to-expect (time, cost, trials) + FAQ.
## Images / diagrams
- How It Works: per-dimension radar (existing asset) at the scoring section.
- Configuration + Anatomy: "what each task file does" mermaid.
- Calibration: mermaid loop flowchart (measure→diagnose→fix→re-measure).
- Token & Cost: section retitled "Quality vs. cost: the Pareto frontier",
value-led rewrite + image placeholder (pareto.png — TODO, user generates).
- Reading Your Results: placeholder for a real `nasde run` screenshot (TODO).
## Reference
- Configuration: "Quick reference: configuring a variant" checklist at top.
Sidebar: 18 pages across 5 groups (Getting Started · Concepts · Creating
Benchmarks · Reference · Guides). Build green, all internal links valid,
20 pages, 4 mermaid diagrams render.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(website): fix attempts/repetitions accuracy + add real Pareto & CLI images
Accuracy pass after a fact-check (the troubleshooting/results content was
partly written from project memory; this verifies it against the code and
corrects the parts that conflated two distinct knobs).
- **Fix attempts vs eval-repetitions conflation** (troubleshooting + reading-
results): `--attempts`/`-n` is the number of independent AGENT runs per task
(the source of the between-trial `mean ±std`); `--eval-repetitions` is the
number of REVIEWER passes per trial (judge noise). These were blurred into
"trials/repetitions" and "run × repetitions". Now stated precisely, with the
two noise sources mapped to where each appears. (Verified against cli.py:
--attempts/-n and --eval-repetitions are separate flags; timeout_sec 1800,
memory_mb 4096 confirmed in CLAUDE.md/scaffold.)
- **Real images replace placeholders**:
- concepts/token-cost.md — the actual quality-vs-cost / quality-vs-tokens
Pareto chart from a real skill×model matrix, with a caption explaining the
shared cost panel + per-provider token panels.
- getting-started/reading-results.md — the real `nasde run` startup banner;
the section is reworded so the prose matches what the screenshot actually
shows (the config banner) vs. the end-of-run summary table (described in
text, since that's a separate screen).
Build green, all internal links valid, images optimized to webp.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(website): accuracy audit fixes + pricing honesty + heading/pager styling
Systematic technical-accuracy audit of every page against the source code
(cli.py, config.py, runner/evaluator/results_exporter, scaffold). Fixes all
findings; the core claims (effort priority, rubric fingerprint, cost formula,
eval clustering, CLI set, scaffold defaults) audited as correct.
WRONG (a user would hit these):
- reading-results: trial dir is `verifier/` (reward.txt + test-stdout.txt),
not `logs/verifier/` (that's the in-container path test.sh writes to).
- configuration + running-benchmarks: `[nasde.source]` shown as JSON; it's a
TOML table in task.toml. Pasting the JSON would break the file.
- use-cases: the per-task config file is `task.toml`, not `task.json` (×3).
IMPRECISE:
- cli-reference: run options table was missing --all-variants, --attempts/-n,
--eval-repetitions, --max-concurrent-eval, --job-suffix. Added.
- authentication: Codex auth checks file *presence* (sets CODEX_FORCE_AUTH_JSON),
not `auth_mode:"chatgpt"`. Gemini Vertex env vars corrected to the ones the
code actually checks (GOOGLE_API_KEY / GOOGLE_APPLICATION_CREDENTIALS, not
GOOGLE_CLOUD_PROJECT). Opik feedback-score list completed (_std, eval_n).
- configuration: nasde.toml sample now shows eval_repetitions + project_name.
- calibration: .calibration/ list now includes assessment_summary.json.
Pricing honesty:
- token-cost: softened "edit pricing.toml" — the catalog is bundled, so after
a PyPI install editing it is impractical (wiped on upgrade); editing works
from a source checkout, and a per-project/user override is flagged as
planned. (load_pricing() is called with no path at both call-sites, so no
override is wired today — tracked as a follow-up.)
Styling:
- Inline `code` in headings scaled to 0.85em (was oversized).
- Previous/Next pager link titles toned down to h5.
Build green, all internal links valid, verified live (light + dark).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(website): shrink pager link titles to body size
The Previous/Next pager titles (.link-title) were still ~24px — louder than
the page's own text. Target .link-title specifically (the earlier rule hit
the wrong span) and set it to body size (--sl-text-base, weight 600), so the
pager sits quietly under its Previous/Next labels. Verified live.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(website): make the right-hand TOC less cramped
The "On this page" panel read smaller than the left sidebar: its heading was
13px (below its own 14px links) and the rows were tight. Bump the
starlight-toc heading to --sl-text-sm and relax the link rows (line-height +
padding) so the TOC is as legible as the left nav. Verified live.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(website): bump navbar Documentation/Changelog links to base size
The header nav links were --sl-text-sm, reading small next to the rest of the
navbar. Bump to --sl-text-base so they sit as first-class navigation. Verified
live.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(website): rewrite Token & Cost's two dense sections in plain language
Two sections read like expert shorthand — dense, jargon-heavy, assuming the
reader already knows the two noise sources and prompt-cache mechanics. Rewritten
for a newcomer who's never used NASDE.
- "A mean is never reported bare" → "Why scores come with a ± (and why that
matters)". Opens with the concrete question (is A really better than B, or
did it get lucky?), explains "wobble" before using it, splits the two noise
sources into bolded bullets, drops the dense parenthetical jargon.
- "How cost is computed" → leads with what it means for the user (consistent,
comparable cost), explains prompt caching in plain terms and why ignoring it
keeps comparisons fair — instead of stating the billing rule tersely.
No factual changes — same behavior, clearer prose with emphasis on key terms.
Verified live.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(website): document non-interactive mode + Anthropic programmatic-use credit
NASDE drives every agent non-interactively (claude -p / codex exec / Gemini CLI
equivalent) — verified in code (claude_subprocess.py uses `claude -p`; the
agent under test runs headless via Harbor). Document this clearly, since it's a
real current limitation (interactive mode is planned) and it touches Anthropic's
programmatic-use terms.
Per Anthropic's announcement, from 2026-06-15 paid Claude plans include a
dedicated monthly credit for programmatic usage (covering `claude -p`, the
Agent SDK, Claude Code GitHub Actions) — so running NASDE on a paid plan is
supported, not restricted. Framed accurately (a credit, not a block) with a
link to Anthropic's terms.
Added in three places:
- Overview → new "How NASDE drives the agents" subsection under the boundaries.
- Authentication & Opik → a note box at the top.
- Troubleshooting FAQ → "interactive mode?" and "programmatic-use?" Q&As.
Verified live, build green, links valid.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs: fix README links to docs site after the 25→18 page consolidation
The README's Documentation section linked to pre-consolidation slugs that now
404 on the live site:
- concepts/scoring → concepts/how-it-works
- reference/commands → reference/cli-reference
- getting-started/installation (merged into quick-start) → dropped
README lives outside website/, so the build's link-validator never caught
these. Verified every README docs link now resolves to an existing page.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(website): cover ADR-012 — native Codex/Gemini skill injection + frontmatter rule
After rebasing onto main (ADR-012, #65/66/67), the docs needed two additions to
stay accurate with how Codex/Gemini skills now work:
- Plugins & Skills: new "How skills reach each agent" section — Claude
discovers from /app/.claude/skills/, while Codex/Gemini auto-discover only
from a HOME-scoped dir ($HOME/.agents/skills, ~/.gemini/skills), where NASDE
now routes them natively (covers agents_skills/, [[skill]], and plugin skills).
- A caution box on the strict requirement: a Codex/Gemini SKILL.md must START
with a `---` frontmatter line, or the loader silently drops the skill (a
leading comment above the frontmatter is the common trap); NASDE warns at run
time. Cross-linked from Configuration.
Our docs never named the old wrong /app/.agents/skills destination, so nothing
was contradicted — these are additions, not corrections. Build green, links valid.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Szymon Janikowski <szymon.janikowski@itlibrium.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>1 parent cf306c9 commit bc4a2f4
51 files changed
Lines changed: 10005 additions & 1253 deletions
File tree
- .github/workflows
- docs
- website
- .vscode
- public
- src
- assets
- benchmark
- components
- content/docs
- concepts
- creating-benchmarks
- getting-started
- guides
- reference
- styles
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | | - | |
4 | | - | |
5 | | - | |
6 | | - | |
7 | | - | |
8 | | - | |
9 | | - | |
10 | | - | |
11 | | - | |
12 | | - | |
13 | | - | |
14 | | - | |
15 | | - | |
16 | | - | |
17 | | - | |
18 | | - | |
19 | | - | |
20 | | - | |
21 | | - | |
22 | | - | |
23 | | - | |
24 | | - | |
25 | | - | |
26 | | - | |
27 | | - | |
28 | | - | |
29 | | - | |
30 | | - | |
31 | | - | |
32 | | - | |
33 | | - | |
34 | | - | |
35 | | - | |
36 | | - | |
37 | | - | |
38 | | - | |
39 | | - | |
40 | | - | |
41 | | - | |
42 | | - | |
43 | | - | |
44 | | - | |
45 | | - | |
46 | | - | |
47 | | - | |
48 | | - | |
49 | | - | |
50 | | - | |
51 | | - | |
52 | | - | |
53 | | - | |
54 | | - | |
55 | | - | |
56 | | - | |
57 | | - | |
58 | | - | |
59 | | - | |
60 | | - | |
61 | | - | |
62 | | - | |
63 | | - | |
64 | | - | |
65 | | - | |
66 | | - | |
67 | | - | |
68 | | - | |
69 | | - | |
70 | | - | |
71 | | - | |
72 | | - | |
73 | | - | |
74 | | - | |
75 | | - | |
76 | | - | |
77 | | - | |
78 | | - | |
79 | | - | |
80 | | - | |
81 | | - | |
82 | | - | |
83 | | - | |
84 | | - | |
85 | | - | |
86 | | - | |
87 | | - | |
88 | | - | |
89 | | - | |
90 | | - | |
91 | | - | |
92 | | - | |
93 | | - | |
94 | | - | |
95 | | - | |
96 | | - | |
97 | | - | |
98 | | - | |
99 | | - | |
100 | | - | |
101 | | - | |
102 | | - | |
103 | | - | |
104 | | - | |
105 | | - | |
106 | | - | |
107 | | - | |
108 | | - | |
109 | | - | |
110 | | - | |
111 | | - | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
0 commit comments