docs: detect orphaned URLs, restore deleted-tutorial redirects, restructure llms.txt (#23567)

## Why

An agent pointed at docs.aztec.network hit 404s, gave up, fell back to
stale training data, and missed `SingleUseClaim` (the right primitive).
Causes: pages deleted without redirects, and `llms.txt` truncating its
API index at 100 of ~1,300 entries (alphabetical, hiding everything past
`m`).

Three fixes + a CI guard so the deletion case can't recur silently.

## What changed

**1. Orphan-URL detection** — `docs/scripts/check_orphaned_urls.sh`
(new)
- Flags any baseline URL that no longer resolves **and** has no
`netlify.toml` redirect (inverse of `validate_redirect_targets.sh`).
- Handles `*` splat / `:name` placeholders. `FAIL_ON_ORPHAN=1` = hard
gate (default warn-only).

**2. Restored redirects** — `docs/netlify.toml`
- 301s for 5 previously-404 pages (4 deleted in #16788, + bare
`/developers/getting_started`), e.g. `private_voting_contract` →
`state_variables` (where `SingleUseClaim` lives).

**3. llms.txt restructure** — `docs/scripts/append_api_docs_to_llms.js`
- Spec-compliant root: single H1 + `>` summary, `llms-full.txt` link up
top, droppable `## Optional` section
([awesome-aztec](https://github.com/AztecProtocol/awesome-aztec)).
- Hub-and-spoke (Cloudflare/Next/Svelte pattern): root `## API
Reference` is now 2 links, not a ~286-module dump (root API content 84
KB → <1 KB).
- Full per-symbol index moves to scoped files an on-task agent fetches:
`aztec-nr-api/mainnet/llms.txt` (~286 modules, structs/traits/fns inline
— `SingleUseClaim` stays grep-able) and
`typescript-api/mainnet/llms.txt`. Each is its own single-H1 doc.
`llms-full.txt` unchanged.
- Fail-loud sentinel: build aborts if the rustdoc index collapses below
floors (≥50 modules, ≥100 item names; currently 286/662), so a future
nargo HTML change can't silently gut the symbol index.
- Build: bare `docusaurus` → `yarn docusaurus` in `package.json` +
`clean.sh` (use repo-pinned binary, not global PATH).

**4. CI guard** — `docs/bootstrap.sh` +
`docs/snapshots/published-urls.txt`
- `check_orphaned_urls` runs on every docs PR with `FAIL_ON_ORPHAN=1`.
Catches accidental deletion-without-redirect (the incident class above).
- Baseline is read from the **base branch** (`git show
origin/$GITHUB_BASE_REF:…`), not the working tree, so a PR can't weaken
its own gate by editing the snapshot. (This PR introduces the file, so
it falls back to the working-tree snapshot for itself; tamper-resistance
applies to every PR after merge.)
- `refresh_url_snapshot.sh` only grows the protected set (registering
newly-published pages); retiring a page needs just the redirect, not a
snapshot edit.

## Test plan
- [x] `validate:redirects` + `validate:api-ref-links` clean
- [x] `check_orphaned_urls.sh`: 162 resolved, 0 orphaned; flags
synthetic deletions
- [x] Full `yarn build`: root single-H1 + `llms-full.txt` link + `## API
Reference` (2 dynamic-version links) + `## Optional`; scoped indexes
single-H1, `SingleUseClaim` present, 9 TS package links
- [x] No-API build: only `## Optional`, single H1, no scoped files
- [x] Sentinel fires (exit 1) when the rustdoc index is forced below the
floor
- [x] Base-branch baseline sourcing verified (falls back to working-tree
snapshot for this introducing PR, as designed)
- [x] `refresh_url_snapshot.sh` idempotent; Codex reviewed
- [x] [Netlify
preview](https://deploy-preview-23567--aztec-docs-dev.netlify.app): all
5 redirects 301 to targets that resolve 200; `/llms.txt`,
`/llms-full.txt`, `/aztec-nr-api/mainnet/llms.txt`,
`/typescript-api/mainnet/llms.txt` all serve 200; scoped module links
resolve

Refs: #16788, #17168, #23572 (scoped-index split, merged here).

Author: critesjosh

docs/bootstrap.sh

@@ -53,6 +53,54 @@ function test {
   test_cmds | parallelize
 }
 
+function check_orphaned_urls {
+  echo_header "Check orphaned URLs"
+  # Source the baseline from the base branch, not the working tree, so a PR
+  # cannot weaken its own gate by deleting URLs from the snapshot in the same
+  # diff. The committed snapshot only grows the protected set for future PRs;
+  # the check a PR must satisfy is the base-branch version it cannot edit.
+  local snapshot=snapshots/published-urls.txt
+  local base_ref="${GITHUB_BASE_REF:-next}"
+  local tmp
+  tmp=$(mktemp)
+  # shellcheck disable=SC2064
+  trap "rm -f '$tmp'" RETURN
+
+  # Resolve the base branch first, distinguishing "base unreachable" from "base
+  # reachable but snapshot absent". Only the latter is a legitimate fallback (it
+  # happens for the PR that first introduces the snapshot). Treating an
+  # unreachable base as a fallback would silently reopen the bypass.
+  local base_obj=""
+  if git rev-parse --verify -q "origin/${base_ref}" >/dev/null; then
+    base_obj="origin/${base_ref}"
+  elif git fetch -q origin "$base_ref" 2>/dev/null; then
+    base_obj="FETCH_HEAD"
+  fi
+
+  local baseline
+  if [[ -n "$base_obj" ]]; then
+    if git show "${base_obj}:docs/${snapshot}" >"$tmp" 2>/dev/null; then
+      echo "Using base-branch baseline (${base_obj})."
+      baseline="$tmp"
+    else
+      echo "Base branch ${base_ref} has no snapshot yet; using working-tree snapshot (introducing PR)."
+      baseline="$snapshot"
+    fi
+  elif [[ "${CI:-0}" -eq 1 ]]; then
+    # In CI the base must be reachable; downgrading to the PR's own editable
+    # snapshot would reopen the bypass this gate exists to close.
+    echo "ERROR: cannot resolve base branch origin/${base_ref} to source the orphan-check baseline." >&2
+    return 1
+  else
+    echo "Base branch origin/${base_ref} unavailable; using working-tree snapshot (local run)."
+    baseline="$snapshot"
+  fi
+
+  local rc=0
+  FAIL_ON_ORPHAN=1 ./scripts/check_orphaned_urls.sh "$baseline" || rc=$?
+  return $rc
+}
+
 function check_references {
   if [[ "${GITHUB_EVENT_NAME:-}" != "merge_group" ]]; then
     echo "Skipping doc reference check (only runs in merge queue)."
@@ -81,11 +129,13 @@ case "$cmd" in
     build_examples
     build_docs
     test
+    check_orphaned_urls
     check_references
     ;;
   "")
     build_examples
     build_docs
+    check_orphaned_urls
     check_references
     ;;
   "hash")

docs/docs-developers/ai_tooling.md

@@ -22,12 +22,13 @@ For Claude Code, create a `CLAUDE.md` file in your project root. For Codex, crea
 ```markdown
 # Aztec Project
 
-## Critical: Use `aztec` CLI, not `nargo` directly
+## Critical: Use the `aztec` CLI, not `nargo` or `bb` directly
 
-This is an Aztec smart contract project. Always use the `aztec` CLI wrapper instead of calling `nargo` directly:
+This is an Aztec smart contract project. Always use the `aztec` CLI wrapper instead of calling `nargo` or `bb` (the Barretenberg prover) directly:
 
 - **Compile**: `aztec compile` (NOT `nargo compile`). Using `nargo compile` alone produces incomplete artifacts.
 - **Test**: `aztec test` (NOT `nargo test`).
+- **Prove**: NEVER call `bb` directly. Proof generation is handled for you by the PXE through the `aztec` CLI and `aztec.js`. There is no contract-development workflow that runs `bb` by hand.
 - **Other nargo commands** like `aztec-nargo fmt` and `aztec-nargo doc` are fine to use directly. The Aztec installer exposes the bundled `nargo` as `aztec-nargo`; bare `nargo` resolves to your own install (if any), not the bundled one.
 
 ## Error Handling
@@ -57,7 +58,7 @@ This prevents the two most common AI mistakes: using `nargo compile`/`nargo test
 
 ### Why this matters
 
-LLMs have extensive training data for `nargo` (the standalone Noir compiler) but limited exposure to the `aztec` CLI wrapper. Without explicit instructions, they default to `nargo compile`, which produces artifacts missing the AVM transpilation step.
+LLMs have extensive training data for `nargo` (the standalone Noir compiler) and `bb` (the Barretenberg prover CLI) but limited exposure to the `aztec` CLI wrapper. Without explicit instructions, they default to `nargo compile` (which produces artifacts missing the AVM transpilation step) or reach for `bb` to generate proofs. In an Aztec project, compilation and proving both go through the `aztec` tooling.
 
 ## MCP servers
 

docs/docs-words.txt

@@ -441,4 +441,5 @@ reentrancy
 variadic
 Wonderland
 indistinguishability
+llmstxt
 nightlies

docs/llms.txt

@@ -0,0 +1,14 @@
+# Aztec Protocol Documentation
+
+> Privacy-centric zkRollup for Ethereum. This repository builds the developer documentation site; the canonical `llms.txt` is auto-generated from it and published on the docs site.
+
+The published, always-current LLM-friendly files live on the docs site:
+
+- [llms.txt](https://docs.aztec.network/llms.txt): navigational index of the documentation, following the [llmstxt.org](https://llmstxt.org) standard.
+- [llms-full.txt](https://docs.aztec.network/llms-full.txt): the complete documentation as a single file.
+
+These are regenerated on every docs build, so prefer them over this checked-in pointer for the current API surface and guides.
+
+## AI tooling
+
+- [AI Tooling guide](https://docs.aztec.network/developers/ai_tooling): recommended `CLAUDE.md` / `AGENTS.md` instructions, MCP servers, and skills for Aztec and Noir development. Read this first when setting up an AI coding agent for an Aztec project.

docs/netlify.toml

@@ -425,6 +425,37 @@
     from = "/developers/docs/tutorials/faceid_wallet"
     to = "/developers/docs/tutorials/contract_tutorials/counter_contract"
 
+# =============================================================================
+# ACTIVE REDIRECTS: Removed contract tutorials (PR #16788)
+# These pages were deleted without a redirect; LLMs trained on older versions
+# of the docs still link to them. Send each to the closest current page.
+# =============================================================================
+[[redirects]]
+    from = "/developers/docs/tutorials/contract_tutorials/private_voting_contract"
+    to = "/developers/docs/aztec-nr/framework-description/state_variables"
+    status = 301
+
+[[redirects]]
+    from = "/developers/docs/tutorials/contract_tutorials/crowdfunding_contract"
+    to = "/developers/docs/tutorials/contract_tutorials/token_contract"
+    status = 301
+
+[[redirects]]
+    from = "/developers/docs/tutorials/contract_tutorials/nft_contract"
+    to = "/developers/docs/aztec-nr/standards/aip-721"
+    status = 301
+
+[[redirects]]
+    from = "/developers/docs/tutorials/contract_tutorials/write_accounts_contract"
+    to = "/developers/docs/aztec-js/how_to_create_account"
+    status = 301
+
+# Bare /developers/getting_started landed on no page after the local/testnet split
+[[redirects]]
+    from = "/developers/getting_started"
+    to = "/developers/getting_started_on_local_network"
+    status = 301
+
 # Legacy network paths - SPECIFIC PATHS MUST COME BEFORE WILDCARDS
 # (Netlify processes redirects top-to-bottom, first match wins)
 [[redirects]]

docs/package.json

@@ -3,20 +3,21 @@
   "version": "0.0.0",
   "private": true,
   "scripts": {
-    "build": "yarn clean && yarn preprocess && yarn spellcheck && yarn preprocess:move && yarn validate:redirects && yarn validate:api-ref-links && docusaurus build && node scripts/augment_sitemap.js && node scripts/append_api_docs_to_llms.js",
+    "build": "yarn clean && yarn preprocess && yarn spellcheck && yarn preprocess:move && yarn validate:redirects && yarn validate:api-ref-links && yarn docusaurus build && node scripts/augment_sitemap.js && node scripts/append_api_docs_to_llms.js",
     "validate:redirects": "./scripts/validate_redirect_targets.sh",
     "validate:api-ref-links": "./scripts/validate_api_ref_links.sh",
+    "check:orphaned-urls": "./scripts/check_orphaned_urls.sh",
     "clean": "./scripts/clean.sh",
     "dev:netlify": "yarn netlify dev",
     "generate:aztec-nr-api": "./scripts/aztec_nr_docs_generation/generate_aztec_nr_docs.sh",
     "generate:typescript-api": "./scripts/typescript_api_generation/generate_ts_api_docs.sh",
     "generate:node-api-reference": "./scripts/node_api_reference_generation/generate_node_api_reference.sh",
     "preprocess": "yarn node -r dotenv/config ./src/preprocess/index.js",
     "preprocess:move": "sh scripts/move_processed.sh",
-    "serve": "docusaurus serve",
+    "serve": "yarn docusaurus serve",
     "spellcheck": "yarn cspell",
     "spellcheck:appendwords": "yarn cspell --words-only --unique | sort --ignore-case >> docs-words.txt",
-    "start": "yarn clean && yarn preprocess && yarn generate:aztec-nr-api && docusaurus start --host ${HOST:-localhost}",
+    "start": "yarn clean && yarn preprocess && yarn generate:aztec-nr-api && yarn docusaurus start --host ${HOST:-localhost}",
     "test": "node --test src/preprocess/__tests__/*.test.js",
     "test:preprocess": "node --test src/preprocess/__tests__/*.test.js"
   },