CI: Unify pipeline into single main.yml with composite actions and include config for production and testing.

Author: dylan-sutton-chavez

.github/README.md

@@ -0,0 +1,71 @@
+# Edge Python CI/CD
+
+One workflow, [`main.yml`](workflows/main.yml), drives the whole monorepo, so the Actions tab shows a single "CI / CD" run per push/PR with every job as a node in one graph. Each package's logic lives in a **composite action** under [`actions/`](actions); `main.yml` only wires the dependency graph. The composite actions are not workflows and do not appear in the Actions tab.
+
+`compiler-check`, `runtime-lint` and `cli-lint` start at t=0. If any job fails the dependents never run (`needs:`), so a red build stops the deploys. `docs-build` runs on every event so a PR that breaks the docs is caught early; the deploy chain (`cdn → demo → docs-deploy`) is gated to `main` and pinned to the production branch. The `host` and `std` matrices use `fail-fast: false` so one capability / package failure still reports the others. `cli-lint` runs clippy + check once, then the heavy per-target `cli-release` build runs; `cli-test` waits on `host`, `std`, and the release artifacts.
+
+## Composite actions
+
+| Action | Inputs | Role |
+|--------|--------|------|
+| `compiler` | `mode: check\|build` | check: `cargo shear` + clippy (host and wasm targets). build: build + optimize `compiler_lib.wasm`, test, upload the artifact (and attach it to the GitHub Release on tags) |
+| `runtime` | `mode: lint\|test` | lint: `deno lint runtime/`. test: Deno + Playwright suite (Chromium driving `createWorker` against the CDN wasm) |
+| `host` | `capability` | Deno-lints and smoke-tests one capability (`dom`, `network`, `storage`, `time`) in headless Chromium. All JS, no release |
+| `std` | `package` | Clippy + build + optimize + corpus test for one stdpkg (`json`, `re`, `math` as wasm; `test` is pure Edge Python, so it skips the wasm build and only runs the corpus). Stages `<pkg>.wasm` / `<pkg>.py`. No release |
+| `cli` | `mode: lint\|release\|test`, `target` | lint: `cargo clippy -D warnings` + `cargo check` (once). release: `cargo build --release` per target → tarball artifact. test: `cargo test` (drives a real Chromium) |
+| `demo` | CF token + account | Hashes deps into `version.json` (cache-busting), builds Tailwind, deploys `demo/` to `edge-python-demo` |
+| `docs` | `mode: build\|deploy`, CF token + account | build: `npm ci` + `next build` static export (`docs/out`, sitemap via `postbuild`), upload artifact. deploy: pull artifact + push to `edge-python-docs` |
+| `cdn-deploy` | CF token + account | Pulls every artifact, stages `./compiler ./runtime ./std ./host ./cli`, one `wrangler pages deploy` to `edge-python-cdn` |
+
+## Cloudflare Pages
+
+Three **Direct Upload** projects. Actions push prebuilt directories via `wrangler pages deploy`; Cloudflare doesn't clone or build.
+
+| Project | Source | Production URL |
+|---------|--------|----------------|
+| `edge-python-cdn` | `_site/{compiler,runtime,std,host,cli}` (consolidates the old per-package `-runtime` / `-host` / `-std` projects) | `https://edge-python-cdn.pages.dev` |
+| `edge-python-demo` | `demo/` (wasm hashed for `version.json`, not bundled) | `https://edge-python-demo.pages.dev` |
+| `edge-python-docs` | `docs/out` (Nextra static export) | `https://edgepython.com` (custom domain; also `https://edge-python-docs.pages.dev`) |
+
+All deploys run **only on pushes to `main`** and are pinned to the production `main` branch. PRs and tags never deploy; the next `main` push refreshes the projects.
+
+### Cloudflare and GitHub setup
+
+```bash
+# Wrangler CLI (Node 22+)
+npx wrangler login
+npx wrangler pages project create edge-python-cdn --production-branch=main
+npx wrangler pages project create edge-python-demo --production-branch=main
+npx wrangler pages project create edge-python-docs --production-branch=main
+```
+
+`edge-python-docs` serves `edgepython.com` (replacing the old Mintlify docs): after
+the first deploy, add `edgepython.com` as a custom domain on the project
+(Pages -> Custom domains) and remove it from Mintlify.
+
+Repo secrets (*Settings -> Secrets and variables -> Actions*):
+
+- `CLOUDFLARE_API_TOKEN`, `Account -> Cloudflare Pages -> Edit`. Create via dashboard: <https://dash.cloudflare.com/profile/api-tokens>.
+- `CLOUDFLARE_ACCOUNT_ID`, from `npx wrangler whoami` or any dashboard sidebar.
+
+Rotate: create new token -> update secret -> revoke old token.
+
+## Releases
+
+Pushing a `v*` tag runs the pipeline; the `compiler` build job uploads `compiler_lib.wasm` to the matching Release. Tag must match workspace version.
+
+1. Bump `version` under `[workspace.package]` in root `Cargo.toml` (every crate inherits via `version.workspace = true`). Run `cargo check` to refresh `Cargo.lock`, commit.
+2. Tag and push:
+
+```bash
+git tag v0.1.0
+git push origin v0.1.0
+```
+
+On tag push: `compiler-check` lints, then the `compiler` build job optimizes the artifact and attaches it to a fresh Release with auto-generated notes. The CDN, demo and docs deploys do not run on tags; they already deployed from the preceding `main` push.
+
+Nothing is published to crates.io, distribution is the `.wasm` on the Release. `starter-module` carries its own version and isn't bumped with the workspace.
+
+Consumer crates pick up the release automatically: `compiler/Cargo.toml` declares `links = "compiler_lib"` and `compiler/build.rs` downloads `<repository>/releases/download/v<version>/compiler_lib.wasm` into `OUT_DIR`. Downstreams read `DEP_COMPILER_LIB_WASM` in their own `build.rs`, see [root README](../../README.md#consume-the-release-from-a-rust-host). Tag bumps flow via `cargo update`.
+
+Gated behind the default-on `prebuilt` feature. Producer-side compiler steps pass `--no-default-features` to avoid fetching the asset that this same pipeline uploads later.

.github/actions/cdn-deploy/action.yml

@@ -0,0 +1,65 @@
+name: CDN deploy
+description: Stage every package output under one tree and deploy it to edge-python-cdn.
+
+inputs:
+  cloudflare-api-token:
+    description: Cloudflare API token.
+    required: true
+  cloudflare-account-id:
+    description: Cloudflare account id.
+    required: true
+
+runs:
+  using: composite
+  steps:
+    - name: Download compiler wasm
+      uses: actions/download-artifact@v8
+      with:
+        name: compiler_lib_wasm
+        path: /tmp/compiler/
+
+    - name: Download std artifacts
+      uses: actions/download-artifact@v8
+      with:
+        pattern: dist-*
+        path: /tmp/std/
+        merge-multiple: true
+
+    - name: Download cli artifacts
+      uses: actions/download-artifact@v8
+      with:
+        pattern: edge-*
+        path: /tmp/cli/
+        merge-multiple: true
+
+    - name: Stage site
+      shell: bash
+      run: |
+        mkdir -p _site/compiler _site/runtime _site/std _site/host _site/cli
+
+        # compiler: the wasm module.
+        cp /tmp/compiler/compiler_lib.wasm _site/compiler/
+
+        # runtime: JS sources plus the wasm it bundles.
+        cp -r runtime/. _site/runtime/
+        cp /tmp/compiler/compiler_lib.wasm _site/runtime/
+
+        # std: <pkg>.wasm / <pkg>.py.
+        cp -r /tmp/std/. _site/std/
+
+        # host: flatten each capability's src/ into _site/host/<cap>/.
+        for dir in host/*/src; do
+          cap="$(basename "$(dirname "$dir")")"
+          mkdir -p "_site/host/$cap"
+          cp -r "$dir"/. "_site/host/$cap/"
+        done
+
+        # cli: release tarballs.
+        cp -r /tmp/cli/. _site/cli/
+
+    - name: Deploy to Cloudflare Pages
+      uses: cloudflare/wrangler-action@v4
+      with:
+        apiToken: ${{ inputs.cloudflare-api-token }}
+        accountId: ${{ inputs.cloudflare-account-id }}
+        command: pages deploy _site --project-name=edge-python-cdn --branch=main

.github/actions/cli/action.yml

@@ -0,0 +1,89 @@
+name: CLI
+description: cli/ is its own workspace. lint = clippy + check; release = build per target; test = cargo test.
+
+inputs:
+  mode:
+    description: "lint | release | test"
+    required: true
+  target:
+    description: Rust target triple (release mode).
+    required: false
+    default: ""
+
+runs:
+  using: composite
+  steps:
+    - name: Toolchain (lint)
+      if: inputs.mode == 'lint'
+      uses: dtolnay/rust-toolchain@stable
+      with:
+        components: clippy
+
+    - name: Cache Rust (lint)
+      if: inputs.mode == 'lint'
+      uses: Swatinem/rust-cache@v2
+      with:
+        workspaces: cli -> cli/target
+
+    - name: Clippy and check
+      if: inputs.mode == 'lint'
+      shell: bash
+      working-directory: cli
+      run: |
+        cargo clippy --all-targets -- -D warnings
+        cargo check
+
+    - name: Toolchain (release)
+      if: inputs.mode == 'release'
+      uses: dtolnay/rust-toolchain@stable
+      with:
+        targets: ${{ inputs.target }}
+
+    - name: Install musl tools
+      if: inputs.mode == 'release' && contains(inputs.target, 'linux-musl')
+      shell: bash
+      run: sudo apt-get update && sudo apt-get install -y musl-tools
+
+    - name: Cache Rust (release)
+      if: inputs.mode == 'release'
+      uses: Swatinem/rust-cache@v2
+      with:
+        workspaces: cli -> cli/target
+        key: ${{ inputs.target }}
+
+    - name: Build release
+      if: inputs.mode == 'release'
+      shell: bash
+      working-directory: cli
+      run: cargo build --release --target "${{ inputs.target }}"
+
+    - name: Tar
+      if: inputs.mode == 'release'
+      shell: bash
+      working-directory: cli
+      run: tar -C "target/${{ inputs.target }}/release" -czf "edge-${{ inputs.target }}.tar.gz" edge
+
+    - name: Upload release artifact
+      if: inputs.mode == 'release'
+      uses: actions/upload-artifact@v6
+      with:
+        name: edge-${{ inputs.target }}
+        path: cli/edge-${{ inputs.target }}.tar.gz
+        retention-days: 1
+
+    - name: Toolchain (test)
+      if: inputs.mode == 'test'
+      uses: dtolnay/rust-toolchain@stable
+
+    - name: Cache Rust (test)
+      if: inputs.mode == 'test'
+      uses: Swatinem/rust-cache@v2
+      with:
+        workspaces: cli -> cli/target
+
+    # Drives a real Chromium; ubuntu-latest ships google-chrome-stable.
+    - name: Test
+      if: inputs.mode == 'test'
+      shell: bash
+      working-directory: cli
+      run: cargo test

.github/actions/compiler/action.yml

@@ -0,0 +1,139 @@
+name: Compiler
+description: Rust compiler crate. check = shear + clippy; build = wasm build/opt + test + upload.
+
+inputs:
+  mode:
+    description: "check | build"
+    required: true
+  github-token:
+    description: Token for the tag Release upload (build mode, tags only).
+    required: false
+    default: ""
+
+runs:
+  using: composite
+  steps:
+    - name: Cache Rust
+      uses: Swatinem/rust-cache@v2
+
+    - name: Toolchain (clippy)
+      if: inputs.mode == 'check'
+      uses: dtolnay/rust-toolchain@stable
+      with:
+        components: clippy
+
+    - name: Install cargo-shear
+      if: inputs.mode == 'check'
+      shell: bash
+      run: cargo install cargo-shear
+
+    # Detects declared but unused dependencies across the workspace.
+    - name: Shear
+      if: inputs.mode == 'check'
+      shell: bash
+      run: cargo shear
+
+    # --no-default-features skips the prebuilt wasm download in build.rs.
+    - name: Clippy (host)
+      if: inputs.mode == 'check'
+      shell: bash
+      run: cargo clippy --all-targets --no-default-features -- -D warnings
+
+    - name: Install wasm target (check)
+      if: inputs.mode == 'check'
+      shell: bash
+      run: rustup target add wasm32-unknown-unknown
+
+    - name: Clippy (wasm)
+      if: inputs.mode == 'check'
+      shell: bash
+      run: cargo clippy --lib --target wasm32-unknown-unknown -p edge-python -p slugify-mod -- -D warnings
+
+    # build-std needs nightly plus rust-src.
+    - name: Toolchain (nightly)
+      if: inputs.mode == 'build'
+      uses: dtolnay/rust-toolchain@nightly
+      with:
+        components: rust-src
+
+    - name: Install wasm target (build)
+      if: inputs.mode == 'build'
+      shell: bash
+      run: rustup target add wasm32-unknown-unknown
+
+    # apt ships an old binaryen, so fetch the upstream release.
+    - name: Install wasm-opt
+      if: inputs.mode == 'build'
+      shell: bash
+      run: |
+        curl -sSL https://github.com/WebAssembly/binaryen/releases/download/version_121/binaryen-version_121-x86_64-linux.tar.gz \
+          | tar -xz --strip-components=2 -C /usr/local/bin binaryen-version_121/bin/wasm-opt
+        wasm-opt --version
+
+    - name: Build
+      if: inputs.mode == 'build'
+      shell: bash
+      run: |
+        RUSTFLAGS="-Z location-detail=none -Z fmt-debug=none -Z unstable-options -C panic=immediate-abort" \
+        cargo +nightly build \
+          --target wasm32-unknown-unknown \
+          --lib \
+          --release \
+          -p edge-python \
+          -Z build-std=std,panic_abort
+
+    - name: Size (unoptimized)
+      if: inputs.mode == 'build'
+      shell: bash
+      run: ls -lh target/wasm32-unknown-unknown/release/compiler_lib.wasm
+
+    # Two passes: -Oz with traps-never-happen, then reflatten for a fresh CFG.
+    - name: Optimize
+      if: inputs.mode == 'build'
+      shell: bash
+      run: |
+        INPUT=target/wasm32-unknown-unknown/release/compiler_lib.wasm
+
+        wasm-opt -Oz --converge \
+          --generate-global-effects \
+          --strip-debug --strip-producers \
+          --enable-bulk-memory-opt \
+          --enable-nontrapping-float-to-int \
+          --enable-sign-ext \
+          -tnh \
+          -o /tmp/wasm_stage1.wasm "$INPUT"
+
+        wasm-opt --flatten --rereloop -Oz -Oz \
+          --enable-bulk-memory-opt \
+          --enable-nontrapping-float-to-int \
+          --enable-sign-ext \
+          -o "$INPUT" /tmp/wasm_stage1.wasm
+
+        rm /tmp/wasm_stage1.wasm
+
+    - name: Size (optimized)
+      if: inputs.mode == 'build'
+      shell: bash
+      run: ls -lh target/wasm32-unknown-unknown/release/compiler_lib.wasm
+
+    - name: Test
+      if: inputs.mode == 'build'
+      shell: bash
+      run: cargo test -p edge-python --no-default-features
+
+    - name: Upload wasm artifact
+      if: inputs.mode == 'build'
+      uses: actions/upload-artifact@v6
+      with:
+        name: compiler_lib_wasm
+        path: target/wasm32-unknown-unknown/release/compiler_lib.wasm
+        retention-days: 1
+
+    - name: Upload WASM Release
+      if: inputs.mode == 'build' && startsWith(github.ref, 'refs/tags/')
+      uses: softprops/action-gh-release@v2
+      with:
+        token: ${{ inputs.github-token }}
+        files: target/wasm32-unknown-unknown/release/compiler_lib.wasm
+        fail_on_unmatched_files: true
+        generate_release_notes: true