Merge branch 'main' into chore/sentry-dashboard

Author: EnderOfWorlds007

.github/workflows/ci.yml

@@ -18,3 +18,15 @@ jobs:
                   node-version: "22"
             - run: pnpm install
             - run: pnpm format:check
+
+    unit-tests:
+        name: Unit Tests
+        runs-on: ubuntu-latest
+        steps:
+            - uses: actions/checkout@v4
+            - uses: pnpm/action-setup@v4
+            - uses: actions/setup-node@v4
+              with:
+                  node-version: "22"
+            - run: pnpm install
+            - run: pnpm test

.github/workflows/e2e.yml

@@ -0,0 +1,47 @@
+name: E2E Tests
+
+on:
+    push:
+        branches: [main]
+    pull_request:
+    # Daily at 06:00 UTC — catches testnet regressions
+    schedule:
+        - cron: "0 6 * * *"
+    workflow_dispatch:
+
+jobs:
+    e2e:
+        name: E2E Tests
+        runs-on: ubuntu-latest
+        timeout-minutes: 30
+        steps:
+            - uses: actions/checkout@v4
+
+            - uses: pnpm/action-setup@v4
+
+            - uses: actions/setup-node@v4
+              with:
+                  node-version: "22"
+
+            - uses: oven-sh/setup-bun@v2
+              with:
+                  bun-version: latest
+
+            - name: Install Kubo (IPFS)
+              run: |
+                  wget -q https://dist.ipfs.tech/kubo/v0.32.1/kubo_v0.32.1_linux-amd64.tar.gz
+                  tar xf kubo_v0.32.1_linux-amd64.tar.gz
+                  sudo mv kubo/ipfs /usr/local/bin/
+                  ipfs init --profile=test
+                  ipfs version
+
+            - run: pnpm install
+
+            - name: Run E2E tests
+              run: pnpm test:e2e
+              env:
+                  TEST_TEMPLATE_DOMAIN: dot-cli-mod-fixture.dot
+                  TEST_TEMPLATE_REPO: https://github.com/paritytech/Rock-Paper-Scissors
+                  DOT_DEPLOY_VERBOSE: "1"
+                  DOT_TAG: e2e-ci
+                  DOT_TELEMETRY: "1"

.gitignore

@@ -4,3 +4,5 @@ dist/
 .cdm/
 docs-internal/
 .worktrees/
+reference-repos/
+.tokensave

AGENTS.md

@@ -19,24 +19,55 @@ Read `CLAUDE.md` alongside this file when you need the full rationale for repo-s
 
 ## Repository Conventions
 
-- Tests live next to the source as `*.test.ts`.
-- Every user-facing PR needs a changeset in `.changeset/*.md`.
+- Tests live next to the source as `*.test.ts`. `vitest.config.ts` only picks up `.test.ts`; update it before adding `.tsx` tests.
+- Pure logic embedded in `.tsx` components should be lifted into sibling `.ts` files so tests can import it without pulling React and Ink into the runner.
+- Every user-facing PR needs a changeset in `.changeset/*.md`. Refactors, tests, and internal tooling-only changes can skip one.
 - Do not add AI/tool attribution, signatures, or provenance to commits, PRs, generated files, or repo content.
 - Do not commit design docs, brainstorming notes, context dumps, or scratch files.
 - Do not mock `polkadot-api` primitives such as `Enum` or encoders in tests.
-- Long-lived resources such as session signers and chain clients must be explicitly destroyed in cleanups.
+- Long-lived resources such as `TerminalAdapter`, session signers, and chain clients must be explicitly destroyed in cleanups.
 
-## Architecture Boundaries
+## Dependency Rules
 
-- All chain URLs and contract addresses belong in `src/config.ts`.
-- `src/bootstrap.ts` is the first import and owns the ambient Sentry handoff to `bulletin-deploy`. Keep `DOT_TELEMETRY` as the privacy gate for both CLI telemetry and `BULLETIN_DEPLOY_TELEMETRY`.
+- Do not upgrade `polkadot-api` or `@polkadot-api/sdk-ink` past the current pins unless `@polkadot-apps/chain-client` is bumped and compatibility is verified.
 - Keep `bulletin-deploy` pinned to an explicit version. Do not switch it to `latest`.
+- When upgrading `bulletin-deploy`, check public API changes for `deploy()`, DotNS methods, `DeployOptions`, `jsMerkle`, signer options, RPC handling, and attributes.
+
+## Architecture Boundaries
+
+- All chain URLs and contract addresses belong in `src/config.ts`. Do not inline websocket URLs or `0x...` addresses elsewhere.
+- Signer mode selection lives in `src/utils/deploy/signerMode.ts`; keep mainnet/testnet signer changes isolated there.
 - `src/utils/deploy/*` and `src/utils/build/*` are SDK-facing surfaces for WebContainer consumers and must not import React or Ink.
 - CLI/TUI code lives under `src/commands/*`.
+- `src/bootstrap.ts` is the first import in `src/index.ts` and owns the ambient Sentry handoff to `bulletin-deploy`. Keep `DOT_TELEMETRY` as the privacy gate for both CLI telemetry and `BULLETIN_DEPLOY_TELEMETRY`.
 - The Bun compiled-binary stdin warm-up in `src/index.ts` is intentional. Do not remove it until Bun compiled TTY stdin works reliably with Ink.
 
+## Deploy And Storage Invariants
+
+- Deploy delegates storage hardening to `bulletin-deploy`: chunking, retries, pool accounts, nonce fallback, DAG-PB, and DotNS commit-reveal stay there.
+- The CLI owns `registry.publish()` because the registry contract must record the user as `env::caller()`.
+- Do not call `bulletin-deploy.deploy()` just to store playground metadata JSON. Use `@polkadot-apps/bulletin::upload()` so it submits `TransactionStorage.store` directly and returns the CID.
+- Metadata uploads need a dedicated Bulletin client with `heartbeatTimeout: 300_000`, destroyed immediately after upload.
+- `dot deploy` currently relies on the Kubo binary path and must not pass `jsMerkle: true` until bulletin-deploy's pure-JS merkleizer preserves DAG-PB directory/file blocks correctly.
+- `dot init` installs `ipfs`, so deploys can rely on the Kubo CLI after setup.
+
 ## Runtime Safety
 
+- `getSessionSigner()` returns an adapter that keeps the Node event loop alive. Every caller must call the returned `destroy()`.
 - New long-running commands should register cleanup through `onProcessShutdown()` and use the process guard where appropriate.
 - `startMemoryWatchdog()` runs for both `dot deploy` and `dot mod`; add it to new top-level commands that do meaningful I/O.
 - TUI info updates must be throttled or coalesced. Do not stream raw high-volume logs directly into Ink state.
+- `DeployLogParser.feed()` must not emit catch-all events per log line. Keep events limited to meaningful phase/progress matches.
+
+## Mod And GitHub Behavior
+
+- `dot mod` is GitHub-tarball-only. Do not reintroduce `git clone`, `gh repo fork`, or tooling requirements for the public-repo path.
+- `ensureGhAuthed()` must not shell out to `gh auth login` from inside Ink. Fail with the existing actionable message instead.
+- `metadata.repository` is written only when `--modable` is explicitly opted in.
+- When `--modable` needs a repository, deploy either pushes to existing `origin` or creates a public repo with `gh repo create --public --push`; failed pushes must fail the deploy.
+
+## Sentry Telemetry
+
+- Helpers in `src/telemetry.ts`, `src/utils/deploy/phase.ts`, `src/cli-runtime.ts`. Every command wraps with `runCliCommand`; every deploy phase wraps with `withDeployPhase`. Don't reimplement the boilerplate.
+- Dashboards are JSON snapshots in `sentry/dashboards/`. Run `./sentry/backup-dashboards.sh` before any change. Use `./sentry/patch-dashboard.py` and `./sentry/create-dashboard.py` for edits and new dashboards.
+- E2E test runs are tagged `cli.tag:e2e-ci` (CI) or `cli.tag:e2e-local` (helper default). Production widgets exclude `cli.tag:e2e-*`; the E2E Health dashboard (id 2216096) targets it.

CHANGELOG.md

@@ -1,5 +1,41 @@
 # playground-cli
 
+## 0.15.4
+
+### Patch Changes
+
+- 7151157: Avoid GitHub auth and `git push` during `dot deploy --modable` when the project already has an `origin`; the existing repository URL is recorded directly.
+
+## 0.15.3
+
+### Patch Changes
+
+- 9d0a0ba: Load the logged-in account for `dot deploy --signer dev --playground` so Playground registry publishes can be signed by the app owner.
+
+## 0.15.2
+
+### Patch Changes
+
+- f2f43c4: Suppress the non-fatal `ReviveApi_trace_call` compatibility stack during Playground registry contract dry-runs.
+
+## 0.15.1
+
+### Patch Changes
+
+- 2de1408: Resolve the Playground registry address from the live CDM meta-registry before publishing or browsing apps.
+
+## 0.15.0
+
+### Minor Changes
+
+- 86abd07: `dot --suri` now accepts a BIP-39 mnemonic in addition to the dev names (Alice, Bob, Charlie, Dave, Eve, Ferdie). An optional `//<path>` derivation suffix is supported, e.g. `dot deploy --suri "<12-word phrase>//0"`. The dev-name fast path is unchanged.
+
+### Patch Changes
+
+- 934c0db: Add E2E integration test suite covering install, build, init, session, deploy, mod, and diagnostic commands. Tests spawn the CLI as a child process via execa and assert on stdout/stderr/exit codes. Deploy tests verify contract detection for Foundry, Hardhat, and CDM backends. Includes CI workflow, fixture projects, and chain query helpers for Paseo testnet validation.
+- 21481ba: Fix `dot deploy` exiting 1 after a successful deploy. polkadot-api's `client.destroy()` can fire a `DisjointError: ChainHead disjointed` from a still-in-flight chainHead operation after the WS has closed, which surfaces as an unhandled rejection and forced the process to exit 1 even though the deploy completed and printed "Deploy complete". Now suppressed alongside the existing benign-teardown filter for `UnsubscriptionError: Not connected`.
+- ba63fec: Fix `dot deploy` and `dot mod` exiting 0 on failures. Previously the CLI's entry point unconditionally called `process.exit(0)` after the action returned, overwriting the non-zero `process.exitCode` set by `scheduleHardExit()` (deploy preflight, e.g. `SignerNotAvailableError` from a corrupt session) and never set by `dot mod` at all on `runSetup` failures (e.g. registry miss). Both paths now propagate a non-zero exit code so shell scripts and CI pipelines can rely on the result.
+
 ## 0.14.1
 
 ### Patch Changes

CLAUDE.md

@@ -24,8 +24,8 @@ These are things that aren't self-evident from reading the code and have bitten
 - **Process-guard safety net** (`src/utils/process-guard.ts`) — deploy pipelines open several long-lived WebSockets + child processes and any one of them can keep the event loop alive after the TUI visibly finishes, turning `dot` into a zombie that accumulates retry buffers indefinitely (seen climbing past 25 GB). We defend in depth: (1) `installSignalHandlers()` catches SIGINT/TERM/HUP + `unhandledRejection` and forces cleanup + exit within 3 s; (2) `scheduleHardExit()` installs an `unref`'d timer that kills the process if the event loop doesn't drain within a grace period; (3) `startMemoryWatchdog()` aborts if RSS exceeds 4 GB — a generous cap because legit deploys on Bun SEA binaries routinely touch 1–1.5 GB from runtime-metadata decoding + Bun's JSC heap + Ink yoga. Do NOT re-add a per-window growth detector: we tried 300 MB / 3 s and it false-positived on the single-burst metadata-loading spike, aborting deploys that would have succeeded. Set `DOT_MEMORY_TRACE=1` to stream per-sample RSS/heap/external stats — useful when diagnosing a real leak report. **Telemetry bootstrap** (`src/bootstrap.ts`) is the FIRST import in `src/index.ts`. It sets `BULLETIN_DEPLOY_USE_AMBIENT_SENTRY=1` and `BULLETIN_DEPLOY_HOST_APP=playground-cli` before `bulletin-deploy` can evaluate, then maps `DOT_TELEMETRY`/internal-context detection to `BULLETIN_DEPLOY_TELEMETRY`. Do not leave `BULLETIN_DEPLOY_TELEMETRY` unset while setting the host app: `bulletin-deploy@0.7.6` treats `playground-cli` as an internal host, which would enable deploy telemetry for external users. `BULLETIN_DEPLOY_MEM_REPORT` is not forced off by default anymore because upstream guards the Bun-incompatible memory-report path. Any new long-running command should register a cleanup hook via `onProcessShutdown()`.
 - **Parser MUST NOT emit an event per log line.** `DeployLogParser.feed()` is called for every console line bulletin-deploy prints — hundreds per deploy on the happy path, thousands if retries fire. We intentionally emit events ONLY for phase-banner matches and `[N/M]` chunk progress. Everything else returns `null`. Adding a catch-all `info` emit turns the parser into a firehose that allocates ~200 bytes × thousands of lines, and was a measurable contributor to chunk-upload memory pressure.
 - **`dot mod` is GitHub-tarball-only and must stay that way.** `src/utils/mod/source.ts` downloads from `codeload.github.com` (no auth, no `git`/`gh` required for the public-repo case) and extracts via `node:zlib` + the pure-JS `tar` package. Do NOT re-introduce `git clone` or `gh repo fork` paths — both would re-add a hard tooling requirement and the fork path was specifically removed because GitHub caps you to one fork per source-repo per account, which broke "mod the same starter twice." A non-modable app (no `metadata.repository`) returns a hard error from `dot mod`; the interactive picker filters those out so the user never sees an unmoddable option.
-- **`ensureGhAuthed()` does NOT shell out to `gh auth login`.** Even from the interactive deploy, Ink owns stdout + raw-mode stdin and a `stdio: "inherit"` child would race Ink's `useInput` for keystrokes — producing garbled output and dropped key events. Both interactive and non-interactive paths fail with the same actionable message: run `gh auth login` once outside `dot`, then retry. Properly suspending Ink to hand off stdio is possible but requires unmounting + re-rendering with state preservation, and we deemed that complexity not worth it for a one-time-per-machine speedbump. If you ever do implement it, the wiring point is `ModablePreflightStage` in `src/commands/deploy/DeployScreen.tsx`.
-- **`metadata.repository` is set ONLY when `--modable` is opted in.** Older code in `publishToPlayground` would silently probe `git remote get-url origin` and stuff whatever it found into the metadata, which surprised users who didn't realise their fork was being advertised. The new contract: `runDeploy` takes an explicit `repositoryUrl: string | null`, and `publishToPlayground` writes the field iff that param is non-null. The CLI command is responsible for resolving the URL upstream via `src/utils/deploy/modable.ts::resolveRepositoryUrl()`, which either pushes to existing `origin` (failing the deploy on push error — stale URLs would mislead modders) or runs `gh repo create --public --push` to set up a fresh public repo. Re-deploys never delete user repos.
+- **`ensureGhAuthed()` does NOT shell out to `gh auth login`.** Even from the interactive deploy, Ink owns stdout + raw-mode stdin and a `stdio: "inherit"` child would race Ink's `useInput` for keystrokes — producing garbled output and dropped key events. Both interactive and non-interactive paths that need GitHub repo creation fail with the same actionable message: run `gh auth login` once outside `dot`, then retry. Existing `origin` URLs do not require `gh` auth. Properly suspending Ink to hand off stdio is possible but requires unmounting + re-rendering with state preservation, and we deemed that complexity not worth it for a one-time-per-machine speedbump. If you ever do implement it, the wiring point is `ModablePreflightStage` in `src/commands/deploy/DeployScreen.tsx`.
+- **`metadata.repository` is set ONLY when `--modable` is opted in.** Older code in `publishToPlayground` would silently probe `git remote get-url origin` and stuff whatever it found into the metadata, which surprised users who didn't realise their fork was being advertised. The new contract: `runDeploy` takes an explicit `repositoryUrl: string | null`, and `publishToPlayground` writes the field iff that param is non-null. The CLI command is responsible for resolving the URL upstream via `src/utils/deploy/modable.ts::resolveRepositoryUrl()`, which uses an existing `origin` URL without pushing, or runs `gh repo create --public --push` to set up a fresh public repo when no origin exists. Re-deploys never delete user repos.
 - **`startMemoryWatchdog()` runs for both `dot deploy` and `dot mod`.** Mod's tarball download is a streaming pipe through `node:zlib` + `tar.extract()`, and a stuck IPFS gateway or a malformed tarball can leak buffers. Same 4 GB cap, same worker-thread sampler. Any new top-level command that does meaningful I/O should also call `startMemoryWatchdog()` and register `stopWatchdog` via `onProcessShutdown()`.
 
 ## Repo conventions
@@ -37,3 +37,17 @@ These are things that aren't self-evident from reading the code and have bitten
 - Do NOT commit design docs, brainstorming notes, or context dumps (e.g. `context.md`) to the repo. They belong in tickets or scratch files outside the tree.
 - Don't mock primitives from `polkadot-api` (`Enum`, encoders) in tests — doing so turns intended coverage into tautology.
 - Long-lived resources (`TerminalAdapter`, `PaseoClient`) have explicit `destroy()` / `destroyConnection()` — always release them, especially from React `useEffect` cleanups. The WebSocket keeps the event loop alive; forgetting a destroy manifests as `dot <cmd>` hanging after its work is visibly finished.
+
+## Sentry telemetry
+
+- DSN: `src/telemetry-config.ts::PLAYGROUND_SENTRY_DSN`. Region: EU (`https://de.sentry.io`).
+- Org slug: `paritytech`. API token: macOS keychain service `sentry-api-token` (member of paritytech org with `org:read` + `org:write`).
+- Attribute prefix: `cli.` (see `getCliRootAttributes` in `src/telemetry-config.ts`). Spec: `sentry-instrumentation-spec.md` at the repo root (untracked — keep there).
+- **Helpers (don't reimplement):** `src/telemetry.ts` exports `withCommandTelemetry`, `withRootSpan`, `withSpan` (2-arg + 4-arg overloads), `captureWarning`, `captureException`, `errorMessage`, `sanitizedErrorMessage`. `src/utils/deploy/phase.ts` exports `withDeployPhase` for deploy-phase orchestration. `src/cli-runtime.ts` exports `runCliCommand` for the standard CLI scaffolding (telemetry + watchdog + hard-exit). Every command's `.action()` body should be one `runCliCommand(name, options, async () => { ... })` call — do not re-add try/finally + `scheduleHardExit` boilerplate.
+- **Dashboards** live as JSON snapshots under `sentry/dashboards/<id>.json`:
+  - `2143100.json` — **Playground CLI Health** (production filter `!cli.tag:e2e-*`).
+  - `2216067.json` — **Playground CLI Failures** (per-error-type drill-downs).
+  - `2216096.json` — **Playground CLI E2E Health** (inverse filter, `cli.tag:e2e-*`).
+- **Workflow:** run `./sentry/backup-dashboards.sh` BEFORE any change. Use `./sentry/patch-dashboard.py <id> <patch.json>` for surgical edits (supports `replace`, `patch_query`, `set_description` ops) or full widget replacement. Use `./sentry/create-dashboard.py <payload.json>` for new dashboards. Per spec §15f, do NOT include a `projects` field in POST payloads. Per spec §15g, PUT replaces the whole widget list — backup first.
+- **E2E tagging:** every spawn from `e2e/cli/helpers/dot.ts` injects `DOT_TAG=e2e-local` (default) and `DOT_TELEMETRY=1`. CI sets `DOT_TAG=e2e-ci` in `.github/workflows/e2e.yml` so production health widgets filter cleanly via `!cli.tag:e2e-*`.
+- **SAD% propagation** is verified by a regression test in `src/telemetry.test.ts` ("SAD% propagation through transaction envelope"). It confirms `captureWarning` flips `cli.sad="true"` on the root transaction. If that test fails, the SAD% dashboard widget on Dashboard 1 will silently degrade to a duplicate of the unexpected-failure rate.