diff --git a/.claude/README.md b/.claude/README.md new file mode 100644 index 000000000..7c334d7c3 --- /dev/null +++ b/.claude/README.md @@ -0,0 +1,55 @@ +# Agent harness + +This directory makes the template **agent-manageable** with Claude Code. It ships skills, subagents, +workflows, and a permission allowlist for the recurring maintenance jobs this project needs. + +Because the template is consumed as a **rolling master** (forks/seeds track branches, not releases), +keeping the harness *in the repo* means projects seeded from it **inherit agent-driven management for +free**. Existing projects copy `.claude/` once (see +[CONTRIBUTING.md → Agent harness](../CONTRIBUTING.md#agent-harness)). + +## Contents + +| Path | What | +| --- | --- | +| [`settings.json`](settings.json) | Allowlist for the read-mostly `make` / `docker` / `git` / `gh` / `composer validate` calls these jobs use, to cut permission prompts. **`make:*` is allowed intentionally** — `make` is this project's primary interface (it includes `make clean`, so review destructive runs). | +| `skills/` | User-invokable runbooks (`/`) wrapping existing `make` targets | +| `agents/` | Subagent definitions used by the skills/workflows | +| `workflows/` | Deterministic multi-step Workflow scripts for the heavy jobs | + +## Skills (`/`) + +- [`review-app-triage`](skills/review-app-triage/SKILL.md) — read pipeline + Behat/Lighthouse/watchdog + output and localize failures. +- [`drupal-upgrade`](skills/drupal-upgrade/SKILL.md) — execute the + [upgrade runbook](../docs/upgrading.md) (D10 → 11 → 12). +- [`dep-bump`](skills/dep-bump/SKILL.md) — bump a dependency; NewRelic agent is the first-class case. +- [`backlog-grooming`](skills/backlog-grooming/SKILL.md) — triage open PRs/issues, regenerate + [docs/backlog.md](../docs/backlog.md). +- [`observability-up`](skills/observability-up/SKILL.md) — bring the logs/traces stack up and print the + Grafana URL. + +## Agents + +- [`upgrade-validator`](agents/upgrade-validator.md) — runs `upgradestatusval` + `drupalrectorval`, + reports blockers (read/much-analysis). +- [`ci-log-analyzer`](agents/ci-log-analyzer.md) — read-only triage of CI / container logs. +- [`dep-bumper`](agents/dep-bumper.md) — edits a single dependency pin and validates. + +## Workflows + +Run via the Workflow tool (or `/workflows`). Opt-in — they only run when invoked. + +- [`drupal-upgrade.workflow.js`](workflows/drupal-upgrade.workflow.js) — bump → remove dropped modules + → rector → `upgrade_status` → iterate, in an isolated worktree. +- [`newrelic-bump.workflow.js`](workflows/newrelic-bump.workflow.js) — bump the pinned NewRelic agent + version and validate. + +## Conventions + +- Project context lives in this repo's [`CLAUDE.md`](../CLAUDE.md) and [`docs/`](../docs/). The + parent-directory `CLAUDE.md` (a "Plasma" doc) is **unrelated** — don't rely on it here. +- Skills wrap **existing `make` targets** rather than reimplementing logic; keep that contract so the + harness stays correct as the Makefile evolves. +- Heavy/parallel jobs run in **git worktrees** (see [parallel-environments.md](../docs/parallel-environments.md)) + so multiple branches can build at once without colliding. diff --git a/.claude/agents/ci-log-analyzer.md b/.claude/agents/ci-log-analyzer.md new file mode 100644 index 000000000..6c0c00925 --- /dev/null +++ b/.claude/agents/ci-log-analyzer.md @@ -0,0 +1,35 @@ +--- +name: ci-log-analyzer +description: Read-only triage of CI pipeline output and Docker/Drupal logs for a failing review app. Finds the first real error, the failing job/target, and the likely root cause. Use when a pipeline or local stack is red and you need the cause, not a fix. +tools: Bash, Read, Grep, Glob +--- + +You are a read-only log triager for the skilld-docker-container review-app pipeline. You find root +causes; you do not edit code. + +Sources you can read: +- GitLab job logs (if given a URL/ID, use `gh`/`glab` or the provided text) and JUnit artifacts + (`junit/*.xml`). +- A running stack: `docker compose logs --tail=300 php`, `docker compose ps`, + `make drush -- watchdog:show --severity=Error --count=100`. +- The repo, to map a failing job to its `make` target via docs/ci-pipeline.md. + +Method: +1. Identify the failing **stage/job** and its underlying `make` target. +2. Find the **first** error (not the cascade after it). For Behat, read the JUnit failure message and + the matching screenshot: Behat writes PNGs to `features/` (FailAid `screenshot.directory` in + `behat.default.yml`); in CI the `test:behat` job moves them to `web/screenshots/` and serves them on + the review URL. +3. Correlate with recent changes (`git log --oneline -15`, `git diff` against last green) to point at a + suspect file/commit. +4. Note environment-specific causes: SAPI mismatch (Unit vs FrankenPHP), DB engine, basic-auth on the + review URL, missing CI variable (REVIEW_DOMAIN, runner tag, RA_BASIC_AUTH). + +Report: +- Failing job + `make` target. +- The exact first error (quoted). +- Most-likely root cause and the file/commit to look at. +- A reproduction command (`make ` locally). + +Do not propose code edits beyond naming the file/line; do not run destructive commands. If logs are +insufficient, say what additional log/artifact is needed. diff --git a/.claude/agents/dep-bumper.md b/.claude/agents/dep-bumper.md new file mode 100644 index 000000000..6b8345ff9 --- /dev/null +++ b/.claude/agents/dep-bumper.md @@ -0,0 +1,29 @@ +--- +name: dep-bumper +description: Mechanically bumps a single dependency pin (composer package, Docker image tag, or the NewRelic agent version) and validates it. Use for one-at-a-time, reviewable dependency updates. +tools: Bash, Read, Edit, Grep, Glob +--- + +You apply one dependency bump at a time for the skilld-docker-container template and validate it. + +Given a target (package + version, image tag, or "latest NewRelic agent"): +1. Locate the pin: + - composer package → `composer.json` `require`/`require-dev`. + - PHP image → `IMAGE_PHP` in `.env.default` (and the CI default in `.gitlab-ci.yml`). + - NewRelic agent → the version string in `scripts/makefile/newrelic.sh`. +2. For composer, check feasibility first: `composer why-not `. +3. Make the **minimal** edit (one pin). +4. Validate: + - composer: `composer validate` then `composer update --with-dependencies` (in the php + container via `make` if a stack is up). + - image/agent: confirm the tag/version exists, then `make provision reload` (image) or + `make newrelic reload` (agent) and smoke-check. +5. Report the diff, the validation output, and whether it's safe to commit. + +Constraints: +- Exactly one dependency per run; keep the diff small and reviewable. +- Never add a committed patch file (`extra.patches` must reference upstream URLs only — `make patchval` + enforces this). +- If the bump pulls a Drupal major (D11/D12), stop and defer to the `drupal-upgrade` skill — don't do + a core jump as a "dependency bump". +- Don't bump unrelated packages to make resolution work; report the conflict instead. diff --git a/.claude/agents/upgrade-validator.md b/.claude/agents/upgrade-validator.md new file mode 100644 index 000000000..85032eaee --- /dev/null +++ b/.claude/agents/upgrade-validator.md @@ -0,0 +1,30 @@ +--- +name: upgrade-validator +description: Runs the Drupal upgrade validation gates (upgrade_status + rector + config schema) and reports blockers with file-level detail. Use during a Drupal 10→11→12 upgrade to check whether the current tree is ready. +tools: Bash, Read, Grep, Glob +--- + +You validate Drupal upgrade-readiness for the skilld-docker-container template. You do not change +production code; you run the project's own gates and report precisely. + +Run, in this order, and capture full output: +1. `make upgradestatusval` — the authority. It enables `upgrade_status`, runs + `drush upgrade_status:analyze --all --ignore-contrib --ignore-uninstalled`, and fails on any `FILE:` + line. Extract each reported file + deprecation. +2. `make drupalrectorval` — `rector` dry-run over `web/modules/custom` + `web/themes/custom` using + `rector.php`. List each suggested change (these are mostly auto-fixable in custom code). +3. `make cinsp` — config schema errors (run before anything that uninstalls modules). + +Then report: +- **Blockers**: deprecations `upgrade_status` flags in custom code, grouped by file, each with the + deprecated API and the D11/D12 replacement. +- **Auto-fixable**: what rector can rewrite (note: `vendor/bin/rector process` would apply them). +- **Contrib/core gaps**: anything pointing at contrib or core constraints (feed back to the upgrade + skill — e.g. a module that needs a D11 release, or the druxxy profile gate). +- **Verdict**: READY / NOT READY, with the shortest path to green. + +Constraints: +- Don't edit `composer.json` or run `composer update` — that's the upgrade skill's job; you only + validate the current state. +- If a gate can't run because the stack isn't installed, say so and stop — don't guess. +- Quote real output; never claim green without the target exiting 0. diff --git a/.claude/settings.json b/.claude/settings.json new file mode 100644 index 000000000..f01b8b00b --- /dev/null +++ b/.claude/settings.json @@ -0,0 +1,25 @@ +{ + "$schema": "https://json.schemastore.org/claude-code-settings.json", + "permissions": { + "allow": [ + "Bash(make:*)", + "Bash(docker compose:*)", + "Bash(docker ps:*)", + "Bash(docker logs:*)", + "Bash(docker inspect:*)", + "Bash(git status:*)", + "Bash(git log:*)", + "Bash(git diff:*)", + "Bash(git branch:*)", + "Bash(git worktree:*)", + "Bash(gh pr list:*)", + "Bash(gh pr view:*)", + "Bash(gh pr diff:*)", + "Bash(gh issue list:*)", + "Bash(gh issue view:*)", + "Bash(composer validate:*)", + "Bash(composer show:*)", + "Bash(composer why-not:*)" + ] + } +} diff --git a/.claude/skills/backlog-grooming/SKILL.md b/.claude/skills/backlog-grooming/SKILL.md new file mode 100644 index 000000000..5f9c9c4e7 --- /dev/null +++ b/.claude/skills/backlog-grooming/SKILL.md @@ -0,0 +1,38 @@ +--- +name: backlog-grooming +description: Triage the open GitHub PRs and issues — classify into adopt-now / rebase / fold-into-docs / close, dedupe, and regenerate docs/backlog.md. Use for "groom the backlog", "triage issues", "what should we do with the open PRs". +--- + +# Backlog grooming + +Keep [docs/backlog.md](../../../docs/backlog.md) current and the queue actionable. Read-only against +GitHub except for editing `docs/backlog.md`. + +## Steps + +1. **Enumerate** (read-only): + - `gh pr list --state open --limit 100` + - `gh issue list --state open --limit 200` +2. **Inspect** the substantive ones: `gh pr view --comments`, `gh pr diff --stat`, + `gh issue view --comments`. For each, note: what it does, files touched, staleness, and whether + master already solved it. +3. **Classify** into the buckets used in `docs/backlog.md`: + - **adopt-now** — small, low-risk, still applies. + - **rebase & finish** — valuable but stale/conflicting. + - **fold into docs** — no code needed; the answer is documentation. + - **close** — obsolete, superseded, or abandoned. +4. **Dedupe** — link PRs to their issues (e.g. an "auto shm size" PR and its issue) and collapse. +5. **Cross-check against the repo**: before recommending adopt, confirm the change isn't already in + master (`git log`, current `composer.json`/`Makefile`). Several issues are already resolved (e.g. + #467 Unit default, #341 shm size). +6. **Regenerate** `docs/backlog.md` with the updated table and a suggested order. Keep the markdown + table shape. + +## Output +- Updated `docs/backlog.md`. +- A short summary: counts per bucket, and the top 3 recommended actions. + +## Guardrails +- Don't close/comment on GitHub from here — produce the recommendation; a human (or an explicit + follow-up) acts on it. +- Note any PR that conflicts with in-flight work (e.g. #403 k3s vs the worktree helper). diff --git a/.claude/skills/dep-bump/SKILL.md b/.claude/skills/dep-bump/SKILL.md new file mode 100644 index 000000000..ad4a6b26b --- /dev/null +++ b/.claude/skills/dep-bump/SKILL.md @@ -0,0 +1,43 @@ +--- +name: dep-bump +description: Bump a dependency and validate it. First-class case is the recurring NewRelic PHP agent version bump; also handles composer packages and Docker image tags. Use for "bump NewRelic", "update dependency X", "upgrade the php image". +--- + +# Dependency bump + +Small, validated version bumps — the steady-state maintenance this template needs most (NewRelic +agent bumps dominate its git history). + +## NewRelic agent (first-class) + +The pinned version lives in [`scripts/makefile/newrelic.sh`](../../../scripts/makefile/newrelic.sh). + +1. Find the latest agent release (NewRelic PHP agent release notes / download index). +2. Update the version string in `scripts/makefile/newrelic.sh` (and any matching version in + `.env.default` / docker config if present). +3. Validate: `composer validate`, and if a stack is up, `make newrelic reload` then confirm the agent + loads (`make drush -- php-eval "var_dump(extension_loaded('newrelic'));"` or check + `docker compose logs php`). +4. Commit with the release tag in the message (matches the repo's existing + `NR: ` convention — see `git log`). + +For an unattended run use the [`newrelic-bump` workflow](../../workflows/newrelic-bump.workflow.js). + +## Composer package + +1. `composer why-not ` to see blockers. +2. Edit the constraint in `composer.json`; `composer update --with-dependencies`. +3. `composer validate` + `make tests` (or at least `make cinsp upgradestatusval`). +4. If it's a Drupal contrib bump for D11, route through [`drupal-upgrade`](../drupal-upgrade/SKILL.md). + +## Docker PHP image + +`IMAGE_PHP` in `.env.default` (and the CI default in `.gitlab-ci.yml`). Confirm the tag exists +(`skilldlabs/php` tags), bump, `make provision reload`, smoke-test the site. + +## Guardrails +- One dependency per change; keep the diff reviewable. +- Never commit a local patch file (`test:patch` forbids it) — patches go to upstream URLs in + `composer.json`'s `extra.patches`. +- Delegate the mechanical edit+validate to the [`dep-bumper`](../../agents/dep-bumper.md) subagent when + doing several. diff --git a/.claude/skills/drupal-upgrade/SKILL.md b/.claude/skills/drupal-upgrade/SKILL.md new file mode 100644 index 000000000..78da5c2d6 --- /dev/null +++ b/.claude/skills/drupal-upgrade/SKILL.md @@ -0,0 +1,44 @@ +--- +name: drupal-upgrade +description: Move the template toward Drupal 11 (transitional ^10.3 || ^11) or stage Drupal 12. Bumps core constraints, removes modules dropped from core, updates rector, and iterates to a clean upgrade_status. Use for "upgrade to Drupal 11", "D11 compatibility", "prepare D12". +--- + +# Drupal upgrade + +Execute the runbook in [docs/upgrading.md](../../../docs/upgrading.md). This skill is the interactive +entry point; for an unattended, isolated run use the +[`drupal-upgrade` workflow](../../workflows/drupal-upgrade.workflow.js). + +## Before you start — check the gate +The default profile `skilldlabs/druxxy` gates D11. Verify first: + +```sh +composer show skilldlabs/druxxy --all # is there a release allowing drupal/core ^11? +``` + +As of the last check, the newest druxxy (`v1.5.1`) only allows `^10.6` — **no D11 yet**. If still +blocked, either obtain a D11-capable druxxy major or temporarily set `PROFILE_NAME=standard` on a +throwaway branch to validate the rest. + +## Steps (transitional D11) + +1. Create/checkout a throwaway branch (ideally an isolated worktree — + [parallel-environments.md](../../../docs/parallel-environments.md)). +2. `composer.json`: core scaffold/hardening → `^10.3.1 || ^11`; bump druxxy when available. +3. Remove `drupal/ckeditor`, `drupal/color`, `drupal/seven` (dropped from D11 core). Switch the admin + theme to **claro**, rely on core **CKEditor 5**. `git grep` for stragglers first. +4. Bump `palantirnet/drupal-rector` (current `^0.20.3` is old → `^0.21`/`1.0.x`); add `Drupal11SetList` + to `rector.php` **only if** the installed release ships it. +5. Re-verify or drop the `drupal/default_content` patch in `composer.json`. +6. `composer update --with-all-dependencies`, then run the gates and iterate: + `make upgradestatusval` (authority) → `make drupalrectorval` → `make cinsp` → `make tests`. + Delegate the run/parse loop to the [`upgrade-validator`](../../agents/upgrade-validator.md) subagent. +7. Open an MR; the definition of done is a **green D11 review app**. + +## D12 (staged, separate branch) +Switch `IMAGE_PHP` to a PHP 8.4 image (`skilldlabs/php:84-unit` exists), widen constraints to include +`^12`, add the D12 rector set if present, re-run the gates. Don't fold D12 into the D11 branch. + +## Guardrails +- Never bypass `upgrade_status`/rector to "make it green" — fix the deprecations. +- Keep the change transitional (`|| ^11`) so downstream D10.3 projects keep resolving. diff --git a/.claude/skills/observability-up/SKILL.md b/.claude/skills/observability-up/SKILL.md new file mode 100644 index 000000000..dc41b0f3a --- /dev/null +++ b/.claude/skills/observability-up/SKILL.md @@ -0,0 +1,41 @@ +--- +name: observability-up +description: Get logs and traces out of the running stack. Tier 1 is Docker logs (no extra services); Tier 2 brings up the opt-in OpenTelemetry trace stack (Tempo + Grafana) and prints the Grafana URL. Use for "show me the logs", "enable tracing", "bring up observability". +--- + +# Observability up + +Implements [docs/observability.md](../../../docs/observability.md). + +## Tier 1 — logs (always available) + +```sh +docker compose logs -f php # access + PHP error log (stdout/stderr) +make drush -- watchdog:tail # live Drupal log +make watchdogval # fail on Emergency|Alert|Critical|Error +``` + +Mail: open Mailpit at `mail-${MAIN_DOMAIN_NAME}`. This is the default answer to "get me the logs". + +## Tier 2 — traces (opt-in OpenTelemetry) + +Trace tooling comes from PR #466 (Tempo + Grafana + the OTel PHP extensions). Steps: + +1. Ensure the OTel pieces are present (rebase/land PR #466 if not): Tempo + Grafana services in + `docker/docker-compose.override.yml`, `docker/tempo.yml`, `docker/grafana-datasources.yml`, the + `php83-pecl-opentelemetry|grpc|protobuf` extensions in `ADDITIONAL_PHP_PACKAGES`, and the `OTEL_*` + env vars on the `php` service. +2. Enable the opt-in profile/flag (keep it **off by default** so plain review apps stay cheap), then + `make provision reload` (or `docker compose up -d`) to start Tempo + Grafana and install the + extensions. +3. Print the Grafana URL (via `make info` / the Traefik host label) and tell the user to open + **Explore → Tempo**. +4. Generate a trace: load a page, then search in Grafana. + +## Hosted alternative +NewRelic: set `NEW_RELIC_LICENSE_KEY`, `make newrelic reload`. Bumping the agent is the +[`dep-bump`](../dep-bump/SKILL.md) skill's job. + +## Guardrails +- Don't enable Tier 2 by default in CI/review apps — it adds two services per app. +- If PR #466 isn't landed yet, say so and offer to run the rebase rather than hand-rolling config. diff --git a/.claude/skills/review-app-triage/SKILL.md b/.claude/skills/review-app-triage/SKILL.md new file mode 100644 index 000000000..18c8ea9e1 --- /dev/null +++ b/.claude/skills/review-app-triage/SKILL.md @@ -0,0 +1,42 @@ +--- +name: review-app-triage +description: Triage a failing review app or local stack — pull pipeline, Behat (junit), Lighthouse, and watchdog output, then localize the failure and propose a fix. Use when a review app build or a test:* job is red, or when "why did the pipeline fail". +--- + +# Review-app triage + +Goal: turn a red pipeline / broken stack into a localized root cause and a concrete fix. Wraps the +project's own validation targets — see [docs/ci-pipeline.md](../../../docs/ci-pipeline.md) and +[docs/review-apps.md](../../../docs/review-apps.md). + +## Inputs +- A GitLab pipeline/job URL or ID, **or** a local stack that is misbehaving. +- The branch/MR under review. + +## Steps + +1. **Identify the failing stage.** Map the job to its `make` target using the table in + [ci-pipeline.md](../../../docs/ci-pipeline.md#job--make-target--script-map). The likely culprits: + - `build:*` red → install failed: re-run `make all_ci` (or `make all` locally) and read the first + error; usually composer resolution or `drush si`. + - `test:behat` red → read the **JUnit** artifact (`junit/*.xml`) and the screenshots (Behat writes + them to `features/`; in CI they are moved to `web/screenshots/` and served on the review URL). + Reproduce locally with `make behat`. + - `test:cinsp` / `test:upgradestatus` / `test:statusreport` / `test:watchdog` red → run the same + target locally (`make cinsp`, `make upgradestatusval`, …); each prints the offending file/config/log. + - `test:lighthouse` red → a category dropped below the `lighthouserc.yml` threshold; read + `web/lighthouseci/*.html`. +2. **Pull logs.** For a running stack: `docker compose logs --tail=200 php` and + `make drush -- watchdog:show --severity=Error --count=50`. See + [docs/observability.md](../../../docs/observability.md). +3. **Reproduce locally** in an isolated worktree if possible + ([parallel-environments.md](../../../docs/parallel-environments.md)) so you don't disturb other work. +4. **Localize** to a file/commit. Use `git log -p` on the suspect path; compare against the last green + pipeline. +5. **Report**: the failing job, the exact error, the root-cause file/line, and a proposed fix. For deep + log reading, delegate to the [`ci-log-analyzer`](../../agents/ci-log-analyzer.md) subagent. + +## Notes +- Respect the test-ordering constraint (#323): if you re-order jobs, keep `cinsp` before the + schema-mutating checks. +- Don't mark anything fixed until the relevant `make ` passes locally. diff --git a/.claude/workflows/drupal-upgrade.workflow.js b/.claude/workflows/drupal-upgrade.workflow.js new file mode 100644 index 000000000..cfe6d5055 --- /dev/null +++ b/.claude/workflows/drupal-upgrade.workflow.js @@ -0,0 +1,104 @@ +export const meta = { + name: 'drupal-upgrade', + description: 'Transitional Drupal 11 upgrade: check the druxxy gate, bump core constraints, remove modules dropped from D11 core, update rector, then iterate to a clean upgrade_status — all in an isolated worktree.', + phases: [ + { title: 'Gate', detail: 'check druxxy D11 availability' }, + { title: 'Apply', detail: 'composer constraints + remove ckeditor/color/seven + rector bump' }, + { title: 'Validate', detail: 'upgrade_status + rector + cinsp, iterate until ready' }, + ], +} + +// --- schemas --- +const GATE = { + type: 'object', + required: ['blocked', 'reason'], + properties: { + blocked: { type: 'boolean' }, + reason: { type: 'string' }, + druxxyLatest: { type: 'string' }, + workaround: { type: 'string', description: 'e.g. temporarily set PROFILE_NAME=standard' }, + }, +} +const APPLY = { + type: 'object', + required: ['composerResolved', 'summary'], + properties: { + composerResolved: { type: 'boolean' }, + summary: { type: 'string' }, + removedModules: { type: 'array', items: { type: 'string' } }, + openProblems: { type: 'array', items: { type: 'string' } }, + }, +} +const VERDICT = { + type: 'object', + required: ['ready', 'blockers'], + properties: { + ready: { type: 'boolean' }, + blockers: { type: 'array', items: { type: 'string' } }, + autoFixable: { type: 'array', items: { type: 'string' } }, + }, +} + +// The harness invokes this with the workflow context (phase/agent/log/budget helpers). +export async function run({ phase, agent, log, budget }) { + // --- Gate: is D11 unblocked? --- + phase('Gate') + const gate = await agent( + `Determine whether a Drupal 11 upgrade is currently unblocked for the skilld-docker-container repo. +Run \`composer show skilldlabs/druxxy --all\` (the default install profile) and check whether any +release allows drupal/core ^11. Also skim composer.json require/require-dev for other obvious D11 +blockers. Return blocked=true if druxxy (or core) cannot resolve on ^11, with a one-line reason and, +if blocked, a workaround (e.g. temporarily set PROFILE_NAME=standard on the throwaway branch).`, + { phase: 'Gate', schema: GATE }, + ) + if (gate?.blocked) { + log(`D11 gate BLOCKED: ${gate.reason}. Proceeding on the throwaway branch with workaround: ${gate.workaround || 'PROFILE_NAME=standard'}.`) + } + + // --- Apply the transitional changes in an isolated worktree --- + phase('Apply') + const apply = await agent( + `In an isolated git worktree, apply the TRANSITIONAL Drupal 11 changes from docs/upgrading.md: +1. composer.json: set drupal/core-composer-scaffold and drupal/core-vendor-hardening to "^10.3.1 || ^11"; + bump skilldlabs/druxxy to a D11-capable release IF ${gate?.blocked ? 'one exists (it may not — if blocked, leave druxxy and instead set PROFILE_NAME=standard in .env.default for validation only)' : 'available'}. +2. Remove drupal/ckeditor, drupal/color, drupal/seven. git grep for stragglers; switch admin theme to claro. +3. Bump palantirnet/drupal-rector (from ^0.20.3) and add Drupal11SetList to rector.php ONLY if the + installed rector release exposes that class. +4. Re-verify the drupal/default_content patch still applies; drop it if upstreamed. +5. Run \`composer update --with-all-dependencies\` (via the php container) and report whether it resolved. +Make minimal, reviewable edits. Return composerResolved, a summary, the removed modules, and any open problems.`, + { phase: 'Apply', schema: APPLY, isolation: 'worktree' }, + ) + log(`Apply: ${apply?.summary || 'no result'}`) + + // --- Validate and iterate to ready --- + phase('Validate') + let verdict = null + const MAX_ITERS = budget.total ? 6 : 3 + for (let i = 1; i <= MAX_ITERS; i++) { + verdict = await agent( + `Validate Drupal upgrade-readiness of the current worktree. Run \`make upgradestatusval\` (authority), +\`make drupalrectorval\`, and \`make cinsp\`. Report ready=true only if upgrade_status prints no FILE: +lines. List remaining blockers (custom-code deprecations, by file) and which are rector-auto-fixable.`, + { phase: 'Validate', label: `validate#${i}`, schema: VERDICT, agentType: 'upgrade-validator' }, + ) + if (verdict?.ready) { log(`Ready after ${i} iteration(s).`); break } + if (!verdict?.blockers?.length) { log('No blockers reported but not marked ready — stopping to avoid a loop.'); break } + log(`Iteration ${i}: ${verdict.blockers.length} blocker(s). Attempting fixes.`) + await agent( + `Fix these Drupal 11 deprecations in custom code only (web/modules/custom, web/themes/custom): +${verdict.blockers.map((b, n) => `${n + 1}. ${b}`).join('\n')} +Prefer \`vendor/bin/rector process web/modules/custom web/themes/custom\` for the auto-fixable ones, +then hand-edit the rest. Do not touch contrib/core. Keep edits minimal.`, + { phase: 'Validate', label: `fix#${i}`, isolation: 'worktree' }, + ) + } + + return { + gate, + apply, + finalVerdict: verdict, + done: !!verdict?.ready, + note: 'Definition of done is a green D11 review app — open an MR and run build:review + the test:* jobs.', + } +} diff --git a/.claude/workflows/newrelic-bump.workflow.js b/.claude/workflows/newrelic-bump.workflow.js new file mode 100644 index 000000000..90472ebc5 --- /dev/null +++ b/.claude/workflows/newrelic-bump.workflow.js @@ -0,0 +1,62 @@ +export const meta = { + name: 'newrelic-bump', + description: 'Bump the pinned NewRelic PHP agent version in scripts/makefile/newrelic.sh to the latest release, validate it loads, and prepare a commit following the repo NR: convention.', + phases: [ + { title: 'Find', detail: 'latest NewRelic PHP agent version' }, + { title: 'Bump', detail: 'edit pin + validate' }, + ], +} + +const LATEST = { + type: 'object', + required: ['version'], + properties: { + version: { type: 'string', description: 'e.g. 12.7.0.36' }, + releaseUrl: { type: 'string' }, + current: { type: 'string', description: 'version currently pinned in the repo' }, + }, +} +const RESULT = { + type: 'object', + required: ['bumped', 'summary'], + properties: { + bumped: { type: 'boolean' }, + summary: { type: 'string' }, + commitMessage: { type: 'string' }, + }, +} + +// The harness invokes this with the workflow context (phase/agent/log helpers). +export async function run({ phase, agent, log }) { + phase('Find') + const latest = await agent( + `Find the latest NewRelic PHP agent release version. Read the current pin from +scripts/makefile/newrelic.sh (the NEW_RELIC_AGENT_VERSION default). Determine the newest available +release version and its release-notes URL. Return version, releaseUrl, and current.`, + { phase: 'Find', schema: LATEST }, + ) + log(`NewRelic agent: current ${latest?.current || '?'} → latest ${latest?.version || '?'}`) + + if (!latest?.version) { + return { bumped: false, summary: 'Could not determine the latest NewRelic agent version; aborting.' } + } + if (latest.version === latest.current) { + return { bumped: false, summary: `Already at the latest agent version (${latest.version}).` } + } + + phase('Bump') + const result = await agent( + `Bump the NewRelic PHP agent to ${latest?.version}. +1. Edit the version string in scripts/makefile/newrelic.sh (and any matching version in .env.default or + docker config if present). Make a minimal diff. +2. Run \`composer validate\`. If a stack is up, run \`make newrelic reload\` and confirm the agent loads + (extension_loaded('newrelic') or docker compose logs php). If no stack is up, note that runtime + validation was skipped. +3. Prepare a commit message in the repo's existing convention: "NR: " + (see \`git log --oneline | grep '^... NR:'\`). Do NOT commit unless asked. +Return bumped, a summary, and the proposed commitMessage.`, + { phase: 'Bump', schema: RESULT, agentType: 'dep-bumper' }, + ) + + return { ...result, from: latest?.current, to: latest?.version, releaseUrl: latest?.releaseUrl } +} diff --git a/.claude/workflows/package.json b/.claude/workflows/package.json new file mode 100644 index 000000000..3dbc1ca59 --- /dev/null +++ b/.claude/workflows/package.json @@ -0,0 +1,3 @@ +{ + "type": "module" +} diff --git a/.github/workflows/validation.yml b/.github/workflows/validation.yml new file mode 100644 index 000000000..607cf4c9c --- /dev/null +++ b/.github/workflows/validation.yml @@ -0,0 +1,61 @@ +name: Validation + +on: + pull_request: + push: + branches: + - master + workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: validation-${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + harness: + name: Harness static gate + runs-on: ubuntu-latest + + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Set up Node.js + uses: actions/setup-node@v4 + with: + node-version: 'lts/*' + + - name: Validate docs and agent harness + run: node scripts/ci/validate-harness.mjs + + sniffers: + name: Makefile sniffers + runs-on: ubuntu-latest + + env: + IMAGE_PHP: skilldlabs/php:83 + IMAGE_PHPCS: skilldlabs/docker-phpcs-drupal:10 + + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Docker diagnostics + run: | + docker --version + docker compose version + + - name: Validate base config langcode + run: make clang + + - name: Validate composer.json + run: make compval + + - name: Validate coding standards + run: make phpcs + + - name: Validate trailing newline + run: make newlineeof diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 2131f53a8..e729c9d03 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -132,6 +132,25 @@ sniffers:newlineeof: changes: - {{ project.path }}/**/* +# Static gate for docs + the .claude/ agent harness (Tier 0 of docs/testing.md). +# Node-only: runs in node:lts-alpine, no Drupal stack needed. Locally: `make harnessval`. +sniffers:harness: + stage: sniffers + image: node:lts-alpine + script: + - node --version + - node scripts/ci/validate-harness.mjs + rules: + - if: $CI_PIPELINE_SOURCE == 'parent_pipeline' + changes: + - {{ project.path }}/**/* + - if: $CI_PIPELINE_SOURCE == 'merge_request_event' + changes: + - {{ project.path }}/**/* + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + changes: + - {{ project.path }}/**/* + prepare:back: stage: prepare diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 000000000..ad5a17e02 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,12 @@ +# AGENTS.md + +Read [CLAUDE.md](CLAUDE.md) first; it is the source of truth for this repository's Docker/Makefile workflow, docs, agent harness, and validation rules. + +## Codex Environments + +- **SDC** (`skilld-labs/skilld-docker-container`): use the `master` default branch; start with the lightweight harness/static gates before Docker-heavy review-app work. +- **druxxy** (`skilld-labs/druxxy`): use its `1.x` default branch; validate profile/core compatibility there before changing SDC's `PROFILE_NAME=druxxy` assumptions. + +## Review Guidelines + +Focus GitHub reviews on real P0/P1 risks: CI breakage, portability regressions, security issues, and docs/runtime drift. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 000000000..a49e5882a --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,140 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## What this is + +A developer starterkit / Composer project template for **Drupal 10** sites (`skilld-labs/sdc`), run +entirely inside Docker Compose and driven by a **Makefile**. It is not a ready-to-run application — +it scaffolds a Drupal install (default profile `druxxy`), seeds demo content, and bundles a full +sniffer/test pipeline used both locally and in GitLab CI. + +## Golden rule: everything runs in containers + +You almost never invoke `php`, `composer`, `drush`, `yarn`, or `phpcs` directly on the host. The +Makefile wraps them so they execute inside the `php` service container as the host user: + +- `php = docker compose exec -T --user $(CUID):$(CGID) php ` — run as the host user. +- `php-0 = docker compose exec -T --user 0:0 php ` — run as root (package installs, chmod). +- `frontexec` (in `front.mk`) runs yarn tasks in a separate throwaway `IMAGE_FRONT` node container, + mounting only `web/themes/custom/$(THEME_NAME)`. + +Containers must be up (`make all` or `make provision`) before most targets work. Run `make` with no +target for the auto-generated help (each `## comment` above a target becomes its help line). + +The dummy rule `%: ; @:` lets trailing words be passed as arguments — that is how `make drush ` +and `make xdebug on` forward extra tokens to the wrapped command. + +## Common commands + +| Command | Purpose | +| --- | --- | +| `make all` | Full build from scratch: provision → back → front → si → localize → hooks → info | +| `make allfast` | Same, but DB lives in `/dev/shm` (RAM, non-persistent, faster) | +| `make clean` | Tear down containers/network and delete composer-installed code + DB data | +| `make si` | (Re)install the Drupal site | +| `make dev` | Enable devel/kint, Twig debug, disable caches & aggregation | +| `make exec` / `make exec0` | Open a shell in the php container (user / root) | +| `make drush ` | Run drush; pass flags after `--`, e.g. `make drush en devel -- -y` | +| `make phpcs` / `make phpcbf` | Check / autofix coding standards (custom code only) | +| `make front` / `make lint` / `make storybook` | Theme build / lint+fix / Storybook (need `THEME_NAME`) | +| `make tests` | Run the full validation + test suite (see below) | +| `make behat` | Run Behat browser tests | +| `make xdebug on\|off\|status` / `make blackfire` / `make newrelic` | Profiling/debug PHP extensions — `xdebug` toggles on/off/status; `blackfire` and `newrelic` enable only (`newrelic` needs `NEW_RELIC_LICENSE_KEY`) | + +### Tests & validations + +`make tests` chains: `sniffers` (`clang` config-langcode, `compval` composer validate, `phpcs`, +`newlineeof`) → `cinsp` (config schema) → `drupalrectorval` → `upgradestatusval` → `behat` → +`watchdogval` → `statusreportval` → `patchval`. Each is its own target in `scripts/makefile/tests.mk` +and can be run individually (e.g. `make phpcs`, `make cinsp`). + +- **PHPCS** runs in a dedicated `skilldlabs/docker-phpcs-drupal` image against **only** + `web/modules/custom` and `web/themes/custom`, using the `Drupal` + `DrupalPractice` standards. +- **Behat** (`make behat`): copies `behat.default.yml` → `behat.yml`, substitutes the live site URL, + starts a headless Chromium driver container (`IMAGE_DRIVER`) sharing the php container's network on + remote-debugging port 9222, then runs `vendor/bin/behat` (auto-`--rerun` on failure). Step + definitions live in `features/bootstrap/FeatureContext.php` (extends `RawDrupalContext`); features + in `features/*.feature`. To run a single feature, `make exec` then + `vendor/bin/behat features/your.feature`. + +## Architecture & layout + +- **Makefile is modular**: the root `Makefile` does `include scripts/makefile/*.mk`. Add + project-specific targets by dropping a new `scripts/makefile/.mk` (see `backup.mk` as a model); + do not bloat the root Makefile. +- **Config files are generated from `.default` templates** on first `make` run and must not be + committed: `.env` ← `.env.default`, `docker/docker-compose.override.yml` ← + `docker/docker-compose.override.yml.default`. `docker/docker-compose.yml` is the immutable base (a single + `php` service + network); all customization (DB engine, mail, Traefik labels, extra services) goes + in the override file. `make diff` shows how your local copies drift from the templates. +- **Drupal web root is `web/`.** Composer installs core/contrib/profiles/themes there + (`installer-paths` in `composer.json`); all of that is gitignored. **Your code lives in + `web/modules/custom/` and `web/themes/custom/` only** — those are the only tracked code paths and + the only ones sniffed/tested. +- **App server is pluggable.** The php image (`IMAGE_PHP`, default `skilldlabs/php:83-unit`) can run + under NGINX Unit (`docker/unit.json`), FrankenPHP/Caddy (`docker/Caddyfile`), or php-fpm. + `scripts/makefile/reload.sh` detects the running SAPI (`/proc/1/comm`) and reloads it correctly — + use `make reload` after changing server config, never assume a specific server. +- **Database** defaults to SQLite (`DB_URL` in `.env`). MySQL/PostgreSQL are opt-in by uncommenting + the relevant service in the override file and changing `DB_URL`; `system-detection.mk` + + `DB_MOUNT_DIR` logic in the Makefile handle persistent volume paths per engine. +- **Install modes** (`PROJECT_INSTALL` env): empty → install from the `PROFILE_NAME` profile; + `config` → `drush si --existing-config` then `drush cim` (config-driven install from `config/sync`). +- **Content seeding** (`make content`): enables `default_content` (+ `project_default_content`) then + `migrate_generator`, which builds migrations from CSVs in `content/`, imports them (tag `mgg`), then + uninstalls the generator modules to leave a clean site. +- **Settings injection**: `make si` appends `settings/settings.local.php` (private files path, mail) + into Drupal's `settings.php` and uncomments its include; `settings.redis.php` is appended only when + Redis is enabled. `settings.dev.php` is for the dev profile. + +## Conventions & gotchas + +- **Use Composer, not Drush, for code management.** `drush/policy.drush.inc` blocks `pm-download`, + `pm-update`, `pm-updatecode` — run `composer require` / `composer update` instead. +- `COMPOSE_PROJECT_NAME` is auto-sanitized to lowercase alphanumerics in `.env` on every make run; + the default `projectname` triggers an interactive prompt during `provision`. +- A pre-push git hook (symlinked by `make hooksymlink` to `scripts/git_hooks/sniffers.sh`) runs the + `sniffers` target. Bypass with `git push --no-verify`. +- No `composer.lock` is committed (intentional — avoids downstream merge conflicts); + `ScriptHandler::checkComposerVersion` enforces a recent Composer. +- Front targets silently no-op unless `THEME_NAME` (in `.env`) points to a real + `web/themes/custom/` directory; CI front jobs need `THEME_PATH` set in `.gitlab-ci.yml`. +- GitLab CI (`.gitlab-ci.yml`) reuses these same make targets across stages + (sniffers → prepare → build → update → tests), deploying review environments behind Traefik; + keep CI behavior and Makefile targets in sync when changing either. + +## Delivery / extras + +`scripts/` also holds opt-in helpers, each with its own README: `delivery-{archive,docker,git}/` +(release packaging), `mirroring/` (repo mirroring), and `multisite/` (config_split-based site +switching). They are examples to copy into a real project's CI, not active by default. Indexed in +[docs/delivery-and-ops.md](docs/delivery-and-ops.md). + +## Review apps & CI + +This template's main job is spinning up **GitLab CI review apps** (ephemeral per-MR sites behind +Traefik) and running a large sniffer/test pipeline. `.gitlab-ci.yml` is itself a **parent-pipeline +template** included by downstream projects. Full narrative docs live in [`docs/`](docs/): + +- [docs/review-apps.md](docs/review-apps.md) — review-app lifecycle, URLs, TTLs, teardown, required + CI/CD variables. **Start here.** +- [docs/ci-pipeline.md](docs/ci-pipeline.md) — every job, the job → `make` target → script map, the + validation gates, and the `cinsp`-before-schema-mutators ordering rule (#323). +- [docs/architecture.md](docs/architecture.md), [docs/local-development.md](docs/local-development.md), + [docs/upgrading.md](docs/upgrading.md) (D10→11→12), [docs/observability.md](docs/observability.md), + [docs/parallel-environments.md](docs/parallel-environments.md) (git-worktree parallel stacks). + +## Working with agents here + +A Claude Code harness lives in [`.claude/`](.claude/README.md) so this template — and projects seeded +from it — can be maintained by agents. Prefer these over ad-hoc work: + +- **Skills** (`/`): `drupal-upgrade`, `dep-bump` (NewRelic-first), `review-app-triage`, + `backlog-grooming`, `observability-up`. +- **Subagents**: `upgrade-validator`, `ci-log-analyzer`, `dep-bumper`. +- **Workflows**: `drupal-upgrade.workflow.js`, `newrelic-bump.workflow.js`. + +Skills wrap **existing `make` targets** — keep that contract. Run heavy/parallel agent work in git +worktrees ([docs/parallel-environments.md](docs/parallel-environments.md)). Contributor-facing rules +are in [CONTRIBUTING.md](CONTRIBUTING.md). diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 000000000..01c7e0d09 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,75 @@ +# Contributing + +This repo is a **template** that downstream Drupal projects are seeded from, and it powers their GitLab +CI review apps. Changes here ripple outward, so favor small, well-tested, well-documented changes. + +New to the codebase? Read [docs/architecture.md](docs/architecture.md) and +[docs/local-development.md](docs/local-development.md) first. + +## Ground rules + +- **Everything runs in containers.** Don't run `composer`/`drush`/`phpcs`/`yarn` on the host — use the + Makefile wrappers (`make drush …`, `make phpcs`, `make exec`). See [CLAUDE.md](CLAUDE.md). +- **`docker/docker-compose.yml` is immutable.** Customize via `docker/docker-compose.override.yml` (generated + from its `.default`). Same for `.env` ← `.env.default`. `make diff` shows your drift; never commit + `.env` or the override (they're gitignored). +- **Composer manages code, not Drush.** `drush/policy.drush.inc` blocks `pm-download`/`pm-update`/ + `pm-updatecode`. Use `composer require` / `composer update`. +- **No committed patches.** `composer.json`'s `extra.patches` must reference upstream URLs + (drupal.org / GitHub issues), never a local file — `make patchval` (and `test:patch` in CI) enforces + this. +- **Custom code only in** `web/modules/custom/` and `web/themes/custom/`. Everything else under `web/` + is Composer-installed and gitignored. + +## Branching & commits + +- Branch off `master` (the template ships as a rolling `master`; there are no semver releases). +- Keep one logical change per PR. Dependency bumps: one dependency per PR (see the + [`dep-bump`](.claude/skills/dep-bump/SKILL.md) skill). +- NewRelic agent bumps follow the existing message convention — `NR: ` (see + `git log`). + +## Quality gates (run before pushing) + +```sh +make hooksymlink # once: symlinks .git/hooks/pre-push → scripts/git_hooks/sniffers.sh +make sniffers # clang + composer validate + phpcs + newlineeof (what the pre-push hook runs) +make tests # full suite: sniffers + cinsp + rector + upgrade_status + behat + watchdog + … +``` + +The pre-push hook runs `make sniffers` automatically; bypass once with `git push --no-verify` only when +you must. CI runs the same `make` targets — see [docs/ci-pipeline.md](docs/ci-pipeline.md), and mind the +`cinsp`-before-schema-mutators ordering constraint (#323). + +If you change docs or the `.claude/` harness, also run the static gate (`make harnessval`, or the +`sniffers:harness` CI job) and follow [docs/testing.md](docs/testing.md) for the full pre-merge plan. + +## Documentation + +User-facing docs live in [`docs/`](docs/) and travel with the code. If you change CI, the install flow, +the app server, or the upgrade path, update the matching doc in the same PR. The +[`backlog-grooming`](.claude/skills/backlog-grooming/SKILL.md) skill keeps +[docs/backlog.md](docs/backlog.md) current. + +## Agent harness + +This repo ships a Claude Code harness under [`.claude/`](.claude/README.md) (skills, subagents, +workflows, permission allowlist) for the recurring maintenance jobs — Drupal upgrades, dependency/ +NewRelic bumps, CI/review-app triage, backlog grooming, and observability. + +- **Forks/seeds inherit it for free** (it's committed). +- **Existing projects adopt it** by copying `.claude/` once: + ```sh + cp -r path/to/skilld-docker-container/.claude ./ + ``` + then review `.claude/settings.json` (note `make:*` is allowed intentionally — `make` is the project's + primary interface and includes `make clean`). +- Project context for agents lives in this repo's [`CLAUDE.md`](CLAUDE.md) and [`docs/`](docs/), **not** + in any parent-directory `CLAUDE.md`. +- Heavy or parallel agent work should run in **git worktrees** + ([docs/parallel-environments.md](docs/parallel-environments.md)) so multiple branches build at once + without colliding. + +## License + +By contributing you agree your contributions are licensed under the repository's MIT license. diff --git a/README.md b/README.md index 4cd4f5699..9de5ecffa 100644 --- a/README.md +++ b/README.md @@ -23,6 +23,16 @@ **Skilld docker container** is a developer starterkit for your Drupal project. +## Documentation + +Full guides live in [`docs/`](docs/README.md): + +* [Review apps](docs/review-apps.md) — how GitLab CI builds ephemeral per-MR environments. +* [CI pipeline](docs/ci-pipeline.md) — every stage/job and the `make`-target mapping. +* [Architecture](docs/architecture.md) · [Local development](docs/local-development.md) · [Parallel environments](docs/parallel-environments.md) +* [Upgrading (D10→11→12)](docs/upgrading.md) · [Observability](docs/observability.md) · [Delivery & ops](docs/delivery-and-ops.md) +* [Contributing](CONTRIBUTING.md) · [Agent harness](.claude/README.md) · [Backlog](docs/backlog.md) + ## What is this? * This is a developer starterkit which can be used for local drupal development or/and integration into your CI/CD processes. @@ -39,9 +49,9 @@ * Check post-installation steps for Linux version 18.06.0 or later * Install Docker Compose V2 version **2.0** or later -* Copy **.env.default** to **.env**, more information about enviroment file can be found docs.docker.com -* Copy **docker-compose.override.yml.default** to **docker-compose.override.yml**, update parts you want to overwrite. - * **docker-compose.yml** contains the base requirements of a working Drupal site. It should not be updated. +* Copy **.env.default** to **.env**, more information about environment files can be found docs.docker.com +* Copy **docker/docker-compose.override.yml.default** to **docker/docker-compose.override.yml**, update parts you want to overwrite. + * **docker/docker-compose.yml** contains the base requirements of a working Drupal site. It should not be updated. * Update **.gitlab-ci.yml** `variables` section THEME_PATH to make front gitlab CI works. * Run `make all` @@ -65,18 +75,18 @@ | ADDITIONAL_PHP_PACKAGES | Additional php extensions and tools to install | `graphicsmagick` | | IMAGE_NGINX | Image to use for nginx container | `skilldlabs/nginx:1.24` | | IMAGE_APACHE | Image to use for apache container | `skilldlabs/skilld-docker-apache` | -| IMAGE_FRONT | Image to use for front tasks | `skilldlabs/frontend:zen` | +| IMAGE_FRONT | Image to use for front tasks | `node:lts-alpine` | | IMAGE_DRIVER | Image to use for automated testing webdriver | `zenika/alpine-chrome` | | MAIN_DOMAIN_NAME | Domain name used for traefik | `docker.localhost` | -| DB_URL | Url to connect to database | `sqlite:///dev/shm/db.sqlite` | -| DB_DATA_DIR | Full path to database storage | `/dev/shm` | -| CLEAR_FRONT_PACKAGES | Set it to `no` to keep `/node_nodules` directory in theme after `make front` task to save build time. | yes | +| DB_URL | URL to connect to database | `sqlite://./../.cache/db.sqlite` | +| DB_DATA_DIR | Full path to database storage | `../.cache` | +| CLEAR_FRONT_PACKAGES | Set to `yes` to remove theme `node_modules` and `dist` during `make clean`; `no` keeps frontend packages between runs. | no | | RA_BASIC_AUTH | username:hashed-password format defining BasicAuth in Traefik. Password hashed using `htpasswd -nibB username password!` as [described here](https://doc.traefik.io/traefik/middlewares/basicauth/#general) | - | #### Persistent Mysql * By default sqlite storage used, which is created inside php container, if you need persistent data to be saved: - * Update `docker-compose.override.yml`, set + * Update `docker/docker-compose.override.yml`, set ```yaml php: depends_on: @@ -92,7 +102,7 @@ #### Network -* Every time project built, it take new available IP address, if you want to have persistent IP, uncomment lines from `docker-compose.override.yml` +* Every time project is built, it takes a new available IP address. If you want to have a persistent IP, uncomment lines from `docker/docker-compose.override.yml` ```yaml networks: front: @@ -120,7 +130,7 @@ networks: * `make front` - Builds frontend tasks. * `make lint` - Runs frontend linters. * `make storybook` - Runs storybook in current theme. -* `make blackfire` - Adds and enables blackfire.io php extension, needs [configuration](https://blackfire.io/docs/configuration/php) in docker-compose.override.yml. +* `make blackfire` - Adds and enables blackfire.io php extension, needs [configuration](https://blackfire.io/docs/configuration/php) in `docker/docker-compose.override.yml`. * `make newrelic` - Adds and enables newrelic.com php extension, needs [configuration](https://docs.newrelic.com/docs/agents/php-agent/getting-started/introduction-new-relic-php#configuration) `NEW_RELIC_LICENSE_KEY` environment variable defined with valid license key. * `make xdebug (on|off|status)` - Enable, disable or report status of [Xdebug](https://xdebug.org/docs/) PHP extension. diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 000000000..a98afa58b --- /dev/null +++ b/docs/README.md @@ -0,0 +1,40 @@ +# Documentation + +Narrative documentation for the Skilld Docker Container — a Composer + Makefile + Docker template for +Drupal projects, used to spin up **GitLab CI review apps** and run a full sniffer/test pipeline. + +Start with the root [README.md](../README.md) for quickstart, then dive in here: + +## Operating the template + +- **[review-apps.md](review-apps.md)** — how ephemeral per-MR environments are built, addressed + (Traefik URLs), and torn down; the required GitLab CI/CD variables. **Start here.** +- **[ci-pipeline.md](ci-pipeline.md)** — every CI stage and job, the job → `make` target → script + map, the validation gates, and the `cinsp` ordering constraint. +- **[architecture.md](architecture.md)** — container topology, the pluggable app server (Unit / + FrankenPHP / fpm), database engines, install flow, and the modular Makefile. +- **[local-development.md](local-development.md)** — running the stack on your machine, including + macOS specifics. +- **[parallel-environments.md](parallel-environments.md)** — run several review-app-like stacks at + once locally with `git worktree`; the basis for parallel agent runs. +- **[delivery-and-ops.md](delivery-and-ops.md)** — index of the opt-in delivery, mirroring, and + multisite helpers under `scripts/`. +- **[observability.md](observability.md)** — getting logs and traces out of Docker (logs tier + + opt-in OpenTelemetry traces). + +## Maintaining the template + +- **[upgrading.md](upgrading.md)** — the Drupal 10 → 11 → 12 upgrade runbook. +- **[testing.md](testing.md)** — how to validate the project + harness before merge (static gate, + harness smoke, Docker functional) and the merge gate. +- **[backlog.md](backlog.md)** — triaged open PRs and issues (adopt / rebase / fold-into-docs / + close-as-stale). +- **[../CONTRIBUTING.md](../CONTRIBUTING.md)** — branching, quality gates, conventions, and how + downstream projects inherit the agent harness. + +## Agent-managed maintenance + +This repo ships a Claude Code harness under [`.claude/`](../.claude) so the template — and projects +seeded from it — can be maintained by agents (Drupal upgrades, dependency/NewRelic bumps, CI/review-app +triage, backlog grooming, observability). See [CONTRIBUTING.md](../CONTRIBUTING.md#agent-harness) and +[`.claude/README.md`](../.claude/README.md). diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 000000000..f800df0bd --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,100 @@ +# Architecture + +This template assembles a Drupal 10 site with Composer and runs it in Docker, orchestrated by a +Makefile. This document covers the moving parts you need to understand before changing anything. For +day-to-day commands see the root [README.md](../README.md) and [CLAUDE.md](../CLAUDE.md). + +## Container topology + +The committed base, [`docker/docker-compose.yml`](../docker/docker-compose.yml), defines a **single +`php` service** (container `${COMPOSE_PROJECT_NAME}_web`) on a `front` bridge network, with the repo +mounted at `/var/www/html`. Everything else (databases, mail, alternate web servers, profiling, +observability) is opt-in and lives in +[`docker/docker-compose.override.yml.default`](../docker/docker-compose.override.yml.default), copied +to `docker/docker-compose.override.yml` on first run. + +> **Rule:** `docker-compose.yml` is the immutable base — never edit it. Customize via the override +> file. `make diff` shows how your local `.env` / override drift from the `.default` templates. + +Two Make helpers wrap container execution: + +- `php = docker compose --env-file .env exec -T --user $(CUID):$(CGID) php …` (as the host user) +- `php-0 = … --user 0:0 …` (as root, for package installs / chmod) + +## Pluggable app server (one image, three SAPIs) + +The `php` image (`IMAGE_PHP`, default `skilldlabs/php:83-unit`) can serve Drupal under three runtimes, +abstracted by [`scripts/makefile/reload.sh`](../scripts/makefile/reload.sh), which detects the running +SAPI from `/proc/1/comm` and reloads it correctly: + +| SAPI | Image tag | Config | Reload | +| --- | --- | --- | --- | +| **Nginx Unit** (default) | `:83-unit` | [`docker/unit.json`](../docker/unit.json) | `PUT` config to the Unit control socket | +| **FrankenPHP / Caddy** | `:83-frankenphp` | [`docker/Caddyfile`](../docker/Caddyfile) | `frankenphp reload` | +| **php-fpm** (legacy) | — | — | `kill -USR2 1` | + +Use `make reload` after changing server config; never assume a specific server. **Nginx Unit is the +current default** — FrankenPHP image builds are problematic on arm64 (QEMU), which is why the default +was reverted to Unit (issue #467). Both `unit.json` and the `Caddyfile` implement the same Drupal +hardening (deny `*.php` outside `index.php`, block dotfiles/backups, long-cache static assets, gzip). + +## Web root and code layout + +The Drupal web root is `web/`. Composer's `installer-paths` (in [`composer.json`](../composer.json)) +place core/contrib/profiles/themes under `web/{core,modules/contrib,profiles/contrib,themes/contrib}`, +and **all of that is gitignored**. The only tracked, sniffed, and tested code lives in: + +- `web/modules/custom/` — custom modules (e.g. `project_default_content`) +- `web/themes/custom/` — the project theme (built separately in a node container, see `front.mk`) + +## Database engines + +`DB_URL` in `.env` selects the engine; SQLite is the default (no extra container): + +| Engine | `DB_URL` | Persistence | +| --- | --- | --- | +| SQLite (default) | `sqlite://./../.cache/db.sqlite` | File under `.cache/` (or `/dev/shm` with `make fast`, non-persistent) | +| MySQL | `mysql://db:db@mysql/db` | Uncomment the `mysql` service in the override; volume under `DB_DATA_DIR` | +| PostgreSQL | `pgsql://db:dbroot@postgresql/db` | Uncomment `postgresql` + `load-extension.sh` (pg_trgm) | + +The Makefile derives `DB_MOUNT_DIR` per engine and per project (`COMPOSE_PROJECT_NAME` + `CURDIR`), so +distinct projects/worktrees never collide on DB storage — the basis for +[parallel environments](parallel-environments.md). + +## Install flow + +`make all` chains: `provision → back → front → si → localize → hooksymlink → info`. In CI, +`make all_ci` skips `back`/`front` (prebuilt as artifacts). Key steps: + +- **provision** — sanitize `COMPOSE_PROJECT_NAME`, bring up containers, install any + `ADDITIONAL_PHP_PACKAGES`, enable NewRelic if `NEW_RELIC_LICENSE_KEY` is set, `make reload`. +- **si** (`PROJECT_INSTALL` env): + - empty → `drush si $(PROFILE_NAME)` (default profile `druxxy`) with site name/mail/locale. + - `config` → `drush si --existing-config` then `drush cim` (config-driven install). + - Then `make content`, and creates a `tester` user with the `contributor` role. +- **content** — enables `default_content` + `project_default_content`, then `migrate_generator`, which + builds migrations from the CSVs in [`content/`](../content/) (tag `mgg`), imports them, and uninstalls + the generator modules to leave a clean site. +- **localize** — `drush locale:check/update` + import custom `translations/*.po`. +- **settings injection** — `make si` copies `settings/settings.local.php` into Drupal's `settings.php` + (private files path, mail) and uncomments its include; `settings/settings.redis.php` is appended only + when Redis is enabled. + +## Modular Makefile + +The root `Makefile` does `include scripts/makefile/*.mk`. Add project-specific targets by dropping a +new `scripts/makefile/.mk` (see `backup.mk` as a template); do not bloat the root Makefile. The +extra word after a target is forwarded as an argument (the `%: ; @:` rule), which is how +`make drush ` and `make xdebug on` work. + +## Health checks (opt-in) + +There is no committed `healthcheck:` today (issue #152). Nginx Unit can expose a status/health route, +making a lightweight liveness probe cheap to add in the override file for the `php` service. This pairs +with [observability.md](observability.md). + +## See also + +- [review-apps.md](review-apps.md) / [ci-pipeline.md](ci-pipeline.md) — how this stack is deployed in CI. +- [local-development.md](local-development.md) — running it on your machine (incl. macOS). +- [upgrading.md](upgrading.md) — moving the stack to Drupal 11 / 12. diff --git a/docs/backlog.md b/docs/backlog.md new file mode 100644 index 000000000..54dd9c0b8 --- /dev/null +++ b/docs/backlog.md @@ -0,0 +1,53 @@ +# Backlog triage + +A decision list for the open PRs and issues on +[skilld-labs/skilld-docker-container](https://github.com/skilld-labs/skilld-docker-container), so the +queue is actionable rather than unbounded. Snapshot taken during the documentation effort; regenerate +with the [`backlog-grooming`](../.claude/skills/backlog-grooming/SKILL.md) skill. + +Buckets: **adopt-now** (small, low-risk), **rebase & finish** (valuable, stale), **fold into docs** +(no code needed), **close** (stale/obsolete/superseded). + +## Pull requests + +| PR | Title | Bucket | Note | +| --- | --- | --- | --- | +| #466 | Open Telemetry tracing | **rebase & finish** | Observability Tier 2; finish TODOs (RA collector, optional flag, dashboards). See [observability.md](observability.md) | +| #290 | Add local composer cache | **adopt-now** | One line; low risk | +| #189 | Lock node image version | **adopt-now** | Reported mergeable; verify the pin is still sensible | +| #279 | Yarn cache on local + RA | **rebase** | Complements #290; re-test against current `.gitlab-ci.yml` | +| #342 | Auto-set shm size | **rebase / maybe close** | Largely superseded by compose `shm_size` (closed #451); keep only if it adds value | +| #403 | k3s alternative to compose | **defer** | Large, invasive Makefile refactor; **conflicts with the worktree helper** — decide its fate before Workstream 6 lands | +| #271 | Add modules to install profile | **needs info / close** | Unclear module list; revisit post-D11 | +| #141 | Try GitHub Actions | **close** | 7-yr-old probe; GitLab CI is the platform | +| #38 | macOS shared folder `cached` | **close** | Based on a deprecated Docker Desktop option; macOS guidance now in [local-development.md](local-development.md) | + +## Issues + +| Issue | Title | Bucket | Action | +| --- | --- | --- | --- | +| #461 | Prepare Drupal 11 upgrade | **adopt-now** | Tracked by [upgrading.md](upgrading.md); **blocked on a D11-capable druxxy** | +| #467 | Nginx Unit as default | **close (done)** | Unit is already the default again (commit `35a7f38`); rationale documented in [architecture.md](architecture.md) | +| #323 | Order for tests | **fold into docs** | `cinsp`-before-schema-mutators captured in [ci-pipeline.md](ci-pipeline.md#test-ordering-constraint-issue-323) | +| #287 | Dotenv for dynamic environments | **adopt** | `artifacts:reports:dotenv` for the review URL; noted in [ci-pipeline.md](ci-pipeline.md#opportunities-tracked-in-the-backlog) | +| #295 | Adopt code-quality for GitLab | **adopt** | Emit `artifacts:reports:codequality` from phpcs/rector | +| #152 | Add health checks | **adopt** | Unit health route → `healthcheck:`; see [observability.md](observability.md) and [architecture.md](architecture.md) | +| #400 | macOS local setup | **fold into docs (done)** | Captured in [local-development.md](local-development.md#macos-issue-400) | +| #70 | Inspection workflow (proposed `insp` target) | **consider** | Consolidate inspection targets; aligns with the agent skills | +| #341 | Auto-set shm size | **close** | Solved by closed #451 (compose `shm_size`) | +| #210 | composer-lock-diff | **fold into docs** | Optional CI nicety | +| #195 | Static storybook build | **fold into docs** | Covered by `make build-storybook` + #287 link injection | +| #164 | New core templates/scaffold | **fold into docs** | Re-evaluate against D11 scaffold | +| #160 | Revamp `clean` task | **consider** | Constraints on `web/sites/*` cleanup | +| #18 | Windows support | **fold into docs** | Path-separator / `-d` notes | +| #10 | Access sites by domain name | **fold into docs** | DNS / `/etc/hosts` pattern; relates to [parallel-environments.md](parallel-environments.md) | +| #144, #142, #179 | Drush 9 config / profile hook / node image | **close** | Obsolete (Drush 13 now) | +| #414, #59, #58, #26, #25, #14, #13, #50 | misc 2016–2022 | **close** | Stale; no concrete scope | + +## Suggested order + +1. **adopt-now**: #290, #189 (quick wins). +2. **fold into docs**: already done for #323/#400/#467; do #287/#295 next (small CI templating). +3. **rebase & finish**: #466 (observability) — highest-value stale PR. +4. **decide #403** before the worktree helper to avoid a Makefile rebase collision. +5. **close** the stale set in one grooming pass. diff --git a/docs/ci-pipeline.md b/docs/ci-pipeline.md new file mode 100644 index 000000000..b91520ac7 --- /dev/null +++ b/docs/ci-pipeline.md @@ -0,0 +1,98 @@ +# CI pipeline reference + +This is a job-by-job reference for [`.gitlab-ci.yml`](../.gitlab-ci.yml). For *how review apps work* +(URLs, TTLs, teardown), read [review-apps.md](review-apps.md) first. + +Every job runs on a **shell runner** (it shells out to `docker` / `docker compose` / `make`), selected +by the `.runner_tag_selection` tag anchor. The base image is `$IMAGE_PHP` (CI default +`skilldlabs/php:83`). The shared `before_script` just prints diagnostics (`date`, `id`, `env`, …). + +## Stages + +``` +sniffers → prepare → build → update → tests → more tests +``` + +- **sniffers** — static checks on the changed code; no running site needed. +- **prepare** — build backend (composer) and frontend (yarn) artifacts, cached and passed forward. +- **build** — deploy the review app (see [review-apps.md](review-apps.md)). +- **update** — optional: simulate a production update against the last tag's DB/files. +- **tests / more tests** — functional and quality gates run against the deployed review app. + +## Job → `make` target → script map + +| Job | Stage | Runs | Underlying script / definition | +| --- | --- | --- | --- | +| `sniffers:clang` | sniffers | `make clang` | `scripts/makefile/baseconfig-langcode.sh` (base-config langcode) | +| `sniffers:compose` | sniffers | `composer validate --profile` | composer built-in | +| `sniffers:front` | sniffers | `make front-install` + `make lintval` | `scripts/makefile/front.mk` (yarn lint) — only if `$THEME_PATH` | +| `sniffers:phpcs` | sniffers | `make phpcs` | `scripts/makefile/tests.mk` → `skilldlabs/docker-phpcs-drupal` (Drupal+DrupalPractice, custom code only) | +| `sniffers:newlineeof` | sniffers | `make newlineeof` | `scripts/makefile/newlineeof.sh` | +| `sniffers:harness` | sniffers | `node scripts/ci/validate-harness.mjs` | docs + `.claude/` static gate (see [testing.md](testing.md)) | +| `prepare:back` | prepare | `composer install … create-required-files` | inline docker run; caches `vendor/`, `web/core`, contrib, drush | +| `prepare:front` | prepare | `make front-install` + `make front-build` | `scripts/makefile/front.mk` (yarn build → theme `dist/`) | +| `build:review/master/tag` | build | `make all_ci` | `Makefile:61` (provision → si → localize → hooksymlink → info) | +| `stop_review` | build | `make clean` | `Makefile` (drop stack + build dir) | +| `generate:logins` | build | `make info` | `Makefile` (login links + IPs) | +| `test:deploy` | update | `make drush deploy` vs last tag | inline; needs `TEST_UPDATE_DEPLOYMENTS=TRUE` + API token | +| `test:storybook` | tests | `make build-storybook` | `scripts/makefile/front.mk` — only if `$STORYBOOK_PATH` | +| `test:behat` | tests | `make behat` | `scripts/makefile/tests.mk` (Behat + headless Chromium; junit report) | +| `test:cinsp` | tests | `make cinsp` | `scripts/makefile/config-inspector-validation.sh` | +| `test:drupalrector` | tests | `make drupalrectorval` | `scripts/makefile/tests.mk` + `rector.php` (dry-run) | +| `test:lighthouse` | tests | `lhci collect/assert` | `lighthouserc.yml`; runs in `cypress/browsers` container | +| `test:contentgen` | tests | `make contentgen` | `scripts/makefile/contentgen.sh` — `when: manual` | +| `test:patch` | tests | `make patchval` | `scripts/makefile/patchval.sh` — skip with `RUN_PATCHVAL_CI_JOB=FALSE` | +| `test:statusreport` | more tests | `make statusreportval` | `scripts/makefile/status-report-validation.sh` | +| `test:upgradestatus` | more tests | `make upgradestatusval` | `scripts/makefile/upgrade-status-validation.sh` | +| `test:watchdog` | more tests | `make watchdogval` | `scripts/makefile/watchdog-validation.sh` | + +## What the validation gates actually assert + +These are the same targets `make tests` runs locally, so a green pipeline is reproducible on a laptop. + +- **`cinsp`** — enables `config_inspector`, fails if `drush config:inspect --only-error` reports any + schema error. **Ordering matters (see below).** +- **`drupalrector`** — `vendor/bin/rector process --dry-run` over `web/modules/custom` + + `web/themes/custom`, using the set lists in [`rector.php`](../rector.php) (currently Drupal 8/9/10). +- **`upgradestatus`** — enables `upgrade_status`, fails if + `drush upgrade_status:analyze --all --ignore-contrib --ignore-uninstalled` prints any `FILE:` line. +- **`statusreport`** — fails on any severity-2 (error) row in `/admin/reports/status`, except the + ignored `Trusted Host Settings`. +- **`watchdog`** — fails if any `Emergency|Alert|Critical|Error` log was written during the run. +- **`patch`** — fails if `composer.json`'s `extra.patches` contains a **non-remote** (committed) patch + file; patches must be hosted upstream (drupal.org/GitHub issue URLs). +- **`behat`** — copies `behat.default.yml` → `behat.yml`, injects the review URL, boots a headless + Chromium driver container, runs `vendor/bin/behat`, and emits a JUnit report consumed by GitLab. + +### Test-ordering constraint (issue #323) + +`test:cinsp` is intentionally meant to run **before** the jobs that toggle modules +(`test:upgradestatus` enables then uninstalls `upgrade_status`; `test:cinsp` enables `config_inspector`). +Config schema must be collected while the dev/inspection modules' schema is still present in the +container. When customizing stage/job order in a downstream project, **keep `cinsp` ahead of the +schema-mutating jobs** or you will get spurious schema errors. + +## Artifacts & reports + +- `prepare:back` → `vendor/`, `web/`, `drush/` (1 day); `prepare:front` → theme `dist/` + `node_modules/`. +- `build:tag` → `web/sites/*/files/` + `.cache` (the DB) for 1 week — the input to `test:deploy`. +- `test:behat` → `junit/*.xml` (surfaced as GitLab **JUnit test report**) + screenshots under + `web/screenshots/` on the review URL. +- `test:lighthouse` → HTML reports moved to `web/lighthouseci/`, linked from the job log. + +## Update-path simulation (`test:deploy`) + +Opt-in via `TEST_UPDATE_DEPLOYMENTS=TRUE`. It calls the GitLab API to find the **last tag**, downloads +that tag's `build:tag` artifact (production-like DB + files), swaps them into the running review app, +disables `config_ignore` if present, and runs `drush deploy` — i.e. it rehearses what deploying the MR +on top of the current release would do. Requires `GITLAB_PROJECT_ACCESS_TOKEN` (and +`GITLAB_PROJECT_BASIC_AUTH` if the repo is behind basic auth). + +## Opportunities (tracked in the backlog) + +- **GitLab Code Quality reports** (issue #295) — emit `artifacts:reports:codequality` from + `sniffers:phpcs`/`test:drupalrector` so findings annotate the MR diff inline. +- **Dynamic environment URL via dotenv** (issue #287) — have the deploy job write the review URL to a + `artifacts:reports:dotenv` file so downstream jobs reuse it instead of recomputing it. + +See [backlog.md](backlog.md) for the full triage. diff --git a/docs/delivery-and-ops.md b/docs/delivery-and-ops.md new file mode 100644 index 000000000..2a420abd9 --- /dev/null +++ b/docs/delivery-and-ops.md @@ -0,0 +1,44 @@ +# Delivery & ops helpers + +Beyond review apps, `scripts/` ships **opt-in** recipes for delivering builds and mirroring repos. +Each is an *example* meant to be copied into a consuming project's pipeline — none runs by default. +This page indexes them; each linked README has the exact CI job and variables. + +All of them exist for the same reasons: built **artifacts (composer/yarn output) are not versioned in +git**, GitLab's built-in mirroring/registry **doesn't work when the repo is behind basic auth**, and +**multi-target mirroring is often a paid feature**. + +## Delivery (on tag, to a release target) + +| Helper | Delivers | Extra file needed | Key CI/CD variables | README | +| --- | --- | --- | --- | --- | +| **Archive** | Current tag as a `.tar.gz` to a raw file registry | — | `DELIVERY_REPOSITORIES_RAW_REGISTRY_DOMAIN_1`, `…_USERNAME`, `…_PASSWORD` | [scripts/delivery-archive](../scripts/delivery-archive/README.md) | +| **Docker** | Current tag as a Docker image to a registry | `scripts/delivery-docker/Dockerfile` | `DELIVERY_REPOSITORIES_DOCKER_REGISTRY_DOMAIN_1`, `…_USERNAME`, `…_PASSWORD` | [scripts/delivery-docker](../scripts/delivery-docker/README.md) | +| **Git** | Current tag pushed to another git repo (e.g. Platform.sh) | `scripts/delivery-git/deliver_current_tag_via_git.sh` | `DELIVERY_REMOTE_REPO_{IP,PRIVATE_KEY,TYPE,URL_1,BRANCH}`, `GIT_USER_{EMAIL,NAME}` | [scripts/delivery-git](../scripts/delivery-git/README.md) | + +Each supports **multiple targets at once** by adding more jobs (`…_DOMAIN_2`, `…_URL_2`, …). Position +the delivery job after `prepare:back`/`prepare:front` and use `dependencies:` so the built artifacts +are present. + +## Mirroring (on every branch) + +| Helper | Action | Extra file | Key variables | README | +| --- | --- | --- | --- | --- | +| **Mirroring** | Mirror the current branch to other repos; also deletes remote branches absent locally | `mirror_current_branch.sh` | `MIRRORING_REMOTE_REPO_{IP,PRIVATE_KEY,TYPE,URL_1}`, `GIT_USER_{EMAIL,NAME}` | [scripts/mirroring](../scripts/mirroring/README.md) | + +## Multisite (config_split switching) + +[scripts/multisite](../scripts/multisite/README.md) lets a multisite setup swap config sets quickly, +locally and in review apps: + +1. `composer require drupal/config_split` and create splits in the UI (no split for the shared + `default` case). +2. Move `config_split.mk` + `config_split_disable_all.sh` into `scripts/makefile/`. +3. Locally: `make split first` / `make split default`. In CI: manual jobs per split; review apps build + with the `default` split. + +## Relationship to review apps + +These run in the **same pipeline** as the review-app jobs ([ci-pipeline.md](ci-pipeline.md)) but in +their own stage (`deliver` / `mirror`), typically on tags. Review apps are for *validation before +merge/release*; delivery/mirroring is for *shipping the validated result onward*. diff --git a/docs/local-development.md b/docs/local-development.md new file mode 100644 index 000000000..0a068695c --- /dev/null +++ b/docs/local-development.md @@ -0,0 +1,71 @@ +# Local development + +Quickstart lives in the root [README.md](../README.md); this page covers the things that bite people: +the `.default` template convention, day-to-day commands, and macOS specifics (issue #400). + +## First run + +```sh +cp .env.default .env # auto-created on first `make` too +cp docker/docker-compose.override.yml.default docker/docker-compose.override.yml +# set COMPOSE_PROJECT_NAME (or you'll be prompted), THEME_NAME if you have a theme +make all +``` + +`make all` builds everything and prints (`make info`) the site URL plus one-click admin/tester logins. + +> **Config files are generated, not committed.** `.env` ← `.env.default` and +> `docker/docker-compose.override.yml` ← its `.default`. The committed `docker/docker-compose.yml` is +> the immutable base — never edit it. `make diff` shows your drift from both templates. + +## Everyday commands + +| Command | Purpose | +| --- | --- | +| `make all` / `make allfast` | Full build (allfast = DB in `/dev/shm`, faster, non-persistent) | +| `make si` | Reinstall the site | +| `make dev` | Devel + kint, Twig debug, caches/aggregation off | +| `make exec` / `make exec0` | Shell into the php container (user / root) | +| `make drush ` | Run drush; flags after `--`, e.g. `make drush cr`, `make drush en devel -- -y` | +| `make phpcs` / `make phpcbf` | Check / autofix Drupal coding standards (custom code only) | +| `make sniffers` / `make tests` | The pre-push gate / the full validation+test suite | +| `make front` / `make lint` / `make storybook` | Theme build / lint+fix / Storybook (need `THEME_NAME`) | +| `make xdebug on\|off\|status` | Toggle Xdebug | +| `make clean` | Tear the stack down and remove built code + DB | + +To run **several stacks at once** (one per branch), see +[parallel-environments.md](parallel-environments.md). + +## macOS (issue #400) + +Docker volume performance and paths differ on macOS. Recommended `.env` overrides: + +```sh +# Keep the SQLite DB on a bind mount, not /dev/shm +DB_URL=sqlite://./../.cache/db.sqlite +DB_DATA_DIR=../.cache +# Match your host user so files aren't root-owned +CUID=1000 +CGID=1000 +``` + +In `docker/docker-compose.override.yml` for the `php` service: + +- Use the `:cached` volume flag instead of `:z` for the bind mount. +- If you need to hit the site without Traefik, publish a port (e.g. `ports: ["8090:80"]`). +- If DNS resolution misbehaves inside containers, add `dns: 8.8.8.8`. + +The Makefile already special-cases Darwin in `scripts/makefile/system-detection.mk` (sets `CUID/CGID` +to 1000 and a longer compose timeout). **[OrbStack](https://docs.orbstack.dev/features)** is a +lighter-weight Docker Desktop alternative many on the team use on macOS. + +## Troubleshooting + +- **Port already in use / can't reach the site** — locally the `php` service publishes no host port by + default; `make info` prints the container IP, or publish a port as above. +- **Permission errors on `web/sites` or `.cache`** — run the failing step via `make exec0` (root) once, + or re-run `make si` which re-applies the settings permissions. +- **Stale containers from another branch** — each stack is keyed by `COMPOSE_PROJECT_NAME`; `make clean` + in the right checkout, or see [parallel-environments.md](parallel-environments.md). +- **Pre-push hook rejects a push** — it runs `make sniffers`; fix the reported issues or bypass once + with `git push --no-verify`. diff --git a/docs/observability.md b/docs/observability.md new file mode 100644 index 000000000..3335268c2 --- /dev/null +++ b/docs/observability.md @@ -0,0 +1,74 @@ +# Observability + +Two tiers, by cost. **Tier 1 (logs)** needs no extra services and works today. **Tier 2 (traces)** is +an opt-in OpenTelemetry stack, default-off so review apps stay cheap. + +## Tier 1 — Logs (zero extra services) + +Whatever SAPI the `php` container runs, it logs to **stdout/stderr**, so Docker captures everything: + +- Nginx Unit writes `access_log` and app stdout/stderr to the container log + ([`docker/unit.json`](../docker/unit.json): `"access_log": "/dev/stdout"`, app `stdout`/`stderr`). +- FrankenPHP/Caddy logs to stdout as well. + +Useful commands: + +```sh +docker compose logs -f php # live web/app log (access + PHP errors) +make drush -- watchdog:tail # live Drupal log (dblog) +make watchdogval # fail if any Emergency|Alert|Critical|Error was logged +``` + +Mail is captured by **Mailpit** (the `mailhog` service in the override), browsable at +`mail-${MAIN_DOMAIN_NAME}` — no real mail leaves the box. + +This already satisfies "logs easy to get from Docker." A handy addition (issue #152): expose Nginx +Unit's status/health route as a liveness probe in the override file's `php` service `healthcheck:`. + +## Tier 2 — Traces (opt-in OpenTelemetry) + +Distributed tracing is prototyped in **PR #466** (`otel` branch). Rather than reinvent it, the plan is +to **rebase and finish that PR**, then gate it behind a flag. What it wires up (none of these files/ +settings exist on this branch yet — they land with PR #466): + +| Piece | Where (added by PR #466) | +| --- | --- | +| Tempo (trace store) + Grafana (UI) services | `docker/docker-compose.override.yml.default` | +| Tempo config | `docker/tempo.yml` | +| Grafana datasource (→ Tempo) | `docker/grafana-datasources.yml` | +| PHP extensions | `ADDITIONAL_PHP_PACKAGES`: `php83-pecl-opentelemetry`, `php83-pecl-grpc`, `php83-pecl-protobuf` | +| Drupal/OTel glue | composer: `mladenrtl/opentelemetry-auto-drupal`, `open-telemetry/exporter-otlp` | +| Wiring | env: `OTEL_PHP_AUTOLOAD_ENABLED=true`, `OTEL_TRACES_EXPORTER=otlp`, `OTEL_EXPORTER_OTLP_ENDPOINT=http://tempo:4318`, `OTEL_EXPORTER_OTLP_PROTOCOL=http/json` | + +**Open TODOs to land it** (carried from the PR): a review-app **collector** path, a **make-it-optional +flag** (a Compose profile / env switch so it is off by default), and starter **Grafana dashboards**. +Rebase onto current master first — the PR predates the Unit-default revert and recent NewRelic bumps. + +The [`observability-up`](../.claude/skills/observability-up/SKILL.md) skill wraps bringing the stack +up/down and printing the Grafana URL; it reuses the existing `ADDITIONAL_PHP_PACKAGES` + `make reload` +install path. + +### Verifying traces + +```sh +# with the opt-in profile enabled: +docker compose up -d # brings up tempo + grafana alongside php +# hit a page, then open Grafana (printed by observability-up) → Explore → Tempo → search traces +``` + +A single page request should produce at least one trace spanning the PHP request and Drupal's +bootstrap/render. + +## Hosted alternative — NewRelic + +For hosted APM instead of self-managed traces, set `NEW_RELIC_LICENSE_KEY` and run `make newrelic` +(enables the agent and the `newrelic` daemon service in the override). The agent version is pinned in +[`scripts/makefile/newrelic.sh`](../scripts/makefile/newrelic.sh) and bumped frequently — that bump is +automated by the [`dep-bump`](../.claude/skills/dep-bump/SKILL.md) skill / +[`newrelic-bump`](../.claude/workflows/newrelic-bump.workflow.js) workflow. + +## Choosing a tier + +- **Just debugging a review app?** Tier 1 logs. +- **Performance / where-is-the-time questions?** Tier 2 traces (or NewRelic if you have a license). +- **Production-like APM?** NewRelic. diff --git a/docs/parallel-environments.md b/docs/parallel-environments.md new file mode 100644 index 000000000..e42db0755 --- /dev/null +++ b/docs/parallel-environments.md @@ -0,0 +1,63 @@ +# Parallel local environments (git worktrees) + +You can run **several full stacks at once on one machine** — one per branch — the same way CI runs one +review app per MR. This is the local equivalent of review apps, and it is what lets agents work on two +branches concurrently (each in its own worktree booting its own stack). + +## Why it already works + +The Makefile keys every Docker resource off `COMPOSE_PROJECT_NAME`: + +- containers are named `${COMPOSE_PROJECT_NAME}_web`, the network is `${COMPOSE_PROJECT_NAME}_front`; +- `DB_MOUNT_DIR` is derived from `COMPOSE_PROJECT_NAME` + the working directory; +- compose calls are pinned to the local `.env` (`docker compose --env-file .env …`). + +`.env` is per-checkout (gitignored, created from `.env.default`). So two checkouts with **different +`COMPOSE_PROJECT_NAME` values don't collide** — different containers, network, and DB storage. +`git worktree` gives you those separate checkouts cheaply, sharing one `.git`. + +## Recipe + +```sh +# from your main checkout +git worktree add ../sdc-feature-x feature-x # new worktree for branch feature-x +cd ../sdc-feature-x +cp ../skilld-docker-container/.env .env 2>/dev/null || cp .env.default .env + +# give THIS worktree a unique project name (or use the helper below) +sed -i 's/^COMPOSE_PROJECT_NAME=.*/COMPOSE_PROJECT_NAME=sdc_feature_x/' .env + +make all # boots an isolated stack for feature-x +``` + +The original checkout's stack keeps running; `make info` in each prints its own URL/logins. Tear one +down with `make clean` in that worktree, then `git worktree remove ../sdc-feature-x`. + +### Optional helper + +[`scripts/makefile/worktree.mk`](../scripts/makefile/worktree.mk) provides `make worktree-name`, which +derives a sanitized `COMPOSE_PROJECT_NAME` from the current git branch (or worktree directory) and +writes it into `.env` — so you don't hand-edit it per worktree. Run it once after creating the +worktree, before `make all`. + +## Caveats (shared host resources) + +A few things are **not** namespaced by `COMPOSE_PROJECT_NAME`; vary them per worktree to avoid clashes: + +| Resource | Collision | Fix per worktree | +| --- | --- | --- | +| `MAIN_DOMAIN_NAME` (Traefik `Host` rule) | Two stacks claiming `docker.localhost` | Set a distinct host, e.g. `featurex.docker.localhost` | +| `/dev/shm` SQLite (`make fast`/`allfast`) | Fixed path `sqlite:///dev/shm/db.sqlite` shared on the host | Use the default per-dir `.cache` DB (don't use `fast`) for parallel stacks | +| Published host `ports:` (if you uncommented any, e.g. macOS `8090:80`) | Same host port twice | Give each worktree a different host port | +| Host RAM / CPU | N stacks = N× resource use | Keep the number sane; `make clean` idle ones | + +If you stick to the defaults (Traefik routing by hostname, per-dir SQLite, no published ports), the +only thing you must set per worktree is `COMPOSE_PROJECT_NAME` (and `MAIN_DOMAIN_NAME` if you use +Traefik locally). + +## Agent tie-in + +Claude Code can run agents in **isolated worktrees** (`Agent` with `isolation: "worktree"`, or the +Workflow tool). Combined with the above, each agent boots its own stack, so the `drupal-upgrade` branch +and a fix branch can build and be tested **at the same time** without stepping on each other. See +[`.claude/README.md`](../.claude/README.md) and [CONTRIBUTING.md](../CONTRIBUTING.md#agent-harness). diff --git a/docs/review-apps.md b/docs/review-apps.md new file mode 100644 index 000000000..7128ab04e --- /dev/null +++ b/docs/review-apps.md @@ -0,0 +1,133 @@ +# Review apps + +A **review app** is an ephemeral, fully-installed Drupal site that this project builds in GitLab CI +for every merge request (and for the default branch and tags). It lets reviewers click through the +exact code under review at a stable URL, and it is the environment the automated `test:*` jobs run +against. + +This document explains how review apps are created, addressed, torn down, and configured. The source +of truth is [`.gitlab-ci.yml`](../.gitlab-ci.yml); the stage-by-stage job reference lives in +[ci-pipeline.md](ci-pipeline.md). + +> **This `.gitlab-ci.yml` is a template.** It is designed to be **included by a downstream project's +> pipeline** as a child pipeline — that is why the jobs gate on `$CI_PIPELINE_SOURCE == 'parent_pipeline'` +> and why the `changes:` rules contain a `{{ project.path }}/**/*` placeholder that the parent +> substitutes. A consuming project includes this file, sets a handful of CI/CD variables (below), and +> inherits the whole review-app lifecycle. + +## Lifecycle at a glance + +```mermaid +sequenceDiagram + participant Dev as Developer + participant CI as GitLab CI + participant Runner as Shell runner (docker+compose+traefik) + participant Traefik + participant Rev as Review app (compose stack) + + Dev->>CI: push / open MR + CI->>Runner: sniffers (phpcs, compose, clang, newlineeof) + CI->>Runner: prepare:back (composer) + prepare:front (yarn) + Note over Runner: artifacts: vendor/, web/, theme dist/ + Dev->>CI: click "build:review" (manual) + CI->>Runner: rsync code to $BUILD_DIR, write .env.default + Runner->>Rev: make all_ci (provision, install, localize) + Rev->>Traefik: container exposes Host(MAIN_DOMAIN_NAME) + Traefik-->>Dev: https://. + CI->>Rev: test:behat / cinsp / lighthouse / watchdog ... + Dev->>CI: click "stop_review" (or auto_stop_in expires) + CI->>Rev: make clean (drop containers, network, volumes) +``` + +## How a review app is built + +All three deploy jobs share one YAML anchor, `.deploy_template` (`.gitlab-ci.yml:205`). Its script: + +1. `mkdir -p ${BUILD_DIR}` and `rsync` the checkout into `${BUILD_DIR}` (excluding `.git` and + `.cache`). **`BUILD_DIR` is provided by the runner/parent environment**, not by this file — it is + the per-app working directory on the runner host. +2. Append two lines to `.env.default` so the stack is uniquely named and addressable: + - `COMPOSE_PROJECT_NAME=${CI_PROJECT_NAME}-review-${CI_COMMIT_REF_SLUG}` + - `MAIN_DOMAIN_NAME=${CI_ENVIRONMENT_SLUG}-${CI_PROJECT_PATH_SLUG}.${REVIEW_DOMAIN}` +3. `make all_ci` (`Makefile:61`) — the CI install path: `provision → si → localize → hooksymlink → + info`. Unlike `make all`, it does **not** run `back`/`front`, because those artifacts were already + built in the `prepare:*` stage and rsynced in. +4. `make drush config-set system.site name '${CI_COMMIT_REF_SLUG}'` so the site name shows the branch. +5. `chmod` and copy `.cache/` (the SQLite DB) and `web/sites/` back to `${CI_PROJECT_DIR}` so GitLab + can capture them as artifacts. + +The `after_script` prunes dangling Docker networks/containers on the runner. + +## How a review app is addressed + +The public URL is always: + +``` +https://${CI_ENVIRONMENT_SLUG}-${CI_PROJECT_PATH_SLUG}.${REVIEW_DOMAIN} +``` + +Routing is done by **Traefik**, configured through container labels in +[`docker/docker-compose.override.yml.default`](../docker/docker-compose.override.yml.default): + +- `traefik.http.routers.web-${COMPOSE_PROJECT_NAME}.rule=Host(\`${MAIN_DOMAIN_NAME}\`)` — match the + review-app hostname. +- `…tls.certresolver=dns` + `…tls=true` — TLS via the runner's DNS cert resolver. +- `…middlewares.web-${COMPOSE_PROJECT_NAME}.basicauth.users=${RA_BASIC_AUTH}` — optional HTTP basic + auth. `RA_BASIC_AUTH` is `username:hashed-password` (`htpasswd -nibB user 'pass'`). When set, every + review URL is protected; `test:lighthouse` authenticates with `RA_BASIC_AUTH_USERNAME/PASSWORD`. + +The Mailpit container is published similarly at `mail-${MAIN_DOMAIN_NAME}`. + +## The three deploy jobs and their TTLs + +The environment block (URL, name, `on_stop`, `auto_stop_in`) comes from one of three TTL anchors that +each deploy job merges on top of `.deploy_template`: + +| Job | Trigger | `when` | TTL anchor | `auto_stop_in` | Notes | +| --- | --- | --- | --- | --- | --- | +| `build:review` | MR pipeline (`$CI_MERGE_REQUEST_IID`) | `manual` | `…ttl_mid` | **1 week** | The everyday reviewer flow | +| `build:master` | default branch | `always` | `…ttl_long` | **1 month** | Auto-deploys the integration env | +| `build:tag` | tag pipeline (`$CI_COMMIT_TAG`) | `manual` | `…ttl_short` | **1 day** | Also archives `web/sites/*/files/` + `.cache` as a 1-week artifact (used by `test:deploy`) | + +All three point `environment.name` at `review/${CI_COMMIT_REF_NAME}` with `on_stop: stop_review`, so +GitLab shows a single environment per ref with a stop button. + +## Tearing a review app down + +- **`stop_review`** (`.gitlab-ci.yml:290`) — `when: manual`, `GIT_STRATEGY: none`, + `environment.action: stop`. Runs `make clean` inside `${BUILD_DIR}` (drops containers, network, + volumes, composer-installed code, DB data) then `rm -rf ${BUILD_DIR}`. It is the `on_stop` handler, + so GitLab also calls it automatically when `auto_stop_in` expires. +- **`generate:logins`** (`.gitlab-ci.yml:316`) — `when: manual`, runs `make info` to print one-click + admin/tester login links and the container IPs for a running app. + +## Required GitLab CI/CD variables + +Set these in the **consuming project** under *Settings → CI/CD → Variables*: + +| Variable | Required? | Purpose | +| --- | --- | --- | +| `REVIEW_DOMAIN` | **Yes** | DNS domain of the review runner (`docker + compose + traefik`); forms the URL | +| Runner tag (`.runner_tag_selection`) | **Yes** | The shell runner that has docker/compose/traefik (edit the `XXX` tag) | +| `THEME_PATH` | If a theme | Enables `sniffers:front` / `prepare:front` / `test:storybook` (`web/themes/custom/XXX`) | +| `STORYBOOK_PATH` | If storybook | Enables `test:storybook` and prints the storybook URL | +| `RA_BASIC_AUTH` | Optional | `user:hashed-pass` for Traefik basic auth on review URLs | +| `RA_BASIC_AUTH_USERNAME` / `RA_BASIC_AUTH_PASSWORD` | If basic auth | Lets `test:lighthouse` reach a protected app | +| `TEST_UPDATE_DEPLOYMENTS` | Optional | `"TRUE"` enables `test:deploy` (update-path simulation) | +| `GITLAB_PROJECT_ACCESS_TOKEN` | For `test:deploy` | Token with `read_api`+`read_repository` to fetch the last tag's artifacts | +| `GITLAB_PROJECT_BASIC_AUTH` | For `test:deploy` | Encoded creds if the repo itself is behind basic auth | +| `RUN_PATCHVAL_CI_JOB` | Optional | Set `"FALSE"` to skip `test:patch` (e.g. unavoidable private-package patches) | +| `NEW_RELIC_LICENSE_KEY` | Optional | Enables the NewRelic PHP agent in the build (`make newrelic`) | +| `IMAGE_PHP` | Optional | Override the PHP image (CI default `skilldlabs/php:83`) | + +## Local equivalent + +The same Makefile mechanics power local development — `make all` builds the identical stack, and a +unique `COMPOSE_PROJECT_NAME` is all that separates two stacks. To run **several review-app-like +stacks at once locally** (one per branch), see [parallel-environments.md](parallel-environments.md). + +## See also + +- [ci-pipeline.md](ci-pipeline.md) — every stage and job, and the job → `make` target → script map. +- [architecture.md](architecture.md) — what runs inside the review-app container. +- [observability.md](observability.md) — logs and traces from a running app. diff --git a/docs/testing.md b/docs/testing.md new file mode 100644 index 000000000..9be3832b7 --- /dev/null +++ b/docs/testing.md @@ -0,0 +1,87 @@ +# Testing & validation + +How to validate this template — both the **project** (Drupal/Docker/Makefile/CI machinery) and the +**agent harness** (`.claude/`) — before merging changes. Three tiers, increasing cost; the merge gate +at the bottom says which must pass. + +The existing `make sniffers`/`make tests` cover the Drupal code. This page adds the **Tier 0 static +gate** for the docs and harness (which `sniffers` does not touch: `clang` is a no-op without +`config/sync`, `phpcs` only sniffs `web/{modules,themes}/custom`, and `newlineeof` only checks +`.env.default`). + +## Tier 0 — Static gate (no Docker) + +`scripts/ci/validate-harness.mjs` (Node, no deps). Run it: + +```sh +node scripts/ci/validate-harness.mjs # directly (needs node) +make harnessval # or containerised (node:lts-alpine, no local node needed) +``` + +In GitLab CI it runs as the **`sniffers:harness`** job (`image: node:lts-alpine`). The lightweight +GitHub Actions gate ([`.github/workflows/validation.yml`](../.github/workflows/validation.yml)) runs the +same Node validator plus the basic `make clang` / `make compval` / `make phpcs` / `make newlineeof` +sniffer set on pull requests. It checks: + +| ID | Check | +| --- | --- | +| T0.1 | Every relative Markdown link in `docs/`, `AGENTS.md`, `CLAUDE.md`, `CONTRIBUTING.md`, `README.md`, `.claude/**` resolves | +| T0.2 | `.claude/settings.json` is valid JSON with a non-empty `permissions.allow` | +| T0.3 | Each `.claude/workflows/*.js` is a valid ES module exporting `meta{name,description}` + an async `run()` | +| T0.4 | Each skill/agent has frontmatter (`name`+`description`; agents also `tools`); `name` matches its path | +| T0.5 | Doc↔reality: every `` `make X` `` in the docs is a real target; CI-doc targets exist in `.gitlab-ci.yml`; key CI vars are documented in `review-apps.md` and used in CI | +| T0.6 | `AGENTS.md` and new files under `docs/`, `.claude/`, `scripts/` end with a newline | +| T0.7 | `.claude/README.md` mentions every skill/agent/workflow on disk | + +**Negative test** (prove the gate bites): add a broken link or a fake `` `make nope` `` to a scratch +file and confirm the gate exits non-zero. + +## Tier 1 — Harness functional (live-invoke) + +Actually run each harness artifact and check behavior. Run anything that edits files in a **throwaway +git worktree** and discard it. + +**Skills** — invoke `/` and verify: +- `review-app-triage` maps a failing job → `make` target, names the first error, proposes a fix; nothing destructive. +- `drupal-upgrade` checks the **druxxy gate first**, reports BLOCKED (`v1.5.1 → core ^10.6`), offers the `PROFILE_NAME=standard` workaround; no unprompted core bump. +- `dep-bump` (NewRelic) edits only the pin in `scripts/makefile/newrelic.sh`, one dependency, validates. +- `backlog-grooming` reads `gh` (no writes), classifies into buckets, regenerates `docs/backlog.md`. +- `observability-up` prints log commands (Tier 1) and flags the PR #466 dependency for traces. + +**Workflows** — run via the Workflow tool (throwaway worktree, discard): +- `drupal-upgrade` → Gate `blocked:true`, Apply in an isolated worktree, Validate spawns `upgrade-validator`, no commits. +- `newrelic-bump` → Find current vs latest; equal → `bumped:false`; else proposes an `NR: ` message, no commit. + +**Subagents** — invoke via the Agent tool: +- `upgrade-validator` runs the gates, reports READY/NOT-READY with file-level blockers; no `composer.json` edits. +- `ci-log-analyzer` is read-only; names job + first error + repro command. +- `dep-bumper` does one pin and refuses a core-major jump. + +## Tier 2 — Project functional (Docker) + regression + +On a Docker host. Because docs/harness changes touch no runtime code, **results must match a `master` +baseline.** + +```sh +make all # T2.1 build: site installs; `make info` gives a reachable URL + logins +make sniffers # T2.2 +make tests # T2.3 full suite (cinsp, rector, upgrade_status, behat, watchdog, …) — green +``` + +- **T2.3** compare `make tests` output to the same run on `master` → identical (no regression). +- **T2.4 worktree isolation** — in two `git worktree`s: `make worktree-name` then `make all`; assert + distinct `COMPOSE_PROJECT_NAME`/network/containers, both URLs up at once; `make clean` both. See + [parallel-environments.md](parallel-environments.md). +- **T2.5 Makefile regression** — `make help` lists `worktree-name`; `make worktree-name` executes + (explicit rule beats the `%: ; @:` catch-all) and edits only `.env`. + +## Merge gate + +Merge a docs/harness PR (e.g. #468) only when **all** hold: + +1. **Tier 0** green (locally + the `sniffers:harness` CI job). +2. **Tier 1** smoke: every skill/workflow/subagent behaves per the criteria above. +3. **Worktree smoke** (T2.4 + T2.5). +4. **Full `make tests` green** (T2.3) and equal to the `master` baseline. + +If a Tier-1 item misbehaves, fix the offending `.claude/` file (not the gate) and re-run. diff --git a/docs/upgrading.md b/docs/upgrading.md new file mode 100644 index 000000000..3194bc13c --- /dev/null +++ b/docs/upgrading.md @@ -0,0 +1,132 @@ +# Upgrade runbook: Drupal 10 → 11 → 12 + +This is the human-readable twin of the `drupal-upgrade` agent workflow +([`.claude/workflows/drupal-upgrade.workflow.js`](../.claude/workflows/drupal-upgrade.workflow.js)). +The strategy is **transitional**: move the template to `^10.3 || ^11` so downstream projects keep +working on D10.3 while D11 becomes possible, then stage D12 separately. + +The verification harness already exists — `make upgradestatusval`, `make drupalrectorval`, +`make cinsp`, and the full `make tests` — so "green" is well defined. See +[ci-pipeline.md](ci-pipeline.md#what-the-validation-gates-actually-assert). + +## ⚠️ Gating dependency: druxxy + +The default install profile is `skilldlabs/druxxy` (external package). **As of this writing, the +latest release `v1.5.1` requires `drupal/core-recommended: ^10.6` — no published druxxy version +supports Drupal 11.** A D11 build is **blocked** until a druxxy major that allows `^11` exists. + +Two paths: +1. **Preferred** — publish/obtain a druxxy major with `drupal/core-recommended: ^10.6 || ^11`, then + proceed below. +2. **Unblock for testing** — temporarily switch `PROFILE_NAME` to a core profile (`standard`/`minimal`) + on a throwaway branch to validate the rest of the upgrade independently of druxxy. + +Re-check druxxy before starting: `composer show skilldlabs/druxxy --all` (or Packagist). + +## Step 1 — Core constraints (transitional) + +In [`composer.json`](../composer.json): + +```diff +- "drupal/core-composer-scaffold": "^10.3.1", +- "drupal/core-vendor-hardening": "^10.3.1", ++ "drupal/core-composer-scaffold": "^10.3.1 || ^11", ++ "drupal/core-vendor-hardening": "^10.3.1 || ^11", +``` + +Bump `skilldlabs/druxxy` to its D11-capable major once available (currently pinned `^1.1`; note even +within D10, `^1.5` exists and tracks `^10.6`). + +## Step 2 — Remove modules dropped from D11 core + +These three were added together for the D10 lock (commit `71f6aba`) and are **removed from Drupal 11 +core**: + +| Remove from `composer.json` | Replacement on D11 | +| --- | --- | +| `drupal/ckeditor` (CKEditor 4) | Core **CKEditor 5** (in core; no module needed) | +| `drupal/color` | None in core — drop unless a contrib successor is genuinely needed | +| `drupal/seven` (admin theme) | Core **Claro** admin theme | + +After removing, grep the repo for references before assuming they are unused: + +```sh +git grep -nE 'ckeditor|\bcolor\b|seven' -- web/ config* settings* .env.default Makefile scripts/ +``` + +Check the `druxxy` profile and any `MODULES`/install code (`.env.default`, `Makefile` `si`/`content`) +for an enabled `seven`/`color`/`ckeditor` dependency, and switch the admin theme to `claro`. + +## Step 3 — Update tooling + +- **drupal-rector** — current pin `^0.20.3` is old. Bump to the latest (`^0.21` or the `1.0.x` line, + which targets Rector 2.x) and add the Drupal 11 set to [`rector.php`](../rector.php) **if** that + release exposes a `Drupal11SetList`: + + ```php + $sets = [ + Drupal8SetList::DRUPAL_8, + Drupal9SetList::DRUPAL_9, + Drupal10SetList::DRUPAL_10, + ]; + // Add the D11 set only if the installed drupal-rector release defines it — + // referencing a missing class would fatal. + if (class_exists(\DrupalRector\Set\Drupal11SetList::class)) { + $sets[] = \DrupalRector\Set\Drupal11SetList::DRUPAL_11; + } + $rectorConfig->sets($sets); + ``` + + If no D11 set ships yet, rely on `upgrade_status` (Step 5) as the authority and keep rector at D10. +- **drush** — already `^13.2`, which supports D11. No change. +- **Other contrib** — `composer why-not drupal/core 11` to surface blockers (e.g. `default_content`, + `migrate_generator`, `imagemagick`); bump each to its D11-compatible release. + +## Step 4 — Re-verify the patch + +[`composer.json`](../composer.json) carries one patch on `drupal/default_content` ("Do not reimport +existing entities"). Confirm it still applies on the D11-compatible release; **drop it if upstreamed** +(`test:patch` only forbids *committed* patch files, not remote URLs, but a stale patch will fail to +apply and break `composer install`). + +## Step 5 — Resolve and validate + +On a throwaway branch (the agent does this in an isolated worktree): + +```sh +composer update --with-all-dependencies +make upgradestatusval # drush upgrade_status:analyze — must report no FILE: lines +make drupalrectorval # rector dry-run over custom code +make cinsp # config schema (run before schema-mutating checks) +make tests # full suite +``` + +Iterate on the `upgrade_status` report until clean. `upgrade_status` is the authority for "is this D11 +ready"; rector helps auto-rewrite deprecated API calls in custom code. + +## Step 6 — Prove it end-to-end + +The real acceptance test is a **green review app on the D11 branch** — open an MR and run +`build:review`, then the `test:*` jobs (see [review-apps.md](review-apps.md)). A passing pipeline is +the definition of done for the transitional step. + +## Step 7 — Drupal 12 readiness (staged, follow-up) + +Do **not** fold D12 into the transitional branch. When D12-compatible druxxy/contrib exist: + +- Switch the PHP image to **8.4** — images already exist: `skilldlabs/php:84-unit` (and + `84-frankenphp`, `84-fpm`). Update `IMAGE_PHP` in `.env.default` and the `IMAGE_PHP` CI default. +- Move core constraints to include `^12`, add the D12 rector set if available, re-run Steps 5–6. +- Revisit `minimum-stability` (currently `dev`) only when pinning for a release. + +## Quick checklist + +- [ ] druxxy D11-capable release available (or profile temporarily swapped) +- [ ] core constraints → `^10.3.1 || ^11` +- [ ] removed `ckeditor` / `color` / `seven`; admin theme → claro; CKEditor 5 verified +- [ ] rector bumped (+ D11 set if present); drush `^13.2` +- [ ] contrib bumped (`composer why-not drupal/core 11` clean) +- [ ] `default_content` patch re-verified or dropped +- [ ] `make upgradestatusval` / `drupalrectorval` / `cinsp` / `tests` green +- [ ] D11 review app builds green +- [ ] D12: PHP 8.4 image + `^12` constraints tracked as a follow-up diff --git a/scripts/ci/validate-harness.mjs b/scripts/ci/validate-harness.mjs new file mode 100644 index 000000000..042610cd2 --- /dev/null +++ b/scripts/ci/validate-harness.mjs @@ -0,0 +1,269 @@ +#!/usr/bin/env node +// Static gate for the docs + .claude/ agent harness (Tier 0 of docs/testing.md). +// Pure Node (ESM), no external deps — runs in node:lts-alpine (the sniffers:harness CI job) +// or locally via `make harnessval`. Exits non-zero on any violation. + +import { readFileSync, existsSync, readdirSync, statSync } from 'node:fs'; +import { join, dirname, normalize, basename, extname } from 'node:path'; +import { pathToFileURL } from 'node:url'; +import { execFileSync } from 'node:child_process'; + +const ROOT = process.cwd(); +const problems = []; +const fail = (check, msg) => problems.push(`${check}: ${msg}`); +let checks = 0; +const did = (check, msg) => { checks++; if (process.env.VERBOSE) console.log(` ok ${check} — ${msg}`); }; + +const read = (p) => readFileSync(join(ROOT, p), 'utf8'); +const exists = (p) => existsSync(join(ROOT, p)); + +function walk(dir, exts) { + const out = []; + const abs = join(ROOT, dir); + if (!existsSync(abs)) return out; + for (const name of readdirSync(abs)) { + const rel = join(dir, name); + const st = statSync(join(ROOT, rel)); + if (st.isDirectory()) out.push(...walk(rel, exts)); + else if (!exts || exts.includes(extname(name))) out.push(rel); + } + return out; +} + +// --------------------------------------------------------------------------- +// T0.1 — Markdown relative links resolve +// --------------------------------------------------------------------------- +function checkLinks() { + const files = [ + ...walk('docs', ['.md']), + ...walk('.claude', ['.md']), + 'AGENTS.md', 'CLAUDE.md', 'CONTRIBUTING.md', 'README.md', + ].filter(exists); + const linkRe = /\]\((?!https?:\/\/|mailto:|#)([^)#]+)(?:#[^)]*)?\)/g; + for (const f of files) { + const base = dirname(f); + const body = read(f); + let m; + while ((m = linkRe.exec(body))) { + const target = normalize(join(base, m[1])); + if (!exists(target)) fail('T0.1', `${f} → broken link "${m[1]}"`); + } + did('T0.1', f); + } +} + +// --------------------------------------------------------------------------- +// T0.2 — settings.json valid + permissions.allow is non-empty strings +// --------------------------------------------------------------------------- +function checkSettings() { + const p = '.claude/settings.json'; + if (!exists(p)) return fail('T0.2', `${p} missing`); + let json; + try { json = JSON.parse(read(p)); } + catch (e) { return fail('T0.2', `${p} invalid JSON: ${e.message}`); } + const allow = json?.permissions?.allow; + if (!Array.isArray(allow) || allow.length === 0) return fail('T0.2', `${p}: permissions.allow must be a non-empty array`); + for (const a of allow) if (typeof a !== 'string' || !a.trim()) fail('T0.2', `${p}: bad allow entry ${JSON.stringify(a)}`); + did('T0.2', p); +} + +// --------------------------------------------------------------------------- +// T0.3 — Workflows: syntax-valid ES modules exporting meta{name,description} + run() +// --------------------------------------------------------------------------- +async function checkWorkflows() { + const files = walk('.claude/workflows', ['.js', '.mjs']); + if (!files.length) return fail('T0.3', 'no workflow files found'); + for (const f of files) { + try { execFileSync('node', ['--check', join(ROOT, f)], { stdio: 'pipe' }); } + catch (e) { + if (e.code !== 'EPERM') { + fail('T0.3', `${f} syntax error: ${String(e.stderr || e).split('\n')[0]}`); + continue; + } + } + try { + const mod = await import(pathToFileURL(join(ROOT, f)).href); + if (!mod.meta || typeof mod.meta.name !== 'string' || typeof mod.meta.description !== 'string') + fail('T0.3', `${f}: must export meta with string name+description`); + if (typeof mod.run !== 'function') + fail('T0.3', `${f}: must export an async run() entry point`); + } catch (e) { fail('T0.3', `${f}: import failed: ${e.message}`); } + did('T0.3', f); + } +} + +// --------------------------------------------------------------------------- +// T0.4 — Skill/agent frontmatter (name+description; agents also tools); name matches path +// --------------------------------------------------------------------------- +function frontmatter(body) { + const m = body.match(/^---\r?\n([\s\S]*?)\r?\n---/); + if (!m) return null; + const fm = {}; + for (const line of m[1].split(/\r?\n/)) { + const mm = line.match(/^([A-Za-z_]+):\s*(.*)$/); + if (mm) fm[mm[1]] = mm[2].trim(); + } + return fm; +} +function checkFrontmatter() { + for (const f of walk('.claude/skills', ['.md']).filter((p) => basename(p) === 'SKILL.md')) { + const fm = frontmatter(read(f)); + if (!fm) { fail('T0.4', `${f}: missing frontmatter`); continue; } + if (!fm.name) fail('T0.4', `${f}: missing name`); + if (!fm.description) fail('T0.4', `${f}: missing description`); + const dir = basename(dirname(f)); + if (fm.name && fm.name !== dir) fail('T0.4', `${f}: name "${fm.name}" != dir "${dir}"`); + did('T0.4', f); + } + for (const f of walk('.claude/agents', ['.md'])) { + const fm = frontmatter(read(f)); + if (!fm) { fail('T0.4', `${f}: missing frontmatter`); continue; } + if (!fm.name) fail('T0.4', `${f}: missing name`); + if (!fm.description) fail('T0.4', `${f}: missing description`); + if (!fm.tools) fail('T0.4', `${f}: agent missing tools`); + const stem = basename(f, '.md'); + if (fm.name && fm.name !== stem) fail('T0.4', `${f}: name "${fm.name}" != file "${stem}"`); + did('T0.4', f); + } +} + +// --------------------------------------------------------------------------- +// T0.5 — Doc ↔ reality cross-checks +// --------------------------------------------------------------------------- +function realMakeTargets() { + const set = new Set(); + // Match rule definitions `name:` / `name::` but not `name :=` variable assignments. + const re = /^([A-Za-z0-9][A-Za-z0-9_-]*):(?!=)/gm; + // Scan all scripts/**/*.mk so opt-in helper targets (e.g. multisite `split`) count as real. + const mkFiles = ['Makefile', ...walk('scripts', ['.mk'])]; + for (const f of mkFiles) { + if (!exists(f)) continue; + let m; const body = read(f); + while ((m = re.exec(body))) set.add(m[1]); + } + return set; +} +function makeTokensInCodeSpans(body) { + const tokens = new Set(); + // Strip escaped backticks, then match fenced (```) blocks and inline (`) spans + // separately so prose between two code blocks is never mistaken for a span. + const cleanBody = body.replace(/\\`/g, ''); + const codeRe = /`{3,}([\s\S]*?)`{3,}|`([^`]+)`/g; + let match; + while ((match = codeRe.exec(cleanBody))) { + const inner = match[1] || match[2]; + if (!inner) continue; + const mr = /\bmake\s+([a-z][a-z0-9_-]*)/g; + let sub; + while ((sub = mr.exec(inner))) tokens.add(sub[1]); + } + return tokens; +} +// Strip shell/YAML `#` comments (respecting quotes) so a variable that appears +// only in a comment doesn't count as "referenced" by runtime config. +function stripHashComments(body) { + return body + .split('\n') + .map((line) => { + let quote = null; + for (let i = 0; i < line.length; i++) { + const ch = line[i]; + if ((ch === '"' || ch === "'") && line[i - 1] !== '\\') { + quote = quote === ch ? null : (quote || ch); + } + if (ch === '#' && !quote) return line.slice(0, i); + } + return line; + }) + .join('\n'); +} +function checkDocReality() { + const targets = realMakeTargets(); + // (a) make targets referenced in docs must exist + const docFiles = ['AGENTS.md', 'CLAUDE.md', 'CONTRIBUTING.md', ...walk('docs', ['.md'])].filter(exists); + for (const f of docFiles) { + for (const t of makeTokensInCodeSpans(read(f))) + if (!targets.has(t)) fail('T0.5a', `${f}: \`make ${t}\` is not a real target`); + did('T0.5a', f); + } + // (b) make targets the CI-pipeline doc cites must actually appear in .gitlab-ci.yml, + // except the local-only aggregators (CI runs their decomposed sub-jobs, not these). + if (exists('docs/ci-pipeline.md') && exists('.gitlab-ci.yml')) { + const ci = read('.gitlab-ci.yml'); + const localAggregators = new Set(['tests', 'sniffers']); + for (const t of makeTokensInCodeSpans(read('docs/ci-pipeline.md'))) + if (!localAggregators.has(t) && !ci.includes(`make ${t}`)) + fail('T0.5b', `docs/ci-pipeline.md cites \`make ${t}\` but no "make ${t}" in .gitlab-ci.yml`); + did('T0.5b', '.gitlab-ci.yml'); + } + // (c) key user-facing review-app variables are documented and backed by runtime config. + const required = [ + 'REVIEW_DOMAIN', 'THEME_PATH', 'STORYBOOK_PATH', 'RA_BASIC_AUTH', + 'TEST_UPDATE_DEPLOYMENTS', 'GITLAB_PROJECT_ACCESS_TOKEN', 'GITLAB_PROJECT_BASIC_AUTH', + 'RUN_PATCHVAL_CI_JOB', 'NEW_RELIC_LICENSE_KEY', 'IMAGE_PHP', + ]; + if (exists('docs/review-apps.md')) { + const doc = read('docs/review-apps.md'); + const runtimeFiles = [ + '.gitlab-ci.yml', '.env.default', 'Makefile', + ...walk('scripts/makefile'), ...walk('docker'), + ].filter((f) => exists(f) && statSync(join(ROOT, f)).isFile()); + const runtimeConfig = runtimeFiles.map((f) => stripHashComments(read(f))).join('\n'); + for (const v of required) { + if (!doc.includes(v)) fail('T0.5c', `review-apps.md does not document CI var ${v}`); + if (!runtimeConfig.includes(v)) fail('T0.5c', `${v} listed as required but not referenced in runtime config`); + } + did('T0.5c', 'review-apps vars'); + } +} + +// --------------------------------------------------------------------------- +// T0.6 — EOF newline on new tree files (covers make newlineeof's blind spot) +// --------------------------------------------------------------------------- +function checkEof() { + const files = [ + ...walk('docs'), ...walk('.claude'), ...walk('scripts/ci'), + 'AGENTS.md', 'scripts/makefile/worktree.mk', 'scripts/makefile/validate.mk', + ].filter((f) => exists(f) && statSync(join(ROOT, f)).isFile()); + for (const f of files) { + const buf = readFileSync(join(ROOT, f)); + if (buf.length && buf[buf.length - 1] !== 0x0a) fail('T0.6', `${f}: no trailing newline`); + did('T0.6', f); + } +} + +// --------------------------------------------------------------------------- +// T0.7 — .claude/README.md mentions every skill/agent/workflow on disk +// --------------------------------------------------------------------------- +function checkSelfConsistency() { + const p = '.claude/README.md'; + if (!exists(p)) return fail('T0.7', `${p} missing`); + const readme = read(p); + const names = [ + ...walk('.claude/skills', ['.md']).filter((f) => basename(f) === 'SKILL.md').map((f) => basename(dirname(f))), + ...walk('.claude/agents', ['.md']).map((f) => basename(f, '.md')), + ...walk('.claude/workflows', ['.js', '.mjs']).map((f) => basename(f)), + ]; + for (const n of names) if (!readme.includes(n)) fail('T0.7', `${p} does not mention "${n}"`); + did('T0.7', `${names.length} components`); +} + +// --------------------------------------------------------------------------- +async function main() { + checkLinks(); + checkSettings(); + await checkWorkflows(); + checkFrontmatter(); + checkDocReality(); + checkEof(); + checkSelfConsistency(); + + if (problems.length) { + console.error(`\n✗ harness static gate: ${problems.length} problem(s) across ${checks} checks:\n`); + for (const p of problems) console.error(` - ${p}`); + process.exit(1); + } + console.log(`✓ harness static gate: all ${checks} checks passed.`); +} + +main().catch((e) => { console.error(e); process.exit(2); }); diff --git a/scripts/makefile/validate.mk b/scripts/makefile/validate.mk new file mode 100644 index 000000000..c3b4955e5 --- /dev/null +++ b/scripts/makefile/validate.mk @@ -0,0 +1,12 @@ +# Static validation of the docs + .claude/ agent harness (Tier 0 of docs/testing.md). +# Needs only Node (no Drupal stack) — runs the validator in the front node image. +# Auto-included by the root Makefile's `include scripts/makefile/*.mk`. + +.PHONY: harnessval + +IMAGE_FRONT ?= node:lts-alpine + +## Validate docs links + the .claude/ agent harness (no containers/site needed) +harnessval: + @echo "Harness static gate (docs links, settings.json, workflows, frontmatter, doc/CI drift)..." + docker run --rm --init -u $(CUID):$(CGID) -v $(CURDIR):/app -w /app $(IMAGE_FRONT) node scripts/ci/validate-harness.mjs diff --git a/scripts/makefile/worktree.mk b/scripts/makefile/worktree.mk new file mode 100644 index 000000000..5e3fa0482 --- /dev/null +++ b/scripts/makefile/worktree.mk @@ -0,0 +1,23 @@ +# Helpers for running several isolated stacks in parallel, one per git worktree. +# See docs/parallel-environments.md. Auto-included by the root Makefile's `include scripts/makefile/*.mk`. +# Note: `sed -i` differs on BSD (macOS) vs GNU (Linux); the recipe branches on uname. + +.PHONY: worktree-name + +## Derive COMPOSE_PROJECT_NAME (and a matching MAIN_DOMAIN_NAME) from the current git branch into .env +worktree-name: + @branch=$$(git rev-parse --abbrev-ref HEAD 2>/dev/null || basename $(CURDIR)); \ + name=$$(echo "$$branch" | tr -cd '[:alnum:]' | tr '[:upper:]' '[:lower:]'); \ + [ -n "$$name" ] || name=$$(basename $(CURDIR) | tr -cd '[:alnum:]' | tr '[:upper:]' '[:lower:]'); \ + [ -f .env ] || cp .env.default .env; \ + if [ "$$(uname)" = "Darwin" ]; then \ + sed -i '' -e "/^COMPOSE_PROJECT_NAME=/ s/=.*/=$$name/" .env; \ + sed -i '' -e "/^MAIN_DOMAIN_NAME=/ s/=.*/=$$name.docker.localhost/" .env; \ + else \ + sed -i -e "/^COMPOSE_PROJECT_NAME=/ s/=.*/=$$name/" .env; \ + sed -i -e "/^MAIN_DOMAIN_NAME=/ s/=.*/=$$name.docker.localhost/" .env; \ + fi; \ + echo "Worktree env set in .env:"; \ + echo " COMPOSE_PROJECT_NAME=$$name"; \ + echo " MAIN_DOMAIN_NAME=$$name.docker.localhost (from branch '$$branch')"; \ + echo "Now run 'make all' to boot this worktree's isolated stack."