Skip to content

Commit e51d603

Browse files
azasypkinazasypkin
committed
chore(dev): update AGENTS.md with dependency upgrade tips
1 parent 26e9cfb commit e51d603

1 file changed

Lines changed: 183 additions & 0 deletions

File tree

‎AGENTS.md‎

Lines changed: 183 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,188 @@
11
# AGENTS.md
22

3+
## Dependency upgrades
4+
5+
This repo's dependency surface spans two Cargo workspaces (root + the `components/retrack`
6+
submodule), four NPM packages (`components/secutils-webui`, `components/secutils-docs`,
7+
`components/retrack` + `components/retrack/components/retrack-web-scraper`, the e2e harness
8+
and the workspace-root `package.json`), six Dockerfiles, an Ory Kratos server image, and
9+
the `playwright-core` / `@playwright/test` / `playwright-python` triple. A naive "bump
10+
everything in one commit" produces an unreviewable diff and an undebuggable failure mode.
11+
Always upgrade in **eleven sequential stages**, each individually committable and verifiable.
12+
13+
### Recommended upgrade order
14+
15+
The order is dictated by data flow: the retrack submodule is consumed as a path-dependency
16+
by the root crate, so anything that touches the retrack Rust API must land before the
17+
parent re-pins it; Node and `playwright-core` versions must be bumped before the Dockerfile
18+
rebuilds (otherwise `npm ci` in the runtime stage validates against a stale lock); Kratos
19+
sits between the auth e2e flows and the webui's `@ory/kratos-client-fetch` and benefits
20+
from being upgraded as a coupled pair.
21+
22+
1. **Retrack Rust crates** — `components/retrack/Cargo.toml`,
23+
`components/retrack/components/retrack-types/Cargo.toml`,
24+
`components/retrack/benches/js-runtime-perf/Cargo.toml`. See
25+
`components/retrack/AGENTS.md` for the in-submodule recipe (insta snapshots,
26+
`.sqlx/` cache, perf harness).
27+
2. **Retrack `.nvmrc`** — bump the Node major; mirror in every `engines.node` and
28+
`@types/node` ^M.x inside the submodule.
29+
3. **Retrack NPM packages** — submodule root + `retrack-web-scraper`. **Pin the
30+
`playwright-core` exact version here** — every other consumer (webui, e2e, the
31+
`playwright-python` git ref baked into `Dockerfile.web-scraper-camoufox`) must be moved
32+
to the same minor in their respective stages.
33+
4. **Retrack Docker base images** — `Dockerfile`, `Dockerfile.web-scraper`,
34+
`Dockerfile.web-scraper-camoufox`. UPX, Camoufox triple, `playwright-python` git ref.
35+
5. **Kratos** — bump `oryd/kratos` server image in `dev/docker/docker-compose.yml` **and**
36+
`@ory/kratos-client-fetch` in `components/secutils-webui/package.json` together. Read
37+
the Ory release notes for any registration/login/recovery flow schema changes; verify
38+
end-to-end against `e2e/tests/registration.spec.ts`.
39+
6. **Root Rust crates** — `Cargo.toml`, `components/secutils-jwt-tools/Cargo.toml`,
40+
`benches/js-runtime-perf/Cargo.toml`. **First** update the retrack submodule pointer
41+
to the SHA you committed in stages 1–4 (`cd components/retrack && git pull` then
42+
`git add components/retrack` from the parent), then `cargo update`.
43+
Refresh `.sqlx/` against the dev Postgres.
44+
7. **Root `.nvmrc`** — bump to the same Node major as stage 2; mirror in `engines.node`
45+
and `@types/node` of all four root-level `package.json` files (root, secutils-webui,
46+
secutils-docs, e2e). Refresh all four lockfiles.
47+
8. **`components/secutils-webui` NPM packages** — read the EUI / React / Parcel release
48+
notes (EUI majors and Parcel resolver behaviour have repeatedly forced workarounds,
49+
see "What to watch for" below). Re-pin `playwright-core` to the same exact version as
50+
stage 3.
51+
9. **`components/secutils-docs` NPM packages** — Docusaurus majors change config schemas
52+
(e.g. `siteConfig.markdown.hooks.onBrokenMarkdownLinks` migration in 3.10), and
53+
`docusaurus-plugin-llms` defaults change between minors. Verify `llms.txt` /
54+
`llms-index.txt` and the per-page `.md` companions resolve, and that the Nginx config
55+
serves them with the right `Content-Type`.
56+
10. **Root Docker base images** — `Dockerfile`, `Dockerfile.docs`, `Dockerfile.webui`. UPX,
57+
distroless runtime, `nginx-unprivileged`. Re-pin SHA256 manifest digests with
58+
`./dev/scripts/docker-pin-digests.sh`. Rebuild the e2e stack and curl-smoke each
59+
service.
60+
11. **`e2e/` harness** — bump `@playwright/test` to match the `playwright-core` from
61+
stages 3 & 8 (within the same minor), refresh ESLint / TypeScript-ESLint / globals.
62+
`npx playwright install chromium`. Run **all three test suites**: standalone
63+
(`make e2e-standalone-test`), full e2e (`make e2e-test`), docs screenshots
64+
(`make docs-screenshots`).
65+
66+
Do **not** reorder. The most common mistake is bumping the root Rust crates (stage 6)
67+
before the retrack submodule pointer is updated — `cargo update` will then either downgrade
68+
the workspace, or fail to compile because the retrack code on disk has already been
69+
upgraded but its public types now mismatch what the parent expects.
70+
71+
### Stage-by-stage verification
72+
73+
Each stage has a hard verification gate before commit. Use the matching `make` target;
74+
do not skip steps even when the previous stage was green.
75+
76+
| Stage | Verify |
77+
|--------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
78+
| 1 (retrack Rust) | `cargo fmt --check && cargo clippy --all-targets -- -D warnings && cargo test && make perf ANALYZE=1 PERF_ITERATIONS=20 PERF_WARMUP=5` |
79+
| 2 (retrack Node) | `npm install && npm run lint --ws --if-present && npm test --ws --if-present && npm run build --ws --if-present` |
80+
| 3 (retrack NPM) | same as stage 2 |
81+
| 4 (retrack Docker) | `make docker-scraper && make docker-scraper-camoufox && make docker-api` (in retrack) |
82+
| 5 (Kratos) | `make e2e-up BUILD=1 && make e2e-test ARGS="tests/registration.spec.ts"` |
83+
| 6 (root Rust) | `cargo +nightly fmt --check && cargo clippy --workspace --all-targets -- -D warnings && cargo sqlx prepare --check && cargo test` |
84+
| 7 (root Node) | `npm --prefix components/secutils-webui run build && npm --prefix components/secutils-docs run build` (and lint/test in each) |
85+
| 8 (webui NPM) | `npm --prefix components/secutils-webui run lint && npm --prefix components/secutils-webui run test && npm --prefix components/secutils-webui run build && npm --prefix components/secutils-webui run analyze` |
86+
| 9 (docs NPM) | `npm --prefix components/secutils-docs run typecheck && npm --prefix components/secutils-docs run build` — then check the build output for `llms.txt`, `llms-index.txt`, and at least one per-page `.md` |
87+
| 10 (root Docker) | `make docker-api && make docker-webui && make docker-docs && make e2e-up BUILD=1` then curl-smoke `/api/status`, `/`, `/docs/` |
88+
| 11 (e2e) | `make e2e-standalone-test && make e2e-test && make docs-screenshots` (full suites — partial runs miss DNS / network regressions, see below) |
89+
90+
### What to watch for
91+
92+
#### Rust (stages 1 & 6)
93+
94+
- **`deno_core`** bumps invalidate the `js_runtime::tests::can_access_deno_apis` snapshot
95+
(same as in retrack) and may pin a transitive `deno_error` patch version that needs
96+
matching in the workspace `Cargo.toml` (e.g. 0.7.1 vs 0.7.3).
97+
- **`sqlx`** macros validate against `.sqlx/`. After any query change or `sqlx` bump:
98+
```bash
99+
docker compose -f dev/docker/docker-compose.yml up -d secutils_db
100+
cargo sqlx prepare
101+
```
102+
CI runs `cargo sqlx prepare --check` and fails when the cache drifts.
103+
- **`serde_json/arbitrary_precision`** is enabled by `secutils` and propagates via Cargo
104+
feature unification to the path-dependent `retrack-types`. `retrack-types`'s snapshots
105+
were recorded without the feature, so `cargo test --workspace` from the **root** repo
106+
will show snapshot diffs (the values become `serde_json::private::Number` instead of
107+
numeric literals). This is a known latent issue; CI runs `cargo test` (not
108+
`--workspace`) so it never trips. Do not "fix" it by re-recording the retrack snapshots
109+
against the unified feature set — that breaks retrack's own CI.
110+
111+
#### Node / NPM (stages 2, 3, 7, 8, 9, 11)
112+
113+
- **Node 24 removed `--experimental-global-webcrypto`** — `globalThis.crypto` is now
114+
always present. Removing the execArgv flag in `worker.ts` is mandatory; **also**
115+
`delete (globalThis as { crypto?: unknown }).crypto;` inside the sandboxed user-script
116+
worker so user code cannot reach the host's `WebCrypto` API.
117+
- **ESLint 10** ships `preserve-caught-error` (rethrow with `{ cause: err }`) and
118+
`eslint-plugin-import@2.32.0` does not yet support v10 as a peer. The whole project is
119+
pinned to **ESLint 9.x** until plugin support catches up. Do not bump `eslint` /
120+
`@eslint/js` past `^9` in any leaf `package.json`.
121+
- **TypeScript 6** deprecates `compilerOptions.baseUrl`. Remove it and migrate any `paths`
122+
entries to direct relative imports. The root project is pinned to **TS 5.9.x** because
123+
Docusaurus and EUI's TS consumers have not validated v6 yet.
124+
- **`eslint-plugin-react-hooks` 7.x** rejects `useCallback(debounce(...))` as improper
125+
memoization. Use `useMemo(() => debounce(...), [])` instead.
126+
- **`@peculiar/x509` v2 + Parcel.** v2 transitively pulls `@peculiar/utils` which uses
127+
`package.json#exports` subpaths (`./bytes`). Parcel 2.16's default resolver does not
128+
honour `exports` subpaths and fails with `Failed to resolve '@peculiar/utils/bytes'`.
129+
Two options: (a) keep `@peculiar/x509` pinned to `^1.x`, or (b) explicitly enable
130+
Parcel's `exports` resolution via:
131+
```json
132+
"@parcel/resolver-default": { "packageExports": true }
133+
```
134+
in the leaf `package.json`. The webui takes option (a) for now to avoid the
135+
`reflect-metadata` polyfill v2 requires.
136+
- **`http-proxy-middleware` v4 is ESM-only**, so it cannot be required from the CommonJS
137+
`.proxyrc.ts`. Stay on v3 for the Parcel dev-server proxy.
138+
- **EUI majors** read the entire CHANGELOG. Common patterns: token renames in CSS-in-JS
139+
(`euiTheme.colors.*`), new required props on data-grid, default ARIA-label changes that
140+
break `getByRole({ name: ... })` selectors in e2e tests.
141+
- **`playwright-core` and `@playwright/test` must share a minor.** The webui pins
142+
`playwright-core` (used at runtime to render preview), retrack pins it for the scraper,
143+
and the e2e harness pins `@playwright/test`. They drive the same Chromium and the same
144+
CDP protocol — a minor mismatch surfaces as "browser closed unexpectedly". After
145+
bumping, run `make e2e-standalone-test` first; the codegen smoke test detects breaking
146+
changes to Playwright's `--target` boilerplate before they corrupt the webui's script
147+
transformer.
148+
149+
#### Docusaurus (stage 9)
150+
151+
- **Nginx `types {}` in server scope replaces, not merges.** When adding `text/markdown
152+
md;` to serve the per-page companions, you must `include /etc/nginx/mime.types;` first
153+
in the same `server { … }` block, otherwise `.txt` (and everything else) regresses to
154+
`application/octet-stream`. `llms.txt` is `text/plain` per spec; the per-page
155+
companions are `text/markdown`.
156+
157+
#### Docker (stages 4 & 10)
158+
159+
- **Re-pin SHA256 digests on every bump** with `./dev/scripts/docker-pin-digests.sh`
160+
(root) or `./dev/scripts/docker-pin-digests.sh` inside the retrack submodule. The
161+
scripts read `FROM image:tag@sha256:...`, drop the digest, query
162+
`docker buildx imagetools inspect`, and rewrite. They always re-pin, even when the
163+
tag is unchanged — rolling tags drift between runs.
164+
- **Disk pressure during the Rust image build is real** — the secutils API image
165+
compiles the full workspace from scratch when the BuildKit cache is cold. If the build
166+
fails with `No space left on device`, run `make docker-prune` (which prunes both
167+
dangling images and BuildKit cache) and retry.
168+
- See the retrack AGENTS.md for the workspace-layout `npm ci` gotcha and the Camoufox
169+
triple — those rules apply identically when bumping the root images that depend on
170+
retrack's runtime images.
171+
172+
#### Submodule pointer & commit hygiene
173+
174+
- Stage 6 is the **only** stage that updates the retrack submodule pointer. Always do
175+
`git submodule update --remote components/retrack` (or pull manually inside the
176+
submodule), then `git add components/retrack` from the parent before running the rest
177+
of stage 6's verification. Forgetting to commit the pointer leaves the parent referring
178+
to a pre-upgrade SHA and CI re-runs the old code.
179+
- The repo enforces **conventional commits** via husky `commit-msg`. Use
180+
`chore(deps): ...` for dependency-only commits, `chore(docker): ...` for image re-pins
181+
and `chore(submodule): ...` for the retrack pointer bump. Commitlint major bumps
182+
(e.g. v20 → v21) only change Node minimum; the existing config keeps working.
183+
- **The performance harness is advisory** (see "Tuning" below). A regression after a
184+
`deno_core` / `tokio` / `reqwest` bump is informational; CI never fails on it.
185+
3186
## End-to-End tests (`e2e/`)
4187

5188
### Overview

0 commit comments

Comments
 (0)