This catalogs the test suites that run in CI for docs-internal: what each one covers, what it needs to run, and how risky it is to admin-merge past it when it's red.
For how to run tests locally, see README.md. For the full CI matrix and per-suite environment variables, see .github/workflows/test.yml.
Admin-merge risk = how dangerous it is to admin-merge past a red result. It's driven mainly by blast radius if the red is a real bug, with flake probability as a secondary factor (flake-prone but contained is lower risk; foundational or security is higher risk even when reds are usually real):
- Low: deterministic/mocked or explicitly non-blocking, contained blast radius. Reds are usually real but isolated. Safe-ish if it's the only red and related checks are green.
- Medium: mixed unit + e2e/local-server/network. A red may be real or infra flake; impact contained to one area. Investigate first.
- High: foundational (affects every page) or a security boundary. A real red means broad user-facing breakage. Don't admin-merge without strong justification.
Note: for build-gating suites (data-directory, content-linter, generated-reference suites), a real red often fails the build/deploy downstream anyway, so admin-merging the PR may just move the failure later.
| Suite | Covers | Prereqs | Admin-merge risk |
|---|---|---|---|
| archives | Deprecated GHES versions: redirects, content serving, asset proxying, and metadata headers for archived releases. | Local test server; some tests fetch external GitHub Pages + Azure Blob URLs. | Medium: reds = broken redirects/assets for deprecated versions; external CDN deps can flake. Contained to deprecated versions. |
| article-api | Article API endpoints (/api/pagelist*, /api/article/body, /api/article/meta) behind hovercards, llms.txt, and markdown delivery, plus the transformers for autogenerated pages. |
Local server with fixture ROOT; no Elasticsearch/GitHub/external network. | Medium: reliable (no external infra) but covers real user/API surfaces. Contained, not site-wide. |
| assets | Dynamic image processing (PNG to WebP, resizing), static asset cache headers keyed by checksum URL, and archived enterprise asset proxying. | Local Express server; checked-in fixture images; external calls mocked with nock. | Low: deterministic, mocked network, clear failures. |
| audit-logs | Audit log event structure, field merging, allowlist filtering, GHES version handling, category notes, and page rendering. | Unit tests run standalone; the rendering test needs the local server (getDOM). | Medium: unit tests reliable; the rendering test can flake. Contained to the audit-log reference. |
| automated-pipelines | The pipeline that auto-generates/syncs content from external sources into markdown, plus autogenerated-page rendering and semver conversion. | Local server for the rendering check (retried HTTP calls); fixtures. | Medium: affects rendered docs; the rendering test's retried HTTP calls can flake. |
| color-schemes | Theme cookie parsing and mapping between CSS and Primer React color modes, including bad-input handling. | None (pure-function unit tests). | Low: pure-function unit tests, no flake. |
| content-linter | Markdown/YAML lint compliance: links, frontmatter schema, Liquid syntax, code annotations, alt text, versioning, and data references. | Changed files (DIFF_FILE) or full repo; markdownlint + custom rules. | Low: deterministic; reds = real content problems in the PR. |
| content-render | The Markdown/Liquid to HTML rendering pipeline: templates, parsing, syntax highlighting, link rewriting, and content transforms. | In-process rendering (unified + LiquidJS); cheerio; no external services. | High: affects every page; reds = real rendering bugs site-wide. |
| data-directory | YAML/JSON/Markdown parsing and loading, JSON schema validation, multi-language retrieval with English fallback, and orphaned-feature detection. | Fixtures + temp dirs; ajv/walk-sync/gray-matter; no external services. | Medium: deterministic, but breakage blocks the site build (usually fails build downstream anyway). |
| early-access | Integration of the private docs-early-access repo content into the main site, and that those pages render. | The private docs-early-access repo cloned via the get-docs-early-access action; only runs on docs-internal. | Medium: reliable; reds = bad merge/missing content, but hidden from main nav. |
| events | Event validation/transform, publishing to Hydro analytics, and comment-quality analysis for survey responses. | App server on :4000; Hydro publishing mocked; external language-guesser library. | Medium: needs the full server; Hydro mocked, so won't catch real endpoint failures. |
| fixtures | Validation of the fixture content/data used by e2e tests across the codebase, plus fixture-data sync and internal-link checks. | Local server (getDOM); copy-fixture-data and update-internal-links scripts. | Low: test-only data; reds = test data needs syncing, not prod breakage. |
| frame | Core page rendering, routing, middleware, HTTP headers (CSP/cache/surrogate), content structure, static serving, and markdown negotiation. | E2e server start/stop; cheerio/express/next/nock. | High: foundational; routing/headers affect every page. Some startup fragility, so confirm a red isn't flake. |
| github-apps | The auto-generated GitHub Apps reference: enabled endpoints and permission requirements across auth types, plus the data transforms behind it. | Mostly standalone unit tests; one rendering test needs the local server (getDOM). | Medium: mostly reliable unit tests + one flaky rendering test. Contained to the GitHub Apps reference. |
| graphql | Auto-generated GraphQL API reference schema integrity and the markdown generation/sync for it, across versions. | In-repo schema data + SSR of reference pages; external graphql-inspector/graphql-tools packages. | Medium: deterministic but depends on external graphql packages. Reds = real data-integrity issues. |
| landings | Landing page rendering and search, plus utilities (fuzzy search, article flattening, octicon validation). | Unit tests standalone; getDOM integration tests need the local server (long timeouts) + fixtures. | Medium: unit tests reliable; e2e prone to timeout/startup false positives. |
| languages | Translated content rendering, language routing, layout, and search across all supported languages, including translator-introduced Liquid corrections. | Clones the translation repos, ENABLED_LANGUAGES=all, and a local Elasticsearch with indexed fixtures; only runs on docs-internal. | High: reds can break routing/rendering/search across all localized docs. Caveat: most flake-prone suite (translation repos + ES), so often infra. Confirm either way. |
| observability | Logging (dev + prod), error handling, logfmt, StatsD runtime metrics, tracing init, Failbot reporting, and request-logging middleware. | None live; StatsD/HTTP mocked (nock), fake timers. | Low: fully mocked, deterministic, reliable. |
| products | Product metadata structure, product-name mappings, and i18n product grouping via octicon translation keys. | Reads real content/index.md frontmatter; ajv schema validation; no external services. | Medium: reads real content files; solid but coupled to frontmatter state. |
| redirects | Redirect handling for legacy paths, enterprise versions, language prefixes, API v3/v4, and deprecated editions. | Unit tests standalone; e2e routing tests; some hit archived enterprise URLs (warmup-remotejson-cache mitigates). | High: broken redirects = widespread 404s across legacy paths, languages, and enterprise. Some network flake, so confirm a red isn't infra. |
| release-notes | GHES/GHAE release-notes rendering, routing, YAML validation, and archived-version fallback. | Local server for routing/rendering (via middleware); YAML/Liquid validation; nock mocks. | Medium: YAML validation solid; e2e fragile to infra. |
| rest | Auto-generated REST API reference from the decorated OpenAPI: schema structure, markdown sync, code examples, and rendering. | In-repo decorated schema data; e2e rendering tests need the local server (long timeout). | Medium: schema validation reliable; e2e rendering can flake on timing. |
| search | Search API (v1), AI search autocomplete, combined search, result rendering, query sanitization, and markdown record processing. | Local Elasticsearch with indexed fixtures for API tests; markup/query unit tests run without ES. | Medium: many tests skip without ES (incomplete signal); required API tests fail loudly on real regressions. |
| secret-scanning | That the auto-generated secret-scanning patterns reference page exists and renders without crashing on malformed URLs. | Live local server on :4000; version-specific YAML pattern data. | Medium: reds block critical docs, but e2e can flake on server startup. |
| shielding | Detection/blocking of malicious HTTP requests: honeypotting, malformed URLs, bad query strings/headers, junk paths, and XSS/open-redirect attempts. | Local server via e2e helpers; observability/frame middleware. | High: broken here = compromised prod abuse/XSS protection. Security boundary. |
| versions | The product versioning system: which versions exist and how version-based content rendering works. | Schema validation + Liquid rendering; semver; no external services. | High: foundational; every page depends on it. Breakage = broken routing/unavailable docs for all. |
| webhooks | The auto-generated webhook events/payloads reference, including complex oneOf parameter structures. | Local server (getDOM); schema data synced from rest-api-description. | Medium: deterministic but coupled to the e2e server + externally-synced schema. |
| workflows | Unit tests for three automation scripts: GitHub Actions workflow validation, PR deploy batch commenting, and llms.txt generation. | None live; Octokit mocked. | Low: isolated unit tests, mocked, no live deps. |
| Suite | Covers | Prereqs | Admin-merge risk |
|---|---|---|---|
| playwright-rendering | Client-side JS behavior: navigation, sidebar, search, platform/tool pickers, code tabs, hovercards, responsive layouts, versioning, and Liquid features. | Built app on :4000, Elasticsearch, and Chromium downloaded from a CDN. | Medium: real client-side regressions, but the CDN Chromium download flakes (disabled before for that). Confirm a red isn't the browser install. |
| playwright-a11y | Accessibility violations on a set of representative pages via axe-core, with and without experimental feature flags. | Local Next.js fixture server; Chromium. | Low: explicitly non-blocking; a red may be real a11y or an axe false positive. Doesn't gate PRs. |
| playwright-secret-scanning | Client-side accessibility and UI behavior of the secret-scanning patterns reference page. | CDN Chromium, dev server on :4000, and ES fixtures. | Medium: real UI/a11y regressions, but CDN Chromium flake adds noise. |
These run as their own PR checks rather than through test.yml. Each self-documents in its workflow header comment.
| Check | Covers | Prereqs | Admin-merge risk |
|---|---|---|---|
| lint-code | ESLint, Prettier, and TypeScript checks across the repo. | None live; runs on PR + merge group. | Low: deterministic; reds = real lint/type errors. |
| content-lint-markdown | Lints changed content/data markdown against the content-linter rules. | Changed files only. | Low: deterministic; reds = real content problems. |
| link-check-on-pr | Checks internal links in changed content files. | Changed files; internal links only. | Low: deterministic, internal-only; reds = real broken links. |
| package-lock-lint | Validates package-lock.json integrity. | Runs only when the lockfile changes. | Low: deterministic; reds = real lockfile issues. |
| orphaned-features-check | Detects feature flags no longer referenced. | Changed paths. | Low: deterministic; reds = real orphaned features. |
| orphaned-files-check | Detects orphaned content/asset files. | Changed paths. | Low: deterministic; reds = real orphaned files. |
| validate-asset-images | Validates added/changed asset images (format, size). | Changed image paths. | Low: deterministic; reds = real invalid images. |
| validate-openapi-check | Validates the decorated OpenAPI schema in a Docker container. | Docker build; changed OpenAPI paths. | Medium: deterministic logic, but Docker setup adds some infra flake surface. |
| article-api-docs | Checks the article-api docs stay in sync with the code. | Changed paths. | Low: deterministic; reds = docs need updating. |
| content-linter-rules-docs | Checks content-linter rule docs stay in sync with the rules. | Changed paths. | Low: deterministic; reds = rule docs need updating. |
| triage-unallowed-contributions | Flags PRs that change files forks aren't allowed to change. | pull_request_target; changed paths. | Low: deterministic guard; a red is usually a legit gate. |
| test-changed-content | Runs targeted tests for the content changed in the PR. | Local server/fixtures for changed content. | Medium: targeted e2e on changed content; can flake like other e2e suites. |
| zizmor | Security lint of the GitHub Actions workflows themselves. | Changed workflow files. | Low: deterministic; reds = real workflow security issues. |
| codeql | CodeQL static security analysis of the codebase. | Runs on PR + push; long-running. | Medium: real security findings, but slow and occasionally infra-flaky. |
| dont-delete-assets | Blocks PRs that delete in-use asset files. | Changed asset paths. | Low: deterministic guard; a red is a real deletion to justify. |
| dont-delete-features | Blocks PRs that delete in-use feature files. | Changed feature paths. | Low: deterministic guard; a red is a real deletion to justify. |
- Adding/removing a vitest suite requires updating branch-protection required checks (see the
test.ymlheader comment). languagesonly runs on github/docs-internal (needs private translation repos).- Some lint/checks also run on a schedule (full-repo link checks, full content/data markdown lint). Those don't gate PRs, so they're omitted here.