feat(Playwright): add a11y regression testing - BED-7788 by TheNando · Pull Request #2859 · SpecterOps/BloodHound

TheNando · 2026-06-03T20:10:41Z

Description

Add accessibility regression testing suite via Playwright

Motivation and Context

How Has This Been Tested?

Tested by manually running new testing suite locally

yarn test:a11y
yarn report:a11y

Screenshots (optional):

Types of changes

New feature (non-breaking change which adds functionality)

Checklist:

I have met the contributing prerequisites
- Assigned myself to this PR
- Added the appropriate labels
- Associated an issue: Issues and Pull Requests README #672
- Read the Contributing guide: https://github.com/SpecterOps/BloodHound/wiki/Contributing
I have ensured that related documentation is up-to-date
- Open API docs
- Code comments (GoDocs / JSDocs)
I have followed proper test practices
- Added/updated tests to cover my changes
- All new and existing tests passed

Summary by CodeRabbit

New Features
- Added CLI commands to run accessibility checks and build/serve Allure reports; introduced a shared accessibility testing package.
Tests
- Automated WCAG A/AA accessibility tests added for multiple UI pages with themed auth snapshots and artifact capture.
Documentation
- New README documenting shared accessibility testing utilities, conventions, and how to add a11y specs.
Chores
- Example env vars added for accessibility runs; test artifacts directory added to ignore list.
Bug Fixes
- Dev startup now tolerates local test-service registration failures and continues rendering.

TheNando · 2026-06-03T20:13:22Z

            <div className={classes.applicationContainer} id='app-root'>
                {showNavBar && <MainNav mainNavData={mainNavData} />}
-                <div className='bg-neutral-1 grow overflow-y-auto overflow-x-hidden'>
+                <div id='content-wrapper' className='bg-neutral-1 grow overflow-y-auto overflow-x-hidden'>


This adds an easy and repeatable selector to only capture the page component content, omitting the Nav menu, which may be tested in isolation. This reduces overhead for Axe Core. This is purely for makeAxeBuilder analyze targeting. User-perspective selectors (byRole, byLabel, etc) are still used to gauge loading and viewable content state.

coderabbitai · 2026-06-03T20:15:14Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds Playwright+ Axe accessibility testing: a shared bh-playwright-testing package (fixtures, Axe helpers, stubs), Playwright a11y config and scripts, global setup to snapshot auth state per theme, multiple a11y specs, docs, minor UI tweaks, and workspace/build integration.

Changes

Accessibility testing framework

Layer / File(s)	Summary
Axe foundation and package metadata `packages/javascript/bh-playwright-testing/{.prettierrc,package.json,tsconfig.json,src/index.ts,src/axe.ts}`, `packages/javascript/bh-playwright-testing/README.md`, `packages/javascript/bh-playwright-testing/eslint.config.cjs`	Core Axe/Playwright test fixture with WCAG tags, formatted violation reporting, optional per-node screenshots, assertion helper, Prettier/TS/package metadata, ESLint flat-config, and top-level README.
Theme management and authentication `packages/javascript/bh-playwright-testing/src/{themes.ts,auth.ts}`	Theme types/constants and `loginAndSnapshotThemes` helper that logs in, toggles dark mode, polls persisted state, and writes per-theme Playwright storageState files.
Route stubs and package entry `packages/javascript/bh-playwright-testing/src/stubs/graph-has-data.ts`, `packages/javascript/bh-playwright-testing/src/index.ts`	Playwright route stub for the graph Cypher endpoint and package entry re-exports with README usage.
Playwright test configuration `cmd/ui/playwright.a11y.config.ts`, `cmd/ui/package.json`, `cmd/ui/.env.example`	Playwright accessibility runner: env-driven baseURL, optional dev webServer, browser/theme project matrix (setup dependency), CI-friendly retries/workers, reporters (console/HTML/Allure), and UI workspace test scripts/devDependencies and env examples.
Test fixtures and global setup `cmd/ui/tests/{fixtures.ts,global.setup.ts}`	Fixture extends Playwright test to install the graph-data stub and re-export a11y assertions; global.setup logs in and snapshots auth state for light/dark themes with best-effort post-login dismissal.
Accessibility test specifications `cmd/ui/tests/a11y/{api-explore,download-collectors,early-access-features,explore,login,no-data-dialog}.a11y.spec.ts`	Six Playwright Axe specs that navigate to pages, wait for stable elements, run scoped Axe scans (typically `#content-wrapper` or dialog scope), and assert no WCAG A/AA violations.
Testing documentation `cmd/ui/tests/README.md`	Documentation for test layout, shared scaffolding, setup behavior, env variables, run commands, artifacts, reporting, and guidance for adding new a11y specs.
UI runtime changes `cmd/ui/src/{App.tsx,main.tsx}`	Adds `id="content-wrapper"` for Axe scoping and wraps dev-only MSW initialization in try/catch to avoid startup failures blocking rendering.
Build and workspace integration `.gitignore`, `cmd/ui/.env.example`, `cmd/ui/tsconfig.{json,node.json}`, root `package.json`, `tools/docker-compose/ui.Dockerfile`, `yarn-workspaces.json`, `packages/go/stbernard/command/license/internal/cmd.go`, `cmd/ui/vite.config.ts`	Ignores Playwright artifacts, adds A11Y env examples, adjusts TS node includes, adds root/UI scripts for a11y test/reporting, copies package.json into Docker build context, registers workspace package, excludes Playwright dir from license scanning, and updates Vitest exclusions.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Suggested labels

user interface

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly describes the main change: adding Playwright-based accessibility regression testing with specific reference to the ticket BED-7788.
Description check	✅ Passed	The PR description provides a clear summary, resolves a tracked issue, includes testing evidence, identifies the change type, and completes the contributor checklist.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch BED-7788-playwright-a11y

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

TheNando · 2026-06-03T20:21:00Z

+
+import { expect, expectNoAccessibilityViolations, test } from 'bh-playwright-testing';
+
+test.describe('Login page accessibility', () => {


An example of an exceptional test. All other tests are pre-authed. But to test login we need to first wipe out the auth.

TheNando · 2026-06-03T20:22:15Z

+const GRAPH_HAS_DATA_QUERY = 'MATCH (A) WHERE NOT A:MigrationData RETURN A LIMIT 1';
+
+test.describe('No Data Available dialog accessibility', () => {
+    test('upload dialog has no detectable WCAG A/AA violations', async ({ page, makeAxeBuilder }, testInfo) => {


Another exceptional test. Cypher graph state is stubbed to ensure "No Data" dialog is not present. To test this dialog, we change the stub to be explicitly empty.

TheNando · 2026-06-03T20:24:33Z

@@ -1 +1,6 @@
 TARGET_PROXY_URL="http://localhost:8080"
+
+A11Y_TEST_URL="http://127.0.0.1:3000"


I need to double check that the README explicitly mentions that adding these .env vars is required for authing test sessions.

TheNando · 2026-06-03T20:26:48Z

        "noFallthroughCasesInSwitch": true,
        "declaration": true,
        "declarationMap": true,
-        "baseUrl": ".",


The only real change here is removing the baseUrl, which is a deprecated practice. The rest is just whitespace.

TheNando · 2026-06-03T21:42:53Z

 			"justfile",
 			filepath.Join("cmd", "api", "src", "api", "static", "assets"),
 			filepath.Join("cmd", "api", "src", "cmd", "testidp", "samlidp"),
+			filepath.Join("cmd", "ui", "playwright"),


This ensures that just prepare-for-codereview ignores any of the generated a11y test artifacts.

urangel · 2026-06-17T15:25:42Z

+// shared UI shell (login form labels "Email Address" / "Password", the LOGIN submit button,
+// the `global_nav-dark-mode` toggle, and the `persistedState` localStorage key written by
+// the global store's throttled subscriber) being identical across consumers.
+export async function loginAndSnapshotThemes(opts: LoginAndSnapshotThemesOptions): Promise<void> {


We can reduce some of the complexity here by only persisting the auth token since the difference between the two persisted files is a toggle on the darkMode boolean.

A helper that produces a certain local storage state similar to what is done in the login spec would allow for more flexibility.

urangel · 2026-06-17T15:30:08Z

+        }
+        return route.fulfill({
+            json: {
+                data: {


There are some mock factory utilities in the ui projects that produce similar shapes of data. It would be nice to share and reuse some of these for related ideas to be coupled and maintained together. I could see us making a different package in packages/javascript specifically for this so that both unit tests and integration tests can consume them.

That would be too much for adding here but I wanted to voice that so it's on our minds.

This is a good consideration. My main concern is dependency shape. bh-playwright-testing is a small, mostly agnostic helper package. Pulling in a dep from bh-shared-ui (where our mocks are) would introduce React, js-client-library, msw, doodle-ui...the whole stack. This could especially get tricky when we try to point Playwright at Doodle directly to do component testing.

One solution we could consider would be to generate the mocks at the app level (at cmd/ui) and pass those values in with installGraphHasDataStub. I'll try that out and see how it goes. The other alternative that comes to mind would changing this playwright lib to be a generalized testing lib, and move all the mocks and fixtures in here.

What do you think?

True, dep shape could get tricky. I was describing having a sibling package in packages/javascript, maybe packages/javascript/mocks, though we probably need to isolate types better in advance. Having response types generated from the open api spec would help get us there.

Passing in the mock sounds like a good solution for this case but no need to make any changes unless you want. My comment was a general observation and thought about similar patterns and maintenance but I don't have a solid idea of what that should look like 😄

urangel

Great work! Looking forward to having this run in CI

TheNando self-assigned this Jun 3, 2026

TheNando requested review from a team as code owners June 3, 2026 20:10

TheNando added documentation Improvements or additions to documentation enhancement New feature or request javascript Pull requests that update javascript code labels Jun 3, 2026

TheNando commented Jun 3, 2026

View reviewed changes

This comment was marked as outdated.

Sign in to view

TheNando commented Jun 3, 2026

View reviewed changes

TheNando force-pushed the BED-7788-playwright-a11y branch from 253075d to d60928d Compare June 3, 2026 21:41

TheNando commented Jun 3, 2026

View reviewed changes

This comment was marked as outdated.

Sign in to view

TheNando force-pushed the BED-7788-playwright-a11y branch from 963cc93 to 0987d6b Compare June 9, 2026 16:33

coderabbitai Bot added the user interface A pull request containing changes affecting the UI code. label Jun 9, 2026

TheNando force-pushed the BED-7788-playwright-a11y branch 2 times, most recently from 763e84f to 3bbf6ea Compare June 9, 2026 19:58

dcairnsspecterops reviewed Jun 10, 2026

View reviewed changes

Comment thread cmd/ui/playwright.a11y.config.ts Outdated

TheNando force-pushed the BED-7788-playwright-a11y branch from 3bbf6ea to 5f2e620 Compare June 11, 2026 16:03

Holocraft reviewed Jun 16, 2026

View reviewed changes

Comment thread packages/javascript/bh-playwright-testing/src/axe.ts

urangel reviewed Jun 17, 2026

View reviewed changes

urangel approved these changes Jun 17, 2026

View reviewed changes

TheNando added 4 commits June 22, 2026 11:38

feat(Playwright): add a11y regression testing - BED-7788

9ea657f

Fix Playwright project lint and run PFC

93992d0

CodeRabbit suggestions and fixes

6e94111

More CodeRabbit updates

e61694b

TheNando added 3 commits June 22, 2026 11:38

Simplify READMEs

cf742be

Change config key style

25e4a50

Simplify auth

fc08cc4

TheNando force-pushed the BED-7788-playwright-a11y branch from 5f2e620 to fc08cc4 Compare June 22, 2026 16:39

TheNando merged commit 002b13c into main Jun 22, 2026
12 checks passed

TheNando deleted the BED-7788-playwright-a11y branch June 22, 2026 16:51

github-actions Bot locked and limited conversation to collaborators Jun 22, 2026


		import { expect, expectNoAccessibilityViolations, test } from 'bh-playwright-testing';

		test.describe('Login page accessibility', () => {

		@@ -1 +1,6 @@
		TARGET_PROXY_URL="http://localhost:8080"

		A11Y_TEST_URL="http://127.0.0.1:3000"

Uh oh!

Conversation

TheNando commented Jun 3, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

How Has This Been Tested?

Screenshots (optional):

Types of changes

Checklist:

Summary by CodeRabbit

Uh oh!

TheNando Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

Changes

Estimated code review effort

Suggested labels

❌ Failed checks (1 warning)

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TheNando Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

urangel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

TheNando commented Jun 3, 2026 •

edited by coderabbitai Bot

Loading

TheNando Jun 3, 2026 •

edited

Loading

coderabbitai Bot commented Jun 3, 2026 •

edited

Loading

TheNando Jun 17, 2026 •

edited

Loading