Skip to content

Commit ce2f468

Browse files
Add dry_run option to the accessibility scanner (#232)
Tracking issue with context: github/accessibility#10757 Based on feedback from `@joseinthearena`: with many open scanner issues there was no way to preview what a scan would do without actually filing, closing, or reopening issues — and no way to do so without mutating the `gh-cache` branch. This adds a `dry_run` input that runs a normal scan and **logs the issues that would be filed**, while making no changes: no issues are opened, closed, or reopened, no Copilot assignment, and the cache is not written. Because dry runs don't update the cache, the next real run behaves exactly as if the dry run never happened. ## Changes - **`action.yml` (root)** — New `dry_run` input (default `false`); forwarded to the `file` step; skips the `Fix`, `Save screenshots`, `Copy results to cache path`, and `Save cached results` steps when `dry_run` is `true` - **`.github/actions/file/action.yml`** — New `dry_run` input declared - **`file/src/index.ts`** — When `dry_run` is set, each filing is classified via the existing `isNewFiling` / `isRepeatedFiling` / `isResolvedFiling` guards and the intended action is logged (early `continue`, so no Octokit calls are made); grouped/tracking-issue creation is skipped; a summary count is logged; `filings_file` is still written so the output contract is unchanged - **`README.md`** — `dry_run` input documented in the inputs table and getting-started example - **`FAQ.md`** — New entry explaining how to preview a scan and noting that dry runs don't write the cache ### Test Updates - Added `file/tests/dryRun.test.ts` — asserts no issues are opened/closed/reopened in dry run, the correct `[dry run] Would …` lines and summary are logged, and `filings_file` is still written; includes a regression guard that mutations still occur when `dry_run` is `false` ## Usage ```yaml - uses: github/accessibility-scanner@v3 with: urls: | https://example.com repository: owner/repo token: ${{ secrets.GH_TOKEN }} cache_key: cached_results.json dry_run: true # Scan and log what would be filed without creating/closing issues or writing the cache ``` When `dry_run` is not provided, the scanner behaves exactly as before. ## Notes - Design decision: cache *restore* is kept (read-only) so dry-run logs can accurately distinguish "would open (new)" from "would reopen (existing)"; only the cache *save* is skipped. - The `file`-action logic is covered by the new tests. The `action.yml` step gating was verified by inspection but not run end-to-end — flagging for confirmation in CI. - Per the tracking issue's second acceptance criterion, this will need a new minor release (and the `v3` tag re-pointed) cut by a maintainer.
2 parents dba0765 + c5538c2 commit ce2f468

6 files changed

Lines changed: 290 additions & 41 deletions

File tree

.github/actions/file/action.yml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,10 @@ inputs:
2424
description: "In the 'file' step, also open grouped issues which link to all issues with the same root cause"
2525
required: false
2626
default: "false"
27+
dry_run:
28+
description: "When true, log the issues that would be filed without opening, closing, or reopening any issues."
29+
required: false
30+
default: "false"
2731

2832
outputs:
2933
filings_file:

.github/actions/file/src/index.ts

Lines changed: 69 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -29,12 +29,14 @@ export default async function () {
2929
? JSON.parse(fs.readFileSync(cachedFilingsFile, 'utf8'))
3030
: []
3131
const shouldOpenGroupedIssues = core.getBooleanInput('open_grouped_issues')
32+
const dryRun = core.getBooleanInput('dry_run')
3233
core.debug(`Input: 'findings_file: ${findingsFile}'`)
3334
core.debug(`Input: 'repository: ${repoWithOwner}'`)
3435
core.debug(`Input: 'base_url: ${baseUrl ?? '(default)'}'`)
3536
core.debug(`Input: 'screenshot_repository: ${screenshotRepo}'`)
3637
core.debug(`Input: 'cached_filings_file: ${cachedFilingsFile}'`)
3738
core.debug(`Input: 'open_grouped_issues: ${shouldOpenGroupedIssues}'`)
39+
core.debug(`Input: 'dry_run: ${dryRun}'`)
3840

3941
const octokit = new OctokitWithThrottling({
4042
auth: token,
@@ -61,50 +63,69 @@ export default async function () {
6163
// Track new issues for grouping
6264
const newIssuesByProblemShort: Record<string, FindingGroupIssue[]> = {}
6365
const trackingIssueUrls: Record<string, string> = {}
66+
const dryRunCounts = {open: 0, reopen: 0, close: 0}
6467

6568
for (const filing of filings) {
6669
let response: OctokitResponse<IssueResponse> | undefined
6770
try {
68-
if (isResolvedFiling(filing)) {
69-
// Close the filing’s issue (if necessary)
70-
response = await closeIssue(octokit, new Issue(filing.issue))
71-
filing.issue.state = 'closed'
72-
} else if (isNewFiling(filing)) {
73-
// Open a new issue for the filing
74-
response = await openIssue(octokit, repoWithOwner, filing.findings[0], screenshotRepo)
75-
;(filing as Filing).issue = {state: 'open'} as Issue
71+
if (dryRun) {
72+
if (isResolvedFiling(filing)) {
73+
dryRunCounts.close++
74+
filing.issue.state = 'closed'
75+
core.info(`[dry run] Would CLOSE issue: ${filing.issue.url}`)
76+
} else if (isNewFiling(filing)) {
77+
dryRunCounts.open++
78+
;(filing as Filing).issue = {state: 'open'} as Issue
79+
core.info(
80+
`[dry run] Would OPEN a new issue for: ${filing.findings[0].problemShort} (${filing.findings[0].url})`,
81+
)
82+
} else if (isRepeatedFiling(filing)) {
83+
dryRunCounts.reopen++
84+
filing.issue.state = 'reopened'
85+
core.info(`[dry run] Would REOPEN issue: ${filing.issue.url}`)
86+
}
87+
} else {
88+
if (isResolvedFiling(filing)) {
89+
// Close the filing's issue (if necessary)
90+
response = await closeIssue(octokit, new Issue(filing.issue))
91+
filing.issue.state = 'closed'
92+
} else if (isNewFiling(filing)) {
93+
// Open a new issue for the filing
94+
response = await openIssue(octokit, repoWithOwner, filing.findings[0], screenshotRepo)
95+
;(filing as Filing).issue = {state: 'open'} as Issue
7696

77-
// Track for grouping
78-
if (shouldOpenGroupedIssues) {
79-
const problemShort: string = filing.findings[0].problemShort
80-
if (!newIssuesByProblemShort[problemShort]) {
81-
newIssuesByProblemShort[problemShort] = []
97+
// Track for grouping
98+
if (shouldOpenGroupedIssues) {
99+
const problemShort: string = filing.findings[0].problemShort
100+
if (!newIssuesByProblemShort[problemShort]) {
101+
newIssuesByProblemShort[problemShort] = []
102+
}
103+
newIssuesByProblemShort[problemShort].push({
104+
url: response.data.html_url,
105+
id: response.data.number,
106+
})
82107
}
83-
newIssuesByProblemShort[problemShort].push({
84-
url: response.data.html_url,
85-
id: response.data.number,
86-
})
108+
} else if (isRepeatedFiling(filing)) {
109+
// Reopen the filing's issue (if necessary) and update the body with the latest finding
110+
response = await reopenIssue(
111+
octokit,
112+
new Issue(filing.issue),
113+
filing.findings[0],
114+
repoWithOwner,
115+
screenshotRepo,
116+
)
117+
filing.issue.state = 'reopened'
118+
}
119+
if (response?.data && filing.issue) {
120+
// Update the filing with the latest issue data
121+
filing.issue.id = response.data.id
122+
filing.issue.nodeId = response.data.node_id
123+
filing.issue.url = response.data.html_url
124+
filing.issue.title = response.data.title
125+
core.info(
126+
`Set issue ${response.data.title} (${repoWithOwner}#${response.data.number}) state to ${filing.issue.state}`,
127+
)
87128
}
88-
} else if (isRepeatedFiling(filing)) {
89-
// Reopen the filing's issue (if necessary) and update the body with the latest finding
90-
response = await reopenIssue(
91-
octokit,
92-
new Issue(filing.issue),
93-
filing.findings[0],
94-
repoWithOwner,
95-
screenshotRepo,
96-
)
97-
filing.issue.state = 'reopened'
98-
}
99-
if (response?.data && filing.issue) {
100-
// Update the filing with the latest issue data
101-
filing.issue.id = response.data.id
102-
filing.issue.nodeId = response.data.node_id
103-
filing.issue.url = response.data.html_url
104-
filing.issue.title = response.data.title
105-
core.info(
106-
`Set issue ${response.data.title} (${repoWithOwner}#${response.data.number}) state to ${filing.issue.state}`,
107-
)
108129
}
109130
} catch (error) {
110131
core.setFailed(`Failed on filing: ${JSON.stringify(filing, null, 2)}\n${error}`)
@@ -114,7 +135,7 @@ export default async function () {
114135

115136
// Open tracking issues for groups with >1 new issue and link back from each
116137
// new issue
117-
if (shouldOpenGroupedIssues) {
138+
if (shouldOpenGroupedIssues && !dryRun) {
118139
for (const [problemShort, issues] of Object.entries(newIssuesByProblemShort)) {
119140
if (issues.length > 1) {
120141
const capitalizedProblemShort = problemShort[0].toUpperCase() + problemShort.slice(1)
@@ -138,6 +159,16 @@ export default async function () {
138159
}
139160
}
140161

162+
if (dryRun) {
163+
core.info('[dry run] Summary of actions that would be taken:')
164+
console.table({
165+
open: dryRunCounts.open,
166+
reopen: dryRunCounts.reopen,
167+
close: dryRunCounts.close,
168+
total: dryRunCounts.open + dryRunCounts.reopen + dryRunCounts.close,
169+
})
170+
}
171+
141172
const filingsPath = path.join(process.env.RUNNER_TEMP || '/tmp', `filings-${crypto.randomUUID()}.json`)
142173
fs.writeFileSync(filingsPath, JSON.stringify(filings))
143174
core.setOutput('filings_file', filingsPath)
Lines changed: 195 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,195 @@
1+
import {describe, it, expect, vi, beforeEach, afterEach} from 'vitest'
2+
3+
// --- Mock the issue-mutating helpers so we can assert they are NEVER called in dry run ---
4+
const openIssue = vi.fn()
5+
const reopenIssue = vi.fn()
6+
const closeIssue = vi.fn()
7+
vi.mock('../src/openIssue.js', () => ({openIssue: (...args: unknown[]) => openIssue(...args)}))
8+
vi.mock('../src/reopenIssue.js', () => ({reopenIssue: (...args: unknown[]) => reopenIssue(...args)}))
9+
vi.mock('../src/closeIssue.js', () => ({closeIssue: (...args: unknown[]) => closeIssue(...args)}))
10+
11+
// --- Mock @actions/core: control inputs, capture logs/outputs ---
12+
const inputs: Record<string, string> = {}
13+
const infoLines: string[] = []
14+
const outputs: Record<string, string> = {}
15+
vi.mock('@actions/core', () => ({
16+
getInput: (name: string) => inputs[name] ?? '',
17+
getBooleanInput: (name: string) => (inputs[name] ?? 'false') === 'true',
18+
setOutput: (name: string, value: string) => {
19+
outputs[name] = value
20+
},
21+
info: (msg: string) => {
22+
infoLines.push(msg)
23+
},
24+
debug: () => {},
25+
warning: () => {},
26+
setFailed: () => {},
27+
}))
28+
29+
// --- Mock fs: feed findings/cached filings in, swallow the output write ---
30+
const files: Record<string, string> = {}
31+
vi.mock('node:fs', () => ({
32+
default: {
33+
readFileSync: (p: string) => files[p],
34+
writeFileSync: (p: string, data: string) => {
35+
files[p] = data
36+
},
37+
},
38+
}))
39+
40+
// --- Stub Octokit so constructing it in index.ts doesn't do anything real ---
41+
const octokitRequest = vi.fn()
42+
vi.mock('@octokit/core', () => ({
43+
Octokit: {
44+
plugin: () =>
45+
class {
46+
request = octokitRequest
47+
},
48+
},
49+
}))
50+
vi.mock('@octokit/plugin-throttling', () => ({throttling: {}}))
51+
52+
import runFileAction from '../src/index.ts'
53+
54+
const finding = {
55+
scannerType: 'axe',
56+
ruleId: 'color-contrast',
57+
url: 'https://example.com/page',
58+
html: '<span>Low contrast</span>',
59+
problemShort: 'elements must meet minimum color contrast ratio thresholds',
60+
problemUrl: 'https://dequeuniversity.com/rules/axe/4.10/color-contrast',
61+
solutionShort: 'ensure sufficient contrast',
62+
}
63+
64+
// A second finding with no matching cached filing -> NEW (open)
65+
const newFinding = {...finding, ruleId: 'heading-order', html: '<h3>Skipped</h3>'}
66+
67+
// A cached filing whose finding matches `finding` -> REPEATED (reopen)
68+
const repeatedCached = {
69+
issue: {id: 1, nodeId: 'N1', url: 'https://github.com/org/repo/issues/1', title: 'repeat'},
70+
findings: [finding],
71+
}
72+
73+
// A cached filing with NO matching finding this run -> RESOLVED (close)
74+
const resolvedCached = {
75+
issue: {id: 2, nodeId: 'N2', url: 'https://github.com/org/repo/issues/2', title: 'resolved'},
76+
findings: [{...finding, ruleId: 'landmark-one-main', html: '<div>old</div>'}],
77+
}
78+
79+
function setup() {
80+
// findings file: includes `finding` (matches repeatedCached) and `newFinding` (brand new)
81+
files['/tmp/findings.json'] = JSON.stringify([finding, newFinding])
82+
// cached filings: one repeated, one resolved (its finding is absent from findings file)
83+
files['/tmp/cached.json'] = JSON.stringify([repeatedCached, resolvedCached])
84+
inputs.findings_file = '/tmp/findings.json'
85+
inputs.cached_filings_file = '/tmp/cached.json'
86+
inputs.repository = 'org/repo'
87+
inputs.token = 'fake-token'
88+
}
89+
90+
describe('file action — dry_run', () => {
91+
beforeEach(() => {
92+
vi.clearAllMocks()
93+
infoLines.length = 0
94+
for (const k of Object.keys(inputs)) delete inputs[k]
95+
for (const k of Object.keys(outputs)) delete outputs[k]
96+
vi.spyOn(console, 'table').mockImplementation(() => {})
97+
})
98+
afterEach(() => {
99+
vi.restoreAllMocks()
100+
})
101+
102+
it('does not open, reopen, or close any issues when dry_run is true', async () => {
103+
setup()
104+
inputs.dry_run = 'true'
105+
106+
await runFileAction()
107+
108+
expect(openIssue).not.toHaveBeenCalled()
109+
expect(reopenIssue).not.toHaveBeenCalled()
110+
expect(closeIssue).not.toHaveBeenCalled()
111+
expect(octokitRequest).not.toHaveBeenCalled()
112+
})
113+
114+
it('logs the intended action for each filing type', async () => {
115+
setup()
116+
inputs.dry_run = 'true'
117+
118+
await runFileAction()
119+
120+
const log = infoLines.join('\n')
121+
expect(log).toContain(
122+
'[dry run] Would OPEN a new issue for: elements must meet minimum color contrast ratio thresholds (https://example.com/page)',
123+
)
124+
expect(log).toContain('[dry run] Would REOPEN issue: https://github.com/org/repo/issues/1')
125+
expect(log).toContain('[dry run] Would CLOSE issue: https://github.com/org/repo/issues/2')
126+
})
127+
128+
it('logs a summary table with counts', async () => {
129+
setup()
130+
inputs.dry_run = 'true'
131+
132+
await runFileAction()
133+
134+
expect(vi.mocked(console.table)).toHaveBeenCalledWith(
135+
expect.objectContaining({open: 1, reopen: 1, close: 1, total: 3}),
136+
)
137+
})
138+
139+
it('still writes the filings_file output in dry run', async () => {
140+
setup()
141+
inputs.dry_run = 'true'
142+
143+
await runFileAction()
144+
145+
expect(outputs.filings_file).toBeDefined()
146+
})
147+
148+
it('updates in-memory issue state for an accurate preview without mutating remotely', async () => {
149+
setup()
150+
inputs.dry_run = 'true'
151+
152+
await runFileAction()
153+
154+
// The path written is `${RUNNER_TEMP||'/tmp'}/filings-<uuid>.json`; grab it from the output.
155+
const writtenPath = outputs.filings_file
156+
const writtenFilings = JSON.parse(files[writtenPath])
157+
158+
// Resolved cached filing (issues/2) -> would be CLOSED
159+
const resolved = writtenFilings.find(
160+
(f: {issue?: {url?: string}}) => f.issue?.url === 'https://github.com/org/repo/issues/2',
161+
)
162+
expect(resolved?.issue.state).toBe('closed')
163+
164+
// Repeated cached filing (issues/1) -> would be REOPENED
165+
const repeated = writtenFilings.find(
166+
(f: {issue?: {url?: string}}) => f.issue?.url === 'https://github.com/org/repo/issues/1',
167+
)
168+
expect(repeated?.issue.state).toBe('reopened')
169+
170+
// New filing -> issue object created with state 'open'
171+
const opened = writtenFilings.find((f: {issue?: {state?: string}}) => f.issue?.state === 'open')
172+
expect(opened).toBeDefined()
173+
174+
// And confirm we still didn't actually mutate anything remotely
175+
expect(openIssue).not.toHaveBeenCalled()
176+
expect(reopenIssue).not.toHaveBeenCalled()
177+
expect(closeIssue).not.toHaveBeenCalled()
178+
})
179+
180+
it('does call the mutating helpers when dry_run is false (regression guard)', async () => {
181+
setup()
182+
inputs.dry_run = 'false'
183+
// helpers return a minimal Octokit-style response so index.ts can read response.data
184+
const resp = {data: {id: 9, node_id: 'N', number: 9, html_url: 'https://github.com/org/repo/issues/9', title: 't'}}
185+
openIssue.mockResolvedValue(resp)
186+
reopenIssue.mockResolvedValue(resp)
187+
closeIssue.mockResolvedValue(resp)
188+
189+
await runFileAction()
190+
191+
expect(openIssue).toHaveBeenCalled()
192+
expect(reopenIssue).toHaveBeenCalled()
193+
expect(closeIssue).toHaveBeenCalled()
194+
})
195+
})

FAQ.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,17 @@ Just keep in mind that resetting the cache means the Action will "forget" what
6060
it's already seen, so it may reopen issues that were previously tracked or
6161
closed.
6262

63+
### How can I preview what the scanner would do without filing issues?
64+
65+
Set the `dry_run` input to `true`. The scanner will run a normal scan and log the
66+
issues it _would_ open, reopen, or close — but it won't actually mutate any data or write to the `gh-cache` branch
67+
assign any issues, and it won't write to the `gh-cache` branch.
68+
69+
This is handy for trying out a new configuration or seeing how many issues a scan
70+
would file, without making any changes to your repository. Because dry runs don't
71+
update the cache, your next real run behaves exactly as if the dry run never
72+
happened.
73+
6374
### Does this work with private repositories?
6475

6576
Yes! The Action works with both public and private repositories. Since it runs

README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,7 @@ jobs:
5555
# skip_copilot_assignment: false # Optional: Set to true to skip assigning issues to GitHub Copilot (or if you don't have GitHub Copilot)
5656
# include_screenshots: false # Optional: Set to true to capture screenshots and include links to them in filed issues
5757
# open_grouped_issues: false # Optional: Set to true to open an issue grouping individual issues per violation
58+
# dry_run: false # Optional: Set to true to scan and log what would be filed without creating/closing issues or writing the cache
5859
# reduced_motion: no-preference # Optional: Playwright reduced motion configuration option
5960
# color_scheme: light # Optional: Playwright color scheme configuration option
6061
# scans: '["axe","reflow-scan"]' # Optional: An array of scans (or plugins) to be performed. If not provided, only Axe will be performed.
@@ -131,6 +132,7 @@ Trigger the workflow manually or automatically based on your configuration. The
131132
| `reduced_motion` | No | Playwright `reducedMotion` setting for scan contexts. Allowed values: `reduce`, `no-preference` | `reduce` |
132133
| `color_scheme` | No | Playwright `colorScheme` setting for scan contexts. Allowed values: `light`, `dark`, `no-preference` | `dark` |
133134
| `scans` | No | An array of scans (or plugins) to be performed. If not provided, only Axe will be performed. | `'["axe", "reflow-scan", ...other plugins]'` |
135+
| `dry_run` | No | When `true`, scan and log the issues that _would_ be filed without opening, closing, reopening, or assigning any issues — and without writing to the `gh-cache` branch. Useful for safely previewing results. Default: `false` | `true` |
134136
| `url_configs` | No | A stringified JSON array of URL config objects. Each object must have a `url` field and may have an optional `excludeSelectors` field (array of CSS selectors to exclude from the Axe scan for that URL). When provided, takes precedence over the `urls` input. | `'[{"url":"https://example.com","excludeSelectors":["iframe","#widget"]}]'` |
135137

136138
---

0 commit comments

Comments
 (0)