feat(core): add runReport to onTestRunEnd payload#1288
Conversation
Reporters and the runner previously each re-derived run-level pass/fail status and counts by filtering raw arrays. This caused subtle drift — e.g. the markdown reporter did not honor `passWithNoTests` when no tests were discovered, while the JSON reporter did. The `Reporter.onTestRunEnd` payload now carries a precomputed `runReport` with `status`, `counts`, `failures`, and serialized `unhandledErrors`. Built-in reporters (default, dot, md, json, junit, github-actions, blob, mergeReports) read from `runReport` directly, and `runTests` derives the process exit code from `runReport.status` so all consumers stay aligned. This matches the contract used by Jest (`AggregatedResult`) and Vitest (`reason: 'passed' | 'interrupted' | 'failed'` + `unhandledErrors`), and exposes `RunReport` / `FailureItem` as public types so custom reporters can consume the same data. The previously documented payload fields (`results`, `testResults`, `duration`, `snapshotSummary`, `unhandledErrors`, etc.) are preserved. Custom reporters that destructure only those fields continue to work without changes.
Deploying rstest with
|
| Latest commit: |
529596f
|
| Status: | 🚫 Build failed. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 529596faed
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Three issues caught by Codex review on the previous commit: 1. Browser hostController called `onTestRunEnd` without `runReport`, so built-in reporters that destructure it (md/json/junit/githubActions/ blob/default/dot) crashed in browser-only runs. Browser package now imports `buildRunReport` from `@rstest/core/browser` and constructs `runReport` at both the early-error path and the main `notifyTestRunEnd` path. This also unblocks `packages/browser` build, which is what was failing CI. 2. Watch-mode "no tests discovered" was setting `process.exitCode = 1` because `buildRunReport` marked the run as `fail`. `reportNoTestFiles` intentionally does not set an exit code in watch mode, so the runner and browser host now pass `passWithNoTests: isWatchMode || config.passWithNoTests` when building the report. 3. `runReport.failures` previously applied `filterRerunTestPaths`, which was derived from node-only `currentEntries`. In unified node+browser runs that filter dropped browser failures from `runReport.failures`, so reporters consuming it (notably GitHub Actions annotations) lost browser failure visibility. The filter is now removed from `buildRunReport` / `collectFailures` and applied in `printSummaryErrorLogs` instead, which is the only place the watch narrowing actually matters.
Rsdoctor Bundle Diff AnalysisFound 12 projects in monorepo, 2 projects with changes. 📊 Quick Summary
📋 Detailed Reports (Click to expand)📁 browserPath:
📦 Download Diff Report: browser Bundle Diff 📁 core/mainPath:
📦 Download Diff Report: core/main Bundle Diff Generated by Rsdoctor GitHub Action |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7ab99687e3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… rerun Two issues caught by Codex review on commit 7ab9968: 1. `MdReporter.onTestRunEnd` consumed unfiltered `runReport.failures`, so in watch mode the markdown report kept rendering stale failures from previous reruns. MdReporter now destructures the existing `filterRerunTestPaths` payload field and runs `filterFailuresByPaths` to narrow the list, matching the behavior already wired through `printSummaryErrorLogs` for DefaultReporter / DotReporter. 2. `RunReport.unhandledErrors` was flattened to `{message, stack, name}`, silently dropping `diff` / `expected` / `actual` / `fullStack` from `FormattedError`. `printError` and reporter output relied on those fields for assertion diffs and `DEBUG=rstest` hints. The type is now `FormattedError[]`, and `buildRunReport` preserves the extra fields. `FormattedError` is added to the public `@rstest/core` type exports. The (Error | FormattedError)[] union initially proposed for `BuildRunReportInput.unhandledErrors` was collapsed to `FormattedError[]` because `Error` is structurally a subtype (only `message` is required on `FormattedError`), eliminating an `as Partial<FormattedError>` cast.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 632db13f8b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- runTests: feed buildRunReport the local per-run results/testResults arrays instead of context.reporterResults (which dedupes by testPath and would drop status from multi-project same-path runs). - mergeReports: when reconstructing Error from blob.unhandledErrors, also copy the FormattedError fields (diff/expected/actual/fullStack) serialized by BlobReporter so sharded runs keep assertion detail.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d442bdbefd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| results, | ||
| testResults, |
There was a problem hiding this comment.
runReport is computed from the per-rerun results/testResults arrays here, but notifyReportersOnTestRunEnd still sends context.reporterResults to onTestRunEnd; in watch reruns these two datasets diverge. That creates contradictory payloads (for example JSON/JUnit can report status: pass and zero failures from runReport while files still include failing entries from earlier runs), which can break downstream tooling that trusts runReport as the canonical run summary.
Useful? React with 👍 / 👎.
| tests: counts.tests, | ||
| failures: counts.failedTests, |
There was a problem hiding this comment.
This reporter now fills <testsuites tests/failures/skipped> from runReport.counts, while individual <testsuite>/<testcase> blocks are still generated from the results payload. In watch reruns (where runReport summarizes the current rerun but results can still be the accumulated reporter state), the root counters can disagree with the actual emitted testcases, which causes inconsistent JUnit artifacts for CI parsers and trend tooling.
Useful? React with 👍 / 👎.
1 participant
Summary
Adds a precomputed
runReportfield to theReporter.onTestRunEndpayload, so reporters and the CLI exit-code logic share one run-level summary instead of each filtering the rawresults/testResultsarrays. Matches the contract style used by Jest (AggregatedResult) and Vitest (reason+unhandledErrors).What this gives custom reporters
runReport: RunReportfield carryingstatus('pass' | 'fail'), pre-aggregatedcounts, the flattenedfailureslist, the serializedunhandledErrors, andduration/snapshot.@rstest/core:RunReport,FailureItem,FormattedError.onTestRunEndpayload fields (results,testResults,duration,snapshotSummary,unhandledErrors,filterRerunTestPaths) are preserved — custom reporters that destructure only those keep compiling and behaving the same.What bug this prevents
Before this PR each reporter computed "did this run pass?" itself, and they had drifted: the markdown reporter ignored
passWithNoTestsin the no-tests case while the JSON reporter honored it, so the same run could reportpassandfailin different outputs. WithbuildRunReportas the single source of truth this class of drift is no longer possible — adding a new reporter (or a new no-tests / unhandled-error edge case) only needs to be expressed once.Behavior changes a reviewer should know about
runReport: RunReportis required on the payload type (matches Jest'sAggregatedResultand Vitest'sreason, which are also required). TS structural typing means existingimplements Reporterclasses that don't destructure it keep compiling; the only break is hand-typingReporter['onTestRunEnd']and manually calling it without the field — an extremely uncommon pattern.process.exitCodealone (it would have been set to1by the new predicate otherwise, contradictingreportNoTestFiles's long-standing watch behavior).runReport.unhandledErrorspreserves the fullFormattedErrorshape (diff/expected/actual/fullStack), so reporters that print assertion diffs for run-level errors keep working.Compatibility
Reportertyping: no changes needed.website/docs/{en,zh}/api/javascript-api/{reporter,types}.mdx.Checklist