Redesign the Compare tab around one row per metric (#4836)

The compare view asked the reader to cross-reference a nine-column statistics table against 24 separate scatter charts to answer one question: did anything change, and by how much? The scatter charts also plotted values against run index, which carries no meaning for independent samples — the actual signal is how the two distributions sit against each other.

Each metric is now a single row: the p-value, a distribution strip showing every baseline and current run on a shared value axis with median ticks, median → median with the delta, and the verdict. A verdict card on top gives the headline count with chips that jump to the flagged metrics, and significant rows are tinted by direction — regressions red, improvements green (the old layout showed a significant improvement in error-red). When both medians are zero but runs differ, the row falls back to means so flaky CPU long tasks stop looking identical. The old chart anchors stay alive as row ids, and the per-series colors were replaced with a colorblind-validated pair (the old tan failed contrast on the card surface).

Co-authored-by: Claude Fable 5 noreply@anthropic.com

Author: soulgalore

AGENTS.md

@@ -176,6 +176,7 @@ The report has accumulated several non-obvious behaviours that an agent should k
 - **Extending pug templates can't hold top-level code.** A template that `extends ./layout.pug` may only have named blocks and mixins at the top level — a top-level `- const …` throws `PUG:UNEXPECTED_NODES_IN_EXTENDING_ROOT` at compile time. Put constants inside `block content`, and pass them to mixins as arguments: mixins compile to standalone functions and do NOT see variables declared in the block that calls them.
 - **Assets page overview vs table.** The `aggregateassets.summary` handler in `lib/plugins/html/index.js` computes the overview-strip totals (transfer/size/type counts/biggest asset) over ALL unique assets before the list is capped to `--maxAssets` (default 100, an undeclared-in-yargs top-level option) — mind the `splice` there, it mutates the list, so totals must be computed first. The cap SELECTION uses the same weight rule the table sorts by (`transferSize || size`, descending) — selecting by request count let one-off heavy assets fall off the page entirely. The per-URL asset shape (`lib/plugins/assets/aggregator.js`) carries `size` = contentSize and `transferSize` = wire bytes; `transferSize` can be missing (Safari HARs, old data), hence the fallback. The capped list feeds only the HTML Assets page; `analysisstorer` and `graphite` consume the raw uncapped message.
 - **The Rendering tab tells the full render story.** It absorbed the old Video, Filmstrip and Screenshots tabs plus the "why was rendering delayed" diagnostics: `templates/url/rendering/index.pug` renders (in order) the playback card + filmstrip (driven by `templates/url/includes/renderingView.js`, styled by `components/rendering.scss`), the per-run screenshot gallery, and the why-section — the recalculate-style card (browsertime `renderBlocking.recalculateStyle`, same run-picking as the Metrics tab: `pageSummary.renderBlocking[medianRun.runIndex - 1]` on summary, `run.renderBlocking` on iteration) and the render-blocking requests card (`rendering/blocking.pug`, reads a `pagexray` local built the same way the PageXray tab builds it: `run || pageSummary`). `#video`, `#filmstrip`, `#screenshots` and `#requests-render-blocking` are a URL contract: `tabScripts.js` maps all four hashes to `#rendering` and anchors with those ids stay inside the section — keep them alive if you rename anything. The PLAYBACK view degrades by CONFIGURATION, not file sniffing: `hasVideo` comes from `--video` (plus `--enableVideoRun` on the summary page only, where the extra video is always `data/video/1.mp4`), `hasFilmstrip` from the createFilmstrip/trace-screenshots gate; filmstrip-without-video swaps the player for a frame preview and drops speed buttons + download link; with NEITHER, no player/preview/transport/timeline renders at all (the milestone timeline is interaction-coupled to a playback medium, so it never renders standalone). The gallery and why-cards are gated on DATA, and the `hasRendering` const in `url/summary/index.pug` + `url/iteration/index.pug` mirrors all of it — the tab exists for video OR filmstrip OR screenshots (iteration only; screenshots are per-run, the summary page never had them) OR a slow TTFB (`timings.ttfb` ≥ 800 ms — the Web Vitals good threshold, same field as the Metrics tab's TTFB tile; renders as the leading why-note with share-of-FCP and a HAR-gated Waterfall cross-link — note the HAR lives at `browsertime.har` on summary but `browsertime.run.har` on iteration pages) OR pagexray `renderBlocking` OR recalculate-style data OR the trace-derived CPU render signals (forced reflows with `startTime` before the run's latest paint milestone — max of FCP and LCP renderTime, since LCP can land before FCP — or any non-composited animations — `cpu.forcedReflows` / `cpu.nonCompositedAnimations`, per-run under `pageSummary.cpu[medianRun.runIndex - 1]` on summary, `run.cpu` on iteration). Those two can exist while recalculate-style is empty (recalc needs an FCP/LCP trace event; the other trace analyses don't), hence their own gate clause; the reflow card windows to before the latest paint milestone with per-milestone FCP/LCP tiles and a share-of-milestone note, and only renders when the pre-paint total is ≥ 10 ms (materiality floor, mirrored in both page gates — keep in sync; no paint milestone → no card — whole-load numbers live on the CPU tab), counts sub-millisecond reflows in the totals but drops them from the offender list, and both why-items deep-link to the CPU tab's `#cpu-forced-reflows` / `#cpu-animations` anchors via an inline `selectTab` call (tabScripts.js maps those hashes to `#cpu` for reloads/shared links). The `--cpu` hint for the recalculate metrics lives here now (still Chromium-only — Gecko profiles don't carry `UpdateLayoutTree.elementCount`); the Metrics tab keeps its `renderBlocking` const declared solely because the LCP card (`metrics/lcp.pug`) keeps its one-line recalculate-before-LCP entry. Milestone flags use the median RUN's own values (`pageSummary.browserScripts[medianRun.runIndex - 1]`), not cross-run medians — they must line up with the recording — and are sorted purely by time (LCP before First Visual Change happens in real runs; don't reorder or clamp). Per-frame completeness steps from the most recent VisualProgress sample `<=` the frame timestamp (see the VisualProgress note below). Keyboard handling only fires while the section is visible (`view.offsetParent !== null`) and never when typing in a form control.
+- **The Compare tab is one row per metric, and its anchors are a contract.** `lib/plugins/compare/pug/index.pug` renders a one-line setup strip, a collapsed `<details>` Run setup (screenshots/video — the foundation strips the native disclosure triangle, so the SCSS draws its own), the content-breakdown table, a verdict card with jump chips, then group cards with one row per metric: name + p-value, a distribution strip (baseline/current runs as DOM dots on a shared value axis — run order carries no meaning for independent samples, so there is NO run-index axis), median → median with Δ%, and a chip. Row ids keep the old `#chart-<group>_<metric>` scheme — old deep links must keep landing. Direction tones: a significant change is tinted by direction (faster = ok-green, slower = error-red), derived from `--compare.alternative` and, for two-sided tests, from the per-metric Cliff's delta SIGN (positive = current ranks higher = slower; all compared metrics are lower-is-better). When both medians are 0 but runs differ (flaky CPU long tasks), the Δ column falls back to means — don't "fix" that to show 0 → 0. Units are derived from group + metric name (helper.js ships a fixed group set; unknown groups like user extras get NO unit rather than a guessed one), and rows guard on `v.baseline && v.current && Array.isArray(...)` because extras metrics can arrive half-shaped. Series colours are validated pairs (light `#b25e12`/`#2a78d6`, dark `#c9701f`/`#4a90e0` — the dark pair is brightened for the dark surface; keep the two theme files in sync). The graphite score export (`metrics.*.statisticalTestU` paths in `lib/plugins/compare/index.js`) is untouched by the HTML layout — don't confuse the two.
 - **Folder path aliases accept Unicode.** Non-ASCII URL components used to collapse to `-` in result paths (issue #3880). The path-alias regex now permits Unicode letters and digits. Don't tighten it back. [#4759](https://github.com/sitespeedio/sitespeed.io/pull/4759).
 - **CSS rebuilds are required.** Editing SCSS under `lib/plugins/html/src/sass/` does nothing on its own — run `npm run build:css` and commit the compiled `*.min.css` under `lib/plugins/html/assets/css/`. The build step writes both light and dark themes.
 - **Per-URL data is keyed by message type, stored nested.** `dataCollector.addDataForUrl` uses `set()` from `lib/support/objectPath.js`, which splits the message-type path on `.`. So `browsertime.pageSummary` and `crux.pageSummary` land as `pageInfo.data.browsertime.pageSummary` / `pageInfo.data.crux.pageSummary` — nested objects, not literal dotted keys. Read them with `h.get(pageInfo.data, 'crux.pageSummary…')` (also dot-splitting) or direct nested access; a literal `pageInfo.data['crux.pageSummary']` lookup returns nothing.

docs/documentation/sitespeed.io/compare/index.md

@@ -77,34 +77,22 @@ docker run -v "$(pwd)":/baseline  sitespeedio/sitespeed.io:{% include version/si
 
 ## Results
 
-When you run your test it will create a new tab in the HTML results. This tab includes **a results table** showing the result data and, for each metric, **comparison graphs** comparing the baseline (previous data) with the latest run.
+When you run your test it will create a new tab in the HTML results. The tab opens with the test setup on one line, a verdict ("2 of 24 metrics are significantly slower compared to the baseline") with chips that jump to the flagged metrics, and then one row per compared metric, grouped by area (timings, visual metrics, Google Web Vitals, CPU and so on).
 
-
-The result table looks something like this:
-
-![Compare result]({{site.baseurl}}/img/compare-table.jpg){:loading="lazy"}
+![Compare rows]({{site.baseurl}}/img/compare-rows.jpg){:loading="lazy"}
 {: .img-thumbnail}
 
-The columns:
-1. **Metric Name**: This column lists the names of the performance metrics that were tested. These could be timings for different events (like Time to First Byte, load event completion, first contentful paint, etc.), or CPU-related metrics (like total duration of CPU tasks, duration of the last long task, etc.).
-2. **Score (mannwhitneyu/wilcox)**: This column shows the Mann-Whitney U/Wilcox scores for the comparisons between baseline and current test runs for each metric. A lower score typically indicates more significant differences between the two groups being compared.
-3. **Baseline Mean**: The average (mean) value for the baseline test run for each metric.
-4. **Current Mean**: The average (mean) value for the current test run for each metric.
-5. **Baseline Median**: The median value for the baseline test run for each metric. The median is the middle value when all the results are ordered from lowest to highest.
-6. **Current Median**: The median value for the current test run for each metric.
-7. **Baseline Std Dev**: Standard deviation for the baseline test run for each metric. This measures the amount of variation from the average.
-8. **Current Std Dev**: Standard deviation for the current test run for each metric.
-9. **Significant Change?**: This column indicates whether the change between the baseline and current test runs is statistically significant for each metric. If the change is statistically significant we use [Cliff's Delta](https://en.wikipedia.org/wiki/Effect_size#Effect_size_for_ordinal_data) to tell if the change is small, medium or large.
-
+Each row shows:
+1. **Metric name and p-value**: the metric that was tested and the score from the Mann-Whitney U/Wilcoxon test. A score below 0.05 means the difference is statistically significant.
+2. **The distribution strip**: every baseline run (orange) and every current run (blue) as dots on a shared value axis, with a taller tick marking each median. Baseline and current are independent samples, so the strip shows the two distributions directly — a real shift is visible as the two clusters separating. Hover a dot for the exact run value.
+3. **Median → median**: the baseline and current medians with the difference in absolute numbers and percent. When both medians are zero but the runs differ (flaky CPU long tasks do this), the means are shown instead.
+4. **The verdict**: whether the change is statistically significant. If it is, we use [Cliff's Delta](https://en.wikipedia.org/wiki/Effect_size#Effect_size_for_ordinal_data) to tell if the change is small, medium or large, and the row is tinted by direction — regressions in red, improvements in green.
 
-And the compare graphs will look like this for every metric:
-
-![Compare graphs]({{site.baseurl}}/img/compare-graph.jpg){:loading="lazy"}
-{: .img-thumbnail}
+The screenshots and video of the baseline and current runs live in the collapsed *Run setup* section above the verdict.
 
 ### Understanding Significant Changes
 
-In the results table, you'll see a list of all metrics with their corresponding scores.
+Every metric row carries its score (the p-value).
 
 If a score is below 0.05 it indicates a statistically significant difference between the baseline and the current data. By default we test if the current data is greater than the baseline. You can change that using `--compare.alternative`. You can test if it's smaller or if there is any difference between the two ('two-sided').