Skip to content

chore(ci): e2e report improve#2355

Merged
universal-itengineer merged 19 commits into
mainfrom
chore/ci/e2e-report-improve
May 18, 2026
Merged

chore(ci): e2e report improve#2355
universal-itengineer merged 19 commits into
mainfrom
chore/ci/e2e-report-improve

Conversation

@universal-itengineer
Copy link
Copy Markdown
Member

@universal-itengineer universal-itengineer commented May 15, 2026

Description

Improve the E2E messenger report generated by CI and reorganize the
report scripts so the same logic is easier to follow.

User-facing changes:

  • Add a Reason column to the failed-tests thread table and rename
    Test group to Tests.
  • Link cluster names in failed-tests thread replies to the matching
    workflow/job URL.
  • Hide the Errors column in the main test-results table when there
    are no Ginkgo errors.
  • Calculate Success Rate from executed specs only:
    passed / (passed + failed + errors).
  • Add table header emojis to the main report.
  • Use the full Ginkgo Failure.Message text as the failed-test reason.
  • Stop wrapping *exec.ExitError with %w in
    test/e2e/internal/framework/ssh.go so Ginkgo no longer dumps Go
    internals into the report when an SSH command fails.
  • Capture Ginkgo stdout/stderr in the reusable pipeline and parse it as
    a fallback report source so suite-level failures such as
    SynchronizedBeforeSuite are still surfaced when the JSON report is
    missing or contains only setup failure data.

Internal refactors (no behavior change, covered by existing tests):

  • Move the shared withTempDir / createCore jest helpers into
    shared/test-utils.js to remove duplication across three test files.
  • Extract escapeRegExp in shared/report-model.js; consolidate
    buildStatusMessage into a status → template lookup table; remove a
    redundant ternary in buildClusterStatus.
  • Unify the Ginkgo source lookup (findGinkgoReport /
    findGinkgoOutput → one findGinkgoSource) and call
    parseGinkgoFile once with the descriptor picked up front.
  • Iterate over a list of required keys in
    requireClusterReportConfig instead of three identical if blocks.
  • Pre-filter cluster reports once by storage key and merge
    renderClusterFailuresSection / renderMissingReportsSection into
    a single renderBulletSection helper. Output stays byte-identical;
    verified by existing messenger-report tests.
  • Pass the loop credentials object as-is to postToLoopApi so the
    two call sites collapse to one-liners.

Why do we need it, and what problem does it solve?

The current E2E notification thread only lists failed test groups and
does not show why a test failed. This forces engineers to open CI
artifacts and logs even for simple failures.

The main report also counted skipped specs in the Success Rate
denominator, which made Ginkgo runs look worse than the executed-spec
result. SSH command failures contained noisy Go internals in
Failure.Message because *exec.ExitError was wrapped with %w.

When E2E fails in SynchronizedBeforeSuite, Ginkgo may skip all specs
and the JSON report may be absent or contain no regular test failures.
Previously the messenger report could only show a generic storage
failure (for example nfs with E2E TEST FAILED), without the actual
setup failure reason. The pipeline now captures Ginkgo stdout into
e2e_output_<storage>_<date>.log and the report parses suite-level
failures out of it as a fallback.

The follow-up refactors remove duplicated helpers, collapse parallel
branches, and replace inline regex/if-chain logic with declarative
data, making the report scripts easier to read and extend without
changing the rendered output.

What is the expected result?

  1. Run the E2E report generation with a Ginkgo JSON report containing
    failed specs.
  2. Check the main messenger report:
    • the Errors column is hidden when all error counts are zero;
    • Success Rate is calculated without skipped specs;
    • table headers include emojis.
  3. Check the failed-tests thread:
    • cluster names are links;
    • the table has Tests and Reason columns;
    • Reason contains the Ginkgo Failure.Message text.
  4. SSH command failures are rendered as useful text instead of dumping
    *exec.ExitError internals.
  5. If Ginkgo fails in SynchronizedBeforeSuite and no JSON report is
    created, the report is still generated from the captured Ginkgo
    output and the failed-tests thread shows the suite-level failure
    reason. Suite-level failures do not change spec metrics such as
    passed, failed, errors, or total.
  6. The jest suite under .github/scripts/js/e2e/report/ still passes
    (37 tests) and tmp/test-ci/report/run.sh produces the same
    markdown for the bundled fixtures.

Checklist

  • The code is covered by unit tests.
  • e2e tests passed.
  • Documentation updated according to the changes.
  • Changes were tested in the Kubernetes cluster manually.

Changelog entries

section: ci
type: fix
summary: Improve E2E messenger report failure reasons, suite setup failure reporting, and success-rate calculation.
impact_level: low

Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
@universal-itengineer universal-itengineer added this to the v1.9.0 milestone May 15, 2026
@universal-itengineer universal-itengineer marked this pull request as ready for review May 15, 2026 15:08
Capture Ginkgo output as a fallback report source so BeforeSuite failures are surfaced even when the JSON report is missing.

Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Comment thread .github/scripts/js/e2e/report/messenger/markdown.js Outdated
Comment thread .github/scripts/js/e2e/report/messenger/markdown.js Outdated
- Merge Ginkgo JSON and output parsers behind a single parseGinkgoFile
  helper with declarative source descriptors.
- Build a single suiteNodePattern and extract small predicates for the
  Ginkgo stdout reason block.
- Show the suite node name (e.g. SynchronizedAfterSuite) as the failed
  group in the messenger thread instead of "Unknown".

Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
The previous suite node regex was too permissive: it matched a plain
"[SynchronizedBeforeSuite]" section header that appears in Ginkgo stdout
before the actual "[FAILED]" line. As a result, the log-only fallback
could not extract suite setup failures from real CI output.

Tighten the pattern to require either "[X] [FAILED]" (body) or
"[FAIL] [X]" (summary) and add a regression test based on a real
BeforeSuite timeout from CI.

Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
- Describe the test-results table as a single columns array with header,
  alignment, and value functions so the "Errors" column is just one
  entry that is appended conditionally, instead of duplicating the row
  template for the with- and without-errors variants.
- Move getFailedTestEntries inside summarizeFailedTestGroups so the
  caller passes a single report argument instead of pre-extracting
  entries on every call site.

Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Document the new helpers introduced by recent refactors:

- cluster-report.js: findGinkgoOutput, emptyParsedReport, parseGinkgoFile
  plus a GinkgoSourceDescriptor typedef used by ginkgoJsonSource and
  ginkgoOutputSource.
- shared/ginkgo-report-utils.js: findFailedSuiteNode,
  extractFailureReasonFromOutput, parseGinkgoOutput.
- messenger/markdown.js: buildTestResultsColumns, buildMarkdownRow,
  summarizeFailedTestGroups plus a TestResultsColumn typedef.

Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Three test files (cluster-report.test.js, messenger-report.test.js,
fs-utils.test.js) duplicated the same temp-directory and core-mock
helpers. Move them to a single shared/test-utils.js module and let
each test wrap it with its own prefix via a thin inTempDir helper.

Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
archivedReportPattern and ginkgoOutputPattern both inlined the same
regex-escape expression. Move it to a tiny escapeRegExp helper so the
two callers share one implementation.

Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
The six-branch if-chain was a pure status → template mapping. Replace
it with a statusMessageTemplates object that makes the symmetry between
statuses obvious and keeps the function body to a single line.

Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
normalizeJobResult guarantees one of success/cancelled/skipped/failure,
so once the loop has filtered out success, stageResult already matches
the status name we want to emit. The nested ternary was an identity
mapping; use stageResult directly.

Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
findGinkgoReport and findGinkgoOutput were two thin wrappers around
findSingleMatchingFile that only differed in the file-name regex and
the human-readable label. The same duplication leaked into the call
site, which picked the source descriptor twice (once via the if/else
on rawReportPath, once when calling parseGinkgoFile).

Move the file-name pattern into the source descriptor itself, expose
a single findGinkgoSource(config, source) lookup, and call
parseGinkgoFile once with the descriptor selected up front.

Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
Replace the three near-identical if blocks with a loop over a list of
required keys. The error messages stay in the same
"buildClusterReport requires <key>" format, so existing tests and CI
behaviour are unchanged.

Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
splitReportsBySection re-checked getReportClusterKey for every filter.
Filter once up front and chain the three category predicates against
the pre-filtered list.

renderClusterFailuresSection and renderMissingReportsSection only
differed in the heading and the per-report message lookup. Extract a
single renderBulletSection helper and pass per-section getMessage
functions for cluster failures and missing reports. The emitted
markdown is byte-for-byte identical to the previous output, which the
existing messenger-report tests verify.

Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
The previous signature unpacked apiUrl/channelId/token out of the loop
object only to be reassembled at every call site. Take the loop
credentials object as-is and accept (loop, message, rootId, core) so
the two call sites in makeThreadedReportInLoop become one-line calls.

Signed-off-by: Nikita Korolev <nikita.korolev@flant.com>
@universal-itengineer universal-itengineer merged commit 79ad6ba into main May 18, 2026
44 of 46 checks passed
@universal-itengineer universal-itengineer deleted the chore/ci/e2e-report-improve branch May 18, 2026 10:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants