Skip to content

Report E2E results to GitHub#363

Draft
kieran-osgood-shopify wants to merge 1 commit into
kieran-osgood/e2e-browserstack/phase-5-browserstack-maestrofrom
kieran-osgood/e2e-browserstack/phase-6-github-reporting
Draft

Report E2E results to GitHub#363
kieran-osgood-shopify wants to merge 1 commit into
kieran-osgood/e2e-browserstack/phase-5-browserstack-maestrofrom
kieran-osgood/e2e-browserstack/phase-6-github-reporting

Conversation

@kieran-osgood-shopify

@kieran-osgood-shopify kieran-osgood-shopify commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Stack Context

This is Phase 6 of the BrowserStack Maestro E2E stack for Checkout Kit, stacked on BrowserStack Maestro execution.

Parent issue: https://github.com/shop/issues-checkout-kit/issues/1100
Epic: https://github.com/shop/issues-checkout-kit/issues/1084

What?

Adds GitHub reporting for E2E results:

  • per-run commit status payloads with stable matrix contexts
  • aggregate GitHub Check Run summary
  • sticky PR failure comment with a marker
  • failure-only PR comment behavior
  • actionable failure markdown with device selector, resolved device, suite path, BrowserStack build URL, video, screenshots, Maestro commands/logs, device logs, and network logs
  • Bitrise report workflow wiring
  • setup docs for the required GitHub token

Why?

Developers should not need to search BrowserStack manually to understand a failed E2E run. Failures should link directly to the actionable artifacts, while green runs should avoid comment noise.

Validation

shadowenv exec -- ruby e2e/test/github_reporter_test.rb
shadowenv exec -- ruby e2e/test/bitrise_pipeline_test.rb
shadowenv exec -- ruby e2e/test/e2e_matrix_test.rb
shadowenv exec -- ruby e2e/test/browserstack_device_resolver_test.rb
shadowenv exec -- ruby e2e/test/maestro_suite_package_test.rb
shadowenv exec -- ruby -c e2e/lib/e2e_github_reporter.rb
shadowenv exec -- ruby -c e2e/scripts/report_e2e_results
shadowenv exec -- bitrise validate -c e2e/bitrise.yml
shadowenv exec -- ruby -e 'require "yaml"; YAML.safe_load_file("e2e/bitrise.yml"); puts "e2e bitrise yaml parsed"'

kieran-osgood-shopify commented Jun 29, 2026

Copy link
Copy Markdown
Contributor Author

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

@kieran-osgood-shopify kieran-osgood-shopify force-pushed the kieran-osgood/e2e-browserstack/phase-5-browserstack-maestro branch from 1646c03 to d99cf73 Compare June 29, 2026 16:04
@kieran-osgood-shopify kieran-osgood-shopify force-pushed the kieran-osgood/e2e-browserstack/phase-6-github-reporting branch from 33b81d9 to dc1c907 Compare June 29, 2026 16:04
@kieran-osgood-shopify kieran-osgood-shopify force-pushed the kieran-osgood/e2e-browserstack/phase-5-browserstack-maestro branch from d99cf73 to eebe02d Compare June 29, 2026 16:38
@kieran-osgood-shopify kieran-osgood-shopify force-pushed the kieran-osgood/e2e-browserstack/phase-6-github-reporting branch 2 times, most recently from ea86286 to e791f00 Compare June 29, 2026 17:01
@kieran-osgood-shopify kieran-osgood-shopify force-pushed the kieran-osgood/e2e-browserstack/phase-5-browserstack-maestro branch 2 times, most recently from 8bc166b to 07c670c Compare June 29, 2026 17:33
@kieran-osgood-shopify kieran-osgood-shopify force-pushed the kieran-osgood/e2e-browserstack/phase-6-github-reporting branch from e791f00 to 3545355 Compare June 29, 2026 17:33
@kieran-osgood-shopify kieran-osgood-shopify force-pushed the kieran-osgood/e2e-browserstack/phase-5-browserstack-maestro branch from 07c670c to ffa1eee Compare June 29, 2026 17:53
@kieran-osgood-shopify kieran-osgood-shopify force-pushed the kieran-osgood/e2e-browserstack/phase-6-github-reporting branch 2 times, most recently from 9bcd74d to 64cf7a4 Compare July 1, 2026 09:54
@kieran-osgood-shopify kieran-osgood-shopify force-pushed the kieran-osgood/e2e-browserstack/phase-5-browserstack-maestro branch 2 times, most recently from 31464ab to e3e8eba Compare July 1, 2026 10:07
@kieran-osgood-shopify kieran-osgood-shopify force-pushed the kieran-osgood/e2e-browserstack/phase-6-github-reporting branch 2 times, most recently from 3895dbe to 9a25f02 Compare July 1, 2026 10:40
@kieran-osgood-shopify kieran-osgood-shopify force-pushed the kieran-osgood/e2e-browserstack/phase-5-browserstack-maestro branch from e3e8eba to 69ba295 Compare July 1, 2026 10:40
@kieran-osgood-shopify kieran-osgood-shopify force-pushed the kieran-osgood/e2e-browserstack/phase-6-github-reporting branch from 9a25f02 to 8477810 Compare July 1, 2026 11:26
@kieran-osgood-shopify kieran-osgood-shopify force-pushed the kieran-osgood/e2e-browserstack/phase-5-browserstack-maestro branch 2 times, most recently from 6484336 to 05c783c Compare July 1, 2026 11:43
@kieran-osgood-shopify kieran-osgood-shopify force-pushed the kieran-osgood/e2e-browserstack/phase-6-github-reporting branch from 8477810 to 3399b8d Compare July 1, 2026 11:43
@kieran-osgood-shopify kieran-osgood-shopify force-pushed the kieran-osgood/e2e-browserstack/phase-5-browserstack-maestro branch from 05c783c to 5153e7b Compare July 1, 2026 13:41
@kieran-osgood-shopify kieran-osgood-shopify force-pushed the kieran-osgood/e2e-browserstack/phase-6-github-reporting branch from 3399b8d to 9bdcf35 Compare July 1, 2026 13:41
@kieran-osgood-shopify kieran-osgood-shopify force-pushed the kieran-osgood/e2e-browserstack/phase-5-browserstack-maestro branch from 5153e7b to 3bdee2e Compare July 1, 2026 13:54
@kieran-osgood-shopify kieran-osgood-shopify force-pushed the kieran-osgood/e2e-browserstack/phase-6-github-reporting branch 2 times, most recently from 3493eeb to bb4aa6c Compare July 1, 2026 13:57
@kieran-osgood-shopify kieran-osgood-shopify force-pushed the kieran-osgood/e2e-browserstack/phase-5-browserstack-maestro branch from 3bdee2e to 93dbc2b Compare July 1, 2026 13:57
@kieran-osgood-shopify kieran-osgood-shopify force-pushed the kieran-osgood/e2e-browserstack/phase-6-github-reporting branch 2 times, most recently from 3e0bbb2 to fbcff21 Compare July 1, 2026 14:32
@kieran-osgood-shopify kieran-osgood-shopify force-pushed the kieran-osgood/e2e-browserstack/phase-5-browserstack-maestro branch 2 times, most recently from 45bc8a3 to fa7a218 Compare July 1, 2026 14:57
@kieran-osgood-shopify kieran-osgood-shopify force-pushed the kieran-osgood/e2e-browserstack/phase-6-github-reporting branch 2 times, most recently from 3e8609a to 96b82a1 Compare July 1, 2026 15:11
@kieran-osgood-shopify kieran-osgood-shopify force-pushed the kieran-osgood/e2e-browserstack/phase-5-browserstack-maestro branch 2 times, most recently from 8fe8410 to b5723fc Compare July 1, 2026 17:12
@kieran-osgood-shopify kieran-osgood-shopify force-pushed the kieran-osgood/e2e-browserstack/phase-6-github-reporting branch from 96b82a1 to 8e8e717 Compare July 1, 2026 17:12
Assisted-By: devx/6c1e3ad5-96c8-4972-b087-da7ff7b195c3
@kieran-osgood-shopify kieran-osgood-shopify force-pushed the kieran-osgood/e2e-browserstack/phase-6-github-reporting branch from 8e8e717 to 6dde7f4 Compare July 1, 2026 17:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

#gsd:50662 Rebase Checkout Kit on UCP

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant