Soft-launch E2E PR checks

Assisted-By: devx/6c1e3ad5-96c8-4972-b087-da7ff7b195c3

Author: kieran-osgood-shopify

e2e/BITRISE.md

@@ -68,6 +68,7 @@ The non-secret E2E defaults are defined in `e2e/bitrise.yml` under `app.envs`. D
 | `E2E_PACKAGE_COMMAND_TIMEOUT_SECONDS` | `300` | Per-command timeout for package-suite matrix and suite commands. |
 | `E2E_REPORT_COMMAND_TIMEOUT_SECONDS` | `300` | Timeout for the GitHub reporting command. |
 | `E2E_RUBY_INSTALL_TIMEOUT_SECONDS` | `1800` | Timeout for installing the exact repository Ruby version from `.ruby-version`. |
+| `E2E_CANCEL_CONCURRENT_PR_BUILDS` | `true` | Cancels older in-progress E2E builds for the same PR when the Bitrise API token secret is available. |
 
 Secrets still need to be configured in Bitrise.io.
 
@@ -85,6 +86,14 @@ BrowserStack credentials are required by the `e2e-run-browserstack` workflow.
 | `BROWSERSTACK_USERNAME` | BrowserStack API username |
 | `BROWSERSTACK_ACCESS_KEY` | BrowserStack API access key |
 
+BrowserStack artifact links in GitHub reports require access to BrowserStack App Automate. Sign in to [BrowserStack App Automate](https://app-automate.browserstack.com/dashboard/v2/builds) before opening build, video, screenshot, or log links.
+
+## Bitrise API token
+
+The package-suite workflow runs a PR concurrency guard before expensive E2E work. Configure `BITRISE_API_TOKEN` as a Bitrise secret so the guard can cancel older in-progress E2E builds for the same PR. If `BITRISE_API_TOKEN` is unavailable, the guard also accepts `BITRISE_PAT`.
+
+If neither token is configured, the guard logs a skip message and the pipeline continues without cancellation.
+
 ## GitHub reporting
 
 The report workflow uses a GitHub API token to create commit statuses, Check Runs, and sticky PR comments. Prefer the short-lived token generated by the Bitrise GitHub App over a long-lived personal access token.
@@ -97,7 +106,7 @@ Green runs update statuses and Check Runs without creating new PR comments. Fail
 
 ## PR trigger
 
-Configure the Bitrise app to run the `e2e` pipeline for pull requests once the skeleton is ready to validate in Bitrise.
+`bitrise.yml` maps pull requests to the `e2e` pipeline with `trigger_map`. Keep the GitHub checks non-required while `E2E_STRICT=false`.
 
 ## Code signing
 

e2e/RUNBOOK.md

@@ -0,0 +1,69 @@
+# Checkout Kit E2E Runbook
+
+## Rollout mode
+
+The Bitrise E2E pipeline starts in soft-fail mode:
+
+```bash
+E2E_STRICT=false
+```
+
+In soft-fail mode, BrowserStack Maestro assertion failures are reported through GitHub commit statuses, Check Runs, and the sticky PR failure comment, but the BrowserStack workflow exits successfully so the initial rollout does not block PR progress.
+
+Set `E2E_STRICT=true` only after the suite has stabilized and the corresponding GitHub contexts are ready to become required checks.
+
+## Retry behavior
+
+BrowserStack API calls retry transient infrastructure responses once by default:
+
+```bash
+E2E_BROWSERSTACK_API_RETRIES=1
+```
+
+Retry applies to HTTP 429 and 5xx responses. Maestro assertion failures are not auto-retried by default so first-failure evidence is preserved.
+
+## Timeouts
+
+BrowserStack polling uses these defaults:
+
+```bash
+E2E_BROWSERSTACK_TIMEOUT_SECONDS=1800
+E2E_BROWSERSTACK_POLL_SECONDS=30
+```
+
+If polling times out, the runner attempts to stop the BrowserStack build before failing or soft-failing according to `E2E_STRICT`.
+
+## Local rerun notes
+
+Use the matrix row from a failure report to identify the app target, platform, OS track, and suite:
+
+```bash
+ruby e2e/scripts/e2e_matrix expand --index <index>
+```
+
+Use the reported resolved device to pin a rerun:
+
+```bash
+E2E_DEVICE_OVERRIDE="<resolved BrowserStack device>" \
+E2E_STRICT=true \
+e2e/scripts/browserstack_maestro --index <index> --suite-zip <suite.zip> --output-dir <results-dir>
+```
+
+The app artifact environment variable for the matrix row must point at the `.apk` or `.ipa` artifact before rerunning.
+
+## Failure triage
+
+Use the GitHub Check Run or sticky PR failure comment first. Failure summaries should include Markdown links to:
+
+- BrowserStack build
+- failed testcase
+- video
+- screenshot
+- Maestro command log
+- Maestro log
+- device log
+- network log when enabled
+
+BrowserStack artifact links require BrowserStack App Automate access. Sign in to [BrowserStack App Automate](https://app-automate.browserstack.com/dashboard/v2/builds) before opening evidence links.
+
+Avoid posting additional PR comments for green runs.

e2e/bitrise.yml

@@ -24,6 +24,11 @@ app:
     - E2E_PACKAGE_COMMAND_TIMEOUT_SECONDS: "300"
     - E2E_REPORT_COMMAND_TIMEOUT_SECONDS: "300"
     - E2E_RUBY_INSTALL_TIMEOUT_SECONDS: "1800"
+    - E2E_CANCEL_CONCURRENT_PR_BUILDS: "true"
+
+trigger_map:
+  - pull_request_source_branch: "*"
+    pipeline: e2e
 
 pipelines:
   e2e:
@@ -57,6 +62,8 @@ workflows:
                 set -euo pipefail
                 source e2e/scripts/bitrise_ci_helpers
                 e2e_prepare_ruby
+                e2e_log "Cancelling older in-progress E2E PR builds"
+                ruby e2e/scripts/cancel_superseded_bitrise_pr_builds
                 e2e_run_with_timeout "${E2E_PACKAGE_COMMAND_TIMEOUT_SECONDS:-300}" ruby e2e/scripts/e2e_matrix validate
                 e2e_log "Expanding E2E matrix"
                 e2e_run_with_timeout "${E2E_PACKAGE_COMMAND_TIMEOUT_SECONDS:-300}" ruby e2e/scripts/e2e_matrix expand > "$BITRISE_DEPLOY_DIR/e2e-matrix.json"
@@ -288,7 +295,7 @@ workflows:
                 : "${E2E_REACT_NATIVE_IOS_APP_PATH:?E2E_REACT_NATIVE_IOS_APP_PATH is required. Check e2e-run-browserstack pull-intermediate-files artifact_sources.}"
                 : "${E2E_REACT_NATIVE_ANDROID_APP_PATH:?E2E_REACT_NATIVE_ANDROID_APP_PATH is required. Check e2e-run-browserstack pull-intermediate-files artifact_sources.}"
                 e2e_log "Running BrowserStack Maestro for matrix row ${run_index}"
-                e2e/scripts/browserstack_maestro \
+                E2E_STRICT="${E2E_STRICT:-false}" e2e/scripts/browserstack_maestro \
                   --index "$run_index" \
                   --suite-zip "$E2E_MAESTRO_SUITE_ZIP" \
                   --output-dir "$results_dir"

e2e/lib/bitrise_concurrency_guard.rb

@@ -0,0 +1,79 @@
+# frozen_string_literal: true
+
+require "json"
+require "net/http"
+
+class BitriseConcurrencyGuard
+  def initialize(client:, app_slug:, current_build_slug:, current_build_number:, pr_number:, workflow_prefix:)
+    @client = client
+    @app_slug = app_slug
+    @current_build_slug = current_build_slug
+    @current_build_number = Integer(current_build_number)
+    @pr_number = pr_number.to_s
+    @workflow_prefix = workflow_prefix
+  end
+
+  def cancel_older_pr_builds
+    return [] if @pr_number.empty?
+
+    older_builds.map do |build|
+      build_slug = build.fetch("slug")
+      @client.abort_build(@app_slug, build_slug)
+      build_slug
+    end
+  end
+
+  private
+
+  def older_builds
+    @client.builds(@app_slug).select do |build|
+      build.fetch("slug") != @current_build_slug &&
+        build.fetch("pull_request_id").to_s == @pr_number &&
+        build.fetch("status_text") == "in-progress" &&
+        build.fetch("triggered_workflow").to_s.start_with?(@workflow_prefix) &&
+        Integer(build.fetch("build_number")) < @current_build_number
+    end
+  end
+end
+
+class BitriseApiClient
+  API_HOST = "api.bitrise.io"
+
+  def initialize(token:)
+    @token = token
+  end
+
+  def builds(app_slug)
+    get_json("/v0.1/apps/#{app_slug}/builds?limit=50").fetch("data")
+  end
+
+  def abort_build(app_slug, build_slug)
+    request = Net::HTTP::Post.new("/v0.1/apps/#{app_slug}/builds/#{build_slug}/abort")
+    request.body = JSON.generate(
+      abort_reason: "Superseded by a newer Checkout Kit E2E PR build",
+      abort_with_success: false,
+      skip_notifications: true
+    )
+    execute_json_request(request)
+  end
+
+  private
+
+  def get_json(path)
+    execute_request(Net::HTTP::Get.new(path))
+  end
+
+  def execute_json_request(request)
+    request["Content-Type"] = "application/json"
+    execute_request(request)
+  end
+
+  def execute_request(request)
+    request["Authorization"] = @token
+    response = Net::HTTP.start(API_HOST, 443, use_ssl: true) { |http| http.request(request) }
+    body = response.body.to_s.empty? ? {} : JSON.parse(response.body)
+    return body if response.is_a?(Net::HTTPSuccess)
+
+    raise "Bitrise request failed #{response.code}: #{body}"
+  end
+end

e2e/scripts/browserstack_maestro

@@ -33,11 +33,15 @@ class BrowserStackMaestro
     sessions = fetch_sessions(build_status)
     result = normalize_result(run, device, app, suite, build, build_status, sessions)
     write_json("result.json", result)
-    exit(result.fetch("passed") ? 0 : 1)
+    exit(result.fetch("passed") || !strict? ? 0 : 1)
   end
 
   private
 
+  def strict?
+    ENV.fetch("E2E_STRICT", "false") == "true"
+  end
+
   def resolve_device(run)
     override = ENV["E2E_DEVICE_OVERRIDE"] || ENV["E2E_#{run.fetch("platform").upcase}_DEVICE_OVERRIDE"]
     if override && !override.empty?
@@ -85,12 +89,21 @@ class BrowserStackMaestro
       response = get_json("/app-automate/maestro/v2/builds/#{build_id}")
       write_json("build-status.json", response)
       return response if TERMINAL_STATUSES.include?(response.fetch("status").to_s.downcase)
-      raise "BrowserStack build timed out: #{build_id}" if Time.now >= deadline
+      if Time.now >= deadline
+        stop_build(build_id)
+        raise "BrowserStack build timed out: #{build_id}"
+      end
 
       sleep ENV.fetch("E2E_BROWSERSTACK_POLL_SECONDS", "30").to_i
     end
   end
 
+  def stop_build(build_id)
+    post_json("/app-automate/maestro/builds/#{build_id}/stop", {})
+  rescue StandardError => error
+    warn "Unable to stop BrowserStack build #{build_id}: #{error.message}"
+  end
+
   def fetch_sessions(build_status)
     build_id = build_status.fetch("id")
     build_status.fetch("devices", []).flat_map do |device|
@@ -155,12 +168,28 @@ class BrowserStackMaestro
   end
 
   def execute_request(request)
-    request.basic_auth(@username, @access_key)
-    response = Net::HTTP.start(API_HOST, 443, use_ssl: true) { |http| http.request(request) }
-    parsed = JSON.parse(response.body)
-    return parsed if response.is_a?(Net::HTTPSuccess)
+    attempts = 0
+    retries = ENV.fetch("E2E_BROWSERSTACK_API_RETRIES", "1").to_i
+    loop do
+      attempts += 1
+      request.basic_auth(@username, @access_key)
+      response = Net::HTTP.start(API_HOST, 443, use_ssl: true) { |http| http.request(request) }
+      parsed = parse_response_body(response)
+      return parsed if response.is_a?(Net::HTTPSuccess)
+      raise "BrowserStack request failed #{response.code}: #{parsed}" unless retryable_browserstack_response?(response) && attempts <= retries
+
+      sleep attempts
+    end
+  end
+
+  def parse_response_body(response)
+    JSON.parse(response.body)
+  rescue JSON::ParserError
+    raise "BrowserStack request returned non-JSON response"
+  end
 
-    raise "BrowserStack request failed #{response.code}: #{parsed}"
+  def retryable_browserstack_response?(response)
+    response.code.to_i == 429 || response.code.to_i >= 500
   end
 
   def custom_id(run, suffix)

e2e/scripts/cancel_superseded_bitrise_pr_builds

@@ -0,0 +1,31 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require_relative "../lib/bitrise_concurrency_guard"
+
+token = ENV["BITRISE_API_TOKEN"] || ENV["BITRISE_PAT"]
+if token.nil? || token.empty?
+  warn "Skipping Bitrise PR concurrency guard because BITRISE_API_TOKEN is not set."
+  exit 0
+end
+
+if ENV.fetch("E2E_CANCEL_CONCURRENT_PR_BUILDS", "true") != "true"
+  warn "Skipping Bitrise PR concurrency guard because E2E_CANCEL_CONCURRENT_PR_BUILDS is not true."
+  exit 0
+end
+
+guard = BitriseConcurrencyGuard.new(
+  client: BitriseApiClient.new(token: token),
+  app_slug: ENV.fetch("BITRISE_APP_SLUG"),
+  current_build_slug: ENV.fetch("BITRISE_BUILD_SLUG"),
+  current_build_number: ENV.fetch("BITRISE_BUILD_NUMBER"),
+  pr_number: ENV.fetch("BITRISE_PULL_REQUEST", ""),
+  workflow_prefix: ENV.fetch("E2E_BITRISE_WORKFLOW_PREFIX", "e2e")
+)
+
+aborted = guard.cancel_older_pr_builds
+if aborted.empty?
+  puts "No older in-progress E2E PR builds to cancel."
+else
+  puts "Cancelled #{aborted.length} older in-progress E2E PR build(s)."
+end

e2e/test/bitrise_concurrency_guard_test.rb

@@ -0,0 +1,74 @@
+# frozen_string_literal: true
+
+require "minitest/autorun"
+require_relative "../lib/bitrise_concurrency_guard"
+
+class BitriseConcurrencyGuardTest < Minitest::Test
+  def test_aborts_older_in_progress_pr_builds
+    client = FakeBitriseClient.new([
+      build("current", 42, "in-progress"),
+      build("older-package", 41, "in-progress"),
+      build("older-android", 41, "in-progress", workflow: "e2e-build-react-native-android"),
+      build("newer", 43, "in-progress"),
+      build("finished", 40, "success"),
+      build("other-pr", 39, "in-progress", pr: "365"),
+    ])
+    guard = BitriseConcurrencyGuard.new(
+      client: client,
+      app_slug: "app-slug",
+      current_build_slug: "current",
+      current_build_number: 42,
+      pr_number: "364",
+      workflow_prefix: "e2e"
+    )
+
+    aborted = guard.cancel_older_pr_builds
+
+    assert_equal ["older-package", "older-android"], aborted
+    assert_equal ["older-package", "older-android"], client.aborted_build_slugs
+  end
+
+  def test_skips_without_pr_number
+    client = FakeBitriseClient.new([build("older", 1, "in-progress")])
+    guard = BitriseConcurrencyGuard.new(
+      client: client,
+      app_slug: "app-slug",
+      current_build_slug: "current",
+      current_build_number: 2,
+      pr_number: "",
+      workflow_prefix: "e2e"
+    )
+
+    assert_empty guard.cancel_older_pr_builds
+    assert_empty client.aborted_build_slugs
+  end
+
+  private
+
+  def build(slug, number, status, pr: "364", workflow: "e2e-package-suite")
+    {
+      "slug" => slug,
+      "build_number" => number,
+      "status_text" => status,
+      "pull_request_id" => pr,
+      "triggered_workflow" => workflow,
+    }
+  end
+
+  class FakeBitriseClient
+    attr_reader :aborted_build_slugs
+
+    def initialize(builds)
+      @builds = builds
+      @aborted_build_slugs = []
+    end
+
+    def builds(_app_slug)
+      @builds
+    end
+
+    def abort_build(_app_slug, build_slug)
+      @aborted_build_slugs << build_slug
+    end
+  end
+end