Soft-launch E2E PR checks

Assisted-By: devx/396ca0c4-44ad-415d-8d80-7200df4e42a4

Author: kieran-osgood-shopify

e2e/BITRISE.md

@@ -91,7 +91,7 @@ Green runs update statuses and Check Runs without creating new PR comments. Fail
 
 ## PR trigger
 
-Configure the Bitrise app to run the `e2e` pipeline for pull requests once the skeleton is ready to validate in Bitrise.
+`bitrise.yml` maps pull requests to the `e2e` pipeline with `trigger_map`. Keep the GitHub checks non-required while `E2E_STRICT=false`.
 
 ## Code signing
 

e2e/RUNBOOK.md

@@ -0,0 +1,67 @@
+# Checkout Kit E2E Runbook
+
+## Rollout mode
+
+The Bitrise E2E pipeline starts in soft-fail mode:
+
+```bash
+E2E_STRICT=false
+```
+
+In soft-fail mode, BrowserStack Maestro assertion failures are reported through GitHub commit statuses, Check Runs, and the sticky PR failure comment, but the BrowserStack workflow exits successfully so the initial rollout does not block PR progress.
+
+Set `E2E_STRICT=true` only after the suite has stabilized and the corresponding GitHub contexts are ready to become required checks.
+
+## Retry behavior
+
+BrowserStack API calls retry transient infrastructure responses once by default:
+
+```bash
+E2E_BROWSERSTACK_API_RETRIES=1
+```
+
+Retry applies to HTTP 429 and 5xx responses. Maestro assertion failures are not auto-retried by default so first-failure evidence is preserved.
+
+## Timeouts
+
+BrowserStack polling uses these defaults:
+
+```bash
+E2E_BROWSERSTACK_TIMEOUT_SECONDS=1800
+E2E_BROWSERSTACK_POLL_SECONDS=30
+```
+
+If polling times out, the runner attempts to stop the BrowserStack build before failing or soft-failing according to `E2E_STRICT`.
+
+## Local rerun notes
+
+Use the matrix row from a failure report to identify the app target, platform, OS track, and suite:
+
+```bash
+ruby e2e/scripts/e2e_matrix expand --index <index>
+```
+
+Use the reported resolved device to pin a rerun:
+
+```bash
+E2E_DEVICE_OVERRIDE="<resolved BrowserStack device>" \
+E2E_STRICT=true \
+e2e/scripts/browserstack_maestro --index <index> --suite-zip <suite.zip> --output-dir <results-dir>
+```
+
+The app artifact environment variable for the matrix row must point at the `.apk` or `.ipa` artifact before rerunning.
+
+## Failure triage
+
+Use the GitHub Check Run or sticky PR failure comment first. Failure summaries should include direct links to:
+
+- BrowserStack build
+- failed testcase
+- video
+- screenshot
+- Maestro command log
+- Maestro log
+- device log
+- network log when enabled
+
+Avoid posting additional PR comments for green runs.

e2e/bitrise.yml

@@ -20,6 +20,10 @@ app:
     - E2E_REPORT_COMMAND_TIMEOUT_SECONDS: "300"
     - E2E_RUBY_INSTALL_TIMEOUT_SECONDS: "1800"
 
+trigger_map:
+  - pull_request_source_branch: "*"
+    pipeline: e2e
+
 pipelines:
   e2e:
     workflows:
@@ -206,7 +210,7 @@ workflows:
                 mkdir -p "$results_dir"
                 envman add --key E2E_BROWSERSTACK_RESULTS_DIR --value "$results_dir"
                 e2e_log "Running BrowserStack Maestro for matrix row ${run_index}"
-                e2e/scripts/browserstack_maestro \
+                E2E_STRICT="${E2E_STRICT:-false}" e2e/scripts/browserstack_maestro \
                   --index "$run_index" \
                   --suite-zip "$E2E_MAESTRO_SUITE_ZIP" \
                   --output-dir "$results_dir"

e2e/scripts/browserstack_maestro

@@ -33,11 +33,15 @@ class BrowserStackMaestro
     sessions = fetch_sessions(build_status)
     result = normalize_result(run, device, app, suite, build, build_status, sessions)
     write_json("result.json", result)
-    exit(result.fetch("passed") ? 0 : 1)
+    exit(result.fetch("passed") || !strict? ? 0 : 1)
   end
 
   private
 
+  def strict?
+    ENV.fetch("E2E_STRICT", "false") == "true"
+  end
+
   def resolve_device(run)
     override = ENV["E2E_DEVICE_OVERRIDE"] || ENV["E2E_#{run.fetch("platform").upcase}_DEVICE_OVERRIDE"]
     if override && !override.empty?
@@ -85,12 +89,21 @@ class BrowserStackMaestro
       response = get_json("/app-automate/maestro/v2/builds/#{build_id}")
       write_json("build-status.json", response)
       return response if TERMINAL_STATUSES.include?(response.fetch("status").to_s.downcase)
-      raise "BrowserStack build timed out: #{build_id}" if Time.now >= deadline
+      if Time.now >= deadline
+        stop_build(build_id)
+        raise "BrowserStack build timed out: #{build_id}"
+      end
 
       sleep ENV.fetch("E2E_BROWSERSTACK_POLL_SECONDS", "30").to_i
     end
   end
 
+  def stop_build(build_id)
+    post_json("/app-automate/maestro/builds/#{build_id}/stop", {})
+  rescue StandardError => error
+    warn "Unable to stop BrowserStack build #{build_id}: #{error.message}"
+  end
+
   def fetch_sessions(build_status)
     build_id = build_status.fetch("id")
     build_status.fetch("devices", []).flat_map do |device|
@@ -155,12 +168,28 @@ class BrowserStackMaestro
   end
 
   def execute_request(request)
-    request.basic_auth(@username, @access_key)
-    response = Net::HTTP.start(API_HOST, 443, use_ssl: true) { |http| http.request(request) }
-    parsed = JSON.parse(response.body)
-    return parsed if response.is_a?(Net::HTTPSuccess)
+    attempts = 0
+    retries = ENV.fetch("E2E_BROWSERSTACK_API_RETRIES", "1").to_i
+    loop do
+      attempts += 1
+      request.basic_auth(@username, @access_key)
+      response = Net::HTTP.start(API_HOST, 443, use_ssl: true) { |http| http.request(request) }
+      parsed = parse_response_body(response)
+      return parsed if response.is_a?(Net::HTTPSuccess)
+      raise "BrowserStack request failed #{response.code}: #{parsed}" unless retryable_browserstack_response?(response) && attempts <= retries
+
+      sleep attempts
+    end
+  end
+
+  def parse_response_body(response)
+    JSON.parse(response.body)
+  rescue JSON::ParserError
+    raise "BrowserStack request returned non-JSON response"
+  end
 
-    raise "BrowserStack request failed #{response.code}: #{parsed}"
+  def retryable_browserstack_response?(response)
+    response.code.to_i == 429 || response.code.to_i >= 500
   end
 
   def custom_id(run, suffix)

e2e/test/soft_launch_test.rb

@@ -0,0 +1,40 @@
+# frozen_string_literal: true
+
+require "minitest/autorun"
+require "yaml"
+
+class E2ESoftLaunchTest < Minitest::Test
+  BITRISE_CONFIG_PATH = "e2e/bitrise.yml"
+
+  def test_bitrise_pr_trigger_runs_e2e_pipeline
+    config = YAML.safe_load_file(BITRISE_CONFIG_PATH)
+    trigger = config.fetch("trigger_map").find { |entry| entry["pull_request_source_branch"] == "*" }
+
+    assert_equal "e2e", trigger.fetch("pipeline")
+  end
+
+  def test_browserstack_runner_supports_soft_fail_and_retry_controls
+    script = File.read("e2e/scripts/browserstack_maestro")
+
+    assert_includes script, "E2E_STRICT"
+    assert_includes script, "E2E_BROWSERSTACK_API_RETRIES"
+    assert_includes script, "retryable_browserstack_response?"
+    assert_includes script, '/app-automate/maestro/builds/#{build_id}/stop'
+  end
+
+  def test_bitrise_browserstack_workflow_defaults_to_soft_fail
+    config = YAML.safe_load_file(BITRISE_CONFIG_PATH)
+    run_script = config.fetch("workflows").fetch("e2e-run-browserstack").fetch("steps").flat_map do |step|
+      step_value = step.values.first
+      next [] unless step.keys.first.start_with?("script")
+
+      step_value.fetch("inputs", []).filter_map { |input| input["content"] }
+    end.join("\n")
+
+    assert_includes run_script, "E2E_STRICT=\"${E2E_STRICT:-false}\""
+  end
+
+  def test_runbook_exists
+    assert File.exist?("e2e/RUNBOOK.md")
+  end
+end