Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -334,6 +334,7 @@ tests/tools/ @MetaMask/qa
tests/websocket/ @MetaMask/qa

# QA Team - CI
.github/guidelines/E2E_DECISION_TREE.md @MetaMask/qa
.github/actions/smart-e2e-selection/ @MetaMask/qa
.github/workflows/ai-pr-risk-analysis.yml @MetaMask/qa
.github/workflows/auto-label-not-ready-for-e2e.yml @MetaMask/qa
Expand Down
1 change: 1 addition & 0 deletions .github/actions/setup-e2e-env/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -281,6 +281,7 @@ runs:
with:
path: |
node_modules
.yarn/install-state.gz
key: ${{ inputs.cache-prefix }}-yarn-${{ inputs.platform }}-${{ runner.os }}-${{ hashFiles('yarn.lock') }}

- name: Install JavaScript dependencies with retry
Expand Down
10 changes: 8 additions & 2 deletions .github/guidelines/E2E_DECISION_TREE.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ flowchart TD
GR -->|PR label: skip-e2e| HS[No E2E]
GR -->|PR label: pr-not-ready-for-e2e| L2[No E2E]
L2 -->|ignorable-only changes| NoBlock[No merge block]
L2 -->|non-ignorable changes| Skip2[Merge blocked]
L2 -->|non-ignorable changes| Skip2[⛔️ Merge blocked]
GR -->|PR ignorable-only changes| Ignorable[No E2E]
GR -->|PR has Android-only changes| Android[Android Build + Tests needed]
GR -->|PR has iOS-only changes| iOS[iOS Build + Test needed]
Expand Down Expand Up @@ -41,9 +41,15 @@ Runs only when all of the following are true:
- No hard E2E skip signal (label `skip-e2e`)
- No `skip-smart-e2e-selection` label

## (Exceptional) skip builds and all E2E tests

- Label `skip-e2e` can be added to the PR to skip E2E tests (and builds) in case of infra issues.
- Using this label should be exceptional in case of CI friction and urgencies. Verify new changes and regressions manually before merging.

## E2E flakiness detection in PRs

Flakiness detection is applied to modified E2E test files in PRs:

- Modified E2E test files run twice
- It applies to existing test files as well as new test files
- It applies to existing test files as well as new test files added in the PR
- It can be disabled by adding the label `skip-e2e-flakiness-detection`. Useful when making large refactors or when changes don't pose flakiness risk.
2 changes: 1 addition & 1 deletion .github/guidelines/LABELING_GUIDELINES.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ Using any of these labels should be exceptional in case of CI friction and urgen

- **skip-sonar-cloud**: The PR will be merged without running SonarCloud checks.
- **skip-e2e**: The PR will be merged without running E2E tests.
- **skip-e2e-quality-gate**: This label will disable the default test retries for E2E test files modified in a PR. Useful when making large refactors or when changes don't pose flakiness risk.
- **skip-e2e-flakiness-detection**: This label will disable the default test retries for E2E test files modified in a PR. Useful when making large refactors or when changes don't pose flakiness risk.

### Skip Smart E2E Selection

Expand Down
7 changes: 7 additions & 0 deletions .github/rules/filter-rules.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,10 @@ low_level_test_files: &low_level_test_files
- '**/*.stories.*'
- '**/*.snap'

# LOCALE TRANSLATION FILES
locale_translation_files: &locale_translation_files
- 'locales/languages/**/*.json'

# CONFIG FILES
config_files: &config_files
- '.eslint*'
Expand All @@ -47,6 +51,7 @@ e2e_ignorable:
- *documentation_files
- *asset_files
- *low_level_test_files
- *locale_translation_files
- *config_files
- *ci_files

Expand Down Expand Up @@ -74,6 +79,7 @@ android_or_ignorable:
- *documentation_files
- *asset_files
- *low_level_test_files
- *locale_translation_files
- *config_files
- *ci_files

Expand All @@ -82,6 +88,7 @@ ios_or_ignorable:
- *documentation_files
- *asset_files
- *low_level_test_files
- *locale_translation_files
- *config_files
- *ci_files

Expand Down
4 changes: 2 additions & 2 deletions .github/scripts/e2e-split-tags-shards.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -108,9 +108,9 @@ async function shouldSkipFlakinessDetection() {
);

const labels = data?.repository?.pullRequest?.labels?.nodes || [];
const labelFound = labels.some((l) => String(l?.name).toLowerCase() === 'skip-e2e-quality-gate');
const labelFound = labels.some((l) => String(l?.name).toLowerCase() === 'skip-e2e-flakiness-detection');
if (labelFound) {
console.log('⏭️ Found "skip-e2e-quality-gate" label → SKIPPING flakiness detection');
console.log('⏭️ Found "skip-e2e-flakiness-detection" label → SKIPPING flakiness detection');
}
return labelFound;
} catch (e) {
Expand Down
13 changes: 13 additions & 0 deletions .github/workflows/auto-label-not-ready-for-e2e.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
# Automatically applies the 'pr-not-ready-for-e2e' label to newly opened PRs,
# but only if the PR is opened between 13:00 and 17:00 UTC (15:00–19:00 CEST).
name: Auto-apply pr-not-ready-for-e2e label

on:
Expand All @@ -16,7 +18,18 @@ jobs:
permissions:
pull-requests: write
steps:
- name: Check current UTC hour
id: time-check
run: |
HOUR=$(date -u +%H)
echo "Current UTC hour: $HOUR"
if [[ $HOUR -ge 13 && $HOUR -lt 17 ]]; then
echo "in_window=true" >> "$GITHUB_OUTPUT"
else
echo "in_window=false" >> "$GITHUB_OUTPUT"
fi
- name: Add pr-not-ready-for-e2e label
if: steps.time-check.outputs.in_window == 'true'
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GH_REPO: ${{ github.repository }}
Expand Down
9 changes: 9 additions & 0 deletions .github/workflows/build-rc-auto.yml
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,15 @@ jobs:
cache: yarn
- name: Install dependencies
run: yarn install --immutable

- name: Download build environment artifacts
continue-on-error: true
uses: actions/download-artifact@v4
with:
pattern: build-env-main-rc-*
path: build-env-artifacts
merge-multiple: false

- name: Post RC Build Comment with Test Plan
run: node -r esbuild-register scripts/build-announce/index.ts
timeout-minutes: 8
Expand Down
10 changes: 10 additions & 0 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -305,6 +305,8 @@ jobs:
eval "$(node scripts/apply-build-config.js ${{ inputs.build_name }} --export)"
# Persist to GITHUB_ENV so later steps (e.g. Build) see CONFIGURATION, IS_SIM_BUILD, etc.
node scripts/apply-build-config.js ${{ inputs.build_name }} --export-github-env >> "$GITHUB_ENV"
# Generate build-env.json capturing actual env values used in this build (non-critical)
node scripts/apply-build-config.js ${{ inputs.build_name }} --write-build-env || true

- name: Validate secrets
env:
Expand Down Expand Up @@ -497,6 +499,14 @@ jobs:
path: ${{ steps.rename.outputs.android_sourcemap_dir }}
if-no-files-found: warn

- name: Upload build environment JSON
if: success()
uses: actions/upload-artifact@v4
with:
name: build-env-${{ inputs.build_name }}-${{ matrix.platform }}
path: build-env.json
if-no-files-found: warn

# Single fan-in job so workflow_call outputs work with matrix `build` (matrix jobs cannot feed workflow outputs directly).
emit-build-metadata:
name: Emit build metadata
Expand Down
51 changes: 36 additions & 15 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -510,11 +510,20 @@ jobs:
if: ${{ !cancelled() && github.event_name != 'merge_group' }}
steps:
- uses: actions/checkout@v6
- name: Restore node_modules cache
id: cache-node-modules
uses: actions/cache@v4
with:
path: |
node_modules
.yarn/install-state.gz
key: ${{ runner.os }}-node-modules-${{ hashFiles('yarn.lock') }}
- uses: actions/setup-node@v6
with:
node-version-file: '.nvmrc'
cache: yarn
- name: Install Yarn dependencies with retry
if: steps.cache-node-modules.outputs.cache-hit != 'true'
uses: nick-fields/retry@ce71cc2ab81d554ebbe88c79ab5975992d79ba08 #v3.0.2
with:
timeout_minutes: 10
Expand Down Expand Up @@ -622,11 +631,20 @@ jobs:
shard: [1, 2]
steps:
- uses: actions/checkout@v6
- name: Restore node_modules cache
id: cache-node-modules
uses: actions/cache@v4
with:
path: |
node_modules
.yarn/install-state.gz
key: ${{ runner.os }}-node-modules-${{ hashFiles('yarn.lock') }}
- uses: actions/setup-node@v6
with:
node-version-file: '.nvmrc'
cache: yarn
- name: Install Yarn dependencies with retry
if: steps.cache-node-modules.outputs.cache-hit != 'true'
uses: nick-fields/retry@ce71cc2ab81d554ebbe88c79ab5975992d79ba08 #v3.0.2
with:
timeout_minutes: 10
Expand Down Expand Up @@ -981,6 +999,8 @@ jobs:
needs:
- get_requirements
- all-jobs-pass
- build-android-apks
- build-ios-apps
- e2e-smoke-tests-android
- e2e-smoke-tests-ios
env:
Expand All @@ -1004,22 +1024,23 @@ jobs:
exit 1
fi

# Check E2E jobs only if they should have run
# Check E2E build + smoke results only if E2E should have run.
# 'skipped' is acceptable — covers merge_group, fork PRs, ignorable-only changes,
# platform-only PRs, and AI selection returning zero tags.
# 'failure'/'cancelled' on any of build or smoke must block merge.
if [[ "${{ needs.get_requirements.outputs.skip_e2e }}" != "true" ]]; then
# Accept both 'success' and 'skipped' as valid results
# 'skipped' occurs during merge_group events or when jobs are intentionally skipped
# Only fail on 'failure' or 'cancelled'
ANDROID_RESULT="${{ needs.e2e-smoke-tests-android.result }}"
if [[ "$ANDROID_RESULT" == "failure" ]] || [[ "$ANDROID_RESULT" == "cancelled" ]]; then
echo "Android E2E tests failed (result: $ANDROID_RESULT)"
exit 1
fi

IOS_RESULT="${{ needs.e2e-smoke-tests-ios.result }}"
if [[ "$IOS_RESULT" == "failure" ]] || [[ "$IOS_RESULT" == "cancelled" ]]; then
echo "iOS E2E tests failed (result: $IOS_RESULT)"
exit 1
fi
for entry in \
"build-android-apks:${{ needs.build-android-apks.result }}" \
"e2e-smoke-tests-android:${{ needs.e2e-smoke-tests-android.result }}" \
"build-ios-apps:${{ needs.build-ios-apps.result }}" \
"e2e-smoke-tests-ios:${{ needs.e2e-smoke-tests-ios.result }}"; do
name="${entry%%:*}"
result="${entry#*:}"
if [[ "$result" == "failure" ]] || [[ "$result" == "cancelled" ]]; then
echo "::error::Required E2E job '$name' did not succeed (result: $result)"
exit 1
fi
done
fi

echo "All required jobs passed"
Expand Down
122 changes: 122 additions & 0 deletions .github/workflows/prod-build-env-notify.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
##############################################################################################
#
# Production Build Environment Notification
#
# Automatically triggers when production builds complete.
# Downloads build-env.json artifact and extracts environment values.
# TODO: Post to Slack (follow-up PR)
#
##############################################################################################
name: Prod Build Env Notify

on:
workflow_run:
workflows: ["Runway iOS Production", "Runway Android Production"]
types: [completed]

jobs:
notify-env:
name: Post Environment Info
runs-on: ubuntu-latest
if: ${{ github.event.workflow_run.conclusion == 'success' }}
steps:
- uses: actions/checkout@v6

- uses: actions/setup-node@v6
with:
node-version-file: '.nvmrc'

- name: Download build-env artifact from triggering workflow
continue-on-error: true
id: download-artifact
uses: actions/download-artifact@v6
with:
name: build-env-main-prod-${{ github.event.workflow_run.name == 'Runway iOS Production' && 'ios' || 'android' }}
path: build-env-artifacts
run-id: ${{ github.event.workflow_run.id }}
github-token: ${{ secrets.GITHUB_TOKEN }}

- name: Extract and display environment values
if: steps.download-artifact.outcome == 'success'
id: extract-env
run: |
JSON_FILE="build-env-artifacts/build-env.json"

if [ ! -f "$JSON_FILE" ]; then
echo "ERROR: build-env.json not found"
exit 1
fi

echo "=== Build Environment Values ==="
cat "$JSON_FILE"
echo ""

# Extract values
BUILD_NAME=$(jq -r '.buildName' "$JSON_FILE")
METAMASK_ENV=$(jq -r '.env.METAMASK_ENVIRONMENT' "$JSON_FILE")
BUILD_TYPE=$(jq -r '.env.METAMASK_BUILD_TYPE' "$JSON_FILE")
REWARDS_URL=$(jq -r '.env.REWARDS_API_URL' "$JSON_FILE")
PORTFOLIO_URL=$(jq -r '.env.MM_PORTFOLIO_URL' "$JSON_FILE")
RAMPS_ENV=$(jq -r '.env.RAMPS_ENVIRONMENT' "$JSON_FILE")

# Map to Remote Feature Flag values
case "$METAMASK_ENV" in
production) REMOTE_FF_ENV="prod" ;;
rc) REMOTE_FF_ENV="rc" ;;
beta) REMOTE_FF_ENV="beta" ;;
test|e2e) REMOTE_FF_ENV="test" ;;
exp) REMOTE_FF_ENV="exp" ;;
*) REMOTE_FF_ENV="dev" ;;
esac

case "$BUILD_TYPE" in
flask) REMOTE_FF_DIST="flask" ;;
*) REMOTE_FF_DIST="main" ;;
esac

# Output for next steps
{
echo "build_name=$BUILD_NAME"
echo "environment=$METAMASK_ENV"
echo "build_type=$BUILD_TYPE"
echo "remote_ff_env=$REMOTE_FF_ENV"
echo "remote_ff_dist=$REMOTE_FF_DIST"
echo "rewards_url=$REWARDS_URL"
echo "portfolio_url=$PORTFOLIO_URL"
echo "ramps_env=$RAMPS_ENV"
} >> "$GITHUB_OUTPUT"

echo ""
echo "=== Extracted Values ==="
echo "Environment: $METAMASK_ENV"
echo "Build Type: $BUILD_TYPE"
echo "Remote Feature Flag Env: $REMOTE_FF_ENV"
echo "Remote Feature Flag Distribution: $REMOTE_FF_DIST"
echo "Rewards API URL: $REWARDS_URL"
echo "MM_PORTFOLIO_URL: $PORTFOLIO_URL"
echo "Ramps Environment: $RAMPS_ENV"

- name: Post to Slack
if: steps.download-artifact.outcome == 'success'
env:
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
PLATFORM: ${{ github.event.workflow_run.name == 'Runway iOS Production' && 'iOS' || 'Android' }}
ENVIRONMENT: ${{ steps.extract-env.outputs.environment }}
BUILD_TYPE: ${{ steps.extract-env.outputs.build_type }}
REMOTE_FF_ENV: ${{ steps.extract-env.outputs.remote_ff_env }}
REMOTE_FF_DIST: ${{ steps.extract-env.outputs.remote_ff_dist }}
REWARDS_URL: ${{ steps.extract-env.outputs.rewards_url }}
PORTFOLIO_URL: ${{ steps.extract-env.outputs.portfolio_url }}
RAMPS_ENV: ${{ steps.extract-env.outputs.ramps_env }}
WORKFLOW_URL: ${{ github.event.workflow_run.html_url }}
run: |
echo "TODO: Post to Slack"
echo "Platform: $PLATFORM"
echo "Environment: $ENVIRONMENT"
echo "Build Type: $BUILD_TYPE"
echo "Remote FF Env: $REMOTE_FF_ENV"
echo "Remote FF Distribution: $REMOTE_FF_DIST"
echo "Rewards URL: $REWARDS_URL"
echo "Portfolio URL: $PORTFOLIO_URL"
echo "Ramps Environment: $RAMPS_ENV"
echo "Workflow URL: $WORKFLOW_URL"
9 changes: 6 additions & 3 deletions .github/workflows/rerun-ci-on-skipped-e2e-labels.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ jobs:
if: >-
github.event.label.name == 'skip-smart-e2e-selection' ||
github.event.label.name == 'skip-e2e' ||
github.event.label.name == 'skip-e2e-quality-gate' ||
github.event.label.name == 'skip-e2e-flakiness-detection' ||
github.event.label.name == 'pr-not-ready-for-e2e'
runs-on: ubuntu-latest
permissions:
Expand Down Expand Up @@ -92,5 +92,8 @@ jobs:
run: |
RUN_ID="${{ steps.find.outputs.run_id }}"
echo "Re-running workflow $RUN_ID..."
gh run rerun "$RUN_ID" --repo "$REPO"
echo "CI workflow re-triggered successfully"
if gh run rerun "$RUN_ID" --repo "$REPO"; then
echo "CI workflow re-triggered successfully"
else
echo "Rerun not possible (run may not be in a retriable state)"
fi
Loading
Loading