diff --git a/docs.json b/docs.json index 08f7416..ee86d84 100644 --- a/docs.json +++ b/docs.json @@ -413,14 +413,6 @@ "qawolf/other-ci-webhook" ] }, - { - "group": "PR Testing", - "icon": "git-merge", - "pages": [ - "qawolf/PR-testing-for-GitHub-Integrations-2db5b2a994fb80a0aa94f3297d1f60c3", - "qawolf/PR-testing-for-GitLab-Integrations-2db5b2a994fb8075b13dd9d5c2e6be8c" - ] - }, { "group": "Mobile build testing", "icon": "smartphone", @@ -494,7 +486,15 @@ "icon": "plug", "pages": [ "qawolf/Test-environments-2dc5b2a994fb800e8ce4fe36ee34764a", - "qawolf/Integrating-with-microsoft-teams" + "qawolf/Integrating-with-microsoft-teams", + { + "group": "PR Testing", + "icon": "git-merge", + "pages": [ + "qawolf/PR-testing-for-GitHub-Integrations-2db5b2a994fb80a0aa94f3297d1f60c3", + "qawolf/PR-testing-for-GitLab-Integrations-2db5b2a994fb8075b13dd9d5c2e6be8c" + ] + } ] } ] diff --git a/qawolf/CI-CD-Overview.mdx b/qawolf/CI-CD-Overview.mdx index 932f2ed..dec8384 100644 --- a/qawolf/CI-CD-Overview.mdx +++ b/qawolf/CI-CD-Overview.mdx @@ -9,10 +9,6 @@ description: "QA Wolf integrates with your CI pipeline to run tests automaticall Run tests when a new build is deployed to an environment. - - Run tests automatically when a pull request is opened or updated. - - Run tests against a new mobile build artifact from your CI pipeline. diff --git a/qawolf/PR-testing-for-GitHub-Integrations-2db5b2a994fb80a0aa94f3297d1f60c3.mdx b/qawolf/PR-testing-for-GitHub-Integrations-2db5b2a994fb80a0aa94f3297d1f60c3.mdx index 3c1a033..b91230f 100644 --- a/qawolf/PR-testing-for-GitHub-Integrations-2db5b2a994fb80a0aa94f3297d1f60c3.mdx +++ b/qawolf/PR-testing-for-GitHub-Integrations-2db5b2a994fb80a0aa94f3297d1f60c3.mdx @@ -1,30 +1,48 @@ --- -title: "How to enable PR testing for GitHub Integrations" +title: "Setting up PR testing" description: "Run QA Wolf tests against preview environments on pull requests and report results as GitHub status checks." sidebarTitle: "GitHub" --- -## What QA Wolf configures + + This guide is for QA Wolf full service customers. If you're unsure whether this applies to you, contact your QA Wolf team. + -PR testing runs QA Wolf tests against preview environments before changes are merged. Tests run when a pull request enters the merge queue, and results are reported back to GitHub as a required status check. +## Configure basic PR testing - - Your [GitHub integration](/qawolf/GitHub-GitHub-Actions-2a15b2a994fb800699f6dc99584915d0) and deploy notifications must already be set up. - + + + Configure the GitHub integration in your team's settings page. + + + Choose one of the following methods to tell QA Wolf when a new preview environment is ready to be tested. + + **GitHub Deployments** + + If your team uses a hosting platform that integrates with GitHub — like Vercel — your deployments are already being reported to GitHub via the [GitHub Deployments API](https://docs.github.com/en/rest/deployments/deployments). QA Wolf listens for those deployment records and automatically acts when one is marked as successful. No additional configuration is required. + + **GitHub Action** + + Configure the [Notify QA Wolf on Deploy](https://github.com/marketplace/actions/notify-qa-wolf-on-deploy) GitHub Action to tell us when a new preview environment is ready to be tested. + + **SDK** -PR testing won't work until QA Wolf configures your workspace to support it. QA Wolf will: + If GitHub Actions aren't an option, the `@qawolf/ci-sdk` npm package offers a flexible alternative. See the [npm package documentation](https://www.npmjs.com/package/@qawolf/ci-sdk) for setup details. + + + QA Wolf will configure a trigger that matches your preview deployments. Reach out to your QA Wolf team to get this set up. + + -- Set up a trigger that matches your preview deployments -- Provide the deployment-type value to use for preview tests -- Confirm the preview URL variable and the status check name (for example, **QA Wolf Test Results**) +--- -## Before you begin +## Advanced: Merge queue setup - - Each pull request must deploy a preview environment that QA Wolf can reach. Your CI should provide a stable preview URL per PR and a step that waits until the preview is live before tests start. - + + This section covers an optional flow for teams using GitHub's merge queue feature. Most teams don't need this. + -## Add a default passing PR check +### Add a default passing PR check Add a GitHub Actions workflow that creates a passing **QA Wolf Test Results** check when a PR is opened or updated. This prevents pull requests from being blocked before the merge queue run happens. @@ -60,7 +78,7 @@ jobs: }); ``` -## Trigger QA Wolf tests in the merge queue +### Trigger QA Wolf tests in the merge queue Add a GitHub Actions workflow that notifies QA Wolf only during merge queue execution. @@ -90,37 +108,17 @@ jobs: - Your CI pipeline already creates a preview environment per pull request. - The preview environment URL is available as `PREVIEW_URL` when this job runs. -- A prior job (such as wait-for-preview-environment) ensures the preview environment is fully deployed and reachable before QA Wolf is notified. +- A prior job (such as `wait-for-preview-environment`) ensures the preview environment is fully deployed and reachable before QA Wolf is notified. -When the pull request enters the merge queue, this workflow sends the preview URL to QA Wolf, which starts a test run against that environment. - -## Require QA Wolf tests before merge +### Require QA Wolf tests before merge - - Open your **GitHub repository** in the browser. - - - Click the **Settings** tab at the top of the repository (next to _Code, Issues, Pull requests_). - In the left sidebar, do one of the following: - Click **Rules** (if your repo uses GitHub's new rulesets), or - Click **Branches** (for classic branch protection rules). - - Create or edit the rule that applies to your main branch. - - - Enable **Require status checks to pass**. - - - Add **QA Wolf Test Results** to the list of required checks. - - - Save the rule. - ## Verify your PR testing setup @@ -136,10 +134,6 @@ When the pull request enters the merge queue, this workflow sends the preview UR In the **QA Wolf app**, go to the **Runs** tab and verify that a new run starts for the preview environment associated with the pull request. - Back in **GitHub**, watch the **QA Wolf Test Results** check update from its placeholder state to the final pass or fail result. - - The merge completes only after this check finishes successfully. + Back in **GitHub**, watch the **QA Wolf Test Results** check update from its placeholder state to the final pass or fail result. The merge completes only after this check finishes successfully. - - -\\ \ No newline at end of file + \ No newline at end of file diff --git a/qawolf/PR-testing-for-GitLab-Integrations-2db5b2a994fb8075b13dd9d5c2e6be8c.mdx b/qawolf/PR-testing-for-GitLab-Integrations-2db5b2a994fb8075b13dd9d5c2e6be8c.mdx index fe96fdc..5cb0a31 100644 --- a/qawolf/PR-testing-for-GitLab-Integrations-2db5b2a994fb8075b13dd9d5c2e6be8c.mdx +++ b/qawolf/PR-testing-for-GitLab-Integrations-2db5b2a994fb8075b13dd9d5c2e6be8c.mdx @@ -1,87 +1,102 @@ --- -title: "How to enable PR testing for GitLab Integrations" +title: "Setting up PR testing" description: "Run QA Wolf tests against preview environments on merge requests and report results back to GitLab." sidebarTitle: "GitLab" --- -## What QA Wolf configures + + This guide is for QA Wolf full service customers. If you're unsure whether this applies to you, contact your QA Wolf team. + -PR testing runs QA Wolf tests against preview environments before changes are merged. Tests run when a merge request pipeline fires, and results are reported back to GitLab as a required pipeline status. +## Configure basic PR testing - - Your GitLab integration and deploy notifications must already be set up. - - -PR testing won't work until QA Wolf configures your workspace to support it. QA Wolf will: - -- Set up a trigger that matches your preview deployments -- Provide the deployment\_type value to use for preview tests -- Confirm how QA Wolf should identify the merge request (for example, by branch name or merge request ID) -- Confirm the preview URL variable name used in your pipeline - -## Before you begin - - - Each merge request must deploy a preview environment that QA Wolf can reach. Your pipeline should provide a stable preview URL for each merge request and include a step that waits until the preview is live before tests start. - - -Your GitLab pipeline must deploy a preview environment during merge request pipelines (often using Review Apps). The preview URL must be available in the job that triggers the QA Wolf notification. - -## Trigger QA Wolf tests from merge request pipelines - -Add a job that notifies QA Wolf after the preview environment is deployed and reachable. Add a `notify_qawolf` job to your `.gitlab-ci.yml` that runs only for merge request pipelines. - -```yaml expandable -stages: - - deploy - - test - -notify_qawolf: - stage: test - image: curlimages/curl:latest - rules: - - if: $CI_PIPELINE_SOURCE == "merge_request_event" - needs: - - deploy_preview - script: - - | - curl -X POST "https://app.qawolf.com/api/webhooks/deploy_success" \ - -H "Authorization: Bearer $QAWOLF_API_KEY" \ - -H "Content-Type: application/json" \ - -d "{ - \"branch\": \"$CI_COMMIT_REF_NAME\", - \"deployment_type\": \"provided-by-qawolf\", - \"deployment_url\": \"$PREVIEW_URL\", - \"sha\": \"$CI_COMMIT_SHA\", - \"hosting_service\": \"GitLab\" - }" -``` - -### What this job assumes - -- Your pipeline already creates a preview environment for each merge request. -- The preview environment URL is available as `PREVIEW_URL` when this job runs. -- A prior job (such as deploy\_preview) deploys the preview environment and verifies it is reachable before notifying QA Wolf. + + + Configure the GitLab integration in your team's settings page. + + + Choose one of the following methods to tell QA Wolf when a new preview environment is ready to be tested. + + **GitLab CI job** + + Add a `notify_qawolf` job to your `.gitlab-ci.yml` that runs only for merge request pipelines, after the preview environment is deployed and reachable. + + ```yaml expandable + stages: + - deploy + - test + + notify_qawolf: + stage: test + image: node:latest + rules: + - if: $CI_PIPELINE_SOURCE == "merge_request_event" + needs: + - deploy_preview + script: + - npm install @qawolf/ci-sdk + - | + node -e " + const { makeQaWolfSdk } = require('@qawolf/ci-sdk'); + const { attemptNotifyDeploy } = makeQaWolfSdk({ apiKey: process.env.QAWOLF_API_KEY }); + attemptNotifyDeploy({ + sha: process.env.CI_COMMIT_SHA, + branch: process.env.CI_COMMIT_REF_NAME, + deploymentType: 'provided-by-qawolf', + deploymentUrl: process.env.PREVIEW_URL, + hostingService: 'GitLab', + }).then(result => { + if (result.outcome !== 'success') process.exit(1); + }); + " + ``` + + **curl** + + If Node isn't available in your pipeline, you can notify QA Wolf using a raw HTTP call instead. + + ```yaml expandable + notify_qawolf: + stage: test + image: curlimages/curl:latest + rules: + - if: $CI_PIPELINE_SOURCE == "merge_request_event" + needs: + - deploy_preview + script: + - | + curl -X POST "https://app.qawolf.com/api/webhooks/deploy_success" \ + -H "Authorization: Bearer $QAWOLF_API_KEY" \ + -H "Content-Type: application/json" \ + -d "{ + \"branch\": \"$CI_COMMIT_REF_NAME\", + \"deployment_type\": \"provided-by-qawolf\", + \"deployment_url\": \"$PREVIEW_URL\", + \"sha\": \"$CI_COMMIT_SHA\", + \"hosting_service\": \"GitLab\" + }" + ``` + + ### What this job assumes + + - Your pipeline already creates a preview environment for each merge request. + - The preview environment URL is available as `PREVIEW_URL` when this job runs. + - A prior job (such as `deploy_preview`) deploys the preview environment and verifies it is reachable before notifying QA Wolf. + + + QA Wolf will configure a trigger that matches your preview deployments. Reach out to your QA Wolf team to get this set up. + + -When the merge request pipeline runs, this job sends the preview URL to QA Wolf, which starts a test run against that environment. +--- -## Require QA Wolf tests before merge +## Advanced: Require QA Wolf tests before merge -In GitLab, merge requests are typically blocked unless pipelines pass. + + In GitLab, merge requests are typically blocked unless pipelines pass. Complete this section if you want to enforce QA Wolf results as a required check. + - - Open your GitLab project. - - - Go to **Settings → Merge requests**. - - - Enable **Pipelines must succeed** (wording varies by GitLab version). - - - Save the setting. - If your project uses approval rules or protected branches, ensure the merge request pipeline is the one required to pass. @@ -96,7 +111,7 @@ If your project uses approval rules or protected branches, ensure the merge requ In GitLab, check the merge request's environment or the Review App link, and confirm that the preview URL is reachable. - In the QA Wolf app, go to the **Runs** tab and verify that a new run starts for the preview environment associated with the merge request. + In the **QA Wolf app**, go to the **Runs** tab and verify that a new run starts for the preview environment associated with the merge request. In GitLab, confirm the pipeline reflects the final result and that the merge request cannot be merged unless the pipeline succeeds.