Skip to content

Move PR testing to full service #155

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
John Gluck (JohnGluck) wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Move PR testing to full service #155

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
432ed4d
Updated mintlify pages
JohnGluck Jun 2, 2026
9a7802d
Updated mintlify pages
JohnGluck Jun 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions docs.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -413,14 +413,6 @@
"qawolf/other-ci-webhook"
]
},
{
"group": "PR Testing",
"icon": "git-merge",
"pages": [
"qawolf/PR-testing-for-GitHub-Integrations-2db5b2a994fb80a0aa94f3297d1f60c3",
"qawolf/PR-testing-for-GitLab-Integrations-2db5b2a994fb8075b13dd9d5c2e6be8c"
]
},
{
"group": "Mobile build testing",
"icon": "smartphone",
Expand Down Expand Up @@ -494,7 +486,15 @@
"icon": "plug",
"pages": [
"qawolf/Test-environments-2dc5b2a994fb800e8ce4fe36ee34764a",
"qawolf/Integrating-with-microsoft-teams"
"qawolf/Integrating-with-microsoft-teams",
{
"group": "PR Testing",
"icon": "git-merge",
"pages": [
"qawolf/PR-testing-for-GitHub-Integrations-2db5b2a994fb80a0aa94f3297d1f60c3",
"qawolf/PR-testing-for-GitLab-Integrations-2db5b2a994fb8075b13dd9d5c2e6be8c"
]
}
]
}
]
Expand Down
4 changes: 0 additions & 4 deletions qawolf/CI-CD-Overview.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,6 @@ description: "QA Wolf integrates with your CI pipeline to run tests automaticall
Run tests when a new build is deployed to an environment.
</Card>

<Card title="PR testing" icon="git-merge" href="/qawolf/PR-testing-for-GitHub-Integrations-2db5b2a994fb80a0aa94f3297d1f60c3">
Run tests automatically when a pull request is opened or updated.
</Card>

<Card title="Mobile build testing" icon="smartphone" href="/qawolf/Fastlane-2db5b2a994fb802186a3d88573899c78">
Run tests against a new mobile build artifact from your CI pipeline.
</Card>
Expand Down
82 changes: 38 additions & 44 deletions qawolf/PR-testing-for-GitHub-Integrations-2db5b2a994fb80a0aa94f3297d1f60c3.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,30 +1,48 @@
---
title: "How to enable PR testing for GitHub Integrations"
title: "Setting up PR testing"
description: "Run QA Wolf tests against preview environments on pull requests and report results as GitHub status checks."
sidebarTitle: "GitHub"
---

## What QA Wolf configures
<Info>
This guide is for QA Wolf full service customers. If you're unsure whether this applies to you, contact your QA Wolf team.
</Info>

PR testing runs QA Wolf tests against preview environments before changes are merged. Tests run when a pull request enters the merge queue, and results are reported back to GitHub as a required status check.
## Configure basic PR testing

<Tip>
Your [GitHub integration](/qawolf/GitHub-GitHub-Actions-2a15b2a994fb800699f6dc99584915d0) and deploy notifications must already be set up.
</Tip>
<Steps>
<Step title="Configure GitHub integration">
Configure the GitHub integration in your team's settings page.
Copy link
Copy Markdown
Contributor

@aldeed Eric Dobbertin (aldeed) Jun 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"team" -> "workspace"

Maybe:

Go to Workspace Settings > Integrations to enable and authorize the GitHub integration.

</Step>
<Step title="Notify QA Wolf of preview deployments">
Choose one of the following methods to tell QA Wolf when a new preview environment is ready to be tested.

**GitHub Deployments**

If your team uses a hosting platform that integrates with GitHub — like Vercel — your deployments are already being reported to GitHub via the [GitHub Deployments API](https://docs.github.com/en/rest/deployments/deployments). QA Wolf listens for those deployment records and automatically acts when one is marked as successful. No additional configuration is required.

**GitHub Action**

Configure the [Notify QA Wolf on Deploy](https://github.com/marketplace/actions/notify-qa-wolf-on-deploy) GitHub Action to tell us when a new preview environment is ready to be tested.

**SDK**

PR testing won't work until QA Wolf configures your workspace to support it. QA Wolf will:
If GitHub Actions aren't an option, the `@qawolf/ci-sdk` npm package offers a flexible alternative. See the [npm package documentation](https://www.npmjs.com/package/@qawolf/ci-sdk) for setup details.
</Step>
<Step title="Ask your QA Wolf team to set up triggers">
QA Wolf will configure a trigger that matches your preview deployments. Reach out to your QA Wolf team to get this set up.
</Step>
</Steps>

- Set up a trigger that matches your preview deployments
- Provide the deployment-type value to use for preview tests
- Confirm the preview URL variable and the status check name (for example, **QA Wolf Test Results**)
---

## Before you begin
## Advanced: Merge queue setup

<Check>
Each pull request must deploy a preview environment that QA Wolf can reach. Your CI should provide a stable preview URL per PR and a step that waits until the preview is live before tests start.
</Check>
<Info>
This section covers an optional flow for teams using GitHub's merge queue feature. Most teams don't need this.
Copy link
Copy Markdown
Contributor

@aldeed Eric Dobbertin (aldeed) Jun 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of "Most teams don't need this.", you could explain why they might? Using a merge queue is common, but I guess we are saying that most customer choose not to run QA Wolf tests in it. Why do some? I don't actually know the answer, but I'm guessing Eric E might.

</Info>

## Add a default passing PR check
### Add a default passing PR check

Add a GitHub Actions workflow that creates a passing **QA Wolf Test Results** check when a PR is opened or updated. This prevents pull requests from being blocked before the merge queue run happens.

Expand Down Expand Up @@ -60,7 +78,7 @@ jobs:
});
```

## Trigger QA Wolf tests in the merge queue
### Trigger QA Wolf tests in the merge queue

Add a GitHub Actions workflow that notifies QA Wolf only during merge queue execution.

Expand Down Expand Up @@ -90,37 +108,17 @@ jobs:

- Your CI pipeline already creates a preview environment per pull request.
- The preview environment URL is available as `PREVIEW_URL` when this job runs.
- A prior job (such as wait-for-preview-environment) ensures the preview environment is fully deployed and reachable before QA Wolf is notified.
- A prior job (such as `wait-for-preview-environment`) ensures the preview environment is fully deployed and reachable before QA Wolf is notified.

When the pull request enters the merge queue, this workflow sends the preview URL to QA Wolf, which starts a test run against that environment.

## Require QA Wolf tests before merge
### Require QA Wolf tests before merge

<Steps>
<Step>
Open your **GitHub repository** in the browser.
</Step>
<Step>
Click the **Settings** tab at the top of the repository (next to _Code, Issues, Pull requests_).
</Step>
<Step>
In the left sidebar, do one of the following:

- Click **Rules** (if your repo uses GitHub's new rulesets), or
- Click **Branches** (for classic branch protection rules).
</Step>
<Step>
Create or edit the rule that applies to your main branch.
</Step>
<Step>
Enable **Require status checks to pass**.
</Step>
<Step>
Add **QA Wolf Test Results** to the list of required checks.
</Step>
<Step>
Save the rule.
</Step>
</Steps>

## Verify your PR testing setup
Expand All @@ -136,10 +134,6 @@ When the pull request enters the merge queue, this workflow sends the preview UR
In the **QA Wolf app**, go to the **Runs** tab and verify that a new run starts for the preview environment associated with the pull request.
</Step>
<Step title="Confirm the check updates before merge.">
Back in **GitHub**, watch the **QA Wolf Test Results** check update from its placeholder state to the final pass or fail result.

The merge completes only after this check finishes successfully.
Back in **GitHub**, watch the **QA Wolf Test Results** check update from its placeholder state to the final pass or fail result. The merge completes only after this check finishes successfully.
</Step>
</Steps>

\\
</Steps>
161 changes: 88 additions & 73 deletions qawolf/PR-testing-for-GitLab-Integrations-2db5b2a994fb8075b13dd9d5c2e6be8c.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,87 +1,102 @@
---
title: "How to enable PR testing for GitLab Integrations"
title: "Setting up PR testing"
description: "Run QA Wolf tests against preview environments on merge requests and report results back to GitLab."
sidebarTitle: "GitLab"
---

## What QA Wolf configures
<Info>
This guide is for QA Wolf full service customers. If you're unsure whether this applies to you, contact your QA Wolf team.
</Info>

PR testing runs QA Wolf tests against preview environments before changes are merged. Tests run when a merge request pipeline fires, and results are reported back to GitLab as a required pipeline status.
## Configure basic PR testing

<Tip>
Your GitLab integration and deploy notifications must already be set up.
</Tip>

PR testing won't work until QA Wolf configures your workspace to support it. QA Wolf will:

- Set up a trigger that matches your preview deployments
- Provide the deployment\_type value to use for preview tests
- Confirm how QA Wolf should identify the merge request (for example, by branch name or merge request ID)
- Confirm the preview URL variable name used in your pipeline

## Before you begin

<Check>
Each merge request must deploy a preview environment that QA Wolf can reach. Your pipeline should provide a stable preview URL for each merge request and include a step that waits until the preview is live before tests start.
</Check>

Your GitLab pipeline must deploy a preview environment during merge request pipelines (often using Review Apps). The preview URL must be available in the job that triggers the QA Wolf notification.

## Trigger QA Wolf tests from merge request pipelines

Add a job that notifies QA Wolf after the preview environment is deployed and reachable. Add a `notify_qawolf` job to your `.gitlab-ci.yml` that runs only for merge request pipelines.

```yaml expandable
stages:
- deploy
- test

notify_qawolf:
stage: test
image: curlimages/curl:latest
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
needs:
- deploy_preview
script:
- |
curl -X POST "https://app.qawolf.com/api/webhooks/deploy_success" \
-H "Authorization: Bearer $QAWOLF_API_KEY" \
-H "Content-Type: application/json" \
-d "{
\"branch\": \"$CI_COMMIT_REF_NAME\",
\"deployment_type\": \"provided-by-qawolf\",
\"deployment_url\": \"$PREVIEW_URL\",
\"sha\": \"$CI_COMMIT_SHA\",
\"hosting_service\": \"GitLab\"
}"
```

### What this job assumes

- Your pipeline already creates a preview environment for each merge request.
- The preview environment URL is available as `PREVIEW_URL` when this job runs.
- A prior job (such as deploy\_preview) deploys the preview environment and verifies it is reachable before notifying QA Wolf.
<Steps>
<Step title="Configure GitLab integration">
Configure the GitLab integration in your team's settings page.
</Step>
<Step title="Notify QA Wolf of preview deployments">
Choose one of the following methods to tell QA Wolf when a new preview environment is ready to be tested.

**GitLab CI job**

Add a `notify_qawolf` job to your `.gitlab-ci.yml` that runs only for merge request pipelines, after the preview environment is deployed and reachable.

```yaml expandable
stages:
- deploy
- test

notify_qawolf:
stage: test
image: node:latest
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
needs:
- deploy_preview
script:
- npm install @qawolf/ci-sdk
- |
node -e "
const { makeQaWolfSdk } = require('@qawolf/ci-sdk');
const { attemptNotifyDeploy } = makeQaWolfSdk({ apiKey: process.env.QAWOLF_API_KEY });
attemptNotifyDeploy({
sha: process.env.CI_COMMIT_SHA,
branch: process.env.CI_COMMIT_REF_NAME,
deploymentType: 'provided-by-qawolf',
deploymentUrl: process.env.PREVIEW_URL,
hostingService: 'GitLab',
}).then(result => {
if (result.outcome !== 'success') process.exit(1);
});
"
```

**curl**

If Node isn't available in your pipeline, you can notify QA Wolf using a raw HTTP call instead.

```yaml expandable
notify_qawolf:
stage: test
image: curlimages/curl:latest
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
needs:
- deploy_preview
script:
- |
curl -X POST "https://app.qawolf.com/api/webhooks/deploy_success" \
-H "Authorization: Bearer $QAWOLF_API_KEY" \
-H "Content-Type: application/json" \
-d "{
\"branch\": \"$CI_COMMIT_REF_NAME\",
\"deployment_type\": \"provided-by-qawolf\",
\"deployment_url\": \"$PREVIEW_URL\",
\"sha\": \"$CI_COMMIT_SHA\",
\"hosting_service\": \"GitLab\"
}"
```

### What this job assumes

- Your pipeline already creates a preview environment for each merge request.
- The preview environment URL is available as `PREVIEW_URL` when this job runs.
- A prior job (such as `deploy_preview`) deploys the preview environment and verifies it is reachable before notifying QA Wolf.
</Step>
<Step title="Ask your QA Wolf team to set up triggers">
QA Wolf will configure a trigger that matches your preview deployments. Reach out to your QA Wolf team to get this set up.
</Step>
</Steps>

When the merge request pipeline runs, this job sends the preview URL to QA Wolf, which starts a test run against that environment.
---

## Require QA Wolf tests before merge
## Advanced: Require QA Wolf tests before merge

In GitLab, merge requests are typically blocked unless pipelines pass.
<Info>
In GitLab, merge requests are typically blocked unless pipelines pass. Complete this section if you want to enforce QA Wolf results as a required check.
</Info>

<Steps>
<Step>
Open your GitLab project.
</Step>
<Step>
Go to **Settings → Merge requests**.
</Step>
<Step>
Enable **Pipelines must succeed** (wording varies by GitLab version).
</Step>
<Step>
Save the setting.
</Step>
</Steps>

If your project uses approval rules or protected branches, ensure the merge request pipeline is the one required to pass.
Expand All @@ -96,7 +111,7 @@ If your project uses approval rules or protected branches, ensure the merge requ
In GitLab, check the merge request's environment or the Review App link, and confirm that the preview URL is reachable.
</Step>
<Step title="Confirm QA Wolf starts a test run.">
In the QA Wolf app, go to the **Runs** tab and verify that a new run starts for the preview environment associated with the merge request.
In the **QA Wolf app**, go to the **Runs** tab and verify that a new run starts for the preview environment associated with the merge request.
</Step>
<Step title="Confirm the merge request is blocked on failure.">
In GitLab, confirm the pipeline reflects the final result and that the merge request cannot be merged unless the pipeline succeeds.
Expand Down