Skip to content

Commit 3b66f3f

Browse files
Skn0ttCopilot
andcommitted
Build driver from source instead of downloading from CDN
Stop downloading the prebuilt Playwright driver bundle from cdn.playwright.dev. Instead, clone microsoft/playwright at the tag matching driver_version and build the per-platform bundles from source via upstream's utils/build/build-playwright-driver.sh. - scripts/build_driver.sh: portable bash script (shareable across the language forks) that clones, builds (npm ci && npm run build), runs the upstream driver build, and stages all 6 platform zips into driver/. - setup.py: ensure_driver_bundle() shells out to the script when a bundle is missing; stale extract-dir cleanup before extraction. - CI/release: add Node 24 (matching the bundled runtime major) wherever wheels are built — ci.yml, publish_docker.yml, test_docker.yml host, Azure NodeTool; meta.yaml conda host gains git + nodejs. - Docs: ROLLING.md, CONTRIBUTING.md, CLAUDE.md, playwright-roll SKILL.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 7a387f2 commit 3b66f3f

11 files changed

Lines changed: 243 additions & 35 deletions

File tree

.azure-pipelines/publish.yml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,10 @@ extends:
4040
inputs:
4141
versionSpec: '3.9'
4242
displayName: 'Use Python'
43+
- task: NodeTool@0
44+
inputs:
45+
versionSpec: '24.x'
46+
displayName: 'Use Node.js'
4347
- script: |
4448
python -m pip install --upgrade pip
4549
pip install -r local-requirements.txt

.claude/skills/playwright-roll/SKILL.md

Lines changed: 17 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ description: Roll Playwright Python to a new driver version. Walks the upstream
55

66
# Rolling Playwright Python
77

8-
The goal of a roll is to move `driver_version` in `setup.py` to a new release, port every public API change introduced upstream during that interval, and suppress the rest, so that `./scripts/update_api.sh` runs clean and the test suite still passes.
8+
The goal of a roll is to move the driver pin in `DRIVER_SHA` to a new release, port every public API change introduced upstream during that interval, and suppress the rest, so that `./scripts/update_api.sh` runs clean and the test suite still passes.
99

1010
The previous human-facing summary lives in `../../../ROLLING.md`. This skill is the operational playbook — read it end to end before starting.
1111

@@ -15,7 +15,7 @@ The Python port is hand-written code in `playwright/_impl/`, plus a generator (`
1515

1616
1. introspects the Python `_impl` classes via `inspect`,
1717
2. emits typed wrapper classes into `playwright/{async,sync}_api/_generated.py`, and
18-
3. diffs the introspected surface against `playwright/driver/package/api.json` (downloaded inside the new driver wheel).
18+
3. diffs the introspected surface against `playwright/driver/package/api.json` (built into the new driver from source).
1919

2020
Anything in `api.json` that is missing or differently typed in `_impl/` causes generation to fail. Three resolutions:
2121

@@ -52,18 +52,27 @@ There is sometimes no `vX.Y.0` tag for the latest release (the bots cut release
5252
- If `python3-venv` is missing system-wide, use `uv venv env` instead, then `uv pip install --python env/bin/python --upgrade pip`. Don't try to `apt install` — sudo is denied in the harness.
5353
- Always activate the venv before any `pip`, `pytest`, `mypy`, or `pre-commit` invocation.
5454

55-
### 2. Bump the driver and download it
55+
### 2. Bump the driver and build it from source
5656

5757
```sh
58-
# Edit setup.py
59-
driver_version = "<new>" # e.g. "1.59.1"
58+
# Edit DRIVER_SHA (repo root): replace with the microsoft/playwright commit SHA
59+
# for the new release, e.g. the commit that v<new> points at.
60+
# 87bb9ddbd78f329df18c2b24847bc9409240cd07
61+
# Update the "# microsoft/playwright @ v<new>" comment in scripts/build_driver.sh too.
6062

6163
source env/bin/activate
62-
python -m build --wheel # downloads the new driver from cdn.playwright.dev
64+
python -m build --wheel # clones microsoft/playwright @ DRIVER_SHA and builds the driver from source
6365
playwright install chromium # NOT --with-deps; sudo is denied
6466
```
6567

66-
The wheel build prints `Fetching https://cdn.playwright.dev/builds/driver/playwright-<new>-linux.zip` and unpacks the driver under `playwright/driver/package/`. From this point, `playwright/driver/package/api.json` reflects the new release.
68+
The wheel build clones `microsoft/playwright` at the commit in `DRIVER_SHA`
69+
into `driver/playwright-src`, runs `npm ci && npm run build`, and runs upstream's
70+
`utils/build/build-playwright-driver.sh` to produce the per-platform driver
71+
bundles (`driver/playwright-<sha>-*.zip`), then unpacks the driver under
72+
`playwright/driver/package/`. From this point,
73+
`playwright/driver/package/api.json` reflects the new release. This requires
74+
**Node.js, npm, git and bash** on PATH; the first build is slow (full upstream
75+
build + per-platform Node downloads).
6776

6877
### 3. Identify the commit range
6978

@@ -240,7 +249,7 @@ For each PORT, add one async test and a matching sync test. Conventions:
240249

241250
### 7. Update existing high-touch artifacts
242251

243-
- `setup.py`: already done in step 2.
252+
- `DRIVER_SHA` (and the version comment in `scripts/build_driver.sh`): already done in step 2.
244253
- `README.md`: gets the chromium/firefox/webkit version table updated automatically by `scripts/update_versions.py` (called from `update_api.sh`). Don't edit by hand.
245254
- The "Backport changes" tracking issue on GitHub (filed by `microsoft-playwright-automation`) is the *intent* tracker, but it's frequently out of sync with what's actually been ported. Treat it as a starting point, not the source of truth — the `docs/src/api/` commit walk is authoritative.
246255

.github/workflows/ci.yml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,10 @@ jobs:
2626
uses: actions/setup-python@v6
2727
with:
2828
python-version: "3.10"
29+
- name: Set up Node.js
30+
uses: actions/setup-node@v4
31+
with:
32+
node-version: '24'
2933
- name: Install dependencies & browsers
3034
run: |
3135
python -m pip install --upgrade pip
@@ -96,6 +100,10 @@ jobs:
96100
uses: actions/setup-python@v6
97101
with:
98102
python-version: ${{ matrix.python-version }}
103+
- name: Set up Node.js
104+
uses: actions/setup-node@v4
105+
with:
106+
node-version: '24'
99107
- name: Install dependencies & browsers
100108
run: |
101109
python -m pip install --upgrade pip
@@ -143,6 +151,10 @@ jobs:
143151
uses: actions/setup-python@v6
144152
with:
145153
python-version: "3.10"
154+
- name: Set up Node.js
155+
uses: actions/setup-node@v4
156+
with:
157+
node-version: '24'
146158
- name: Install dependencies & browsers
147159
run: |
148160
python -m pip install --upgrade pip

.github/workflows/publish_docker.yml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,10 @@ jobs:
2828
uses: actions/setup-python@v6
2929
with:
3030
python-version: "3.10"
31+
- name: Set up Node.js
32+
uses: actions/setup-node@v4
33+
with:
34+
node-version: '24'
3135
- name: Set up Docker QEMU for arm64 docker builds
3236
uses: docker/setup-qemu-action@v4
3337
with:

.github/workflows/test_docker.yml

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,10 @@ jobs:
3535
uses: actions/setup-python@v6
3636
with:
3737
python-version: "3.10"
38+
- name: Set up Node.js
39+
uses: actions/setup-node@v4
40+
with:
41+
node-version: '24'
3842
- name: Install dependencies
3943
run: |
4044
python -m pip install --upgrade pip
@@ -53,6 +57,10 @@ jobs:
5357
docker exec "${CONTAINER_ID}" pip install -r local-requirements.txt
5458
docker exec "${CONTAINER_ID}" pip install -r requirements.txt
5559
docker exec "${CONTAINER_ID}" pip install -e .
60+
# build.sh (above) already built the driver bundles into driver/ on the
61+
# host. The repo is bind-mounted into the container, so this in-container
62+
# wheel build reuses those bundles (setup.py skips the source build when
63+
# the zip already exists) and therefore needs no Node.js in the image.
5664
docker exec "${CONTAINER_ID}" python -m build --wheel
5765
docker exec "${CONTAINER_ID}" xvfb-run pytest tests/sync/
5866
docker exec "${CONTAINER_ID}" xvfb-run pytest tests/async/

CLAUDE.md

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -13,19 +13,20 @@ Python bindings for [Playwright](https://playwright.dev). The Python client talk
1313
- `scripts/generate_api.py`, `scripts/generate_async_api.py`, `scripts/generate_sync_api.py`, `scripts/documentation_provider.py` — codegen and validation. They diff the Python implementation against the driver's `playwright/driver/package/api.json` and abort if either side is out of sync.
1414
- `scripts/expected_api_mismatch.txt` — explicit allowlist of "documented in JS, not in Python" or "named differently in Python" gaps. Lines that no longer apply must be removed.
1515
- `tests/async/`, `tests/sync/` — pytest suites. Most new tests are added to the async file with a sync mirror.
16-
- `setup.py``driver_version = "X.Y.Z"` is the source of truth for which driver build is downloaded from `cdn.playwright.dev`.
16+
- `DRIVER_SHA` — the single source of truth for which Playwright commit the driver is built from (one line, the 40-char `microsoft/playwright` commit SHA). Read by `setup.py`, `scripts/build_driver.sh`, and CI. The wheel build clones `microsoft/playwright` at this commit and builds the driver from source (via `scripts/build_driver.sh` + upstream's `utils/build/build-playwright-driver.sh`). The SHA is baked into the staged bundle filenames (`driver/playwright-<sha>-<suffix>.zip`), so it doubles as the build cache key.
17+
- `scripts/build_driver.sh` — clones and builds the upstream driver bundles into `driver/`. A portable bash script (shareable with the other language forks) that needs Node.js, npm, git and bash; invoked from `setup.py`'s `bdist_wheel`. Reads the pin from `DRIVER_SHA`; takes no arguments.
1718
- `ROLLING.md`, `CONTRIBUTING.md` — human-facing setup and roll docs.
1819

1920
## Setup
2021

21-
`CONTRIBUTING.md` has the full sequence. The short version:
22+
`CONTRIBUTING.md` has the full sequence. The short version (needs Node.js, npm, git and bash for the driver build):
2223

2324
```sh
2425
python3 -m venv env && source env/bin/activate
2526
pip install --upgrade pip
2627
pip install -r local-requirements.txt
2728
pip install -e .
28-
python -m build --wheel # downloads the driver listed in setup.py
29+
python -m build --wheel # clones microsoft/playwright @ DRIVER_SHA and builds the driver from source
2930
pre-commit install
3031
```
3132

CONTRIBUTING.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,12 @@ pip install -r local-requirements.txt
2121

2222
Build and install drivers:
2323

24+
The driver is built from upstream `microsoft/playwright` source, so building a
25+
wheel requires **Node.js (with npm), git and bash** on your PATH. The commit to
26+
build from is pinned in the `DRIVER_SHA` file. The first
27+
`python -m build --wheel` clones that commit and runs its full
28+
build, which is slow.
29+
2430
```sh
2531
pip install -e .
2632
python -m build --wheel

DRIVER_SHA

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
87bb9ddbd78f329df18c2b24847bc9409240cd07

ROLLING.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,8 @@ pip install -r local-requirements.txt
1111
pre-commit install
1212
pip install -e .
1313
```
14-
* change driver version in `setup.py`
15-
* download new driver: `python -m build --wheel`
14+
* change the driver pin in `DRIVER_SHA` (the `microsoft/playwright` commit SHA to build from)
15+
* build the new driver from source: `python -m build --wheel` (clones `microsoft/playwright` at that commit and builds it; requires Node.js, npm, git and bash)
1616
* generate API: `./scripts/update_api.sh`
1717
* commit changes & send PR
1818
* wait for bots to pass & merge the PR

scripts/build_driver.sh

Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
#!/usr/bin/env bash
2+
# Copyright (c) Microsoft Corporation.
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
16+
# Build the Playwright driver bundles from upstream source.
17+
#
18+
# This script checks out microsoft/playwright at the commit pinned in the
19+
# DRIVER_SHA file (repo root) and runs upstream's
20+
# utils/build/build-playwright-driver.sh. That script cross-builds the
21+
# per-platform bundles, which this script stages into driver/ as
22+
# playwright-<sha>-<suffix>.zip for setup.py to embed into the platform wheels.
23+
#
24+
# The pin is an immutable commit SHA (tags can be moved upstream) and lives in
25+
# the neutral DRIVER_SHA file so setup.py and CI can read it without parsing
26+
# this script. The SHA is baked into the staged bundle filenames, so the
27+
# filename doubles as the build cache key: a roll changes DRIVER_SHA, which
28+
# changes the filenames and invalidates the cache.
29+
#
30+
# A single host builds all platform bundles at once: the upstream script
31+
# downloads the matching Node.js binary for each target, so the host platform
32+
# does not constrain which bundles can be produced.
33+
#
34+
# This is intentionally a shell script (rather than language-specific code) so
35+
# the same build step can be shared across the Playwright language forks.
36+
#
37+
# Usage: scripts/build_driver.sh (reads the pin from DRIVER_SHA; no arguments)
38+
39+
set -euo pipefail
40+
41+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
42+
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
43+
DRIVER_DIR="$REPO_ROOT/driver"
44+
SOURCE_DIR="$DRIVER_DIR/playwright-src"
45+
PLAYWRIGHT_REPO="https://github.com/microsoft/playwright"
46+
47+
# The driver pin: an immutable commit in microsoft/playwright.
48+
# microsoft/playwright @ v1.60.0
49+
EXPECTED_SHA="$(tr -d '[:space:]' < "$REPO_ROOT/DRIVER_SHA")"
50+
if [[ -z "$EXPECTED_SHA" ]]; then
51+
echo "DRIVER_SHA is empty or missing at $REPO_ROOT/DRIVER_SHA" >&2
52+
exit 2
53+
fi
54+
55+
# Bundle suffixes produced by utils/build/build-playwright-driver.sh. Keep in
56+
# sync with the "zip_name" values in setup.py.
57+
SUFFIXES=(mac mac-arm64 linux linux-arm64 win32_x64 win32_arm64)
58+
59+
bundles_present() {
60+
local suffix
61+
for suffix in "${SUFFIXES[@]}"; do
62+
[[ -f "$DRIVER_DIR/playwright-$EXPECTED_SHA-$suffix.zip" ]] || return 1
63+
done
64+
return 0
65+
}
66+
67+
require_tools() {
68+
local missing=()
69+
local tool
70+
for tool in git node npm bash; do
71+
if ! command -v "$tool" >/dev/null 2>&1; then
72+
missing+=("$tool")
73+
fi
74+
done
75+
if [[ ${#missing[@]} -gt 0 ]]; then
76+
echo "Building the Playwright driver from source requires the following tools," >&2
77+
echo "which were not found on PATH: ${missing[*]}." >&2
78+
echo "Install Node.js (with npm), git and bash, then retry. On Windows, run the" >&2
79+
echo "build from a bash shell (e.g. Git Bash)." >&2
80+
exit 1
81+
fi
82+
}
83+
84+
checked_out_sha() {
85+
if [[ -d "$SOURCE_DIR/.git" ]]; then
86+
git -C "$SOURCE_DIR" rev-parse HEAD 2>/dev/null || true
87+
fi
88+
}
89+
90+
clone_source() {
91+
# Reuse an existing checkout's git object store across rolls: only initialize
92+
# a fresh repo when one isn't already present, then fetch and check out the
93+
# pinned commit. This avoids re-cloning the repo (and re-running npm ci from
94+
# scratch) every time the pin changes.
95+
if [[ ! -d "$SOURCE_DIR/.git" ]]; then
96+
rm -rf "$SOURCE_DIR"
97+
mkdir -p "$SOURCE_DIR"
98+
git -C "$SOURCE_DIR" init -q
99+
git -C "$SOURCE_DIR" remote add origin "$PLAYWRIGHT_REPO"
100+
fi
101+
if [[ "$(checked_out_sha)" != "$EXPECTED_SHA" ]]; then
102+
echo "Fetching $PLAYWRIGHT_REPO at $EXPECTED_SHA"
103+
# Shallow-fetch a single commit. GitHub allows fetching an arbitrary commit
104+
# by SHA, so a full clone is unnecessary.
105+
git -C "$SOURCE_DIR" fetch --depth 1 origin "$EXPECTED_SHA"
106+
git -C "$SOURCE_DIR" checkout -q --detach FETCH_HEAD
107+
fi
108+
# Make sure we landed on exactly the pinned commit.
109+
if [[ "$(checked_out_sha)" != "$EXPECTED_SHA" ]]; then
110+
echo "Checked out commit '$(checked_out_sha)' but '$EXPECTED_SHA' was requested." >&2
111+
exit 1
112+
fi
113+
}
114+
115+
build_source() {
116+
echo "Installing Playwright dependencies (npm ci)"
117+
(cd "$SOURCE_DIR" && npm ci)
118+
# Drop build outputs left over from a previously checked-out commit when the
119+
# checkout is reused across rolls (lib/ dirs are gitignored, so switching
120+
# commits doesn't clear them).
121+
echo "Cleaning previous build outputs (npm run clean)"
122+
(cd "$SOURCE_DIR" && npm run clean)
123+
echo "Building Playwright (npm run build)"
124+
(cd "$SOURCE_DIR" && npm run build)
125+
echo "Building driver bundles"
126+
(cd "$SOURCE_DIR" && bash utils/build/build-playwright-driver.sh)
127+
}
128+
129+
copy_bundles() {
130+
local output_dir="$SOURCE_DIR/utils/build/output"
131+
# The output dir also holds build intermediates (downloaded Node.js binaries,
132+
# tgz archives, extracted package dirs), so copy only the bundles. Upstream
133+
# names them playwright-<version>-<suffix>.zip; restage each one with the pin
134+
# SHA in the name so the filename doubles as the build cache key.
135+
local suffix matches
136+
for suffix in "${SUFFIXES[@]}"; do
137+
matches=("$output_dir"/playwright-*-"$suffix".zip)
138+
if [[ ! -f "${matches[0]}" ]]; then
139+
echo "Expected driver bundle for '$suffix' was not produced in $output_dir" >&2
140+
exit 1
141+
fi
142+
cp "${matches[0]}" "$DRIVER_DIR/playwright-$EXPECTED_SHA-$suffix.zip"
143+
done
144+
}
145+
146+
# Fast path: the bundles for this exact pin are already staged, so there is
147+
# nothing to (re)build. This keeps repeat invocations cheap and lets consumers
148+
# that only downloaded the prebuilt bundles skip the build entirely (no Node).
149+
if bundles_present; then
150+
echo "Driver bundles for $EXPECTED_SHA already present in $DRIVER_DIR; skipping build."
151+
exit 0
152+
fi
153+
154+
require_tools
155+
clone_source
156+
build_source
157+
copy_bundles

0 commit comments

Comments
 (0)