Skip to content

Commit e75b62a

Browse files
Skn0ttCopilot
andcommitted
Build driver from source instead of downloading from CDN
Stop downloading the prebuilt Playwright driver bundle from cdn.playwright.dev. Instead, clone microsoft/playwright at the tag matching driver_version and build the per-platform bundles from source via upstream's utils/build/build-playwright-driver.sh. - scripts/build_driver.sh: portable bash script (shareable across the language forks) that clones, builds (npm ci && npm run build), runs the upstream driver build, and stages all 6 platform zips into driver/. - setup.py: ensure_driver_bundle() shells out to the script when a bundle is missing; stale extract-dir cleanup before extraction. - CI/release: add Node 24 (matching the bundled runtime major) wherever wheels are built — ci.yml, publish_docker.yml, test_docker.yml host, Azure NodeTool; meta.yaml conda host gains git + nodejs. - Docs: ROLLING.md, CONTRIBUTING.md, CLAUDE.md, playwright-roll SKILL.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 7a387f2 commit e75b62a

10 files changed

Lines changed: 188 additions & 27 deletions

File tree

.azure-pipelines/publish.yml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,10 @@ extends:
4040
inputs:
4141
versionSpec: '3.9'
4242
displayName: 'Use Python'
43+
- task: NodeTool@0
44+
inputs:
45+
versionSpec: '24.x'
46+
displayName: 'Use Node.js'
4347
- script: |
4448
python -m pip install --upgrade pip
4549
pip install -r local-requirements.txt

.claude/skills/playwright-roll/SKILL.md

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ The Python port is hand-written code in `playwright/_impl/`, plus a generator (`
1515

1616
1. introspects the Python `_impl` classes via `inspect`,
1717
2. emits typed wrapper classes into `playwright/{async,sync}_api/_generated.py`, and
18-
3. diffs the introspected surface against `playwright/driver/package/api.json` (downloaded inside the new driver wheel).
18+
3. diffs the introspected surface against `playwright/driver/package/api.json` (built into the new driver from source).
1919

2020
Anything in `api.json` that is missing or differently typed in `_impl/` causes generation to fail. Three resolutions:
2121

@@ -52,18 +52,25 @@ There is sometimes no `vX.Y.0` tag for the latest release (the bots cut release
5252
- If `python3-venv` is missing system-wide, use `uv venv env` instead, then `uv pip install --python env/bin/python --upgrade pip`. Don't try to `apt install` — sudo is denied in the harness.
5353
- Always activate the venv before any `pip`, `pytest`, `mypy`, or `pre-commit` invocation.
5454

55-
### 2. Bump the driver and download it
55+
### 2. Bump the driver and build it from source
5656

5757
```sh
5858
# Edit setup.py
5959
driver_version = "<new>" # e.g. "1.59.1"
6060

6161
source env/bin/activate
62-
python -m build --wheel # downloads the new driver from cdn.playwright.dev
62+
python -m build --wheel # clones microsoft/playwright @ v<new> and builds the driver from source
6363
playwright install chromium # NOT --with-deps; sudo is denied
6464
```
6565

66-
The wheel build prints `Fetching https://cdn.playwright.dev/builds/driver/playwright-<new>-linux.zip` and unpacks the driver under `playwright/driver/package/`. From this point, `playwright/driver/package/api.json` reflects the new release.
66+
The wheel build clones `microsoft/playwright` at tag `v<new>` into
67+
`driver/playwright-src`, runs `npm ci && npm run build`, and runs upstream's
68+
`utils/build/build-playwright-driver.sh` to produce the per-platform driver
69+
bundles (`driver/playwright-<new>-*.zip`), then unpacks the driver under
70+
`playwright/driver/package/`. From this point,
71+
`playwright/driver/package/api.json` reflects the new release. This requires
72+
**Node.js, npm, git and bash** on PATH; the first build is slow (full upstream
73+
build + per-platform Node downloads).
6774

6875
### 3. Identify the commit range
6976

.github/workflows/ci.yml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,10 @@ jobs:
2626
uses: actions/setup-python@v6
2727
with:
2828
python-version: "3.10"
29+
- name: Set up Node.js
30+
uses: actions/setup-node@v4
31+
with:
32+
node-version: '24'
2933
- name: Install dependencies & browsers
3034
run: |
3135
python -m pip install --upgrade pip
@@ -96,6 +100,10 @@ jobs:
96100
uses: actions/setup-python@v6
97101
with:
98102
python-version: ${{ matrix.python-version }}
103+
- name: Set up Node.js
104+
uses: actions/setup-node@v4
105+
with:
106+
node-version: '24'
99107
- name: Install dependencies & browsers
100108
run: |
101109
python -m pip install --upgrade pip
@@ -143,6 +151,10 @@ jobs:
143151
uses: actions/setup-python@v6
144152
with:
145153
python-version: "3.10"
154+
- name: Set up Node.js
155+
uses: actions/setup-node@v4
156+
with:
157+
node-version: '24'
146158
- name: Install dependencies & browsers
147159
run: |
148160
python -m pip install --upgrade pip

.github/workflows/publish_docker.yml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,10 @@ jobs:
2828
uses: actions/setup-python@v6
2929
with:
3030
python-version: "3.10"
31+
- name: Set up Node.js
32+
uses: actions/setup-node@v4
33+
with:
34+
node-version: '24'
3135
- name: Set up Docker QEMU for arm64 docker builds
3236
uses: docker/setup-qemu-action@v4
3337
with:

.github/workflows/test_docker.yml

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,10 @@ jobs:
3535
uses: actions/setup-python@v6
3636
with:
3737
python-version: "3.10"
38+
- name: Set up Node.js
39+
uses: actions/setup-node@v4
40+
with:
41+
node-version: '24'
3842
- name: Install dependencies
3943
run: |
4044
python -m pip install --upgrade pip
@@ -53,6 +57,10 @@ jobs:
5357
docker exec "${CONTAINER_ID}" pip install -r local-requirements.txt
5458
docker exec "${CONTAINER_ID}" pip install -r requirements.txt
5559
docker exec "${CONTAINER_ID}" pip install -e .
60+
# build.sh (above) already built the driver bundles into driver/ on the
61+
# host. The repo is bind-mounted into the container, so this in-container
62+
# wheel build reuses those bundles (setup.py skips the source build when
63+
# the zip already exists) and therefore needs no Node.js in the image.
5664
docker exec "${CONTAINER_ID}" python -m build --wheel
5765
docker exec "${CONTAINER_ID}" xvfb-run pytest tests/sync/
5866
docker exec "${CONTAINER_ID}" xvfb-run pytest tests/async/

CLAUDE.md

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -13,19 +13,20 @@ Python bindings for [Playwright](https://playwright.dev). The Python client talk
1313
- `scripts/generate_api.py`, `scripts/generate_async_api.py`, `scripts/generate_sync_api.py`, `scripts/documentation_provider.py` — codegen and validation. They diff the Python implementation against the driver's `playwright/driver/package/api.json` and abort if either side is out of sync.
1414
- `scripts/expected_api_mismatch.txt` — explicit allowlist of "documented in JS, not in Python" or "named differently in Python" gaps. Lines that no longer apply must be removed.
1515
- `tests/async/`, `tests/sync/` — pytest suites. Most new tests are added to the async file with a sync mirror.
16-
- `setup.py``driver_version = "X.Y.Z"` is the source of truth for which driver build is downloaded from `cdn.playwright.dev`.
16+
- `setup.py``driver_version = "X.Y.Z"` is the source of truth for which Playwright release is built. The wheel build clones `microsoft/playwright` at tag `vX.Y.Z` and builds the driver from source (via `scripts/build_driver.sh` + upstream's `utils/build/build-playwright-driver.sh`).
17+
- `scripts/build_driver.sh` — clones and builds the upstream driver bundles into `driver/`. A portable bash script (shareable with the other language forks) that needs Node.js, npm, git and bash; invoked from `setup.py`'s `bdist_wheel`.
1718
- `ROLLING.md`, `CONTRIBUTING.md` — human-facing setup and roll docs.
1819

1920
## Setup
2021

21-
`CONTRIBUTING.md` has the full sequence. The short version:
22+
`CONTRIBUTING.md` has the full sequence. The short version (needs Node.js, npm, git and bash for the driver build):
2223

2324
```sh
2425
python3 -m venv env && source env/bin/activate
2526
pip install --upgrade pip
2627
pip install -r local-requirements.txt
2728
pip install -e .
28-
python -m build --wheel # downloads the driver listed in setup.py
29+
python -m build --wheel # clones microsoft/playwright @ vX.Y.Z and builds the driver from source
2930
pre-commit install
3031
```
3132

CONTRIBUTING.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,11 @@ pip install -r local-requirements.txt
2121

2222
Build and install drivers:
2323

24+
The driver is built from upstream `microsoft/playwright` source, so building a
25+
wheel requires **Node.js (with npm), git and bash** on your PATH. The first
26+
`python -m build --wheel` clones the matching Playwright tag and runs its full
27+
build, which is slow.
28+
2429
```sh
2530
pip install -e .
2631
python -m build --wheel

ROLLING.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ pre-commit install
1212
pip install -e .
1313
```
1414
* change driver version in `setup.py`
15-
* download new driver: `python -m build --wheel`
15+
* build the new driver from source: `python -m build --wheel` (clones `microsoft/playwright` at the matching tag and builds it; requires Node.js, npm, git and bash)
1616
* generate API: `./scripts/update_api.sh`
1717
* commit changes & send PR
1818
* wait for bots to pass & merge the PR

scripts/build_driver.sh

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
#!/usr/bin/env bash
2+
# Copyright (c) Microsoft Corporation.
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
16+
# Build the Playwright driver bundles from upstream source.
17+
#
18+
# Instead of downloading pre-built bundles from cdn.playwright.dev, this script
19+
# clones microsoft/playwright at the tag matching the desired version and runs
20+
# upstream's utils/build/build-playwright-driver.sh. That script cross-builds
21+
# the per-platform bundles (playwright-<version>-<suffix>.zip) that setup.py
22+
# embeds into the platform wheels -- the same artifacts the CDN serves.
23+
#
24+
# A single host builds all platform bundles at once: the upstream script
25+
# downloads the matching Node.js binary for each target, so the host platform
26+
# does not constrain which bundles can be produced.
27+
#
28+
# This is intentionally a shell script (rather than language-specific code) so
29+
# the same build step can be shared across the Playwright language forks.
30+
#
31+
# Usage: scripts/build_driver.sh <version>
32+
33+
set -euo pipefail
34+
35+
VERSION="${1:-}"
36+
if [[ -z "$VERSION" ]]; then
37+
echo "usage: scripts/build_driver.sh <version>" >&2
38+
exit 2
39+
fi
40+
41+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
42+
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
43+
DRIVER_DIR="$REPO_ROOT/driver"
44+
SOURCE_DIR="$DRIVER_DIR/playwright-src"
45+
PLAYWRIGHT_REPO="https://github.com/microsoft/playwright"
46+
47+
# Bundle suffixes produced by utils/build/build-playwright-driver.sh. Keep in
48+
# sync with the "zip_name" values in setup.py.
49+
SUFFIXES=(mac mac-arm64 linux linux-arm64 win32_x64 win32_arm64)
50+
51+
require_tools() {
52+
local missing=()
53+
local tool
54+
for tool in git node npm bash; do
55+
if ! command -v "$tool" >/dev/null 2>&1; then
56+
missing+=("$tool")
57+
fi
58+
done
59+
if [[ ${#missing[@]} -gt 0 ]]; then
60+
echo "Building the Playwright driver from source requires the following tools," >&2
61+
echo "which were not found on PATH: ${missing[*]}." >&2
62+
echo "Install Node.js (with npm), git and bash, then retry. On Windows, run the" >&2
63+
echo "build from a bash shell (e.g. Git Bash)." >&2
64+
exit 1
65+
fi
66+
}
67+
68+
cloned_version() {
69+
if [[ -f "$SOURCE_DIR/package.json" ]]; then
70+
(cd "$SOURCE_DIR" && node -p "require('./package.json').version") 2>/dev/null || true
71+
fi
72+
}
73+
74+
clone_source() {
75+
# Reuse an existing checkout only if it is at the exact version we want;
76+
# otherwise wipe it so a stale ref can never leak into the bundles.
77+
if [[ -d "$SOURCE_DIR" && "$(cloned_version)" != "$VERSION" ]]; then
78+
rm -rf "$SOURCE_DIR"
79+
fi
80+
if [[ ! -d "$SOURCE_DIR" ]]; then
81+
mkdir -p "$DRIVER_DIR"
82+
echo "Cloning $PLAYWRIGHT_REPO at v$VERSION"
83+
git clone --depth 1 --branch "v$VERSION" "$PLAYWRIGHT_REPO" "$SOURCE_DIR"
84+
fi
85+
local cloned
86+
cloned="$(cloned_version)"
87+
if [[ "$cloned" != "$VERSION" ]]; then
88+
echo "Cloned Playwright source reports version '$cloned' but '$VERSION' was requested." >&2
89+
exit 1
90+
fi
91+
}
92+
93+
build_source() {
94+
echo "Installing Playwright dependencies (npm ci)"
95+
(cd "$SOURCE_DIR" && npm ci)
96+
echo "Building Playwright (npm run build)"
97+
(cd "$SOURCE_DIR" && npm run build)
98+
echo "Building driver bundles"
99+
(cd "$SOURCE_DIR" && bash utils/build/build-playwright-driver.sh)
100+
}
101+
102+
copy_bundles() {
103+
local output_dir="$SOURCE_DIR/utils/build/output"
104+
# The output dir also holds build intermediates (downloaded Node.js binaries,
105+
# tgz archives, extracted package dirs), so copy only the versioned bundles
106+
# rather than the whole directory.
107+
cp "$output_dir"/playwright-"$VERSION"-*.zip "$DRIVER_DIR/"
108+
# Fail fast if upstream produced fewer bundles than the platforms we ship.
109+
local count
110+
count="$(find "$DRIVER_DIR" -maxdepth 1 -name "playwright-$VERSION-*.zip" | wc -l)"
111+
if [[ "$count" -ne "${#SUFFIXES[@]}" ]]; then
112+
echo "Expected ${#SUFFIXES[@]} driver bundles, found $count in $output_dir" >&2
113+
exit 1
114+
fi
115+
}
116+
117+
require_tools
118+
clone_source
119+
build_source
120+
copy_bundles

setup.py

Lines changed: 19 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -98,24 +98,19 @@ def extractall(zip: zipfile.ZipFile, path: str) -> None:
9898
os.chmod(extracted_path, attr)
9999

100100

101-
def download_driver(zip_name: str) -> None:
102-
zip_file = f"playwright-{driver_version}-{zip_name}.zip"
103-
destination_path = "driver/" + zip_file
101+
def ensure_driver_bundle(zip_name: str) -> None:
102+
destination_path = f"driver/playwright-{driver_version}-{zip_name}.zip"
104103
if os.path.exists(destination_path):
105104
return
106-
url = "https://cdn.playwright.dev/builds/driver/"
107-
if (
108-
"-alpha" in driver_version
109-
or "-beta" in driver_version
110-
or "-next" in driver_version
111-
):
112-
url = url + "next/"
113-
url = url + zip_file
114-
temp_destination_path = destination_path + ".tmp"
115-
print(f"Fetching {url}")
116-
# Don't replace this with urllib - Python won't have certificates to do SSL on all platforms.
117-
subprocess.check_call(["curl", url, "-o", temp_destination_path])
118-
os.rename(temp_destination_path, destination_path)
105+
# Build all platform bundles from source (microsoft/playwright @ the
106+
# matching tag) and stage them into driver/. A single invocation produces
107+
# every platform's bundle, so subsequent calls hit the early return above.
108+
build_script = os.path.join(os.path.dirname(__file__), "scripts", "build_driver.sh")
109+
subprocess.check_call(["bash", build_script, driver_version])
110+
if not os.path.exists(destination_path):
111+
raise RuntimeError(
112+
f"Driver bundle {destination_path} was not produced by the source build."
113+
)
119114

120115

121116
class PlaywrightBDistWheelCommand(BDistWheelCommand):
@@ -152,10 +147,13 @@ def _build_wheel(
152147
assert self.dist_dir
153148
base_wheel_location: str = glob.glob(os.path.join(self.dist_dir, "*.whl"))[0]
154149
without_platform = base_wheel_location[:-7]
155-
download_driver(wheel_bundle["zip_name"])
150+
ensure_driver_bundle(wheel_bundle["zip_name"])
156151
zip_file = f"driver/playwright-{driver_version}-{wheel_bundle['zip_name']}.zip"
152+
extract_dir = f"driver/{wheel_bundle['zip_name']}"
153+
if os.path.exists(extract_dir):
154+
shutil.rmtree(extract_dir)
157155
with zipfile.ZipFile(zip_file, "r") as zip:
158-
extractall(zip, f"driver/{wheel_bundle['zip_name']}")
156+
extractall(zip, extract_dir)
159157
wheel_location = without_platform + wheel_bundle["wheel"]
160158
shutil.copy(base_wheel_location, wheel_location)
161159
with zipfile.ZipFile(
@@ -197,8 +195,10 @@ def _download_and_extract_local_driver(
197195
)
198196
assert len(zip_names_for_current_system) == 1
199197
zip_name = zip_names_for_current_system.pop()
200-
download_driver(zip_name)
198+
ensure_driver_bundle(zip_name)
201199
zip_file = f"driver/playwright-{driver_version}-{zip_name}.zip"
200+
if os.path.exists("playwright/driver"):
201+
shutil.rmtree("playwright/driver")
202202
with zipfile.ZipFile(zip_file, "r") as zip:
203203
extractall(zip, "playwright/driver")
204204

0 commit comments

Comments
 (0)