Skip to content

Commit fd24913

Browse files
hoodmanehoodmane
committed
Add Emscripten/Pyodide support
1 parent 26dd31e commit fd24913

20 files changed

Lines changed: 1216 additions & 14 deletions

File tree

‎.github/workflows/main.yml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,9 +29,36 @@ jobs:
2929
python-version: ${{ matrix.python-version }}
3030
enable-cache: true
3131

32+
- name: Install Chrome for Pyodide tests
33+
if: ${{ matrix.python-version == '3.12' }}
34+
uses: pyodide/pyodide-actions/install-browser@v2
35+
with:
36+
runner: selenium
37+
browser: chrome
38+
browser-version: latest
39+
40+
- name: Install Node for Pyodide tests
41+
if: ${{ matrix.python-version == '3.12' }}
42+
uses: pyodide/pyodide-actions/install-browser@v2
43+
with:
44+
runner: selenium
45+
browser: node
46+
browser-version: 22
47+
48+
- name: Download Pyodide
49+
if: ${{ matrix.python-version == '3.12' }}
50+
uses: pyodide/pyodide-actions/download-pyodide@v2
51+
with:
52+
version: 0.29.3
53+
to: pyodide_dist
54+
3255
- name: Install dependencies
3356
run: scripts/install
3457

58+
- name: Install Pyodide test dependencies
59+
if: ${{ matrix.python-version == '3.12' }}
60+
run: uv pip install -r requirements-emscripten.txt
61+
3562
- name: Run linting checks
3663
run: scripts/check
3764

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,3 +11,4 @@ venv*/
1111
.python-version
1212
build/
1313
dist/
14+
pyodide_dist/

‎docs/advanced/emscripten.md‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
---
2+
template: pyodide.html
3+
---
4+
5+
# Emscripten Support
6+
7+
httpx2 has support for running on WebAssembly / Emscripten using
8+
[Pyodide](https://github.com/pyodide/pyodide/).
9+
10+
Asynchronous requests always use `fetch`. Synchronous requests use the following
11+
methods:
12+
1. If [Javascript Promise Integration](https://github.com/WebAssembly/js-promise-integration/blob/main/proposals/js-promise-integration/Overview.md)
13+
(JSPI) is supported by the JavaScript runtime, the request will be made with
14+
`fetch` and stack switching.
15+
2. Otherwise, if in a browser, the request will be made using a synchronous
16+
`XMLHttpRequest`.
17+
3. Otherwise, if in Node, the request will fail. Synchronous requests in Node
18+
require JSPI.
19+
20+
In Emscripten, all network connections are handled by the enclosing Javascript
21+
runtime. As such, there is limited control over various features. In particular:
22+
23+
- Proxy servers are handled by the runtime, so httpx2 cannot control them.
24+
- httpx2 has no control over connection pooling.
25+
- Certificate handling is done by the browser, so httpx2 cannot modify it.
26+
- Requests are constrained by cross-origin isolation settings in the same way as
27+
any request that is originated by Javascript code.
28+
- Timeouts will not work in the main browser thread unless the browser supports
29+
JSPI because main thread synchronous `XMLHttpRequest` does not support
30+
timeouts.
31+
32+
Setting any of the transport options that depend on these features (`verify`,
33+
`cert`, `http2`, `limits`, `proxy`, `uds`, `local_address`, `retries`, or
34+
`socket_options`) will emit a `UserWarning` and the option will be silently
35+
ignored.
36+
37+
## Try it in your browser
38+
39+
Use the following live example to test httpx2 in your web browser. You can
40+
change the code below and hit run again to test different features or web
41+
addresses.
42+
43+
<div id="pyodide_editor">import httpx2
44+
print("Sending response using httpx2 in the browser:")
45+
print("--------------------------------------------")
46+
r = httpx2.get("http://www.example.com")
47+
print("Status = ", r.status_code)
48+
print("Response = ", r.text[:50], "...")</div>
49+
50+
<div id="pyodide_output"></div>
51+
52+
<div id="pyodide_buttons"></div>
53+
54+
## Build it
55+
56+
Because `httpx2` is a pure python module, building is the same as ever
57+
(`python -m build`), or use the built wheel from PyPI.
58+
59+
## Testing Custom Builds of httpx2 in Emscripten
60+
61+
Once you have a wheel you can test it in your browser. You can do this using the
62+
[Pyodide console](https://pyodide.org/en/stable/console.html), or by hosting
63+
your own web page. You will need version 0.26.2 or later of Pyodide.
64+
65+
1. To test in Pyodide console, serve the wheel file via http (e.g. by calling
66+
python -m `http.server` in the dist directory.) Then in the [Pyodide
67+
console](https://pyodide.org/en/stable/console.html), type the following,
68+
replacing the URL of the locally served wheel.
69+
70+
```python
71+
import pyodide_js as pjs
72+
import ssl, certifi, idna
73+
pjs.loadPackage("<URL_OF_THE_WHEEL>")
74+
import httpx2
75+
# Now httpx2 should work
76+
```
77+
78+
2. To test a custom-built wheel in your own web page, create a page which loads
79+
the Pyodide JavaScript (see the
80+
[instructions](https://pyodide.org/en/stable/usage/index.html) on the
81+
Pyodide website). After starting the Pyodide runtime, run the following code
82+
to load httpx2 and its dependencies:
83+
```js
84+
await pyodide.loadPackage([httpx2_wheel_url, "ssl", "certifi", "idna"])
85+
```
86+
87+
3. To test in Node.js, run `npm i pyodide` or download a Pyodide distribution
88+
download to a known folder, then load Pyodide following the instructions on
89+
the Pyodide website (https://pyodide.org/en/stable/usage/index.html). After
90+
starting the Pyodide runtime, run the following code to load httpx2 and its
91+
dependencies:
92+
```js
93+
await pyodide.loadPackage([httpx2_wheel_url, "ssl", "certifi", "idna"])
94+
```

‎docs/index.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -143,3 +143,8 @@ To include the optional brotli and zstandard decoders support, use:
143143
```shell
144144
pip install 'httpx2[brotli,zstd]'
145145
```
146+
147+
## Pyodide / Emscripten Support
148+
149+
There is experimental support for running in WebAssembly under the Pyodide
150+
runtime. See the [Emscripten](advanced/emscripten.md) page for more details.

‎docs/overrides/pyodide.html‎

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
{% extends "main.html" %}
2+
{% block styles %}
3+
{{ super() }}
4+
<link
5+
href="https://cdn.jsdelivr.net/npm/ace-builds@1.36.2/css/ace.min.css"
6+
rel="stylesheet"
7+
/>
8+
<style>
9+
#pyodide_editor {
10+
width: 100%;
11+
height: 14em;
12+
font-size: 0.8rem;
13+
box-sizing: border-box;
14+
}
15+
#pyodide_editor .ace_gutter,
16+
#pyodide_editor .ace_content,
17+
#pyodide_editor .ace_text-layer {
18+
font-size: 0.8rem;
19+
}
20+
#pyodide_output {
21+
width: 100%;
22+
height: 20em;
23+
white-space: pre-wrap;
24+
background-color: #1e1e1e;
25+
color: #e6e6e6;
26+
font-family: "Courier New", monospace;
27+
font-size: 0.8rem;
28+
overflow-y: scroll;
29+
padding: 0.5em;
30+
box-sizing: border-box;
31+
border-radius: 4px;
32+
}
33+
34+
#pyodide_output span {
35+
display: block;
36+
}
37+
.pyodide_error {
38+
color: #ff6b6b;
39+
}
40+
</style>
41+
{% endblock %}
42+
43+
{% block scripts %}
44+
{{ super() }}
45+
<script src="https://cdn.jsdelivr.net/npm/ace-builds@1.36.2/src-noconflict/ace.min.js"></script>
46+
<script src="https://cdn.jsdelivr.net/pyodide/v0.29.3/full/pyodide.js"></script>
47+
<script language="javascript">
48+
const editor = ace.edit("pyodide_editor");
49+
editor.setTheme("ace/theme/monokai");
50+
editor.session.setMode("ace/mode/python");
51+
editor.setOptions({
52+
fontSize: "0.8rem",
53+
vScrollBarAlwaysVisible: true,
54+
});
55+
const outputDiv = document.getElementById("pyodide_output");
56+
outputDiv.innerText = "Loading python in your browser";
57+
58+
function stdoutLine(line) {
59+
const content = document.createElement("span");
60+
content.textContent = line;
61+
outputDiv.appendChild(content);
62+
}
63+
64+
function stderrLine(line) {
65+
const content = document.createElement("span");
66+
content.className = "pyodide_error";
67+
content.textContent = line;
68+
outputDiv.appendChild(content);
69+
}
70+
71+
async function initPyodide() {
72+
const pyodide = await loadPyodide();
73+
pyodide.setStdout({ batched: stdoutLine });
74+
pyodide.setStderr({ batched: stderrLine });
75+
if (
76+
// document.location.origin.startsWith("http://127.0.0.1") ||
77+
// document.location.origin.startsWith("http://localhost")
78+
false
79+
) {
80+
// If we are on a development machine, use the wheel from the dist folder
81+
await pyodide.loadPackage(["/test.whl", "ssl", "idna", "certifi"]);
82+
} else {
83+
// Otherwise load it from PyPI
84+
// (once a Pyodide-supporting version is on there)
85+
await pyodide.loadPackage("micropip");
86+
await pyodide.runPythonAsync("import micropip; await micropip.install('httpx2')");
87+
}
88+
outputDiv.innerText = "Python runtime ready...\n";
89+
return pyodide;
90+
}
91+
92+
const loadPromise = initPyodide();
93+
94+
async function runCode() {
95+
const pyodide = await loadPromise;
96+
const code = editor.getValue();
97+
await pyodide.loadPackagesFromImports(code);
98+
try {
99+
await pyodide.runPythonAsync(code);
100+
} catch (e) {
101+
stderrLine(e.message);
102+
}
103+
}
104+
105+
function clearOutput() {
106+
outputDiv.innerHTML = "";
107+
}
108+
109+
const run_button = document.createElement("button");
110+
run_button.className = "md-button";
111+
run_button.onclick = runCode;
112+
run_button.innerText = "Run";
113+
const clear_button = document.createElement("button");
114+
clear_button.className = "md-button";
115+
clear_button.onclick = clearOutput;
116+
clear_button.innerText = "Clear output";
117+
clear_button.style = "float:right";
118+
const buttonDiv = document.getElementById("pyodide_buttons");
119+
buttonDiv.appendChild(run_button);
120+
buttonDiv.appendChild(clear_button);
121+
122+
globalThis.runCode = runCode;
123+
</script>
124+
{% endblock %}

‎mkdocs.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ site_description: A next-generation HTTP client for Python.
33

44
theme:
55
name: 'material'
6+
custom_dir: 'docs/overrides'
67
palette:
78
- scheme: 'default'
89
media: '(prefers-color-scheme: light)'
@@ -34,6 +35,7 @@ nav:
3435
- Transports: 'advanced/transports.md'
3536
- Text Encodings: 'advanced/text-encodings.md'
3637
- Extensions: 'advanced/extensions.md'
38+
- Emscripten: 'advanced/emscripten.md'
3739
- Guides:
3840
- Async Support: 'async.md'
3941
- HTTP/2 Support: 'http2.md'

‎pyproject.toml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,12 @@ markers = [
8181
source_pkgs = ["httpx2", "httpcore2", "tests"]
8282
omit = ["src/httpcore2/httpcore2/_sync/*", "tests/test_benchmark.py"]
8383

84+
[tool.coverage.paths]
85+
source = [
86+
"src/httpx2/httpx2",
87+
"*/site-packages/httpx2",
88+
]
89+
8490
[tool.coverage.report]
8591
exclude_also = [
8692
"if TYPE_CHECKING:",

‎requirements-emscripten.txt‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# extra requirements for testing on emscripten
2+
pytest-pyodide >= 0.59.2
3+
selenium

‎scripts/coverage‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,13 @@
22

33
set -x
44

5-
uv run coverage report --show-missing --skip-covered --fail-under=100
5+
uv run coverage combine
6+
if [ -d 'pyodide_dist' ]; then
7+
IGNORE_ARGS=""
8+
else
9+
# if we don't have a pyodide environment set up, then don't test coverage
10+
# for the emscripten transport
11+
IGNORE_ARGS="--omit=src/httpx2/httpx2/_transports/jsfetch.py,tests/httpx2/emscripten/*"
12+
fi
13+
14+
uv run coverage report ${IGNORE_ARGS} --show-missing --skip-covered --fail-under=100

‎scripts/download-pyodide‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
#!/bin/sh -e
2+
set -x
3+
4+
uv pip install -r requirements-emscripten.txt
5+
6+
mkdir -p pyodide_dist
7+
PYODIDE_URL="https://github.com/pyodide/pyodide/releases/download/0.29.3/pyodide-core-0.29.3.tar.bz2"
8+
PYODIDE_OUTPATH="/tmp/pyodide.tar.bz2"
9+
if command -v wget >/dev/null 2>&1; then
10+
wget -q "$PYODIDE_URL" -O "$PYODIDE_OUTPATH"
11+
else
12+
curl -sL "$PYODIDE_URL" -o "$PYODIDE_OUTPATH"
13+
fi
14+
tar -xjf "$PYODIDE_OUTPATH" -C pyodide_dist --strip-components=1
15+
./pyodide_dist/python - <<'EOF'
16+
import pyodide_js as pjs
17+
18+
pjs.loadPackage(
19+
[
20+
"coverage",
21+
"idna",
22+
"micropip",
23+
"pytest",
24+
"sqlite3",
25+
"tblib",
26+
]
27+
)
28+
EOF

0 commit comments

Comments
 (0)