Fetch.enable + --http-proxy deadlocks navigation (subresource events stop firing)

[staylor]

Summary

Calling Fetch.enable (CDP) on a Lightpanda session that was started with --http-proxy deadlocks every subsequent navigation. Specifically, Page.navigate returns successfully but the document never finishes loading -- subresource Network.requestWillBeSent events stop firing, and the navigation hangs until the client-side timeout.

This is the CDP-layer root cause behind a behavior that puppeteer-core's page.setRequestInterception(true) exposes -- puppeteer enables the Fetch domain under the hood when interception is requested.

Reproduction

CDP-level (no puppeteer):

1. Start lightpanda: lightpanda serve --http-proxy http://127.0.0.1:8080 --port 9222 --host 127.0.0.1
   (any HTTP CONNECT proxy on 127.0.0.1:8080 reproduces -- a local mitmproxy / squid is fine; it doesn't have to actually reach the upstream)
2. Connect a CDP client, attach to a target with flatten: true.
3. Send Fetch.enable with patterns: [{ urlPattern: "*" }] (or even Fetch.enable {} with no patterns).
4. Send Page.navigate { url: "https://example.com" }.

Observed: Page.navigate returns a frame id; no Page.frameStoppedLoading, no Page.loadEventFired, no further Network.requestWillBeSent after the initial document request. The navigation hangs until the client-side timeout.

Without --http-proxy: Fetch.enable works correctly -- subresource interception fires, navigation completes normally.

Without Fetch.enable: --http-proxy works correctly -- pages load through the proxy in the expected time.

The deadlock requires both at once.

puppeteer-core surface

// Same shape, easier to reproduce locally
const browser = await puppeteer.connect({ browserWSEndpoint: 'ws://127.0.0.1:9222' });
const page = await browser.newPage();
await page.setRequestInterception(true);  // calls Fetch.enable internally
await page.goto('https://example.com');   // hangs until timeout

I empirically measured this against https://www.allbirds.com/products/womens-wool-runners-dapple-grey through a residential-proxy chain: direct fetch through Lightpanda (with --http-proxy set) completes in ~4.4 s; the same page with setRequestInterception(true) enabled hits the navigation timeout (30 s) and fails.

Why this matters

Fetch.enable is the standard way to filter outbound resource requests at the browser level (block image / font / media / tracking-domain hosts before they fetch). For automation use cases this is critical:

• Scraping at scale: blocking unnecessary subresources cuts page load time 2-5x and saves substantial proxy bandwidth.
• The --http-proxy flag is also critical -- routing through residential proxies / proxy classifiers is a common production setup.

When the two can't coexist, callers have to choose: skip the bandwidth filter (heavier pages, more third-party JS executes -- including widgets like Yotpo that inject hundreds of <style> elements per interaction), or skip the proxy (loses the routing model entirely). Neither is acceptable for production scrape paths.

Suspected mechanism

I haven't traced this in the Lightpanda source, but the symptom (subresource events stop firing, document never completes) suggests something in the CDP Fetch.enable path either:

1. Doesn't propagate the upstream proxy configuration to the intercepted-and-resumed request, so resumed requests stall at connect time, or
2. Holds a lock / pending-completion handle that the proxy round-trip doesn't release, or
3. Expects the Fetch.requestPaused / Fetch.continueRequest round-trip to complete on the direct request path and breaks when the request is going through the proxy

Happy to instrument and bisect if a maintainer can point me at the right entry point in src/cdp/domains/fetch* (or wherever the Fetch domain lives).

Environment

• Lightpanda: 1.0.0-nightly.6240+37391687
• OS: macOS 14, arm64
• Test driver: puppeteer-core via CDP at ws://127.0.0.1:9222

Related

• Affects scraper setups that need both per-request filtering and proxy routing.
• Documented as a known constraint in our codebase (we skip resource filtering on Lightpanda only, with a multi-paragraph comment pointing at this exact deadlock) -- happy to share the workaround code if it's useful as a reference for what surface client code uses.

[staylor]

Significant update with a much sharper repro and a concrete, instrumented characterization.

Correction first

My original framing -- "deadlocks every subsequent navigation" -- is wrong. Pages with low subresource volume (example.com, github.com) work fine through --http-proxy + Fetch.enable. The deadlock fires reliably on heavier real-world pages:

URL                       normal             intercept
example.com               OK 124ms           OK 124ms
github.com                OK 857ms           OK 823ms
shopify.com               OK 1289ms          HANG (20s timeout)
allbirds.com              OK 3361ms          HANG (20s timeout)
nike.com                  OK 4727ms          HANG (20s timeout)

The threshold correlates with subresource count / parse-time, not with the proxy itself. (My local proxy is a vanilla Node forward proxy; same behavior reproduces with mitmproxy and BrightData.)

Concrete instrumentation: what's actually stuck

I added file-based logging to InterceptionLayer.request/continueRequest (PAUSE/CONTINUE counters with running intercepted count) and to CDP.readSocket (per-call byte counts). Reproducer: puppeteer setRequestInterception(true) + simple req.continue() handler against https://www.allbirds.com through a localhost forward proxy.

Both sides measured independently:

Puppeteer side:
  request events seen by puppeteer:  53
  req.continue() succeeded:          12
  req.continue() failed:              0    <- the other 41 promises never resolve

Lightpanda side:
  PAUSE events emitted:              53    (matches puppeteer's request count)
  CONTINUE events received:          12    (matches puppeteer's resolved count)
  CDP.readSocket calls total:        20    (over 20 seconds)

So both sides agree: lightpanda sends 53 Fetch.requestPaused events, puppeteer sends 53 Fetch.continueRequest messages back, but only 12 of those reach lightpanda's CDP processing.

Pattern

Sequential trace of the events as they fire (annotated):

PAUSE id=1   intercepted=1     <- root document
CONTINUE id=1 intercepted=0    <- puppeteer continues, root resumes
PAUSE id=2..12  intercepted=11 <- 11 scripts discovered while root parses
CONTINUE id=2..12 intercepted=0  <- all 11 continues processed
PAUSE id=13..53  intercepted=41 <- 41 more scripts discovered as those 11 run
[no further CONTINUE events for the next 18 seconds, then puppeteer times out]

The first batch (root + 11 scripts) is processed cleanly. The second batch (41 more scripts) PAUSEs but never CONTINUEs. The intercepted counter sticks at 41 until timeout.

Where it stalls

CDP.readSocket gets called only 20 times total across the 20-second hang. After the burst that handles the 12 continues, lightpanda effectively stops draining the CDP WebSocket. Puppeteer's writes either pile up in puppeteer's send buffer or arrive at lightpanda's socket but never get read.

The most likely shape (haven't fully traced yet, would appreciate a maintainer's eyes): something in the runner loop / V8 macrotask processing keeps lightpanda away from readSocket() for long stretches once the second batch of scripts is in flight. The 12 continued scripts complete, their done_callbacks parse the HTML, find more <script> tags, queue more transfers, those pause at the InterceptionLayer -- all in one pass through handles.perform() inside makeRequest. Lightpanda then returns to the runner loop and polls, but by then the 41 PAUSE messages are queued and puppeteer's reply messages are arriving. Something about that state keeps readSocket from being called frequently enough to drain them.

I tried to instrument the runner loop further but lightpanda's structured logger drops messages under load (lots of writes -> only some make it to stderr -- I had to write directly to a file to get reliable counts). That's a separate debuggability paper-cut worth flagging if it's not already known.

What rules out simpler explanations

• Not a libcurl / proxy connectivity issue. Without intercept, the same proxy + same URL completes in 3-5 seconds. With intercept, the first 12 continues' HTTP transfers complete normally. The proxy logs show CONNECT tunnels and data flowing.
• Not a CDP message size limit. The 41 continueRequest messages are ~80 bytes each (~3.3 KB total), well under the 16 KB WebSocket reader buffer.
• Not request volume per se. Synthetic page with 50 local subresources (resolved instantly through the proxy) works fine in both modes. The trigger is parse-time + real network latency for many remote subresources -- which is exactly the realistic Shopify-storefront shape.
• Not specific to one site. Reliably reproduces on shopify.com, allbirds.com, and nike.com. None of these are anything special; they're representative React/Vue + ad-tech-heavy storefronts.

Reproducer

import puppeteer from 'puppeteer-core';

const browser = await puppeteer.connect({
  browserWSEndpoint: 'ws://127.0.0.1:9223',
  defaultViewport: null,
});
const page = await browser.newPage();
await page.setRequestInterception(true);
page.on('request', (req) => req.continue());
try {
  await page.goto('https://www.allbirds.com', { waitUntil: 'load', timeout: 20000 });
  console.log('OK');
} catch (e) {
  console.log('HANG:', e.message);
}

Lightpanda startup:

lightpanda serve --host 127.0.0.1 --port 9223 \
  --http-proxy http://127.0.0.1:8080

Any HTTP CONNECT-capable forward proxy on 127.0.0.1:8080 will do (vanilla Node http.createServer with a connect handler is enough).

Environment

• Lightpanda: 1.0.0-nightly.6240+37391687
• OS: macOS 14, arm64
• Driver: puppeteer-core 23.x via CDP at ws://127.0.0.1:9223

Workaround we're using

Skip setRequestInterception(true) on Lightpanda. We lose host-level resource filtering (which means heavy third-party widgets like Yotpo / Redo / Gladly load fully and inflate per-page DOM by ~2-3x), but pages don't hang. Documented in our codebase with a multi-paragraph comment pointing at this issue. Restoring the filter would close most of the Lightpanda-vs-Chromium throughput gap we still see.