browsertrix-crawler v1.13.2: ~2× slowdown in page processing vs v1.13.1

[axllent]

Summary

After upgrading from browsertrix-crawler v1.13.1 to v1.13.2, full-site crawls of the same JS-heavy site (same seed list, identical crawler configuration) take roughly twice as long to complete. The regression is consistent across runs. Reverting to v1.13.1 fully restores prior throughput.

Environment

• browsertrix-crawler: v1.13.2 (regression) vs v1.13.1 (baseline - has been consistent on all versions in the last 7 months)
• Architecture: linux/amd64 (DigitalOcean Kubernetes cluster)
• Dedicated (to this site) 3 active crawler nodes, 12 concurrent browser windows in total
• Crawler CPU is at ~100% utilisation in both versions during crawls
• No configuration changes between the two runs; only the crawler version differs

Workload

• Single site, JS-heavy on all pages
• 326 pages per full crawl

Observed throughput

Version	End-to-end full crawl time
v1.13.1	~21 minutes
v1.13.2	~41 minutes

CPU saturation is unchanged at ~100% in both versions, so the additional wall-clock time appears to be spent waiting (timeouts / stalls) rather than doing additional work.

Observed log patterns

Comparing crawl logs from the same workload, v1.13.2 shows a large increase in timeout-related messages. Approximate counts per full crawl:

Log message	v1.13.1	v1.13.2
Link extraction timed out	~228	~885
General errors (total)	~365	~1075

The ~4× increase in Link extraction timed out strongly suggests that something in the page-evaluation / link-extraction path is now stalling out more often, which is consistent with the wall-clock doubling despite CPU remaining saturated.

Mitigation

We have pinned to v1.13.1 for now, which restores prior throughput.

Suspected cause

Full changelog: v1.13.1...v1.13.2

Only two changes between v1.13.1 and v1.13.2 appear to touch the page-processing hot path:

1. puppeteer-core: ^24.38.0 → 25.1.0 (major version bump)
2. Brave browser: 1.90.128 → 1.91.175

The remaining changes in v1.13.2 are not on the page-processing path and are unlikely to explain the regression as far as I can tell.

The increase in Link extraction timed out in particular points at one of these two - either Puppeteer 25, or a Chromium-side change in Brave 1.91 affecting page evaluation timing.

Summary

I do not know if there is an obvious culprit causing the issue, however I thought I would post this in case anyone is experiencing the same issue(s). Obviously we cannot remain on a pinned v1.13.1 for ever, that is just buying us some time in the meantime, and I do hope to conduct further testing in the near future.

[ikreymer]

Do you have an example site that we can test with that repros the issue?

[axllent]

@ikreymer Thanks for the quick response.

Before we consider setting up a reproduction case, could you share what changed in v1.13.2 that could potentially affect CPU usage or crawl performance? Based on our own analysis, the most likely candidates appear to be the puppeteer-core major version bump (24.38.0 → 25.1.0) or the Brave browser update (1.90.128 → 1.91.175), both of which touch the page-processing path. That would help us understand whether what we experienced is a known or expected change, or an unintended regression.

We observed a ~60% increase in crawl duration after upgrading to v1.13.2. CPU utilisation remained high throughout the crawl (as is typical for our workload), but the crawls were taking significantly longer to complete. Pinning back to v1.13.1 resolved the duration issue immediately.

We're happy to assist with further investigation if that would be helpful.

[ikreymer]

@axllent what you describe were the only changed, we updated puppeteer and Brave dependency. We also switched to Brave Origin which is a leaner version of Brave, so it should actually run faster. In 1.14.0-beta.0, we further slimmed the Docker image as well and removed an extra puppeteer dependency that was unneeded. You can try building with either the old Brave or old puppeteer version and see if that makes a difference.

[tw4l]

I will add that I did some testing of 1.13.1 and 1.13.2 against a few websites and did not find the same duration increase or increase in link extraction timeout messages. The timing can be a bit inconsistent between crawls, but 1.13.2 was in fact faster on a few of the sites I tested without any noticeable difference in crawl quality.

So it may be that the issue is particular to the site being crawled, or sites that share some common characteristic with it, hence why a reproduction case would be helpful to investigate further.

[axllent]

Thanks for the additional context and for looking into this. The switch to Brave Origin is interesting as our target site makes heavy use of JavaScript rendering via Next.js and React, with Radix UI components and Intersection Observer, which may explain why the regression is site-specific and wasn't reproducible on other sites.

Would it be worth testing 1.14.0-beta.0 given the further dependency cleanup, or would you recommend waiting for a stable release? We're currently pinned to v1.13.1 which is working fine, so there's no immediate urgency on our end.

[ikreymer]

@axllent Can you try with the latest 1.14.0-beta.3 ? There have been a variety of fixes, hopefully this is not something you're seeing anymore.

[axllent]

I'll need to look into this next week @ikreymer - sorry, flat out with work at the moment. I'll report back once I can do a comparative test.

[axllent]

Hi @ikreymer. In the last couple of days I have run various tests using a replica server using the 1.14.0-beta.3 crawler (and various custom debug builds of it). Unfortunately it's still experiencing significant issues, so I have been working to isolate the problem by running multiple tests with different settings, and digging into the browsertrix-crawler code itself.

I have managed to identify what I strongly believe is the issue I'm experiencing (there was some AI involved to further investigate). To debug this, I had to set PAGE_OP_TIMEOUT_SECS= 30 to avoid the timeout errors, and debug the crawler to log timings for each step of the link extraction process. The results are quite clear.

Summary: BehaviorManager.extractLinks scales poorly with link count due to per-URL exposed-function callbacks.

Problem

On link-heavy pages (~300+ anchors) under CPU pressure, extractLinks is taking 20-30 seconds per page, causing frequent "Link extraction timed out" errors against PAGE_OP_TIMEOUT_SECS = 5. This isn't obvious from the source: the extraction code itself is trivial. The cost appears to be entirely in the CDP (Chrome DevTool Protocol) round-trips it generates.

Evidence

I debugged browsertrix-crawler's extractLinks to record per-page timings on the JS-heavy website I am indexing (most having 326-338 anchors per page):

Metric	With __bx_behaviors.extractLinks	With plain frame.evaluate returning string[]
Callbacks arrived at Node	~500 ms	n/a
Node-side Redis writes complete	~640 ms	~500 ms
Total frame.evaluate resolves	~26 000 ms	~300 ms
Links returned	~326	~334
Errors due to 5s timeout	>90% of pages	0

The ~25 second gap sits entirely between "last URL delivered to Node" and "in-page evaluate promise resolves."

Just for reference, the in-page extraction code (bypassing __bx_behaviors.extractLinks) I tested with is:

              .evaluate(
                (sel: string, prop: string, useAttr: boolean) => {
                  const out: string[] = [];
                  document.querySelectorAll(sel).forEach((el: Element) => {
                    // eslint-disable-next-line @typescript-eslint/no-explicit-any
                    const any = el as any;
                    const v = useAttr ? any.getAttribute(prop) : any[prop];
                    if (typeof v === "string" && v) out.push(v);
                  });
                  return out;
                },
                selector,
                extract,
                !!attrOnly,
              )
              .then(async (urls: string[] | undefined) => {
                if (!urls) return;
                await Promise.allSettled(
                  urls.map((url) => callbacks.addLink!(url)),
                );
              })
              .catch((e) =>
                logger.warn("Link Extraction failed in frame", {
                  frameUrl: frame.url,
                  ...logDetails,
                  ...formatErr(e),
                }),
              );

Root cause (in the CDP layer, not in the extraction code itself)

extractLinksActual fires one addLink(url, true) per URL and awaits them with Promise.allSettled. Each addLink is a Puppeteer exposed-function call. Per Puppeteer's Binding.run, each resolution requires one Runtime.callFunctionOn CDP command to resolve the in-page promise. Chromium processes these serially on the renderer main thread. Under CPU pressure the cost scales at roughly 75 ms per call, times N URLs.

Suggested fix

Change extractLinksActual to return Array.from(urls) instead of firing per-URL callbacks, and have the crawler iterate the returned array Node-side. This reduces CDP traffic from 3N messages per page to 2, without changing extraction semantics. Dedup, attribute fallback, and checkToJsonOverride all stay in the library where they belong.

Also noticed while investigating

browsertrix-crawler's exposed function at crawler.ts:867-870 silently drops the second alwaysObeyScope argument that the behaviors library passes as true. The scope-enforcement intent is lost. Unrelated to the timeout issue, but probably worth patching alongside.

--

I hope this is useful, else please just let me know.

[ikreymer]

Thanks for the investigation! Yes, at some point we switched to calling addLink() instead of an array since the arrays could get large, thinking that one URL at a time would be more performant, but have not done the proper benchmarking on it..

Testing on pages with lots of links is perhaps making this problem more evident..

browsertrix-crawler's exposed function at crawler.ts:867-870 silently drops the second alwaysObeyScope argument that the behaviors library passes as true. The scope-enforcement intent is lost. Unrelated to the timeout issue, but probably worth patching alongside.

That was just added in 1.14.0-beta.4 and was not being used until now, so analysis of 1.14.0-beta.3 and latest behaviors would be incorrect (it was using an older version of behaviors).

[axllent]

You're welcome - I'm looking forward to testing a fix when you have one👌

[ikreymer]

Though, of course, it's worth mentioning that this change was not made between 1.13.1 and 1.13.2, so it's odd that this would be significantly different between these two particular versions.. unless puppeteer did something to make it slower.. we can also try updating puppeteer-core in case they fixed it in the later release..

[axllent]

I think it the issue is heavily influenced by CPU usage per browser - ie: a slight increase in CPU usage pushes our instance over the edge, and thus exaggerates the issue tenfold. Just for clarity, on the previous release ( 1.13.1) we are still getting approximately 1 error/failure per successfully crawled page (which is not ideal either). I guess whatever landed in 1.13.2 onwards was enough to do that. Maybe it was an update to Puppeteer, maybe not, however it seems unnecessary to run through every link in the current way (3 x total links).

[axllent]

Just a note, I realised I hadn't pasted the full testing replacement script - I've edited my earlier post (js code)