handleCloudflareChallenge not working as intended

[maxih4]

Which package is this bug report for? If unsure which one to select, leave blank

None

Issue description

Description
I am trying to use handleCloudflareChallenge with Camoufox in Crawlee, but the challenge handling does not properly block execution until Cloudflare verification is complete.

Even when following the E2E example from the Crawlee repository, the request handler continues execution before the Cloudflare challenge is fully resolved.

Reference test:
(https://github.com/apify/crawlee/blob/master/test/e2e/camoufox-cloudflare/actor/main.js)

I also created a minimal reproduction using a simple site (grabjobs.co), and the issue is still reproducible.

Expected behavior
handleCloudflareChallenge should:

• Detect Cloudflare challenge
• Wait until the challenge is fully solved
• Only then continue execution of the request handler

Actual behavior

• Page opens
• Cloudflare challenge is still active (“Just a moment…” / spinner visible)
• Request handler continues immediately after handleCloudflareChallenge resolves
• page.title() is executed too early and returns empty or incomplete value

Environment
Crawlee version: 3.16.0
Playwright version: 1.59.1
Camoufox version: 0.10.2
OS: Ubuntu 24.04

Code sample

import { PlaywrightCrawler, Dataset } from "@crawlee/playwright";
import { firefox } from "playwright";
import { launchOptions } from "camoufox-js";


const crawler = new PlaywrightCrawler({
  launchContext: {
    launcher: firefox,
    launchOptions: await launchOptions({
      headless: true,
    }),
  },
  postNavigationHooks: [
    async ({ handleCloudflareChallenge }) => {
        await handleCloudflareChallenge({
          sleepSecs: 10,
          preChallengeSleepSecs: 15
        });
    },
],

  async requestHandler({ page, request, parseWithCheerio }) {
    console.log(`Visiting: ${request.url}`);

    await page.goto(request.url, {
      waitUntil: "domcontentloaded",
    });

    const isBlocked = await page.evaluate(() => {
      const html = document.body?.innerText?.toLowerCase() || "";
      return (
        html.includes("checking your browser") ||
        html.includes("cloudflare") ||
        html.includes("attention required") ||
        !!document.querySelector(".ray-id")
      );
    });

    let title = "";

    try {
      const $ = await parseWithCheerio();
      title = $("h1").first().text().trim();
    } catch (e) {
      console.log("Cheerio parse failed");
    }

    console.log({ isBlocked, title });

    await Dataset.pushData({
      url: request.url,
      isBlocked,
      title,
    });
  },
});

await crawler.run(["https://grabjobs.co"]);

Package version

├── @apify/tsconfig@0.1.2 ├── @crawlee/impit-client@3.16.0 ├── @crawlee/playwright@3.16.0 ├── @types/node@24.12.2 ├── camoufox-js@0.10.2 ├── crawlee@3.16.0 ├── playwright@1.59.1 ├── tsx@4.21.0 └── typescript@6.0.3

Node.js version

v24.15.0

Operating system

Ubuntu

Apify platform

• Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

No response

[barjin]

Hello @maxih4, and thank you for your interest in this project!

Looking at the code, you can bump the options.sleepSecs to ensure the page fully loads before handleCloudflareChallenge resolves.

crawlee/packages/playwright-crawler/src/internals/utils/playwright-utils.ts

Line 834 in a01fa0b

await sleep((options.sleepSecs ?? 10) * 1000);

Alternatively, you can use Playwright's Locator.waitFor() (docs) to wait for a specific element to appear in the DOM after calling handleCloudflareChallenge.

I'm assuming we're only using the simple delay because it's difficult to programmatically figure out that the challenge passed... is that right @B4nan ? Should we, e.g., consider adding another option (something like successSelector isSucceededCallback) that could replace the sleep call, if set? Wdyt?

[maxih4]

Hello @barjin,

thanks for your reply. I tried already adjusting options.sleepsecs and options.preChallengeSleepSecs but it does not slow down anything at all. At least while using Firefox/camoufox. I investigated this with headless = false. I am not exactly sure if the options do work with chromium.

[Tamplier]

Hi, @maxih4!
As I understand it, the built-in check for CF challenges is not working. Without a reliable indicator, the handleCloudflareChallenge function will not wait properly. Try refining your own isChallengeCallback and clickPositionCallback.

[maxih4]

I dug into why handleCloudflareChallenge is currently broken and found two issues:

1. The selector used in isChallengeCallback is incorrect.
2. The selector used to resolve the position) is also incorrect.

Would you like me to open a PR that fixes both by hardcoding the corrected selectors, or would you prefer a more flexible approach (e.g., allowing users to pass custom CSS selectors)?