Skip to content

README.md

Latest commit

 

History

History
215 lines (154 loc) · 7.77 KB

README.md

File metadata and controls

215 lines (154 loc) · 7.77 KB
Raw

CloudScraper.js

ci node license

A Node.js wrapper for Python-based CloudFlare bypass functionality. This library is a JavaScript port of the popular cloudscraper Python library, designed to help developers bypass CloudFlare protection mechanisms in their Node.js applications.

🎯 Purpose

CloudScraper.js provides a seamless way to make HTTP requests to websites protected by CloudFlare, handling JavaScript challenges, CAPTCHAs, and other protection mechanisms automatically. It's particularly useful for:

  • Web scraping applications
  • API integrations with CloudFlare-protected endpoints
  • Automated testing of protected websites
  • Data collection from sites with anti-bot protection

🚀 Features

  • Automatic CloudFlare bypass: Handles JavaScript challenges and CAPTCHAs
  • Multi-platform support: Works on macOS, Linux, and Windows
  • Safe, explicit setup: no privileged or automatic scripts run on npm install
  • Optional install helper: npm run install-deps sets up Python + a venv when you want it
  • TypeScript support: Full type definitions included
  • Multiple HTTP methods: GET, POST, COOKIE, and TOKENS support

📦 Installation

This library requires Node.js and Python 3 (with the cloudscraper package).

Quick Start

npm install cloudscraper.js
# Install the Python side (opt-in — nothing privileged runs on `npm install`):
pip install cloudscraper          # or: pip install --break-system-packages cloudscraper

Why opt-in? Earlier versions auto-installed Python on postinstall (including sudo / Homebrew / curl | bash). That was removed for safety — installing system packages is now an explicit choice, which keeps the package usable in CI, Docker and locked-down environments. If you'd like the guided helper (detects Python, creates a venv, installs cloudscraper), run:

npm run install-deps

If the cloudscraper Python package is missing at runtime, the library fails fast with a clear message telling you how to install it. See INSTALLATION.md for details.

🐳 Docker

Prefer a container with Node and Python + cloudscraper already wired up? The repo ships a multi-stage Dockerfile:

docker build -t cloudscraper-js .
docker run -i --rm cloudscraper-js          # boots the MCP server on stdio

🔧 Usage

// ES6
import CloudScraper from "cloudscraper.js";

// CommonJS
const CloudScraper = require("cloudscraper.js").default;

const scraper = new CloudScraper({
  usePython3: true, // Set to true if using python3, false for python
  timeoutInSeconds: 10,
});

// Simple GET request
scraper
  .get("https://example.com")
  .then((response) => {
    console.log("Status:", response.status);
    console.log("Data:", response.text());
  })
  .catch((error) => {
    console.error("Error:", error);
  });

// POST request with data
scraper
  .post("https://api.example.com", {
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ key: "value" }),
  })
  .then((response) => {
    console.log(response.json());
  });

Faster: reusable hot sessions (v0.2, experimental)

createScraper() talks to a long-lived daemon that keeps the solved Cloudflare session hot, so repeated requests skip re-solving the challenge:

import { createScraper } from "cloudscraper.js";

const scraper = await createScraper({ retries: 3, rateLimitPerHost: 1 });

const first = await scraper.get("https://www.irishjobs.ie/");     // solves once
const again = await scraper.get("https://www.irishjobs.ie/jobs"); // reuses session (fast)

console.log(first.status, again.status);
await scraper.close();

Supports get/post/put/delete/patch/head/cookies/tokens, plus proxy, retries, rateLimitPerHost and timeoutMs. Requires Python 3 + pip install cloudscraper. This API is new in v0.2 and evolving.

🤖 Use it from an AI agent (v0.2)

Three ways for LLM agents to read anti-bot–protected pages, returning clean markdown by default.

MCP server — point any MCP client (Claude Desktop, IDEs) at the cloudscraper-mcp binary:

{ "mcpServers": { "cloudscraper": { "command": "npx", "args": ["-y", "-p", "cloudscraper.js", "cloudscraper-mcp"] } } }

Tools: fetch_protected_url, get_cookies, solve_challenge.

LangChain / LangGraph (optional peer @langchain/core):

import { createScraper, createCloudScraperTool } from "cloudscraper.js";
const tool = await createCloudScraperTool(await createScraper({ format: "markdown" }));
// bind `tool` to your agent / graph

Function calling (OpenAI / Anthropic) — hand functionSchemas to the model, run the handler:

import { createScraper, fetchProtectedUrl, functionSchemas } from "cloudscraper.js";
const scraper = await createScraper();
const { text } = await fetchProtectedUrl(scraper, { url, format: "markdown" });

The MCP server dynamically loads the ESM MCP SDK, so it needs Node ≥ 22.12 plus Python 3 + pip install cloudscraper. See examples/.

🏗️ Architecture

Today the SDK spawns a short-lived Python process (running the cloudscraper library) per request and reads the result back over stdout:

flowchart LR
  App[Node App] -->|new CloudScraper| SDK[cloudscraper.js]
  SDK -->|spawn python per request| Py[(python + cloudscraper)]
  Py -->|stdout| SDK
  SDK --> App
Loading

v0.2 (in progress) replaces the per-request spawn with a long-lived daemon that keeps solved sessions hot (large speedup on repeated calls), a robust NDJSON IPC protocol, and first-class AI-agent interfaces (an MCP server + a LangChain tool):

flowchart TB
  Agent[LLM Agent] -->|MCP| MCP[MCP Server]
  Dev[Dev SDK] -->|createScraper| SDK[cloudscraper.js SDK]
  MCP --> SDK
  SDK <-->|NDJSON over socket| D[Python Daemon: hot session pool]
  D -->|clean HTML / Markdown| SDK
Loading

🛠️ Development Status

⚠️ Early Development Phase

This library is currently in early development. While functional, it may have limitations and breaking changes as we continue to improve it. We're actively working on:

  • Enhanced error handling
  • Better performance optimization
  • Additional HTTP methods support
  • Improved documentation
  • More robust CloudFlare challenge handling

We plan to release regular updates to improve functionality and maintain compatibility with the latest CloudFlare protection mechanisms.

📚 Documentation

🤝 Credits

This project is based on and inspired by:

All credit for the core CloudFlare bypass functionality goes to the original Python project and its contributors.

📄 License

ISC License - see LICENSE file for details.

🐛 Issues & Contributions

Found a bug or have a feature request? Please open an issue on GitHub. Contributions are welcome!

⚠️ Disclaimer

This library is intended for legitimate use cases such as web scraping, testing, and API integration. Please ensure you comply with the terms of service of any websites you interact with and respect rate limits and robots.txt files.