docs-mintlify/services/smartcrawler.mdx at 64be0e29f4ac7602e65cc5ce281a3f3df42a500b · ScrapeGraphAI/docs-mintlify

title	SmartCrawler
description	AI-powered website crawling and multi-page extraction
icon	spider

Overview

SmartCrawler is our advanced web crawling service that offers two modes:

AI-Powered Extraction: LLM-powered web crawling with intelligent data extraction (10 credits per page)
Markdown Conversion: Cost-effective HTML to markdown conversion without AI/LLM processing (2 credits per page - 80% savings!)

Unlike SmartScraper, which extracts data from a single page, SmartCrawler can traverse multiple pages, follow links, and either extract structured data or convert content to clean markdown from entire websites or sections.

Try SmartCrawler instantly in our [interactive playground](https://dashboard.scrapegraphai.com/)

Getting Started

Quick Start

from scrapegraph_py import Client

client = Client(api_key="your-api-key")

response = client.crawl(
    url="https://scrapegraphai.com/",
    prompt="Extract info about the company",
    depth=2,
    breadth=None,
    max_pages=10,
    rules={
        "include_paths": ["/about", "/services/*"],
        "exclude_paths": ["/admin/*", "/api/**"],
        "same_domain": True
    }
)

import { smartCrawler } from 'scrapegraph-js';

const apiKey = 'your-api-key';
const url = 'https://scrapegraphai.com';
const prompt = 'Extract info about the company';
const depth = 2;
const breadth = null;
const maxPages = 10;
const rules = {
  include_paths: ['/about', '/services/*'],
  exclude_paths: ['/admin/*', '/api/**'],
  same_domain: true
};

try {
  const response = await smartCrawler(apiKey, url, prompt, depth, breadth, maxPages, null, rules);
  console.log(response);
} catch (error) {
  console.error(error);
}

curl -X 'POST' \
  'https://api.scrapegraphai.com/v1/crawl' \
  -H 'accept: application/json' \
  -H 'SGAI-APIKEY: your-api-key' \
  -H 'Content-Type: application/json' \
  -d '{
  "url": "https://scrapegraphai.com/",
  "prompt": "Extract info about the company",
  "depth": 2,
  "breadth": null,
  "max_pages": 10,
  "rules": {
    "include_paths": ["/about", "/services/*"],
    "exclude_paths": ["/admin/*", "/api/**"],
    "same_domain": true
  }
}'

Required Headers

Header	Description
SGAI-APIKEY	Your API authentication key
Content-Type	application/json

Parameters

Parameter	Type	Required	Description
url	string	Yes	The starting URL for the crawl.
prompt	string	No*	Instructions for what to extract (*required when extraction_mode=true).
extraction_mode	bool	No	When `false`, enables markdown conversion mode (default: true).
depth	int	No	How many link levels to follow (default: 1).
breadth	int	No	Maximum number of links to crawl per depth level. If null/undefined, unlimited (default). Controls the 'width' of exploration at each depth. Useful for limiting crawl scope on large sites. Note: max_pages always takes priority. Ignored when sitemap=true.
max_pages	int	No	Maximum number of pages to crawl (default: 20).
schema	object	No	Pydantic or Zod schema for structured output.
rules	object	No	Crawl rules object with optional fields: `exclude` (array of regex URL patterns), `include_paths` (array of path patterns to include, supports wildcards `` and `*`), `exclude_paths` (array of path patterns to exclude, takes precedence over `include_paths`), `same_domain` (boolean, default: true). See below for details.
sitemap	bool	No	Use sitemap.xml for discovery (default: false).
webhook_url	string	No	URL to receive webhook notification on job completion.
wait_ms	int	No	Milliseconds to wait before scraping each page. Useful for pages with heavy JavaScript rendering that need extra time to load (default: 3000).

Get your API key from the [dashboard](https://dashboard.scrapegraphai.com)

Markdown Conversion Mode

For cost-effective content archival and when you only need clean markdown without AI processing, use the markdown conversion mode. This mode offers significant cost savings and is perfect for documentation, content migration, and simple data collection.

Benefits

80% Cost Savings: Only 2 credits per page vs 10 credits for AI mode
No AI/LLM Processing: Pure HTML to markdown conversion
Clean Output: Well-formatted markdown with metadata extraction
Fast Processing: No AI inference delays
Perfect for: Documentation, content archival, site migration

Quick Start - Markdown Mode

from scrapegraph_py import Client

client = Client(api_key="your-api-key")

# Markdown conversion mode - no prompt needed
response = client.crawl(
    url="https://scrapegraphai.com/",
    extraction_mode=False,  # False = Markdown conversion (NO AI/LLM)
    depth=2,
    breadth=None,
    max_pages=5,
    rules={
        "include_paths": ["/docs/*", "/blog/**"],
        "exclude_paths": ["/admin/*"],
        "same_domain": True
    }
)

import { crawl } from 'scrapegraph-js';

const apiKey = 'your-api-key';
const url = 'https://scrapegraphai.com';

try {
  // Markdown conversion mode - no prompt needed  
  const response = await crawl(apiKey, url, null, null, {
    extractionMode: false,  // false = Markdown conversion (NO AI/LLM)
    depth: 2,
    breadth: null,
    maxPages: 5,
    rules: {
      include_paths: ['/docs/*', '/blog/**'],
      exclude_paths: ['/admin/*'],
      same_domain: true
    }
  });
  console.log(response);
} catch (error) {
  console.error(error);
}

curl -X 'POST' \
  'https://api.scrapegraphai.com/v1/crawl' \
  -H 'accept: application/json' \
  -H 'SGAI-APIKEY: your-api-key' \
  -H 'Content-Type: application/json' \
  -d '{
  "url": "https://scrapegraphai.com/",
  "extraction_mode": false,
  "depth": 2,
  "breadth": null,
  "max_pages": 5,
  "rules": {
    "include_paths": ["/docs/*", "/blog/**"],
    "exclude_paths": ["/admin/*"],
    "same_domain": true
  }
}'

Markdown Mode Response

```json { "status": "success", "result": { "status": "done", "pages_processed": 5, "credits_used": 10, "crawled_urls": [ "https://scrapegraphai.com/", "https://scrapegraphai.com/about", "https://scrapegraphai.com/pricing" ], "pages": [ { "url": "https://scrapegraphai.com/", "title": "ScrapeGraphAI - AI-Powered Web Scraping", "markdown": "# Transform Websites into Structured Data\n\nScrapeGraphAI is the most complete web scraping library...", "metadata": { "word_count": 1250, "headers": ["Transform Websites", "Features", "Pricing"], "links_count": 25 } } ] } } ```

Crawl Rules

You can control the crawl behavior with the rules object:

rules = {
    "exclude": ["https://example.com/logout", "https://example.com/admin/*"],  # Regex patterns for full URLs to exclude
    "include_paths": ["/products/*", "/blog/**"],  # Path patterns to include
    "exclude_paths": ["/admin/*", "/api/**", "/private/*"],  # Path patterns to exclude (takes precedence)
    "same_domain": True  # Only crawl links on the same domain
}

Field	Type	Default	Description
exclude	list	[]	List of URL patterns (regex) to exclude from crawling. Matches full URL.
include_paths	list	[]	(Optional) List of path patterns to include (e.g., `["/products/", "/blog/"]`). Supports wildcards: `` matches any characters, `**` matches any path segments. If empty or not specified, all paths are included.
exclude_paths	list	[]	(Optional) List of path patterns to exclude (e.g., `["/admin/", "/api/"]`). Supports wildcards: `` matches any characters, `**` matches any path segments. Takes precedence over `include_paths`. If empty or not specified, no paths are excluded.
same_domain	bool	True	Restrict crawl to the same domain

Both `include_paths` and `exclude_paths` are optional. If `include_paths` is not specified or empty, all paths are included. If `exclude_paths` is not specified or empty, no paths are excluded. The `exclude_paths` patterns take precedence over `include_paths` patterns.

Example Response

```json { "status": "success", "result": { "status": "done", "llm_result": { "company": { "name": "ScrapeGraphAI, Inc", "description": "ScrapeGraphAI is a company that provides web scraping services using artificial intelligence...", "features": ["AI Agent Ready", "Universal Data Extraction", ...], "contact_email": "contact@scrapegraphai.com", "social_links": { "github": "https://github.com/ScrapeGraphAI/Scrapegraph-ai", "linkedin": "https://www.linkedin.com/company/101881123", "twitter": "https://x.com/scrapegraphai" } }, "services": [ {"service_name": "Markdownify", ...}, {"service_name": "Smart Scraper", ...} ], "legal": { "privacy_policy": "https://scrapegraphai.com/privacy", "terms_of_service": "https://scrapegraphai.com/terms" } }, "crawled_urls": [ "https://scrapegraphai.com/", ... ], "pages": [ { "url": "https://scrapegraphai.com/", "markdown": "# Transform Websites into Structured Data\n..." }, ... ] } } ```

llm_result: Structured extraction based on your prompt/schema
crawled_urls: List of all URLs visited
pages: List of objects with url and extracted markdown content

Retrieve a Previous Crawl

You can retrieve the result of a crawl job by its task ID:

result = client.get_crawl_result(task_id="your-task-id")

import { getCrawlResult } from 'scrapegraph-js';
const apiKey = 'your_api_key';
const taskId = 'your-task-id';
const result = await getCrawlResult(apiKey, taskId);

Parameters

Parameter	Type	Required	Description
apiKey	string	Yes	The ScrapeGraph API Key.
taskId	string	Yes	The crawl job task ID.

Custom Schema Example

Define exactly what data you want to extract from every page:

from pydantic import BaseModel, Field

class CompanyData(BaseModel):
    name: str = Field(description="Company name")
    description: str = Field(description="Description")
    features: list[str] = Field(description="Features")

response = client.crawl(
    url="https://example.com",
    prompt="Extract company info",
    schema=CompanyData,
    depth=1,
    breadth=None,
    max_pages=5,
    rules={
        "include_paths": ["/about", "/company/*"],
        "exclude_paths": ["/admin/*", "/private/**"],
        "same_domain": True
    }
)

import { smartCrawler } from 'scrapegraph-js';
import { z } from 'zod';

const CompanySchema = z.object({
  name: z.string().describe('Company name'),
  description: z.string().describe('Description'),
  features: z.array(z.string()).describe('Features')
});

const rules = {
  include_paths: ['/about', '/company/*'],
  exclude_paths: ['/admin/*', '/private/**'],
  same_domain: true
};

const response = await smartCrawler(apiKey, url, prompt, 1, null, 5, CompanySchema, rules);

Async Support

SmartCrawler supports async execution for large crawls:

import asyncio
from scrapegraph_py import AsyncClient

async def main():
    async with AsyncClient(api_key="your-api-key") as client:
        task = await client.crawl(
            url="https://scrapegraphai.com/",
            prompt="Extract info about the company",
            depth=2,
            breadth=None,
            max_pages=10,
            rules={
                "include_paths": ["/about", "/services/*"],
                "exclude_paths": ["/admin/*", "/api/**"],
                "same_domain": True
            }
        )
        # Poll for result
        result = await client.get_crawl_result(task["task_id"])
        print(result)

if __name__ == "__main__":
    asyncio.run(main())

import { AsyncSmartCrawler } from 'scrapegraph-js';
const scraper = new AsyncSmartCrawler(apiKey);
const task = await scraper.crawl({ 
  url, 
  prompt, 
  depth: 2, 
  breadth: null,
  maxPages: 10,
  rules: {
    include_paths: ['/about', '/services/*'],
    exclude_paths: ['/admin/*', '/api/**'],
    same_domain: true
  }
});
const result = await scraper.getResult(task.taskId);

Validation & Error Handling

SmartCrawler performs advanced validation:

Ensures either url or website_html is provided
Validates HTML size (max 2MB)
Checks for valid URLs and HTML structure
Handles empty or invalid prompts
Returns clear error messages for all validation failures

Endpoint Details

POST https://api.scrapegraphai.com/v1/crawl

Required Headers

Header	Description
SGAI-APIKEY	Your API authentication key
Content-Type	application/json

Request Body

Field	Type	Required	Description
url	string	Yes*	Starting URL (*or website_html required)
website_html	string	No	Raw HTML content (max 2MB)
prompt	string	Yes	Extraction instructions
schema	object	No	Output schema
headers	object	No	Custom headers
number_of_scrolls	int	No	Infinite scroll per page
depth	int	No	Crawl depth
breadth	int	No	Maximum number of links to crawl per depth level. If null/undefined, unlimited (default). Controls the 'width' of exploration at each depth. Useful for limiting crawl scope on large sites. Note: max_pages always takes priority. Ignored when sitemap=true.
max_pages	int	No	Max pages to crawl
rules	object	No	Crawl rules object with optional fields: `exclude` (regex URL patterns), `include_paths` (path patterns to include), `exclude_paths` (path patterns to exclude), `same_domain` (boolean)
sitemap	bool	No	Use sitemap.xml
wait_ms	int	No	Milliseconds to wait before scraping each page. Useful for pages with heavy JavaScript rendering (default: 3000).

Response Format

{
  "status": "success",
  "result": {
    "status": "done",
    "llm_result": { /* Structured extraction */ },
    "crawled_urls": ["..."],
    "pages": [ { "url": "...", "markdown": "..." }, ... ]
  }
}

Key Features

Crawl and extract from entire sites, not just single pages Contextual extraction across multiple pages Cost-effective HTML to markdown conversion (80% savings!) Fine-tune what gets crawled and extracted Define custom output schemas for structured results Get notified when crawl jobs complete with signed webhooks

Use Cases

AI Extraction Mode

Site-wide data extraction with smart understanding
Product catalog crawling with structured output
Legal/Privacy/Terms aggregation with AI parsing
Research and competitive analysis with insights
Multi-page blog/news scraping with content analysis

Markdown Conversion Mode

Website documentation archival and migration
Content backup and preservation (80% cheaper!)
Blog/article collection in markdown format
Site content analysis without AI overhead
Fast bulk content conversion for CMS migration

Best Practices

AI Extraction Mode

Be specific in your prompts for better results
Use schemas for structured output validation
Test prompts on single pages first
Include examples in your schema descriptions

Markdown Conversion Mode

Perfect for content archival and documentation
No prompt required - set extraction_mode: false
80% cheaper than AI mode (2 credits vs 10 per page)
Ideal for bulk content migration

General

Set reasonable max_pages and depth limits
Use rules with include_paths and exclude_paths to precisely control which pages to crawl
Use exclude for full URL regex patterns when needed
exclude_paths takes precedence over include_paths
Always handle errors and poll for results
Monitor your credit usage and rate limits

Webhooks

SmartCrawler supports webhook notifications for job completion. When you provide a webhook_url, you'll receive a POST request when the crawl finishes.

Setting Up Webhooks

Configure your webhook secret in the dashboard
Provide a webhook_url in your crawl request
Verify incoming webhooks using the X-Webhook-Signature header

response = client.crawl(
    url="https://example.com",
    prompt="Extract product data",
    webhook_url="https://your-server.com/webhook"
)

Each webhook request includes an `X-Webhook-Signature` header for verification. See the [API Reference](/api-reference/endpoint/smartcrawler/start#webhook-signature-verification) for detailed verification examples.

API Reference

For detailed API documentation, see:

Support & Resources

Comprehensive guides and tutorials Detailed API documentation Join our Discord community Check out our open-source projects Sign up now and get your API key to begin extracting data with SmartCrawler!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Overview

Getting Started

Quick Start

Required Headers

Parameters

Markdown Conversion Mode

Benefits

Quick Start - Markdown Mode

Markdown Mode Response

Crawl Rules

Example Response

Retrieve a Previous Crawl

Parameters

Custom Schema Example

Async Support

Validation & Error Handling

Endpoint Details

Required Headers

Request Body

Response Format

Key Features

Use Cases

AI Extraction Mode

Markdown Conversion Mode

Best Practices

AI Extraction Mode

Markdown Conversion Mode

General

Webhooks

Setting Up Webhooks

API Reference

Support & Resources

Uh oh!

FilesExpand file tree

smartcrawler.mdx

Latest commit

History

smartcrawler.mdx

File metadata and controls

Overview

Getting Started

Quick Start

Required Headers

Parameters

Markdown Conversion Mode

Benefits

Quick Start - Markdown Mode

Markdown Mode Response

Crawl Rules

Example Response

Retrieve a Previous Crawl

Parameters

Custom Schema Example

Async Support

Validation & Error Handling

Endpoint Details

Required Headers

Request Body

Response Format

Key Features

Use Cases

AI Extraction Mode

Markdown Conversion Mode

Best Practices

AI Extraction Mode

Markdown Conversion Mode

General

Webhooks

Setting Up Webhooks

API Reference

Support & Resources