|
| 1 | +--- |
| 2 | +title: 'Claude Code Skill' |
| 3 | +description: 'Give Claude Code and AI coding agents the power of web scraping' |
| 4 | +icon: '/logo/claude-color.svg' |
| 5 | +--- |
| 6 | + |
| 7 | +## Overview |
| 8 | + |
| 9 | +The ScrapeGraphAI [Claude Code Skill](https://github.com/ScrapeGraphAI/skill) gives AI coding agents full access to ScrapeGraphAI's web scraping, search, and crawling APIs. Once installed, agents like Claude Code, Cursor, Copilot, and Cline can scrape websites, extract structured data, and crawl pages — all from natural language prompts. |
| 10 | + |
| 11 | +<Card |
| 12 | + title="GitHub Repository" |
| 13 | + icon="github" |
| 14 | + href="https://github.com/ScrapeGraphAI/skill" |
| 15 | +> |
| 16 | + View the skill source code and documentation |
| 17 | +</Card> |
| 18 | + |
| 19 | +## Installation |
| 20 | + |
| 21 | +<Steps> |
| 22 | + |
| 23 | +### Option 1: Install via skills.sh (Recommended) |
| 24 | + |
| 25 | +The fastest way to install. Requires [Node.js](https://nodejs.org). |
| 26 | + |
| 27 | +```bash |
| 28 | +npx skills add ScrapeGraphAI/skill |
| 29 | +``` |
| 30 | + |
| 31 | +This clones the skill and symlinks it into your `~/.claude/skills/` directory automatically. |
| 32 | + |
| 33 | +### Option 2: Manual install |
| 34 | + |
| 35 | +Clone the repository and create the symlink yourself: |
| 36 | + |
| 37 | +```bash |
| 38 | +git clone https://github.com/ScrapeGraphAI/skill.git ~/.claude/skills/scrapegraphai |
| 39 | +ln -sf ~/.claude/skills/scrapegraphai/SKILL.md ~/.claude/skills/scrapegraphai.md |
| 40 | +``` |
| 41 | + |
| 42 | +### Option 3: Project-level install |
| 43 | + |
| 44 | +Install the skill for a single project only: |
| 45 | + |
| 46 | +```bash |
| 47 | +mkdir -p .claude/skills |
| 48 | +git clone https://github.com/ScrapeGraphAI/skill.git .claude/skills/scrapegraphai |
| 49 | +ln -sf .claude/skills/scrapegraphai/SKILL.md .claude/skills/scrapegraphai.md |
| 50 | +``` |
| 51 | + |
| 52 | +</Steps> |
| 53 | + |
| 54 | +## Setup |
| 55 | + |
| 56 | +Set your ScrapeGraphAI API key as an environment variable: |
| 57 | + |
| 58 | +```bash |
| 59 | +export SGAI_API_KEY="sgai-..." |
| 60 | +``` |
| 61 | + |
| 62 | +<Note> |
| 63 | +Get your API key from the [dashboard](https://dashboard.scrapegraphai.com). |
| 64 | +</Note> |
| 65 | + |
| 66 | +## What's Included |
| 67 | + |
| 68 | +The skill installs the following files: |
| 69 | + |
| 70 | +| File | Description | |
| 71 | +|------|-------------| |
| 72 | +| `SKILL.md` | Main skill file with API reference, examples, and decision guide | |
| 73 | +| `references/api-endpoints.md` | Full parameter tables for all endpoints | |
| 74 | +| `references/sdk-examples.md` | Python and JavaScript SDK examples | |
| 75 | +| `references/advanced-features.md` | Stealth mode, schemas, scrolling, pagination, and more | |
| 76 | + |
| 77 | +## Capabilities |
| 78 | + |
| 79 | +<CardGroup cols={2}> |
| 80 | + <Card title="SmartScraper" icon="wand-magic-sparkles"> |
| 81 | + Extract structured data from any webpage using natural language prompts |
| 82 | + </Card> |
| 83 | + <Card title="SearchScraper" icon="magnifying-glass"> |
| 84 | + Search the web and extract results with AI or as markdown |
| 85 | + </Card> |
| 86 | + <Card title="Markdownify" icon="file-code"> |
| 87 | + Convert any webpage into clean, formatted markdown |
| 88 | + </Card> |
| 89 | + <Card title="SmartCrawler" icon="spider"> |
| 90 | + Crawl multiple pages from a website with depth and path controls |
| 91 | + </Card> |
| 92 | + <Card title="Sitemap" icon="sitemap"> |
| 93 | + Extract all URLs from a website's sitemap |
| 94 | + </Card> |
| 95 | + <Card title="Agentic Scraper" icon="robot"> |
| 96 | + Browser automation — login, click, navigate, fill forms, then extract |
| 97 | + </Card> |
| 98 | +</CardGroup> |
| 99 | + |
| 100 | +## Example Prompts |
| 101 | + |
| 102 | +Once the skill is installed, you can use natural language prompts directly in your AI coding agent: |
| 103 | + |
| 104 | +```text |
| 105 | +Scrape https://example.com/products and extract all product names, prices, and ratings |
| 106 | +``` |
| 107 | + |
| 108 | +```text |
| 109 | +Search the web for "best React UI libraries 2025" and extract the top 5 with pros and cons |
| 110 | +``` |
| 111 | + |
| 112 | +```text |
| 113 | +Convert https://example.com/docs/getting-started to markdown and save it to docs/ |
| 114 | +``` |
| 115 | + |
| 116 | +```text |
| 117 | +Crawl https://example.com/blog with depth 2 and extract the title and summary from each post |
| 118 | +``` |
| 119 | + |
| 120 | +```text |
| 121 | +Get all URLs from the sitemap of https://example.com |
| 122 | +``` |
| 123 | + |
| 124 | +```text |
| 125 | +Log into https://example.com/dashboard, click "Reports", and extract the table data |
| 126 | +``` |
| 127 | + |
| 128 | +The agent will automatically select the right ScrapeGraphAI endpoint, handle authentication, poll for async results, and return structured data. |
| 129 | + |
| 130 | +## Supported Agents |
| 131 | + |
| 132 | +This skill works with any AI coding agent that supports the skills protocol: |
| 133 | + |
| 134 | +- **Claude Code** — Anthropic's CLI agent |
| 135 | +- **Cursor** — AI-powered code editor |
| 136 | +- **GitHub Copilot** — AI pair programmer |
| 137 | +- **Cline** — VS Code AI agent |
| 138 | +- **Windsurf** — AI coding assistant |
| 139 | +- And any other agent supporting [skills.sh](https://skills.sh) |
| 140 | + |
| 141 | +## Support |
| 142 | + |
| 143 | +Need help with the skill? |
| 144 | + |
| 145 | +<CardGroup cols={2}> |
| 146 | + <Card |
| 147 | + title="GitHub Issues" |
| 148 | + icon="github" |
| 149 | + href="https://github.com/ScrapeGraphAI/skill/issues" |
| 150 | + > |
| 151 | + Report bugs and request features |
| 152 | + </Card> |
| 153 | + <Card |
| 154 | + title="Discord Community" |
| 155 | + icon="discord" |
| 156 | + href="https://discord.gg/uJN7TYcpNa" |
| 157 | + > |
| 158 | + Get help from our community |
| 159 | + </Card> |
| 160 | +</CardGroup> |
0 commit comments