You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs: add monitor.activity(), fix API base URL, update skill to just-scrape
- Add monitor.activity() tick-history docs to Python SDK, JS SDK, and MCP server
- Fix API base URL from /v2 to /api/v2 across all SDK and MCP docs
- Rename SGAI_TIMEOUT_S → SGAI_TIMEOUT (with legacy alias noted)
- Update Claude Code skill to reference just-scrape CLI and new install paths
- Bump MCP server tool count from 17 → 18
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: integrations/claude-code-skill.mdx
+45-39Lines changed: 45 additions & 39 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,14 +6,16 @@ icon: '/logo/claude-color.svg'
6
6
7
7
## Overview
8
8
9
-
The ScrapeGraphAI [Claude Code Skill](https://github.com/ScrapeGraphAI/skill) gives AI coding agents full access to ScrapeGraphAI's web scraping, search, and crawling APIs. Once installed, agents like Claude Code, Cursor, Copilot, and Cline can scrape websites, extract structured data, and crawl pages — all from natural language prompts.
9
+
The ScrapeGraphAI Claude Code Skill ships with [just-scrape](https://github.com/ScrapeGraphAI/just-scrape), the official CLI for the **v2 API**. Once installed, agents like Claude Code, Cursor, Copilot, Cline, and Windsurf can scrape websites, extract structured data, search the web, crawl sites, and set up page-change monitors — all from natural language prompts.
10
+
11
+
The skill wires `just-scrape` into your agent's skill directory so the agent knows when and how to invoke the CLI.
Set your ScrapeGraphAI API key as an environment variable:
62
+
Install the CLI and set your ScrapeGraphAI API key:
57
63
58
64
```bash
65
+
npm install -g just-scrape@latest
59
66
export SGAI_API_KEY="sgai-..."
60
67
```
61
68
62
69
<Note>
63
-
Get your API key from the [dashboard](https://scrapegraphai.com/dashboard).
70
+
Get your API key from the [dashboard](https://scrapegraphai.com/dashboard). The CLI also accepts the key via a `.env` file, `~/.scrapegraphai/config.json`, or an interactive prompt.
64
71
</Note>
65
72
66
-
## What's Included
67
-
68
-
The skill installs the following files:
69
-
70
-
| File | Description |
71
-
|------|-------------|
72
-
|`SKILL.md`| Main skill file with API reference, examples, and decision guide |
73
-
|`references/api-endpoints.md`| Full parameter tables for all endpoints |
74
-
|`references/sdk-examples.md`| Python and JavaScript SDK examples |
75
-
|`references/advanced-features.md`| Stealth mode, schemas, scrolling, pagination, and more |
76
-
77
73
## Capabilities
78
74
75
+
The skill maps to the v2 API surface via `just-scrape`:
Search the web and extract results with AI or as markdown
81
+
<Cardtitle="search"icon="magnifying-glass">
82
+
Search the web and extract structured results (`just-scrape search`)
85
83
</Card>
86
-
<Cardtitle="Markdownify"icon="file-code">
87
-
Convert any webpage into clean, formatted markdown
84
+
<Cardtitle="scrape"icon="file-code">
85
+
Fetch a page in 8 formats: markdown, html, screenshot, branding, links, images, summary, json
88
86
</Card>
89
-
<Cardtitle="SmartCrawler"icon="spider">
90
-
Crawl multiple pages from a website with depth and path controls
87
+
<Cardtitle="markdownify"icon="file-lines">
88
+
Convert any webpage into clean markdown (wraps `scrape -f markdown`)
91
89
</Card>
92
-
<Cardtitle="Sitemap"icon="sitemap">
93
-
Extract all URLs from a website's sitemap
90
+
<Cardtitle="crawl"icon="spider">
91
+
Crawl multi-page sites with depth, link, and pattern controls
94
92
</Card>
95
-
<Cardtitle="Agentic Scraper"icon="robot">
96
-
Browser automation — login, click, navigate, fill forms, then extract
93
+
<Cardtitle="monitor"icon="clock">
94
+
Schedule page-change monitors with cron intervals, webhooks, and activity polling
97
95
</Card>
98
96
</CardGroup>
99
97
98
+
<Note>
99
+
Removed from v1: `sitemap`, `agentic_scraper`, `generate-schema`, `validate`. There is no direct replacement on v2.
100
+
</Note>
101
+
100
102
## Example Prompts
101
103
102
104
Once the skill is installed, you can use natural language prompts directly in your AI coding agent:
@@ -118,14 +120,18 @@ Crawl https://example.com/blog with depth 2 and extract the title and summary fr
118
120
```
119
121
120
122
```text
121
-
Get all URLs from the sitemap of https://example.com
123
+
Monitor https://store.example.com/pricing every hour and webhook me when it changes
124
+
```
125
+
126
+
```text
127
+
Create a 30m monitor on https://example.com and poll its activity feed, printing new ticks as they come in
122
128
```
123
129
124
130
```text
125
-
Log into https://example.com/dashboard, click "Reports", and extract the table data
131
+
Fetch a full-page screenshot and branding assets for https://example.com
126
132
```
127
133
128
-
The agent will automatically select the right ScrapeGraphAI endpoint, handle authentication, poll for async results, and return structured data.
134
+
The agent will automatically select the right `just-scrape` command, handle authentication, poll for async results (crawls), and return structured data.
`MonitorActivityRequest` fields: `limit` (1–100, default `20`) and optional `cursor` for pagination. Each `MonitorTickEntry` exposes `id`, `created_at`, `status`, `changed`, `elapsed_ms`, and a `diffs` model with per-format deltas.
Copy file name to clipboardExpand all lines: services/mcp-server.mdx
+12-4Lines changed: 12 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -22,8 +22,8 @@ A production‑ready Model Context Protocol (MCP) server that connects LLMs to t
22
22
23
23
## Key Features
24
24
25
-
- Full v2 API coverage: scrape, extract, search, crawl (+ stop/resume), monitor lifecycle, credits, history, and schema generation
26
-
- Uses the v2 API base URL (`https://api.scrapegraphai.com/v2`) with the `SGAI-APIKEY` header — wire format matches [scrapegraph-py v2](https://github.com/ScrapeGraphAI/scrapegraph-py/pull/84)
25
+
- Full v2 API coverage: scrape, extract, search, crawl (+ stop/resume), monitor lifecycle (+ activity polling), credits, history, and schema generation
26
+
- Uses the v2 API base URL (`https://api.scrapegraphai.com/api/v2`) with the `SGAI-APIKEY` header — wire format matches [scrapegraph-py v2](https://github.com/ScrapeGraphAI/scrapegraph-py/pull/84)
27
27
- Remote HTTP MCP endpoint and local Python server support
28
28
- Works with Cursor, Claude Desktop, and any MCP‑compatible client
29
29
- Robust error handling, timeouts, and production‑tested reliability
@@ -176,9 +176,10 @@ The server reads the ScrapeGraph API key from `SGAI_API_KEY` (local) or the `X-A
176
176
| Variable | Description | Default |
177
177
|---|---|---|
178
178
|`SGAI_API_KEY`| ScrapeGraph API key | — |
179
-
|`SGAI_API_URL`| Override the v2 API base URL |`https://api.scrapegraphai.com/v2`|
180
-
|`SGAI_TIMEOUT_S`| Request timeout in seconds |`120`|
179
+
|`SGAI_API_URL`| Override the v2 API base URL |`https://api.scrapegraphai.com/api/v2`|
180
+
|`SGAI_TIMEOUT`| Request timeout in seconds |`120`|
181
181
|`SCRAPEGRAPH_API_BASE_URL`| Legacy alias for `SGAI_API_URL` (still honored) | — |
182
+
|`SGAI_TIMEOUT_S`| Legacy alias for `SGAI_TIMEOUT` (still honored) | — |
182
183
183
184
## Available Tools
184
185
@@ -316,8 +317,15 @@ monitor_get(monitor_id: str)
316
317
monitor_pause(monitor_id: str)
317
318
monitor_resume(monitor_id: str)
318
319
monitor_delete(monitor_id: str)
320
+
monitor_activity(
321
+
monitor_id: str,
322
+
limit: int|None=None, # 1–100, default 20
323
+
cursor: str|None=None, # pagination cursor
324
+
)
319
325
```
320
326
327
+
`monitor_activity` returns the tick history (`id`, `createdAt`, `status`, `changed`, `elapsedMs`, `diffs`) plus a `nextCursor` when more results are available — mirrors `sgai.monitor.activity()` in the SDKs.
0 commit comments