Skip to content

Commit d5211ae

Browse files
VinciGit00claude
VinciGit00
and
claude
committed
feat: align CLI with scrapegraph-js v2 PR #13 (0738786)
- Replace SDK factory with raw function imports (scrape, extract, search, crawl, monitor, history, getCredits) - Add monitor command (create, list, get, update, delete, pause, resume) - Update crawl to use formats array and crawl.get instead of crawl.status - Update history to use history.list/history.get with new pagination response - Update search to pass query in params, remove nationality flag - Update extract to pass url in params - Make history service filter optional - Update README with monitor docs and v2 migration notes Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 9f98553 commit d5211ae

13 files changed

Lines changed: 452 additions & 145 deletions

File tree

‎README.md‎

Lines changed: 71 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ Made with love by the [ScrapeGraphAI team](https://scrapegraphai.com) 💜
55

66
![Demo Video](/assets/demo.gif)
77

8-
Command-line interface for [ScrapeGraph AI](https://scrapegraphai.com) — AI-powered web scraping, data extraction, search, and crawling. Uses the **v2 API**.
8+
Command-line interface for [ScrapeGraph AI](https://scrapegraphai.com) — AI-powered web scraping, data extraction, search, crawling, and monitoring. Uses the **v2 API**.
99

1010
## Project Structure
1111

@@ -14,7 +14,7 @@ just-scrape/
1414
├── src/
1515
│ ├── cli.ts # Entry point, citty main command + subcommands
1616
│ ├── lib/
17-
│ │ ├── client.ts # ScrapeGraphAI v2 client factory
17+
│ │ ├── client.ts # API key resolver
1818
│ │ ├── env.ts # Env config (API key, JUST_SCRAPE_* → SGAI_* bridge)
1919
│ │ ├── folders.ts # API key resolution + interactive prompt
2020
│ │ └── log.ts # Logger factory + syntax-highlighted JSON output
@@ -24,6 +24,7 @@ just-scrape/
2424
│ │ ├── scrape.ts
2525
│ │ ├── markdownify.ts
2626
│ │ ├── crawl.ts
27+
│ │ ├── monitor.ts
2728
│ │ ├── history.ts
2829
│ │ └── credits.ts
2930
│ └── utils/
@@ -87,22 +88,23 @@ Legacy variables (`JUST_SCRAPE_API_URL`, `JUST_SCRAPE_TIMEOUT_S`, `JUST_SCRAPE_D
8788
All commands support `--json` for machine-readable output. When set, banner, spinners, and interactive prompts are suppressed — only minified JSON on stdout (saves tokens when piped to AI agents).
8889

8990
```bash
90-
just-scrape credits --json | jq '.remainingCredits'
91+
just-scrape credits --json | jq '.remaining'
9192
just-scrape extract https://example.com -p "Extract data" --json > result.json
92-
just-scrape history extract --json | jq '.[].status'
93+
just-scrape history scrape --json | jq '.[].status'
9394
```
9495

9596
---
9697

9798
## Extract
9899

99-
Extract structured data from any URL using AI (replaces `smart-scraper`). [docs](https://docs.scrapegraphai.com/api-reference/extract)
100+
Extract structured data from any URL using AI. [docs](https://docs.scrapegraphai.com/api-reference/extract)
100101

101102
### Usage
102103

103104
```bash
104105
just-scrape extract <url> -p <prompt> # Extract data with AI
105106
just-scrape extract <url> -p <prompt> --schema <json> # Enforce output schema
107+
just-scrape extract <url> -p <prompt> --mode <mode> # HTML mode: normal, reader, prune
106108
just-scrape extract <url> -p <prompt> --scrolls <n> # Infinite scroll (0-100)
107109
just-scrape extract <url> -p <prompt> --mode js --stealth # Anti-bot bypass
108110
just-scrape extract <url> -p <prompt> --cookies <json> --headers <json>
@@ -122,21 +124,21 @@ just-scrape extract https://news.example.com -p "Get all article headlines and d
122124

123125
# Scrape a JS-heavy SPA with stealth mode
124126
just-scrape extract https://app.example.com/dashboard -p "Extract user stats" \
125-
--mode js --stealth
127+
--stealth
126128
```
127129

128130
## Search
129131

130-
Search the web and extract structured data from results (replaces `search-scraper`). [docs](https://docs.scrapegraphai.com/api-reference/search)
132+
Search the web and extract structured data from results. [docs](https://docs.scrapegraphai.com/api-reference/search)
131133

132134
### Usage
133135

134136
```bash
135137
just-scrape search <query> # AI-powered web search
136138
just-scrape search <query> --num-results <n> # Sources to scrape (1-20, default 3)
137139
just-scrape search <query> -p <prompt> # Extraction prompt for results
138-
just-scrape search <query> --schema <json> # Enforce output schema
139-
just-scrape search <query> --country <code> # Geo-target search (e.g. 'us', 'de', 'jp')
140+
just-scrape search <query> --schema <json> # Enforce output schema (requires -p)
141+
just-scrape search <query> --country <code> # Geo-target search (e.g. 'us', 'de', 'jp')
140142
just-scrape search <query> --time-range <range> # past_hour | past_24_hours | past_week | past_month | past_year
141143
just-scrape search <query> --format <markdown|html> # Result format (default markdown)
142144
just-scrape search <query> --headers <json>
@@ -153,6 +155,7 @@ just-scrape search "EU AI act latest news" --time-range past_week --country de
153155

154156
# Structured output with schema
155157
just-scrape search "Top 5 cloud providers pricing" \
158+
-p "Extract provider name and free tier details" \
156159
--schema '{"type":"object","properties":{"providers":{"type":"array","items":{"type":"object","properties":{"name":{"type":"string"},"free_tier":{"type":"string"}}}}}}'
157160
```
158161

@@ -219,12 +222,12 @@ just-scrape markdownify https://blog.example.com/my-article
219222
just-scrape markdownify https://protected.example.com -m js --stealth
220223

221224
# Pipe markdown to a file
222-
just-scrape markdownify https://docs.example.com/api --json | jq -r '.markdown' > api-docs.md
225+
just-scrape markdownify https://docs.example.com/api --json | jq -r '.results.markdown.data[0]' > api-docs.md
223226
```
224227

225228
## Crawl
226229

227-
Crawl multiple pages. The CLI starts the crawl and polls until completion. [docs](https://docs.scrapegraphai.com/api-reference/crawl)
230+
Crawl multiple pages. The CLI starts the crawl and polls until completion. Supports the same format options as scrape. [docs](https://docs.scrapegraphai.com/api-reference/crawl)
228231

229232
### Usage
230233

@@ -235,6 +238,7 @@ just-scrape crawl <url> --max-depth <n> # Crawl depth (default 2)
235238
just-scrape crawl <url> --max-links-per-page <n> # Links per page (default 10)
236239
just-scrape crawl <url> --allow-external # Allow external domains
237240
just-scrape crawl <url> -f html # Page format (default markdown)
241+
just-scrape crawl <url> -f markdown,links # Multi-format (comma-separated)
238242
just-scrape crawl <url> -m js --stealth # Anti-bot bypass
239243
```
240244

@@ -251,21 +255,63 @@ just-scrape crawl https://example.com --max-pages 50
251255
just-scrape crawl https://example.com --json --max-pages 10
252256
```
253257

258+
## Monitor
259+
260+
Create and manage page-change monitors. Monitors periodically scrape a URL and detect changes. [docs](https://docs.scrapegraphai.com/api-reference/monitor)
261+
262+
### Usage
263+
264+
```bash
265+
just-scrape monitor create --url <url> --interval <interval> # Create a monitor
266+
just-scrape monitor create --url <url> --interval 1h --name "My Monitor"
267+
just-scrape monitor create --url <url> --interval 30m --webhook-url <url>
268+
just-scrape monitor create --url <url> --interval 1d -f markdown,screenshot
269+
just-scrape monitor list # List all monitors
270+
just-scrape monitor get --id <id> # Get monitor details
271+
just-scrape monitor update --id <id> --interval 2h # Update interval
272+
just-scrape monitor pause --id <id> # Pause a monitor
273+
just-scrape monitor resume --id <id> # Resume a paused monitor
274+
just-scrape monitor delete --id <id> # Delete a monitor
275+
```
276+
277+
### Examples
278+
279+
```bash
280+
# Monitor a pricing page every hour
281+
just-scrape monitor create --url https://store.example.com/pricing --interval 1h
282+
283+
# Monitor with webhook notification
284+
just-scrape monitor create --url https://example.com \
285+
--interval 30m --webhook-url https://hooks.example.com/notify
286+
287+
# Monitor markdown + screenshot changes daily
288+
just-scrape monitor create --url https://example.com \
289+
--interval 1d -f markdown,screenshot --name "Daily check"
290+
291+
# List all monitors
292+
just-scrape monitor list
293+
294+
# Pause and resume
295+
just-scrape monitor pause --id abc123
296+
just-scrape monitor resume --id abc123
297+
```
298+
254299
## History
255300

256-
Browse request history for any service. Interactive by default — arrow keys to navigate, select to view details, "Load more" for pagination.
301+
Browse request history. Interactive by default — arrow keys to navigate, select to view details, "Load more" for pagination. Service filter is optional.
257302

258303
### Usage
259304

260305
```bash
261-
just-scrape history <service> # Interactive browser
262-
just-scrape history <service> <request-id> # Fetch specific request
263-
just-scrape history <service> --page <n> # Start from page (default 1)
264-
just-scrape history <service> --page-size <n> # Results per page (default 20, max 100)
265-
just-scrape history <service> --json # Raw JSON (pipeable)
306+
just-scrape history # All history (interactive)
307+
just-scrape history <service> # Filter by service
308+
just-scrape history <service> <request-id> # Fetch specific request by ID
309+
just-scrape history --page <n> # Start from page (default 1)
310+
just-scrape history --page-size <n> # Results per page (default 20, max 100)
311+
just-scrape history --json # Raw JSON (pipeable)
266312
```
267313

268-
Services: `scrape`, `extract`, `search`, `monitor`, `crawl`
314+
Services: `scrape`, `extract`, `schema`, `search`, `monitor`, `crawl`
269315

270316
### Examples
271317

@@ -274,10 +320,10 @@ Services: `scrape`, `extract`, `search`, `monitor`, `crawl`
274320
just-scrape history extract
275321

276322
# Jump to a specific request by ID
277-
just-scrape history extract abc123-def456-7890
323+
just-scrape history scrape abc123-def456-7890
278324

279-
# Export crawl history as JSON
280-
just-scrape history crawl --json --page-size 100 | jq '.[].status'
325+
# Export all history as JSON
326+
just-scrape history --json --page-size 100 | jq '.[].status'
281327
```
282328

283329
## Credits
@@ -286,7 +332,7 @@ Check your credit balance.
286332

287333
```bash
288334
just-scrape credits
289-
just-scrape credits --json | jq '.remainingCredits'
335+
just-scrape credits --json | jq '.remaining'
290336
```
291337

292338
---
@@ -301,11 +347,12 @@ Commands have been renamed to match the v2 API:
301347
| `search-scraper` | `search` | Renamed |
302348
| `markdownify` | `markdownify` | Now wraps `scrape --format markdown` |
303349
| `scrape` | `scrape` | Gains `--format` (markdown, html, screenshot, branding, links, images, summary, json), multi-format via comma, `--html-mode`, `--scrolls`, `--prompt`, `--schema` |
304-
| `crawl` | `crawl` | New options: `--max-depth`, `--max-links-per-page`, `--allow-external`, `--format` |
350+
| `crawl` | `crawl` | Now uses `formats` array like scrape, supports multi-format |
305351
| `search` | `search` | New options: `--country`, `--time-range`, `--format` |
352+
| — | `monitor` | **New**: create, list, get, update, delete, pause, resume page-change monitors |
306353
| `--stealth` flag | `--stealth` | Separate boolean flag; fetch mode is now `auto`, `fast`, or `js` |
307354
| `agentic-scraper` | — | Removed from API |
308-
| `generate-schema` | — | Removed from API |
355+
| `generate-schema` | — | Removed from CLI (still available in SDK) |
309356
| `sitemap` | — | Removed from API |
310357
| `validate` | — | Removed from API |
311358

‎bun.lock‎

Lines changed: 4 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎package.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@
2828
"chalk": "^5.4.1",
2929
"citty": "^0.1.6",
3030
"dotenv": "^17.2.4",
31-
"scrapegraph-js": "github:ScrapeGraphAI/scrapegraph-js#016ae8b"
31+
"scrapegraph-js": "github:ScrapeGraphAI/scrapegraph-js#0738786"
3232
},
3333
"devDependencies": {
3434
"@biomejs/biome": "^1.9.4",

‎src/cli.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ const main = defineCommand({
1717
scrape: () => import("./commands/scrape.js").then((m) => m.default),
1818
markdownify: () => import("./commands/markdownify.js").then((m) => m.default),
1919
crawl: () => import("./commands/crawl.js").then((m) => m.default),
20+
monitor: () => import("./commands/monitor.js").then((m) => m.default),
2021
history: () => import("./commands/history.js").then((m) => m.default),
2122
credits: () => import("./commands/credits.js").then((m) => m.default),
2223
},

‎src/commands/crawl.ts‎

Lines changed: 33 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import { defineCommand } from "citty";
2-
import { createClient } from "../lib/client.js";
2+
import { crawl } from "scrapegraph-js";
3+
import { getApiKey } from "../lib/client.js";
34
import * as log from "../lib/log.js";
45

56
const POLL_INTERVAL_MS = 3000;
@@ -23,7 +24,7 @@ export default defineCommand({
2324
type: "string",
2425
alias: "f",
2526
description:
26-
"Page format: markdown (default), html, screenshot, branding, links, images, summary",
27+
"Page format: markdown (default), html, screenshot, branding, links, images, summary. Comma-separate for multi-format.",
2728
},
2829
mode: {
2930
type: "string",
@@ -36,51 +37,58 @@ export default defineCommand({
3637
run: async ({ args }) => {
3738
const out = log.create(!!args.json);
3839
out.docs("https://docs.scrapegraphai.com/api-reference/crawl");
39-
const sgai = await createClient(!!args.json);
40+
const apiKey = await getApiKey(!!args.json);
41+
42+
const requestedFormats = (args.format ?? "markdown")
43+
.split(",")
44+
.map((f) => f.trim())
45+
.filter(Boolean);
46+
47+
const formats = requestedFormats.map((f) => {
48+
if (f === "markdown" || f === "html") return { type: f as "markdown" | "html", mode: "normal" as const };
49+
return { type: f };
50+
});
51+
52+
const params: Record<string, unknown> = { url: args.url, formats };
53+
if (args["max-pages"]) params.maxPages = Number(args["max-pages"]);
54+
if (args["max-depth"]) params.maxDepth = Number(args["max-depth"]);
55+
if (args["max-links-per-page"]) params.maxLinksPerPage = Number(args["max-links-per-page"]);
56+
if (args["allow-external"]) params.allowExternal = true;
4057

41-
const crawlOptions: Record<string, unknown> = {};
42-
if (args["max-pages"]) crawlOptions.maxPages = Number(args["max-pages"]);
43-
if (args["max-depth"]) crawlOptions.maxDepth = Number(args["max-depth"]);
44-
if (args["max-links-per-page"])
45-
crawlOptions.maxLinksPerPage = Number(args["max-links-per-page"]);
46-
if (args["allow-external"]) crawlOptions.allowExternal = true;
47-
if (args.format) crawlOptions.format = args.format;
4858
const fetchConfig: Record<string, unknown> = {};
4959
if (args.mode) fetchConfig.mode = args.mode;
5060
if (args.stealth) fetchConfig.stealth = true;
51-
if (Object.keys(fetchConfig).length > 0) crawlOptions.fetchConfig = fetchConfig;
61+
if (Object.keys(fetchConfig).length > 0) params.fetchConfig = fetchConfig;
5262

5363
out.start("Crawling");
54-
const t0 = performance.now();
5564
try {
56-
const job = await sgai.crawl.start(args.url, crawlOptions as any);
57-
const jobId = (job.data as { id: string }).id;
65+
const job = await crawl.start(apiKey, params as any);
66+
const jobData = job.data as { id: string; status: string } | null;
5867

59-
if (!jobId) {
60-
out.stop(Math.round(performance.now() - t0));
68+
if (!jobData?.id) {
69+
out.stop(job.elapsedMs);
6170
out.result(job.data);
6271
return;
6372
}
6473

65-
// Poll until the crawl completes
6674
while (true) {
6775
await new Promise((r) => setTimeout(r, POLL_INTERVAL_MS));
68-
const status = await sgai.crawl.status(jobId);
69-
const statusData = status.data as { status: string; [key: string]: unknown };
70-
out.poll(statusData.status);
76+
const status = await crawl.get(apiKey, jobData.id);
77+
const statusData = status.data as { status: string; [key: string]: unknown } | null;
78+
out.poll(statusData?.status ?? "unknown");
7179

7280
if (
73-
statusData.status === "completed" ||
74-
statusData.status === "failed" ||
75-
statusData.status === "cancelled"
81+
statusData?.status === "completed" ||
82+
statusData?.status === "failed" ||
83+
statusData?.status === "deleted"
7684
) {
77-
out.stop(Math.round(performance.now() - t0));
85+
out.stop(job.elapsedMs + status.elapsedMs);
7886
out.result(status.data);
7987
return;
8088
}
8189
}
8290
} catch (err) {
83-
out.stop(Math.round(performance.now() - t0));
91+
out.stop(0);
8492
out.error(err instanceof Error ? err.message : String(err));
8593
}
8694
},

‎src/commands/credits.ts‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import { defineCommand } from "citty";
2-
import { createClient } from "../lib/client.js";
2+
import { getCredits } from "scrapegraph-js";
3+
import { getApiKey } from "../lib/client.js";
34
import * as log from "../lib/log.js";
45

56
export default defineCommand({
@@ -12,16 +13,15 @@ export default defineCommand({
1213
},
1314
run: async ({ args }) => {
1415
const out = log.create(!!args.json);
15-
const sgai = await createClient(!!args.json);
16+
const apiKey = await getApiKey(!!args.json);
1617

1718
out.start("Fetching credits");
18-
const t0 = performance.now();
1919
try {
20-
const result = await sgai.credits();
21-
out.stop(Math.round(performance.now() - t0));
20+
const result = await getCredits(apiKey);
21+
out.stop(result.elapsedMs);
2222
out.result(result.data);
2323
} catch (err) {
24-
out.stop(Math.round(performance.now() - t0));
24+
out.stop(0);
2525
out.error(err instanceof Error ? err.message : String(err));
2626
}
2727
},

0 commit comments

Comments
 (0)