Skip to content

Commit be3b8d9

Browse files
FrancescoSaverioZuppichiniFrancescoSaverioZuppichini
committed
feat: add AI SDK tools and crawl pages support
1 parent aeb2ea8 commit be3b8d9

16 files changed

Lines changed: 1828 additions & 558 deletions

‎README.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,16 @@ bun add scrapegraph-js
2121

2222
## Quick Start
2323

24+
### API key
25+
26+
Log in to the [ScrapeGraphAI dashboard](https://scrapegraphai.com/) to create an API key. The dashboard also shows your request history, usage, credits, and crawl/monitor activity.
27+
28+
Set it in your environment:
29+
30+
```bash
31+
export SGAI_API_KEY=...
32+
```
33+
2434
```ts
2535
import { ScrapeGraphAI } from "scrapegraph-js";
2636

@@ -140,6 +150,12 @@ const start = await sgai.crawl.start({
140150
// Check status
141151
const status = await sgai.crawl.get(start.data?.id!);
142152

153+
// Fetch paginated pages with resolved scrape results
154+
const pages = await sgai.crawl.pages(start.data?.id!, {
155+
cursor: 0,
156+
limit: 50,
157+
});
158+
143159
// Control
144160
await sgai.crawl.stop(id);
145161
await sgai.crawl.resume(id);

‎package.json‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,17 @@
11
{
22
"name": "scrapegraph-js",
3-
"version": "2.1.0",
3+
"version": "2.2.0",
44
"description": "Official JavaScript/TypeScript SDK for the ScrapeGraph AI API — smart web scraping powered by AI",
55
"type": "module",
66
"main": "dist/index.js",
77
"types": "dist/index.d.ts",
88
"exports": {
99
".": {
10-
"import": "./dist/index.js",
11-
"types": "./dist/index.d.ts"
10+
"types": "./dist/index.d.ts",
11+
"import": "./dist/index.js"
1212
}
1313
},
14+
"workspaces": ["packages/*"],
1415
"scripts": {
1516
"dev": "tsup --watch",
1617
"build": "tsup",

‎packages/ai-sdk/README.md‎

Lines changed: 200 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,200 @@
1+
# ScrapeGraphAI AI SDK Tools
2+
3+
[![npm version](https://badge.fury.io/js/%40scrapegraphai%2Fai-sdk.svg)](https://www.npmjs.com/package/@scrapegraphai/ai-sdk)
4+
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
5+
6+
<p align="center">
7+
<a href="https://scrapegraphai.com">
8+
<img src="../../media/banner.png" alt="ScrapeGraphAI AI SDK Tools" style="width: 100%;">
9+
</a>
10+
</p>
11+
12+
Vercel [AI SDK](https://ai-sdk.dev/docs/introduction) tools for the ScrapeGraphAI API.
13+
14+
## Install
15+
16+
```bash
17+
npm i @scrapegraphai/ai-sdk ai
18+
# or
19+
bun add @scrapegraphai/ai-sdk ai
20+
```
21+
22+
`ai` is a peer dependency. Install the model provider package you use, for example:
23+
24+
```bash
25+
npm i @ai-sdk/openai
26+
# or
27+
bun add @ai-sdk/openai
28+
```
29+
30+
## Quick Start
31+
32+
### API key
33+
34+
Log in to the [ScrapeGraphAI dashboard](https://scrapegraphai.com/) to create an API key. The dashboard also shows your request history, usage, credits, and crawl/monitor activity.
35+
36+
Set it in your environment:
37+
38+
```bash
39+
export SGAI_API_KEY=...
40+
```
41+
42+
Minimal scrape-only setup:
43+
44+
```ts
45+
import { openai } from "@ai-sdk/openai";
46+
import { generateText, stepCountIs } from "ai";
47+
import { scrapeTool } from "@scrapegraphai/ai-sdk";
48+
49+
const result = await generateText({
50+
model: openai("gpt-5-nano"),
51+
prompt: "Find the main headline on https://example.com",
52+
tools: {
53+
scrape: scrapeTool(),
54+
},
55+
stopWhen: stepCountIs(5),
56+
});
57+
58+
console.log(result.text);
59+
```
60+
61+
Use every ScrapeGraphAI tool group:
62+
63+
```ts
64+
import { openai } from "@ai-sdk/openai";
65+
import { generateText, stepCountIs } from "ai";
66+
import {
67+
crawlTools,
68+
extractTool,
69+
monitorTools,
70+
scrapeTool,
71+
searchTool,
72+
} from "@scrapegraphai/ai-sdk";
73+
74+
const result = await generateText({
75+
model: openai("gpt-5-nano"),
76+
prompt: "Search for ScrapeGraphAI docs, scrape the best page, and summarize it.",
77+
tools: {
78+
scrape: scrapeTool(),
79+
extract: extractTool(),
80+
search: searchTool(),
81+
...crawlTools(),
82+
...monitorTools(),
83+
},
84+
stopWhen: stepCountIs(10),
85+
});
86+
87+
console.log(result.text);
88+
```
89+
90+
Tools read `SGAI_API_KEY` from the environment by default. You can also pass it explicitly:
91+
92+
```ts
93+
const tools = {
94+
scrape: scrapeTool({ apiKey: process.env.SGAI_API_KEY }),
95+
};
96+
```
97+
98+
## Tools
99+
100+
### scrapeTool
101+
102+
Scrape a webpage with ScrapeGraphAI. Supports markdown, html, json extraction, links, images, summary, branding, and screenshots.
103+
104+
```ts
105+
import { scrapeTool } from "@scrapegraphai/ai-sdk";
106+
107+
const tools = {
108+
scrape: scrapeTool(),
109+
};
110+
```
111+
112+
### extractTool
113+
114+
Extract structured JSON from a URL, HTML, or markdown with a natural-language prompt.
115+
116+
```ts
117+
import { extractTool } from "@scrapegraphai/ai-sdk";
118+
119+
const tools = {
120+
extract: extractTool(),
121+
};
122+
```
123+
124+
### searchTool
125+
126+
Search the web and optionally extract structured data from search results.
127+
128+
```ts
129+
import { searchTool } from "@scrapegraphai/ai-sdk";
130+
131+
const tools = {
132+
search: searchTool(),
133+
};
134+
```
135+
136+
### crawlTools
137+
138+
Start, poll, page through, stop, resume, and delete ScrapeGraphAI crawl jobs.
139+
140+
```ts
141+
import { crawlTools } from "@scrapegraphai/ai-sdk";
142+
143+
const tools = {
144+
...crawlTools(),
145+
};
146+
```
147+
148+
Crawl page retrieval is paginated. Use `getCrawl` for status, then `getCrawlPages` for pages and resolved scrape results.
149+
150+
```ts
151+
const tools = {
152+
startCrawl: startCrawlTool(),
153+
getCrawl: getCrawlTool(),
154+
getCrawlPages: getCrawlPagesTool(),
155+
};
156+
```
157+
158+
### monitorTools
159+
160+
Create, list, update, pause, resume, delete, and fetch activity for ScrapeGraphAI monitors.
161+
162+
```ts
163+
import { monitorTools } from "@scrapegraphai/ai-sdk";
164+
165+
const tools = {
166+
...monitorTools(),
167+
};
168+
```
169+
170+
## Examples
171+
172+
| Example | Description |
173+
|---------|-------------|
174+
| [`hacker-news.ts`](examples/hacker-news.ts) | Scrape Hacker News with AI SDK tools |
175+
| [`crawl-blog.ts`](examples/crawl-blog.ts) | Crawl ScrapeGraphAI blog pages, fetch paginated crawl results, and summarize them |
176+
177+
Run an example:
178+
179+
```bash
180+
OPENAI_API_KEY=... SGAI_API_KEY=... bun examples/crawl-blog.ts
181+
```
182+
183+
## Environment Variables
184+
185+
| Variable | Description |
186+
|----------|-------------|
187+
| `SGAI_API_KEY` | ScrapeGraphAI API key |
188+
| `OPENAI_API_KEY` | Required by the OpenAI provider examples |
189+
190+
## Development
191+
192+
```bash
193+
bun install
194+
bun run build
195+
bun run check
196+
```
197+
198+
## License
199+
200+
MIT - [ScrapeGraphAI](https://scrapegraphai.com)

‎packages/ai-sdk/examples/crawl-blog.ts‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
import { openai } from "@ai-sdk/openai";
2+
import { generateText, stepCountIs, type ModelMessage } from "ai";
3+
import { stdin as input, stdout as output } from "node:process";
4+
import { createInterface } from "node:readline/promises";
5+
import { crawlTools } from "../src/index";
6+
7+
const initialPrompt =
8+
"Find 10 https://scrapegraphai.com/ blog posts. Start a crawl, poll its status, fetch crawled pages with getCrawlPages, then summarize what you found.";
9+
const messages: ModelMessage[] = [];
10+
let activeController: AbortController | undefined;
11+
12+
async function run(prompt: string) {
13+
messages.push({ role: "user", content: prompt });
14+
const controller = new AbortController();
15+
activeController = controller;
16+
17+
try {
18+
const result = await generateText({
19+
model: openai("gpt-5-nano"),
20+
messages,
21+
tools: { ...crawlTools() },
22+
stopWhen: stepCountIs(20),
23+
abortSignal: controller.signal,
24+
onStepFinish: ({ text, toolCalls, toolResults }) => {
25+
if (text) {
26+
console.log(`\n[assistant]\n${text}`);
27+
}
28+
29+
for (const toolCall of toolCalls) {
30+
console.log(`\n[tool] ${toolCall.toolName}`);
31+
console.log(JSON.stringify(toolCall.input, null, 2));
32+
}
33+
34+
for (const toolResult of toolResults) {
35+
console.log(`\n[result] ${toolResult.toolName}`);
36+
console.log(JSON.stringify(toolResult.output, null, 2));
37+
}
38+
},
39+
});
40+
41+
messages.push(...result.response.messages);
42+
console.log(`\n${result.text}\n`);
43+
} catch (error) {
44+
if (controller.signal.aborted) {
45+
console.error("[aborted]");
46+
} else {
47+
console.error(error instanceof Error ? error.message : error);
48+
}
49+
} finally {
50+
if (activeController === controller) {
51+
activeController = undefined;
52+
}
53+
}
54+
}
55+
56+
const rl = createInterface({ input, output });
57+
58+
process.on("SIGINT", () => {
59+
output.write("\n");
60+
if (activeController) {
61+
activeController.abort();
62+
return;
63+
}
64+
65+
rl.close();
66+
process.exit(0);
67+
});
68+
69+
await run(initialPrompt);
70+
71+
while (true) {
72+
const prompt = (await rl.question("> ")).trim();
73+
74+
if (prompt) {
75+
await run(prompt);
76+
}
77+
}

‎packages/ai-sdk/examples/hacker-news.ts‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
import { openai } from "@ai-sdk/openai";
2+
import { generateText, stepCountIs } from "ai";
3+
import { scrapeTool } from "../src/index";
4+
5+
const { text } = await generateText({
6+
model: openai("gpt-5-nano"),
7+
prompt:
8+
"Scrape Hacker News and write a short, concise summary of what people are talking about today.",
9+
tools: {
10+
scrape: scrapeTool(),
11+
},
12+
stopWhen: stepCountIs(3),
13+
});
14+
15+
console.log(text);

‎packages/ai-sdk/package.json‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
{
2+
"name": "@scrapegraphai/ai-sdk",
3+
"version": "0.1.0",
4+
"description": "Vercel AI SDK tools integration for ScrapeGraphAI.",
5+
"type": "module",
6+
"main": "dist/index.js",
7+
"types": "dist/index.d.ts",
8+
"exports": {
9+
".": {
10+
"types": "./dist/index.d.ts",
11+
"import": "./dist/index.js"
12+
}
13+
},
14+
"files": ["dist"],
15+
"scripts": {
16+
"dev": "tsup --watch",
17+
"build": "tsup",
18+
"check": "tsc --noEmit",
19+
"prepublishOnly": "bun run build"
20+
},
21+
"keywords": ["scrapegraph", "ai-sdk", "tools", "scraping", "extraction"],
22+
"author": "ScrapeGraph Team",
23+
"license": "MIT",
24+
"homepage": "https://scrapegraphai.com",
25+
"repository": {
26+
"type": "git",
27+
"url": "https://github.com/ScrapeGraphAI/scrapegraph-sdk.git",
28+
"directory": "packages/ai-sdk"
29+
},
30+
"peerDependencies": {
31+
"ai": ">=6"
32+
},
33+
"dependencies": {
34+
"scrapegraph-js": "^2.2.0",
35+
"zod": "^4.3.6"
36+
},
37+
"devDependencies": {
38+
"@ai-sdk/openai": "^3.0.65",
39+
"ai": "^6.0.191",
40+
"tsup": "^8.3.6",
41+
"typescript": "^5.8.2"
42+
}
43+
}

0 commit comments

Comments
 (0)