Skip to content

Commit af1f511

Browse files
VinciGit00claude
VinciGit00
and
claude
authored
docs(langchain): align integration guide with scrapegraph-py 2.0.0 (#40)
- CrawlStartTool: depth -> max_depth - MonitorCreateTool: cron -> interval, prompt is now optional - HistoryTool: document service/page/limit filters - ScrapeTool: list all supported formats Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 5450511 commit af1f511

1 file changed

Lines changed: 160 additions & 52 deletions

File tree

‎integrations/langchain.mdx‎

Lines changed: 160 additions & 52 deletions
Original file line numberDiff line numberDiff line change
@@ -25,81 +25,72 @@ pip install langchain-scrapegraph
2525

2626
## Available Tools
2727

28-
### SmartScraperTool
28+
### ExtractTool
2929

3030
Extract structured data from any webpage using natural language prompts:
3131

3232
```python
33-
from langchain_scrapegraph.tools import SmartScraperTool
33+
from langchain_scrapegraph.tools import ExtractTool
3434

3535
# Initialize the tool (uses SGAI_API_KEY from environment)
36-
tool = SmartscraperTool()
36+
tool = ExtractTool()
3737

3838
# Extract information using natural language
3939
result = tool.invoke({
40-
"website_url": "https://www.example.com",
41-
"user_prompt": "Extract the main heading and first paragraph"
40+
"url": "https://www.example.com",
41+
"prompt": "Extract the main heading and first paragraph"
4242
})
4343
```
4444

4545
<Accordion title="Using Output Schemas" icon="code">
4646
Define the structure of the output using Pydantic models:
4747

4848
```python
49-
from typing import List
5049
from pydantic import BaseModel, Field
51-
from langchain_scrapegraph.tools import SmartScraperTool
50+
from langchain_scrapegraph.tools import ExtractTool
5251

5352
class WebsiteInfo(BaseModel):
54-
title: str = Field(description="The main title of the webpage")
55-
description: str = Field(description="The main description or first paragraph")
56-
urls: List[str] = Field(description="The URLs inside the webpage")
53+
title: str = Field(description="The main title of the page")
54+
description: str = Field(description="The main description")
5755

58-
# Initialize with schema
59-
tool = SmartScraperTool(llm_output_schema=WebsiteInfo)
56+
# Initialize with output schema
57+
tool = ExtractTool(llm_output_schema=WebsiteInfo)
6058

6159
result = tool.invoke({
62-
"website_url": "https://www.example.com",
63-
"user_prompt": "Extract the website information"
60+
"url": "https://example.com",
61+
"prompt": "Extract the title and description"
6462
})
6563
```
6664
</Accordion>
6765

68-
### SearchScraperTool
66+
### SearchTool
6967

70-
Process HTML content directly with AI extraction:
68+
Search the web and extract structured results using AI:
7169

7270
```python
73-
from langchain_scrapegraph.tools import SearchScraperTool
71+
from langchain_scrapegraph.tools import SearchTool
7472

75-
76-
tool = SearchScraperTool()
73+
tool = SearchTool()
7774
result = tool.invoke({
78-
"user_prompt": "Find the best restaurants in San Francisco",
75+
"query": "Find the best restaurants in San Francisco",
7976
})
80-
8177
```
8278

83-
<Accordion title="Using Output Schemas" icon="code">
84-
```python
85-
from typing import Optional
86-
from pydantic import BaseModel, Field
87-
from langchain_scrapegraph.tools import SearchScraperTool
79+
### ScrapeTool
8880

89-
class RestaurantInfo(BaseModel):
90-
name: str = Field(description="The restaurant name")
91-
address: str = Field(description="The restaurant address")
92-
rating: float = Field(description="The restaurant rating")
81+
Scrape a webpage and return it in the desired format. Supported formats: `markdown`, `html`, `screenshot`, `branding`, `links`, `images`, `summary`.
9382

83+
```python
84+
from langchain_scrapegraph.tools import ScrapeTool
9485

95-
tool = SearchScraperTool(llm_output_schema=RestaurantInfo)
86+
tool = ScrapeTool()
9687

97-
result = tool.invoke({
98-
"user_prompt": "Find the best restaurants in San Francisco"
99-
})
88+
# Scrape as markdown (default)
89+
result = tool.invoke({"url": "https://example.com"})
10090

91+
# Scrape as HTML
92+
result = tool.invoke({"url": "https://example.com", "format": "html"})
10193
```
102-
</Accordion>
10394

10495
### MarkdownifyTool
10596

@@ -112,34 +103,151 @@ tool = MarkdownifyTool()
112103
markdown = tool.invoke({"website_url": "https://example.com"})
113104
```
114105

106+
### Crawl Tools
107+
108+
Start and manage crawl jobs with `CrawlStartTool`, `CrawlStatusTool`, `CrawlStopTool`, and `CrawlResumeTool`:
109+
110+
```python
111+
import time
112+
from langchain_scrapegraph.tools import CrawlStartTool, CrawlStatusTool
113+
114+
start_tool = CrawlStartTool()
115+
status_tool = CrawlStatusTool()
116+
117+
# Start a crawl job
118+
result = start_tool.invoke({
119+
"url": "https://example.com",
120+
"max_depth": 2,
121+
"max_pages": 5,
122+
"format": "markdown",
123+
})
124+
print("Crawl started:", result)
125+
126+
# Check status
127+
crawl_id = result.get("id")
128+
if crawl_id:
129+
time.sleep(5)
130+
status = status_tool.invoke({"crawl_id": crawl_id})
131+
print("Crawl status:", status)
132+
```
133+
134+
### Monitor Tools
135+
136+
Create and manage monitors (replaces scheduled jobs) with `MonitorCreateTool`, `MonitorListTool`, `MonitorGetTool`, `MonitorPauseTool`, `MonitorResumeTool`, and `MonitorDeleteTool`:
137+
138+
```python
139+
from langchain_scrapegraph.tools import MonitorCreateTool, MonitorListTool
140+
141+
create_tool = MonitorCreateTool()
142+
list_tool = MonitorListTool()
143+
144+
# Create a monitor (interval accepts cron expressions or shorthand like "1h", "30m")
145+
result = create_tool.invoke({
146+
"url": "https://example.com/products",
147+
"name": "Price Monitor",
148+
"interval": "0 9 * * *", # Daily at 9 AM
149+
"prompt": "Extract current product prices", # optional JSON extraction
150+
})
151+
print("Monitor created:", result)
152+
153+
# List all monitors
154+
monitors = list_tool.invoke({})
155+
print("All monitors:", monitors)
156+
```
157+
158+
### HistoryTool
159+
160+
Retrieve request history, optionally filtered by service with pagination:
161+
162+
```python
163+
from langchain_scrapegraph.tools import HistoryTool
164+
165+
tool = HistoryTool()
166+
167+
# List the most recent requests
168+
history = tool.invoke({})
169+
170+
# Filter to a specific service and page
171+
history = tool.invoke({"service": "scrape", "page": 1, "limit": 20})
172+
```
173+
174+
### GetCreditsTool
175+
176+
Check your remaining API credits:
177+
178+
```python
179+
from langchain_scrapegraph.tools import GetCreditsTool
180+
181+
tool = GetCreditsTool()
182+
credits = tool.invoke({})
183+
```
184+
115185
## Example Agent
116186

117187
Create a research agent that can gather and analyze web data:
118188

119189
```python
120-
from langchain.agents import initialize_agent, AgentType
121-
from langchain_scrapegraph.tools import SmartScraperTool
190+
from langchain.agents import AgentExecutor, create_openai_functions_agent
191+
from langchain_core.messages import SystemMessage
192+
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
122193
from langchain_openai import ChatOpenAI
194+
from langchain_scrapegraph.tools import ExtractTool, GetCreditsTool, SearchTool
123195

124-
# Initialize tools
196+
# Initialize the tools
125197
tools = [
126-
SmartScraperTool(),
198+
ExtractTool(),
199+
GetCreditsTool(),
200+
SearchTool(),
127201
]
128202

129-
# Create an agent
130-
agent = initialize_agent(
131-
tools=tools,
132-
llm=ChatOpenAI(temperature=0),
133-
agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
134-
verbose=True
135-
)
136-
137-
# Use the agent
138-
response = agent.run("""
139-
Visit example.com, make a summary of the content and extract the main heading and first paragraph
140-
""")
203+
# Create the prompt template
204+
prompt = ChatPromptTemplate.from_messages([
205+
SystemMessage(
206+
content=(
207+
"You are a helpful AI assistant that can analyze websites and extract information. "
208+
"You have access to tools that can help you scrape and process web content. "
209+
"Always explain what you're doing before using a tool."
210+
)
211+
),
212+
MessagesPlaceholder(variable_name="chat_history", optional=True),
213+
("user", "{input}"),
214+
MessagesPlaceholder(variable_name="agent_scratchpad"),
215+
])
216+
217+
# Initialize the LLM
218+
llm = ChatOpenAI(temperature=0)
219+
220+
# Create the agent
221+
agent = create_openai_functions_agent(llm, tools, prompt)
222+
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)
223+
224+
# Example usage
225+
response = agent_executor.invoke({
226+
"input": "Extract the main products from https://www.scrapegraphai.com/"
227+
})
228+
print(response["output"])
141229
```
142230

231+
## Migration from v1
232+
233+
If you're upgrading from v1, here are the key changes:
234+
235+
| v1 Tool | v2 Tool |
236+
|---------|---------|
237+
| `SmartScraperTool` | `ExtractTool` |
238+
| `SearchScraperTool` | `SearchTool` |
239+
| `SmartCrawlerTool` | `CrawlStartTool` / `CrawlStatusTool` / `CrawlStopTool` / `CrawlResumeTool` |
240+
| `CreateScheduledJobTool` | `MonitorCreateTool` |
241+
| `GetScheduledJobsTool` | `MonitorListTool` |
242+
| `GetScheduledJobTool` | `MonitorGetTool` |
243+
| `PauseScheduledJobTool` | `MonitorPauseTool` |
244+
| `ResumeScheduledJobTool` | `MonitorResumeTool` |
245+
| `DeleteScheduledJobTool` | `MonitorDeleteTool` |
246+
| `MarkdownifyTool` | `MarkdownifyTool` (unchanged) |
247+
| `GetCreditsTool` | `GetCreditsTool` (unchanged) |
248+
| `AgenticScraperTool` | Removed |
249+
| -- | `HistoryTool` (new) |
250+
143251
## Configuration
144252

145253
Set your ScrapeGraph API key in your environment:
@@ -156,7 +264,7 @@ os.environ["SGAI_API_KEY"] = "your-api-key-here"
156264
```
157265

158266
<Note>
159-
Get your API key from the [dashboard](https://dashboard.scrapegraphai.com)
267+
Get your API key from the [dashboard](https://scrapegraphai.com/dashboard)
160268
</Note>
161269

162270
## Use Cases

0 commit comments

Comments
 (0)