Skip to content

Commit 0b8d10a

Browse files
committed
docs: unify crawl caps and fix runnable examples
Lower the page cap from 50 to 10 across all crawling examples so the browser-based ones finish within the runnable-demo timeout. Make Selenium (snippet too large for the Run-on-Apify URL) and Browser Use (needs an LLM API key) non-runnable with explanatory comments, keep both Scrapling examples runnable, and have the Pydantic example fail cleanly via `Actor.fail` instead of re-raising into a raw traceback.
1 parent 53122e4 commit 0b8d10a

16 files changed

Lines changed: 28 additions & 25 deletions

docs/03_guides/04_selenium.mdx

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,9 @@ title: Browser automation with Selenium
44
description: Build an Apify Actor that scrapes dynamic web pages using Selenium WebDriver.
55
---
66

7-
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
7+
import CodeBlock from '@theme/CodeBlock';
88

9-
import SeleniumExample from '!!raw-loader!roa-loader!./code/04_selenium.py';
9+
import SeleniumExample from '!!raw-loader!./code/04_selenium.py';
1010

1111
In this guide, you'll learn how to use [Selenium](https://www.selenium.dev/) for browser automation and web scraping in your Apify Actors.
1212

@@ -36,9 +36,10 @@ This is a simple Actor that recursively scrapes data from linked pages on the sa
3636

3737
It uses Selenium ChromeDriver to open the pages in an automated Chrome browser, and to extract the title, headings, and links after the pages load.
3838

39-
<RunnableCodeBlock className="language-python" language="python">
39+
{/* Not runnable from the docs: the "Run on Apify" link encodes the whole snippet into the URL, and this Actor (with its inline proxy-auth extension) is large enough to exceed the URL length limit and fail with an HTTP 414. */}
40+
<CodeBlock className="language-python">
4041
{SeleniumExample}
41-
</RunnableCodeBlock>
42+
</CodeBlock>
4243

4344
## Using Apify Proxy
4445

docs/03_guides/06_scrapy.mdx

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,7 @@ For further details, see the [Scrapy migration guide](https://docs.apify.com/cli
7373

7474
The following example shows a Scrapy Actor that scrapes page titles and enqueues links found on each page. This example aligns with the structure provided in the Apify Actor templates.
7575

76+
{/* Not runnable from the docs: a Scrapy Actor is a multi-file project, while the "Run on Apify" runner executes a single self-contained snippet. */}
7677
<Tabs>
7778
<TabItem value="__main__.py" label="__main__.py">
7879
<CodeBlock className="language-python">

docs/03_guides/07_scrapling.mdx

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,11 +4,10 @@ title: Adaptive scraping with Scrapling
44
description: Build an Apify Actor that scrapes web pages using the Scrapling adaptive web scraping library.
55
---
66

7-
import CodeBlock from '@theme/CodeBlock';
87
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
98

109
import ScraplingExample from '!!raw-loader!roa-loader!./code/07_scrapling.py';
11-
import ScraplingBrowserScraper from '!!raw-loader!./code/07_scrapling_browser.py';
10+
import ScraplingBrowserScraper from '!!raw-loader!roa-loader!./code/07_scrapling_browser.py';
1211

1312
In this guide, you'll learn how to use the [Scrapling](https://scrapling.readthedocs.io/) library for adaptive web scraping in your Apify Actors.
1413

@@ -101,9 +100,9 @@ scrapling install
101100

102101
To switch the example from HTTP to a real browser, fetch each page through a browser session instead of `AsyncFetcher`. Opening a fresh browser for every page would be wasteful, so `main` enters an `AsyncDynamicSession` once and reuses it for the whole crawl, while `scrape_page` fetches with `session.fetch`. The parsing API is identical, so the extraction code stays the same:
103102

104-
<CodeBlock className="language-python">
103+
<RunnableCodeBlock className="language-python" language="python">
105104
{ScraplingBrowserScraper}
106-
</CodeBlock>
105+
</RunnableCodeBlock>
107106

108107
Note that:
109108

docs/03_guides/09_browser_use.mdx

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,9 @@ title: Browser AI agents with Browser Use
44
description: Build an Apify Actor that automates a browser with an LLM agent using the Browser Use library.
55
---
66

7-
import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
7+
import CodeBlock from '@theme/CodeBlock';
88

9-
import BrowserUseExample from '!!raw-loader!roa-loader!./code/09_browser_use.py';
9+
import BrowserUseExample from '!!raw-loader!./code/09_browser_use.py';
1010

1111
In this guide, you'll learn how to use the [Browser Use](https://browser-use.com/) library to drive a browser with an LLM agent in your Apify Actors.
1212

@@ -46,9 +46,10 @@ The following Actor runs a Browser Use agent for a single task and stores its st
4646

4747
The whole Actor fits in a single file. A `run_agent_task` helper holds the Browser Use-specific logic: it defines the output schema and builds the LLM, browser, and agent. The `main` coroutine handles the [Actor](https://docs.apify.com/platform/actors) lifecycle, reads the input, sets up [Apify Proxy](https://docs.apify.com/platform/proxy), runs the agent, and stores the result:
4848

49-
<RunnableCodeBlock className="language-python" language="python">
49+
{/* Not runnable from the docs: the agent needs an LLM API key (OPENAI_API_KEY) that the shared example runner does not provide. */}
50+
<CodeBlock className="language-python">
5051
{BrowserUseExample}
51-
</RunnableCodeBlock>
52+
</CodeBlock>
5253

5354
Note that:
5455

docs/03_guides/11_pydantic.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,7 @@ The following Actor declares its input as a Pydantic `BaseModel`, validates the
5656
### About the validation
5757

5858
- `model_validate` parses the raw dictionary into a typed `ActorInput` instance. It fills in defaults and guarantees every field is valid, or raises a `ValidationError` that describes every problem at once.
59-
- Catching that error, logging a readable summary, and re-raising makes the Actor fail fast with a clear explanation right at the start, rather than crashing with an obscure error somewhere deep in the run. Because the body runs inside `async with Actor:`, the re-raised exception automatically marks the run as `FAILED`.
59+
- Catching that error, logging a readable summary, and failing the run with <ApiLink to="class/Actor#fail">`Actor.fail`</ApiLink> marks the run as `FAILED` with a clear status message. It fails fast right at the start with a readable explanation, instead of crashing with a raw traceback deeper in the run.
6060
- The error messages refer to the fields by their input-schema aliases. For invalid input like `{"searchTerms": [], "maxResults": 999, "outputFormat": "xml"}`, the log shows exactly what's wrong:
6161

6262
```text

docs/03_guides/code/01_beautifulsoup_httpx.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,7 @@ async def main() -> None:
8282
await request_queue.add_request(Request.from_url(url))
8383

8484
# Cap the crawl. Raise or remove the limit to follow more pages.
85-
max_requests = 50
85+
max_requests = 10
8686
handled_requests = 0
8787

8888
while handled_requests < max_requests and (

docs/03_guides/code/02_parsel_impit.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,7 @@ async def main() -> None:
8282
await request_queue.add_request(Request.from_url(url))
8383

8484
# Cap the crawl. Raise or remove the limit to follow more pages.
85-
max_requests = 50
85+
max_requests = 10
8686
handled_requests = 0
8787

8888
while handled_requests < max_requests and (

docs/03_guides/code/03_playwright.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -94,7 +94,7 @@ async def main() -> None:
9494
await request_queue.add_request(Request.from_url(url))
9595

9696
# Cap the crawl. Raise or remove the limit to follow more pages.
97-
max_requests = 50
97+
max_requests = 10
9898
handled_requests = 0
9999

100100
Actor.log.info('Launching Playwright...')

docs/03_guides/code/04_selenium.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -151,7 +151,7 @@ async def main() -> None:
151151
await request_queue.add_request(Request.from_url(url))
152152

153153
# Cap the crawl. Raise or remove the limit to follow more pages.
154-
max_requests = 50
154+
max_requests = 10
155155
handled_requests = 0
156156

157157
# Fresh proxy URL for the run (None if no proxy).

docs/03_guides/code/05_crawlee_beautifulsoup.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@ async def main() -> None:
5151
proxy_configuration=proxy_configuration,
5252
request_handler=router,
5353
# Cap the crawl. Remove or increase the limit to follow all links.
54-
max_requests_per_crawl=50,
54+
max_requests_per_crawl=10,
5555
)
5656

5757
await crawler.run(start_urls)

0 commit comments

Comments
 (0)