|
1 | 1 | # Self-Hosting Crawl4AI 🚀 |
2 | 2 |
|
| 3 | +> **🔐 0.9.0 is secure-by-default (breaking changes).** The self-hosted Docker |
| 4 | +> server now requires authentication by default, binds to loopback unless you |
| 5 | +> set a token, validates request bodies against a strict trust boundary, uses |
| 6 | +> declarative hooks instead of inline Python, and returns artifact ids for |
| 7 | +> screenshot/pdf. If you are upgrading from 0.8.x, read the |
| 8 | +> [migration guide](https://github.com/unclecode/crawl4ai/blob/main/deploy/docker/MIGRATION.md) |
| 9 | +> first. Some examples below are being updated for 0.9.0; the migration guide is |
| 10 | +> the authoritative reference for the new defaults. |
| 11 | +
|
3 | 12 | **Take Control of Your Web Crawling Infrastructure** |
4 | 13 |
|
5 | 14 | Self-hosting Crawl4AI gives you complete control over your web crawling and data extraction pipeline. Unlike cloud-based solutions, you own your data, infrastructure, and destiny. |
@@ -442,6 +451,15 @@ Executes JavaScript snippets on the specified URL and returns the full crawl res |
442 | 451 |
|
443 | 452 | ## User-Provided Hooks API |
444 | 453 |
|
| 454 | +> **⚠️ Changed in 0.9.0.** The inline-Python hook API described below was removed. |
| 455 | +> Sending arbitrary Python code to the server is no longer accepted (it was an |
| 456 | +> unauthenticated code-execution surface). 0.9.0 replaces it with **declarative |
| 457 | +> hooks**: a fixed set of safe, server-validated actions (for example |
| 458 | +> `add_cookies`, `set_headers`, `block_resources`) supplied as JSON, with no code |
| 459 | +> execution. See the [migration guide](https://github.com/unclecode/crawl4ai/blob/main/deploy/docker/MIGRATION.md) |
| 460 | +> for the declarative hook format. The inline-code examples in this section apply |
| 461 | +> to 0.8.x only and are kept for reference until this page is fully rewritten. |
| 462 | +
|
445 | 463 | The Docker API supports user-provided hook functions, allowing you to customize the crawling behavior by injecting your own Python code at specific points in the crawling pipeline. This powerful feature enables authentication, performance optimization, and custom content extraction without modifying the server code. |
446 | 464 |
|
447 | 465 | > ⚠️ **IMPORTANT SECURITY WARNING**: |
@@ -2459,6 +2477,11 @@ rate_limiting: |
2459 | 2477 | storage_uri: "memory://" # Use "redis://localhost:6379" if you need persistent/shared limits |
2460 | 2478 |
|
2461 | 2479 | # Security Configuration |
| 2480 | +# NOTE (0.9.0): the defaults below reflect 0.8.x. In 0.9.0 the Docker server is |
| 2481 | +# secure-by-default - authentication is required, the server binds loopback |
| 2482 | +# unless a token is set, and request bodies are validated against a trust |
| 2483 | +# boundary. See the migration guide for the 0.9.0 defaults and config keys: |
| 2484 | +# https://github.com/unclecode/crawl4ai/blob/main/deploy/docker/MIGRATION.md |
2462 | 2485 | security: |
2463 | 2486 | enabled: false # Master toggle for security features |
2464 | 2487 | jwt_enabled: false # Enable JWT authentication (requires security.enabled=true) |
|
0 commit comments