release: merge 0.9.0 secure-by-default Docker server hardening into develop

Squash of security/docker-hardening-2026-06 (the full R1-R7 redesign) plus the
three new 0.9.0 security fixes (download path traversal, streaming-path SSRF,
extra_args RCE) and uniform egress pinning across all browser-fetch endpoints.

Version 0.9.0. Supersedes the 0.8.8/0.8.9 non-breaking patches with their
breaking-by-design variants; the 0.8.8/0.8.9 CHANGELOG and credit ledger
entries are preserved. The old exec-based hook_manager.py is removed in favor
of declarative hook_registry.py. Security suite: 309 passed, 1 xfailed.

See deploy/docker/MIGRATION.md for the breaking-change migration guide.

Author: unclecode

.github/workflows/security.yml

@@ -0,0 +1,65 @@
+name: Security
+
+# Runs the offline Docker-server security suite: behavioral tests for the
+# secure-by-default posture (R1-R7). No network, browser, Redis or Docker needed
+# - it boots the app via TestClient and monkeypatches DNS, so it is fast
+# (~seconds) and deterministic.
+
+on:
+  push:
+    branches: [main, develop, "security/**"]
+    paths:
+      - "deploy/docker/**"
+      - "crawl4ai/async_configs.py"
+      - ".github/workflows/security.yml"
+  pull_request:
+    paths:
+      - "deploy/docker/**"
+      - "crawl4ai/async_configs.py"
+      - ".github/workflows/security.yml"
+
+jobs:
+  security-offline:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.10"
+          cache: pip
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e .
+          pip install -r deploy/docker/requirements.txt
+          pip install pytest pytest-asyncio
+
+      - name: Run security suite
+        run: |
+          pytest deploy/docker/tests/test_security_*.py -q
+
+  posture-gate:
+    # The headline secure-by-default acceptance gate, isolated so it can be a
+    # required status check. xfail-marked items (e.g. the build-gated
+    # --no-sandbox removal) do not fail this job.
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.10"
+          cache: pip
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e .
+          pip install -r deploy/docker/requirements.txt
+          pip install pytest pytest-asyncio
+
+      - name: Default-posture gate
+        run: |
+          pytest deploy/docker/tests/test_security_default_posture.py -q

.gitignore

@@ -308,3 +308,6 @@ HANDOFF-*.md
 
 # Local pipeline/run output
 out/
+
+# entrypoint.sh is a required deployment artifact (Dockerfile COPYs it)
+!deploy/docker/entrypoint.sh

CHANGELOG.md

@@ -5,6 +5,51 @@ All notable changes to Crawl4AI will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.9.0] - 2026-06-18
+
+0.9.0 is a major, secure-by-default release of the Crawl4AI Docker API server. The out-of-the-box deployment is now hardened with defense in depth: authentication is on by default, the server binds loopback unless you give it a token, and the network request body is treated as an untrusted trust boundary. This release contains breaking changes for the self-hosted HTTP server only. The core pip library (SDK / in-process use) is unchanged.
+
+**What changed:** the Docker server moved from an open, trust-the-caller posture to a closed, secure-by-default one. Defaults that used to be permissive (open bind, no auth, request-supplied browser internals, TLS verification off, Redis with no password) are now safe by default and gated behind explicit configuration.
+
+**What you must do:** set `CRAWL4AI_API_TOKEN` and re-issue any tokens, then review whether you relied on any of the request fields or features that are now configured server-side. Most plain "crawl these URLs" users only need the two steps in the "Everyone" section of the migration guide. The full guide is at `deploy/docker/MIGRATION.md`.
+
+### Security
+
+This release completes the secure-by-default hardening of the Docker API server begun in 0.8.7 and 0.8.8. It moves the worst remaining issues from mitigation to architecture: unauthenticated access and request-supplied code/config are eliminated by design rather than patched in place. Every change is hardening; users self-hosting the Docker server should upgrade and follow the migration guide.
+
+- **Authentication on by default, loopback bind**: the server no longer serves an unauthenticated API on `0.0.0.0`. With no token it binds `127.0.0.1` and prints a one-off local token; exposing it requires `CRAWL4AI_API_TOKEN` and `Authorization: Bearer <token>` on every request except `GET /health`.
+- **Request trust boundary**: a crawl request body now carries declarative, scalar options only. Fields that previously let a caller drive browser internals or arbitrary code are rejected at the network boundary.
+- **Declarative hooks replace hook code**: arbitrary Python hook strings are replaced by a fixed set of declarative actions, removing request-supplied code from the server entirely.
+- **Strengthened JWT, admin-scoped monitor actions, deny-by-default CORS, strict security headers, TLS verification on, password-protected loopback-only Redis, bounded job queue, generic error responses with correlation ids, and validated webhook headers** round out the defense-in-depth posture. See the migration guide for the full list.
+- **Download path confinement (CWE-22)**: both download sinks now confine writes with basename plus realpath plus `O_NOFOLLOW`, closing a path-traversal-to-file-write class. Credit: Y4tacker.
+- **SSRF destination validation on the streaming crawl path (CWE-918)**: `/crawl/stream` and `/crawl` with `stream=true` now validate the destination and return HTTP 400 for disallowed targets, matching the non-streaming handlers. Credit: KOH Jun Sheng.
+- **Request-supplied `browser_config.extra_args` rejected (CWE-94)**: launch arguments can no longer be supplied over the network, closing a Chromium launch-arg injection class. Credit: Y4tacker, UDU_RisePho ([hoanggxyuuki](https://github.com/hoanggxyuuki)).
+
+All reporters are credited in `SECURITY-CREDITS.md`. GitHub Security Advisories accompany this release.
+
+### Breaking Changes
+
+These apply to the self-hosted Docker API server only. The pip library is unaffected. See `deploy/docker/MIGRATION.md` for the step-by-step migration and `deploy/docker/SECURITY-VERIFY.md` for the deployment checklist.
+
+- **Auth is on by default**: set `CRAWL4AI_API_TOKEN` and send `Authorization: Bearer <token>`. With no token the server binds loopback only.
+- **Loopback bind by default**: the server no longer binds `0.0.0.0` without a token; put a TLS-terminating reverse proxy in front when you expose it.
+- **Tokens must be re-issued**: the JWT implementation changed and tokens from older versions are no longer valid. Re-mint via `POST /token`.
+- **Request trust boundary**: `js_code`, `js_code_before_wait`, `c4a_script`, `proxy` / `proxy_config`, `extra_args`, `user_data_dir`, `cdp_url`, `cookies`, `headers`, `init_scripts`, `base_url`, `deep_crawl_strategy`, `simulate_user`, `magic`, `process_in_browser`, and nested LLM config objects are rejected with HTTP 400 when sent over the network. Configure them server-side or use the in-process SDK. Unknown fields are dropped; timeouts, viewport, and scroll counts are clamped.
+- **Hooks are declarative**: `hooks.code` is replaced by a fixed action set (`block_resources`, `add_cookies`, `set_headers`, `scroll_to_bottom`, `wait_for_timeout`). See `GET /hooks/info`.
+- **`output_path` removed, replaced by an artifact id**: `/screenshot` and `/pdf` store the result and return `artifact_id` + URL; fetch via authenticated `GET /artifacts/{artifact_id}` (TTL and quota apply).
+- **LLM `base_url` removed**: `/md`, `/llm`, and `/llm/job` select a provider by name only; endpoint and key are configured server-side and constrained by `config.llm.allowed_providers`.
+- **Monitor actions require an admin token**: `POST /monitor/actions/*` and `/monitor/stats/reset` need an admin-scope principal.
+- **CORS deny-by-default**: cross-origin browser requests are denied unless listed in `security.cors_allow_origins`.
+- **TLS verification on**: self-signed / internal TLS targets fail by default. Escape hatches for trusted internal testing: `CRAWL4AI_ALLOW_INSECURE_TLS=true`, `CRAWL4AI_ALLOW_INTERNAL_URLS=true`.
+- **Webhook headers validated**: malformed or hop-by-hop / sensitive headers are rejected with HTTP 422.
+- **Redis requires a password**: in-container Redis is loopback-only, password-protected, and its port is no longer published. For external Redis set `REDIS_PASSWORD`.
+- **Bounded background job queue**: request body size, per-crawl wall clock, queue size, and per-principal concurrency are now capped (configurable; `0` = unbounded).
+- **Generic 5xx responses**: server errors return `{"error": "Internal server error", "correlation_id": "…"}`; match the id in the logs for detail.
+
+### Security Credits
+
+Y4tacker, KOH Jun Sheng, and UDU_RisePho ([hoanggxyuuki](https://github.com/hoanggxyuuki)). See `SECURITY-CREDITS.md`.
+
 ## [0.8.9] - 2026-06-04
 
 0.8.9 is a follow-up, backward-compatible security patch for the self-hosted Docker API server, closing an SSRF path that 0.8.8 did not cover. Upgrade in place; no configuration changes required.

Dockerfile

@@ -1,7 +1,7 @@
 FROM python:3.12-slim-bookworm AS build
 
 # C4ai version
-ARG C4AI_VER=0.8.9
+ARG C4AI_VER=0.9.0
 ENV C4AI_VERSION=$C4AI_VER
 LABEL c4ai.version=$C4AI_VER
 
@@ -193,12 +193,18 @@ COPY deploy/docker/* ${APP_HOME}/
 # copy the playground + any future static assets
 COPY deploy/docker/static ${APP_HOME}/static
 
-# Change ownership of the application directory to the non-root user
-RUN chown -R appuser:appuser ${APP_HOME}
+# /app is root-owned and read-only to the runtime user: a write bug can no
+# longer plant a persistent self-RCE in the application directory.
+RUN chown -R root:root ${APP_HOME} && chmod -R a-w ${APP_HOME}
 
 # give permissions to redis persistence dirs if used
 RUN mkdir -p /var/lib/redis /var/log/redis && chown -R appuser:appuser /var/lib/redis /var/log/redis
 
+# Sandboxed artifact store (server-owned screenshot/PDF outputs), 0700.
+RUN mkdir -p /var/lib/crawl4ai/outputs \
+    && chown -R appuser:appuser /var/lib/crawl4ai \
+    && chmod 700 /var/lib/crawl4ai/outputs
+
 HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
     CMD bash -c '\
     MEM=$(free -m | awk "/^Mem:/{print \$2}"); \
@@ -209,12 +215,15 @@ HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
     redis-cli ping > /dev/null && \
     curl -f http://localhost:11235/health || exit 1'
 
-EXPOSE 6379
+# Redis is in-container only (loopback + requirepass); never expose its port.
+# (was: EXPOSE 6379)
 # Switch to the non-root user before starting the application
 USER appuser
 
 # Set environment variables to ptoduction
 ENV PYTHON_ENV=production 
 
-# Start the application using supervisord
-CMD ["supervisord", "-c", "supervisord.conf"]
+# Start via entrypoint.sh, which resolves the socket-level auth/egress posture
+# (loopback unless a credential is present) and the redis password, then execs
+# supervisord.
+CMD ["bash", "entrypoint.sh"]

README.md

@@ -37,9 +37,11 @@ Limited slots._
 
 Crawl4AI turns the web into clean, LLM ready Markdown for RAG, agents, and data pipelines. Fast, controllable, battle tested by a 50k+ star community.
 
-[✨ Check out latest update v0.8.9](#-recent-updates)
+[✨ Check out latest update v0.9](#-recent-updates)
 
-✨ **New in v0.8.9**: Follow-up security patch for the self-hosted Docker API server, closing an SSRF via proxy settings that 0.8.8 did not cover. Backward compatible. If you run the Docker server, upgrade. A larger secure-by-default release with breaking changes is coming in ~1-2 weeks. [Release notes →](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.8.9.md)
+✨ **New in v0.9**: Major secure-by-default release of the Docker API server. Auth is on by default, the server binds loopback unless given a token, and the request body is now an untrusted trust boundary. Breaking changes for the self-hosted server only; the pip library is unchanged. If you self-host the Docker API, read the [migration guide](https://github.com/unclecode/crawl4ai/blob/main/deploy/docker/MIGRATION.md) before upgrading. [Release notes →](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.9.0.md)
+
+✨ Recent v0.8.7: Security-hardening release. Fixes critical Docker API vulnerabilities (RCE, SSRF, auth bypass, file write, XSS, hardcoded JWT secret), adds DomainMapper, and ships scraping, deep-crawl, and LLM fixes. [Release notes →](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.8.7.md)
 
 ✨ Recent v0.8.6: Security hotfix that replaced `litellm` with `unclecode-litellm` due to a PyPI supply chain compromise.
 
@@ -565,28 +567,17 @@ async def test_news_crawl():
 ## ✨ Recent Updates
 
 <details open>
-<summary><strong>Version 0.8.9 Release Highlights - Proxy SSRF Patch</strong></summary>
-
-A follow-up, backward-compatible security patch for the self-hosted Docker API server: closes an SSRF via proxy settings (`proxy_config.server`, the deprecated `proxy`, `crawler_config.proxy_config`, and proxy/DNS flags in `extra_args`) that 0.8.8 did not cover. Proxy destinations are now validated like crawl URLs. Upgrade in place, no config changes. A larger secure-by-default release with breaking changes is coming in ~1-2 weeks; a migration guide will accompany the pre-announcement.
-
-```bash
-pip install -U crawl4ai
-```
-
-[Full v0.8.9 Release Notes →](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.8.9.md)
+<summary><strong>Version 0.9.0 Release Highlights - Secure-by-Default Docker Server</strong></summary>
 
-</details>
-
-<details>
-<summary><strong>Version 0.8.8 Release Highlights - Docker Server Security Patch</strong></summary>
+A major, secure-by-default release of the Docker API server. The out-of-the-box deployment is hardened with defense in depth: authentication is on by default, the server binds loopback unless you give it a token, and the network request body is treated as an untrusted trust boundary. Request-supplied browser internals and hook code are gone; hooks are declarative, `output_path` is replaced by an artifact store, TLS verification is on, CORS is deny-by-default, and Redis is password-protected and loopback-only.
 
-A focused, backward-compatible security patch for the self-hosted Docker API server: closes SSRF filter gaps (IPv6 transition forms), hardens screenshot/PDF `output_path` against a symlink write, stops LLM credential exfiltration via a request `base_url`, and adds CRLF-safe logging and webhook header validation. Upgrade in place, no config changes.
+This is a breaking release for the self-hosted Docker server only. The core pip library (SDK / in-process use) is unchanged. If you self-host the Docker API, read the migration guide before upgrading.
 
 ```bash
 pip install -U crawl4ai
 ```
 
-[Full v0.8.8 Release Notes →](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.8.8.md)
+[Migration Guide →](https://github.com/unclecode/crawl4ai/blob/main/deploy/docker/MIGRATION.md) · [Full v0.9.0 Release Notes →](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.9.0.md)
 
 </details>
 

SECURITY-CREDITS.md

@@ -8,8 +8,11 @@ We thank the following security researchers for their responsible disclosure:
 | Jeongbean Jeon | wjswjdqls7@gmail.com | File write, SSRF, monitor auth bypass, stored XSS | 2026-04-13 |
 | wulonchia | wulonchia@gmail.com | File write via output_path (independent report) | 2026-04-13 |
 | by111 (August829) | GitHub: [August829](https://github.com/August829) | Hardcoded JWT secret, eval in /config/dump, /execute_js, hook sandbox escape | 2026-04-14 |
-| secsys_codex ([FORIMOC](https://github.com/FORIMOC)) | secsys_codex@163.com | SSRF via /md, /crawl, /llm endpoints (URL destination validation) | 2026-04-18 |
+| secsys_codex | secsys_codex@163.com | SSRF via /md, /crawl, /llm endpoints (URL destination validation) | 2026-04-18 |
 | Velayutham Selvaraj | [LinkedIn](https://www.linkedin.com/in/velayuthamselvaraj) | SSRF via missing host validation in validate_url_scheme (independent report) | 2026-05-06 |
 | IcySun & Yashon | icysun@qq.com, liyaoyin@qq.com | SSRF, file write via output_path, missing auth by default, hook sandbox bypass via asyncio (independent report) | 2026-05-15 |
 | Geo ([geo-chen](https://github.com/geo-chen)) | cve@sageby.com | LLM API key exfiltration via unvalidated base_url (0.8.8) | 2026-06-02 |
 | Geo ([geo-chen](https://github.com/geo-chen)) | cve@sageby.com | SSRF via proxy_config.server bypassing the SSRF check (0.8.9) | 2026-06-04 |
+| Y4tacker | y4tacker@gmail.com | Download path traversal -> file write; Chromium launch-arg injection via extra_args (0.9.0) | 2026-06-18 |
+| KOH Jun Sheng ([seankohjs](https://github.com/seankohjs)) | jskoh.2023@scis.smu.edu.sg | SSRF on the streaming crawl path /crawl/stream (0.9.0) | 2026-06-18 |
+| UDU_RisePho | GitHub: [hoanggxyuuki](https://github.com/hoanggxyuuki) | Chromium launch-flag RCE class via extra_args (0.9.0) | 2026-06-18 |

crawl4ai/__version__.py

@@ -1,7 +1,7 @@
 # crawl4ai/__version__.py
 
 # This is the version that will be used for stable releases
-__version__ = "0.8.9"
+__version__ = "0.9.0"
 
 # For nightly builds, this gets set during build process
 __nightly_version__ = None