feat(connectors): add url source connector

[paulkarayan]

Part of making "playground" (the Test / workbench flow) run on the mainstream processing path instead of its own bespoke code.

Today playground ingests uploaded/sample files through a hidden, one-off plugin (plugin-utic-playground) with its own indexer + downloader. This adds a real url source connector so ingest can instead be a normal connector — "fetch this list of {url, filename}" — the same way every other source works.

What it does

• UrlIndexer: yields FileData for a configured list of {url, filename}.
• UrlDownloader: SSRF-safe fetch — validates every resolved address (is_global allowlist), pins the httpx transport to the validated IP, revalidates each redirect hop (no DNS-rebind TOCTOU). Private-IP guard is connector config (allow_private_ips), not an env var.
• Registered in the source registry; httpx is a lazy url extra. 14 unit tests.

Once released, playground's uploaded-file ingest becomes a plain url source — no special plugin.

Linkages

• platform-libs pk/url-source-connector-type declares this as a SourceConnectorType (needed for it to be usable in workflows).
• platform-plugins pk/playground-notify handles the uploader side of the same goal.
• Downstream: platform-api rewrites the playground DAG to emit this url source (separate PR, once this is released and plugin manifests reference it).

🤖 Generated with Claude Code

[cubic-dev-ai]

All reported issues were addressed across 5 files

<sub>Shadow auto-approve: would not auto-approve because issues were found.

Re-trigger cubic</sub>

[paulkarayan]

Addressed cubic:

• P1 (path traversal): fixed — _safe_filename reduces caller filenames to a basename and rejects pure-traversal names; tested.
• Duplicate filenames: fixed — precheck rejects duplicate basenames.
• Coverage (default guard not exercised through the pipeline): added test_download_blocks_private_by_default — runs UrlDownloader.run with allow_private_ips=False and asserts the fetch is blocked.
• Streaming/memory (P2): deferred — the only current caller is playground (20 MB cap), and streaming with per-redirect revalidation adds real complexity. Worth doing if this connector is used for large general downloads; filing as follow-up rather than in this PR.
• Spike redirect test (P2): the spike is a smoke script; the real per-hop revalidation coverage is test_redirect_target_is_revalidated (monkeypatches _validate_and_pin, allow_private=False, asserts the post-redirect host is validated).

[cubic-dev-ai]

0 issues found across 2 files (changes from recent commits).

<sub>Shadow auto-approve: would not auto-approve. Auto-approval blocked by 5 unresolved issues from previous reviews.

Re-trigger cubic</sub>

[aballman]

The SSRF design is right and I verified it: the socket pins to the validated IP with no connect-time re-resolution (no TOCTOU), every resolved record is validated, each redirect hop is revalidated, and Host/SNI stay on the real hostname. allow_private_ips defaults to deny.

Blocking:

• Lint is red on 3.11/3.12/3.13, and changelog/check-version are failing. Needs green CI and a CHANGELOG entry.
• The module docstring says "NOT YET REGISTERED ... follow-ups", but this PR adds the add_source_entry("url", ...) registration in connectors/__init__.py. Fix the docstring.
• Is scripts/url_connector_spike.py meant to ship, or is it a leftover spike?

Worth addressing:

• is_global alone misses IPv6 transition addresses: 64:ff9b::a9fe:a9fe (NAT64), 2002:a9fe:a9fe::1 (6to4), and ::7f00:1 (IPv4-compatible) all embed a private/metadata IPv4 but return is_global == True. Only reachable behind NAT64/DNS64, but I'd reject reserved addresses or re-check the embedded IPv4. The same guard in platform-plugins#1668 has the identical gap, which is the argument for factoring _validate_and_pin into one shared util instead of two copies.
• is_global's handling of CGNAT 100.64.0.0/10 and IPv4-mapped is Python-patch dependent (correct on >=3.11.9/3.12.4). This connector supports 3.11+, so either raise the floor or note the caveat.
• The response body is read fully into memory with no size cap. A hostile or oversized URL can OOM the worker.

[cubic-dev-ai]

1 issue found across 5 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="unstructured_ingest/processes/connectors/url.py">

<violation number="1" location="unstructured_ingest/processes/connectors/url.py:231">
P2: Streaming to disk mid-download can leave a partial file on failure. Previously, `_ssrf_safe_get` buffered the full response in memory and wrote the file only after a successful download, so an error never produced a partial file on disk. Now that the file is opened and written incrementally inside `_ssrf_safe_download`, a mid-stream failure (connection drop, timeout, server error after partial body) will leave a truncated file at `download_path`. On retry, the file is truncated and overwritten (safe). But on a non-retried failure, the partial file persists and could be picked up by a subsequent pipeline pass as if it were a complete download. Consider writing to a temporary path and renaming on success, or cleaning up the partial file on failure.</violation>
</file>

<sub>Shadow auto-approve: would not auto-approve because issues were found.
Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: Streaming to disk mid-download can leave a partial file on failure. Previously, _ssrf_safe_get buffered the full response in memory and wrote the file only after a successful download, so an error never produced a partial file on disk. Now that the file is opened and written incrementally inside _ssrf_safe_download, a mid-stream failure (connection drop, timeout, server error after partial body) will leave a truncated file at download_path. On retry, the file is truncated and overwritten (safe). But on a non-retried failure, the partial file persists and could be picked up by a subsequent pipeline pass as if it were a complete download. Consider writing to a temporary path and renaming on success, or cleaning up the partial file on failure.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At unstructured_ingest/processes/connectors/url.py, line 231:

<comment>Streaming to disk mid-download can leave a partial file on failure. Previously, `_ssrf_safe_get` buffered the full response in memory and wrote the file only after a successful download, so an error never produced a partial file on disk. Now that the file is opened and written incrementally inside `_ssrf_safe_download`, a mid-stream failure (connection drop, timeout, server error after partial body) will leave a truncated file at `download_path`. On retry, the file is truncated and overwritten (safe). But on a non-retried failure, the partial file persists and could be picked up by a subsequent pipeline pass as if it were a complete download. Consider writing to a temporary path and renaming on success, or cleaning up the partial file on failure.</comment>

<file context>
@@ -217,17 +226,15 @@ def run(self, file_data: FileData, **kwargs: Any) -> DownloadResponse:
-        data = _ssrf_safe_get(
+        download_path = self.get_download_path(file_data=file_data)
+        download_path.parent.mkdir(parents=True, exist_ok=True)
+        _ssrf_safe_download(
             url,
+            download_path,
</file context>

[cubic-dev-ai]

0 issues found across 2 files (changes from recent commits).

<sub>Shadow auto-approve: would not auto-approve. Auto-approval blocked by 4 unresolved issues from previous reviews.

Re-trigger cubic</sub>

[paulkarayan]

Dropped scripts/url_connector_spike.py in 17685cf — throwaway spike; the connector is covered by test/unit/connectors/test_url.py.

[paulkarayan]

Addressed the SSRF address-validation gaps (1fd4891):

• IPv6 transition addrs: _is_public_address now rejects IPv6 that embeds a private/metadata IPv4 — NAT64 64:ff9b::/96, 6to4 2002::/16, IPv4-mapped ::ffff:…, IPv4-compatible ::a.b.c.d. Tests cover NAT64/6to4/mapped/compat + a public IPv6.
• CGNAT version-dependence: rejected 100.64.0.0/10 explicitly, so it no longer depends on the Python patch level of is_global.

On the shared-util suggestion: I mirrored the same fix into platform-plugins#1668 rather than factor a cross-repo util, because that downloader is being retired by the playground collapse (its index/downloader are replaced by this url connector). Not worth a shared dependency for code that's going away.

[cubic-dev-ai]

1 issue found across 3 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="unstructured_ingest/processes/connectors/url.py">

<violation number="1" location="unstructured_ingest/processes/connectors/url.py:134">
P1: SSRF protection can still miss NAT64/DNS64 addresses that use an operator Network-Specific Prefix: `_NAT64_PREFIX` only covers the Well-Known Prefix `64:ff9b::/96`, so a global NSP embedding `169.254.169.254` can pass as public IPv6. Consider decoding configured RFC6052 NAT64 prefixes or failing closed for known translation prefixes before treating these IPv6 addresses as public.</violation>
</file>

<sub>Shadow auto-approve: would not auto-approve because issues were found.
Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

P1: SSRF protection can still miss NAT64/DNS64 addresses that use an operator Network-Specific Prefix: _NAT64_PREFIX only covers the Well-Known Prefix 64:ff9b::/96, so a global NSP embedding 169.254.169.254 can pass as public IPv6. Consider decoding configured RFC6052 NAT64 prefixes or failing closed for known translation prefixes before treating these IPv6 addresses as public.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At unstructured_ingest/processes/connectors/url.py, line 134:

<comment>SSRF protection can still miss NAT64/DNS64 addresses that use an operator Network-Specific Prefix: `_NAT64_PREFIX` only covers the Well-Known Prefix `64:ff9b::/96`, so a global NSP embedding `169.254.169.254` can pass as public IPv6. Consider decoding configured RFC6052 NAT64 prefixes or failing closed for known translation prefixes before treating these IPv6 addresses as public.</comment>

<file context>
@@ -131,6 +131,42 @@ class UrlDownloaderConfig(DownloaderConfig):
 # each hop is revalidated.
 
 
+_NAT64_PREFIX = ipaddress.ip_network("64:ff9b::/96")
+_CGNAT = ipaddress.ip_network("100.64.0.0/10")
+
</file context>

[aballman]

The IPv6 transition hardening looks right. _is_public_address extracts and re-checks the embedded IPv4 for NAT64/6to4/compat, the explicit CGNAT reject makes it Python-version-independent, and the tests cover all four cases (64:ff9b::, 2002::, ::7f00:1, 100.64/10). Streaming to disk, the spike removal, and the docstring fix are all addressed too.

One low nit: 192.0.0.0/24 special-use (e.g. 192.0.0.9) still passes is_global. It isn't a metadata or internal target, so I wouldn't hold on it. Otherwise this looks good to release.