Skip to content

Catalog URL-ingest over-refuses public pages that embed slow/cached iframes #2083

Description

@atomantic

Summary

pickMainFrameHops in server/services/browserService.js (the pinned CDP SSRF gate added in #2068) classifies a request as a top-level navigation using only p.type === 'Document' && p.requestId === p.loaderId. But CDP sets requestId === loaderId for the main resource of every frame, including cross-origin sub-frames (iframes).

Impact (over-refusal only — never a bypass)

  • A sub-frame document whose response hasn't arrived by settle-end lands in pendingMainRequestIdsssrfPinRefusalReason returns "a top-level navigation was still in flight" and ingest is refused.
  • A cached iframe document (empty remoteIPAddress, likely under the persistent shared browser profile) lands in hops → trips the "unverifiable address" refusal.

Either way a safe public article that merely embeds a slow or cached iframe is falsely refused ingestion. This cannot weaken the gate — every connection IP is still independently verified by collectConnectedIps; the extra pending/empty-IP entries only make refusal stricter. Fails safe.

Proposed fix

Capture the top frame's frameId from the Page.navigate response (msg.id === 2 && msg.result?.frameId in navigateToUrlPinned) and thread it into pickMainFrameHops / ssrfPinRefusalReason. Only add a requestId to mainRequestIds when p.frameId === topFrameId. Leave collectConnectedIps / collectWebSocketHosts unchanged so sub-frame and sub-resource IPs stay fully verified — this only stops sub-frames from gating the top-level pending/empty-IP checks.

Make topFrameId an optional param so the existing pure unit tests keep passing; add a focused test for the iframe case. Include a defensive fallback (if no request matches topFrameId, retain the old requestId === loaderId classification) so a frameId-format surprise can never regress the feature to refuse-all.

Why deferred from the v2.26.0 release

Found during the v2.26.0 release-gate review. Deferred (not fixed in the release) because it is IMPROVEMENT-severity, fails safe, and lives in the SSRF gate — which can't be exercised end-to-end without a live Chrome/CDP navigation, so a hasty change risks a worse regression than the bug it fixes. Best handled as its own PR with the CDP path exercised.

https://claude.ai/code/session_01CXDnRojmjgdxYwwMsbMNaM

Metadata

Metadata

Assignees

No one assigned

    Labels

    planTracked by /do:replan

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions