Skip to content

Add --allow-remote-refs to disable HTTP fetching of $ref by default#3051

Merged
koxudaxi merged 12 commits into
koxudaxi:mainfrom
butvinm:fix/allow-remote-refs
Apr 1, 2026
Merged

Add --allow-remote-refs to disable HTTP fetching of $ref by default#3051
koxudaxi merged 12 commits into
koxudaxi:mainfrom
butvinm:fix/allow-remote-refs

Conversation

@butvinm

@butvinm butvinm commented Mar 13, 2026

Copy link
Copy Markdown
Contributor

Fixes #3039

When a JSON schema has a $id with an HTTP URL and a $ref pointing to a non-existent local file, the tool silently resolved the ref against the $id URL, made an HTTP request, and produced a confusing YAML parse error on the HTML response.

This PR adds --allow-remote-refs / --no-allow-remote-refs with a deprecation path: remote fetching still works by default but emits a FutureWarning when it happens implicitly. In a future major version, the default will flip to disabled.

Before (bug)

$ datamodel-codegen --input schema.json
yaml.scanner.ScannerError: mapping values are not allowed in this context
  in "<unicode string>", line 35, column 28

After (fix)

$ datamodel-codegen --input schema.json
FutureWarning: Fetching remote $ref without --allow-remote-refs: https://...
In a future version, remote $ref fetching will be disabled by default.
Pass --allow-remote-refs explicitly to silence this warning.
HTTP 404 error fetching https://...

--allow-remote-refs / --no-allow-remote-refs (BooleanOptionalAction)

  • Not set (default): allows remote refs but emits FutureWarning — backward compatible
  • --allow-remote-refs: explicit opt-in, no warning
  • --no-allow-remote-refs: blocks remote refs with clear error
  • --url input: implicitly enables --allow-remote-refs (no warning)

Other improvements

  • Better local ref errors: "$ref file not found: <path>" instead of raw FileNotFoundError
  • HTTP response validation: checks status codes and Content-Type before parsing (via SchemaFetchError)
  • Transport error wrapping: DNS/timeout/connection errors wrapped in SchemaFetchError

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features

    • Added --allow-remote-refs option with automatic enablement when a URL input is used.
    • New public SchemaFetchError exception for remote schema fetch failures.
  • Bug Fixes

    • Improved HTTP fetching: clearer errors for transport failures, non-200 responses, and unexpected HTML content.
    • Parser now respects allow_remote_refs (blocks remote $ref fetching by default and warns when implicit fetching occurs).
  • Tests

    • Expanded tests for remote refs, HTTP response handling, and fetch error scenarios.

butvinm and others added 3 commits March 13, 2026 21:58
Remote $ref fetching over HTTP/HTTPS is now disabled by default.
When a $ref resolves to an HTTP(S) URL and --allow-remote-refs is not set,
a clear error message is shown instead of silently fetching remote content.

file:// URLs are still allowed without the flag since they are local.
--url input implicitly enables --allow-remote-refs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When a $ref points to a local file that doesn't exist, the error now
clearly states "$ref file not found: <path>" instead of raising a raw
FileNotFoundError.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Check status codes and Content-Type headers when fetching remote $ref
targets over HTTP. Returns clear error messages for HTTP errors (4xx/5xx)
and unexpected HTML responses instead of cryptic YAML parse errors.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Mar 13, 2026

Copy link
Copy Markdown

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Adds an opt-in --allow-remote-refs config/CLI flag, a SchemaFetchError exception, stricter HTTP/content-type checks for remote schema fetching, and blocks or warns on remote $ref resolution unless allowed; tests updated to exercise these flows and HTTP mock metadata.

Changes

Cohort / File(s) Summary
Core exception
src/datamodel_code_generator/__init__.py
Added new public exception SchemaFetchError.
CLI & config
src/datamodel_code_generator/__main__.py, src/datamodel_code_generator/arguments.py, src/datamodel_code_generator/cli_options.py, src/datamodel_code_generator/config.py, src/datamodel_code_generator/_types/generate_config_dict.py, src/datamodel_code_generator/_types/parser_config_dicts.py
Introduced allow_remote_refs config field and --allow-remote-refs CLI option; auto-enable when a URL input is provided; propagated through generation call sites and typed dicts.
HTTP utility & tests
src/datamodel_code_generator/http.py, tests/test_http.py
get_body() now wraps transport errors, enforces status-code checks and rejects HTML Content-Type, raising SchemaFetchError; unit tests added/adjusted for these cases.
Parser behavior & imports
src/datamodel_code_generator/parser/base.py, src/datamodel_code_generator/parser/jsonschema.py, tests/parser/*
Parser stores allow_remote_refs; JSON Schema parser blocks or warns for remote $ref based on flag, improves missing-file and remote-fetch error handling, and updates imports to use new error/parse helpers; parser tests updated to enable/verify remote-ref flows.
Public API & test baselines
tests/main/test_public_api_signature_baseline.py, tests/main/jsonschema/*, tests/main/openapi/*, tests/main/test_main_json.py, tests/test_main_kr.py, tests/data/expected/.../config_class.py
Baseline signatures updated to include allow_remote_refs; many tests updated to pass the flag and to set mocked responses' status_code and headers; expected typed-dict outputs updated.

Sequence Diagram

sequenceDiagram
    participant User
    participant CLI
    participant Config
    participant Parser
    participant HTTP
    participant FileSystem

    User->>CLI: invoke generator (maybe with --allow-remote-refs)
    CLI->>Config: set allow_remote_refs
    Config->>Parser: init parser(allow_remote_refs)
    Parser->>Parser: encounter $ref
    alt $ref is remote URL
        Parser->>Parser: check allow_remote_refs
        alt enabled
            Parser->>HTTP: fetch URL
            HTTP-->>HTTP: validate status & Content-Type
            alt error (HTTP >=400 or HTML)
                HTTP-->>Parser: raise SchemaFetchError
                Parser-->>User: surface error
            else valid content
                HTTP-->>Parser: return body
                Parser->>Parser: parse returned schema
            end
        else disabled
            Parser-->>User: raise Error instructing to enable remote refs
        end
    else local path
        Parser->>FileSystem: read file
        alt not found
            FileSystem-->>Parser: file not found
            Parser-->>User: raise Error with attempted path
        else found
            FileSystem-->>Parser: return content
            Parser->>Parser: parse local schema
        end
    end
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

  • #2934 — Overlaps changes to JSON Schema parser and public API exports (similar modules updated).
  • #2711 — Related changes to Parser constructor surface and propagation of parser-level options.
  • #2791 — Related modifications to HTTP fetching behavior in http.py and fetch/error handling.

Suggested labels

breaking-change-analyzed

Suggested reviewers

  • ilovelinux

Poem

🐰 I hopped along the schema trail,
Guarding refs from an accidental sail,
SchemaFetchError stands gentle and wise,
"Enable me" it nudges with careful eyes,
A tiny rabbit patch beneath the skies.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the main change: introducing a --allow-remote-refs flag to control HTTP fetching of $ref by default.
Linked Issues check ✅ Passed The PR fully addresses all coding requirements from issue #3039: introduces --allow-remote-refs flag for opt-in HTTP fetching, improves error messages for missing local refs, validates HTTP responses, wraps transport errors in SchemaFetchError, and implements a deprecation path with FutureWarning.
Out of Scope Changes check ✅ Passed All changes are directly related to implementing the --allow-remote-refs feature and addressing requirements from issue #3039; no unrelated modifications detected.
Docstring Coverage ✅ Passed Docstring coverage is 86.27% which is sufficient. The required threshold is 80.00%.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
📝 Coding Plan
  • Generate coding plan for human review comments

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Tip

CodeRabbit can use Trivy to scan for security misconfigurations and secrets in Infrastructure as Code files.

Add a .trivyignore file to your project to customize which findings Trivy reports.

Comment thread tests/main/jsonschema/test_main_jsonschema.py Fixed
Comment thread tests/main/jsonschema/test_main_jsonschema.py Fixed
@codspeed-hq

codspeed-hq Bot commented Mar 13, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 11 untouched benchmarks
⏩ 98 skipped benchmarks1


Comparing butvinm:fix/allow-remote-refs (031ca0e) with main (7e1a5c7)

Open in CodSpeed

Footnotes

  1. 98 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@codecov

codecov Bot commented Mar 13, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (76439c3) to head (031ca0e).
⚠️ Report is 8 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##              main     #3051    +/-   ##
==========================================
  Coverage   100.00%   100.00%            
==========================================
  Files           86        87     +1     
  Lines        18031     18165   +134     
  Branches      2112      2081    -31     
==========================================
+ Hits         18031     18165   +134     
Flag Coverage Δ
unittests 100.00% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

butvinm and others added 2 commits March 17, 2026 02:21
The module-level import already covers these usages.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
CLI reference docs will be auto-generated by CI from this marker.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@butvinm butvinm force-pushed the fix/allow-remote-refs branch from 3afe40b to 88c5c74 Compare March 17, 2026 00:53
butvinm and others added 2 commits March 17, 2026 04:13
Remove isinstance/hasattr guards from http.get_body() and instead
fix all test mocks to set status_code=200 and headers={}, matching
real httpx.Response behavior.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add SchemaFetchError(Error) for HTTP error status codes and unexpected
HTML responses. This ensures errors are caught by the existing
`except Error` handler in __main__.py and shown as clean messages
instead of unhandled tracebacks.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@butvinm butvinm marked this pull request as ready for review March 17, 2026 06:16

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Nitpick comments (3)
tests/test_http.py (1)

17-60: Add one transport-error regression test.

Great coverage on status/content-type checks. Please also cover httpx.get transport failures (e.g., timeout/connect error) to ensure they’re surfaced as SchemaFetchError.

✅ Suggested additional test
+def test_get_body_raises_on_httpx_transport_error(mocker: MockerFixture) -> None:
+    """Test that transport-level httpx errors are wrapped consistently."""
+    import httpx
+
+    request = httpx.Request("GET", "https://example.com/schema.json")
+    mocker.patch("httpx.get", side_effect=httpx.ConnectError("boom", request=request))
+
+    with pytest.raises(SchemaFetchError, match="fetching"):
+        get_body("https://example.com/schema.json")
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/test_http.py` around lines 17 - 60, Add a regression test that mocks
httpx.get to raise a transport error and verifies get_body raises
SchemaFetchError: create a new test (e.g.,
test_get_body_raises_on_transport_error) using MockerFixture to patch
"httpx.get" to raise an httpx.TransportError (or httpx.ConnectTimeout) when
called, then call get_body("https://example.com/schema.json") and assert it
raises SchemaFetchError (matching an appropriate message); reference the
existing tests and the get_body and SchemaFetchError symbols so the new test
follows the same style and assertions as test_get_body_raises_on_http_error and
test_get_body_succeeds_with_json_response.
tests/parser/test_jsonschema.py (1)

58-59: Use concrete Content-Type values in the mocked success responses.

The parser now validates response metadata before parsing remote schemas, but headers = {} means these tests don't actually pin the allowed JSON/YAML media types. Setting representative Content-Type headers here would make the success-path coverage match the contract introduced by this PR.

Also applies to: 101-102, 141-142, 174-175

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/parser/test_jsonschema.py` around lines 58 - 59, The mocked HTTP
responses in tests use mock_get.return_value.headers = {} so they don't assert
the parser's Content-Type validation; update the tests to set concrete
Content-Type header values on mock_get.return_value.headers (e.g.,
'Content-Type': 'application/json' for JSON-schema success cases and
'Content-Type': 'application/x-yaml' or 'text/yaml' for YAML cases) at the
occurrences where mock_get.return_value.status_code and headers are set (the
mock_get.return_value.headers assignments around the blocks at lines referenced
in the review: 58-59, 101-102, 141-142, and 174-175) so the success-path tests
exercise the media-type checks used by the parser.
tests/main/jsonschema/test_main_jsonschema.py (1)

816-821: Consider a shared mock-response helper for the new HTTP validation contract.

These tests now repeat the same status_code/headers scaffolding in many places. A tiny factory/helper would make future response-validation changes easier to update consistently.

Also applies to: 857-862, 892-893, 911-912, 942-947, 2116-2117, 2260-2261, 7199-7200, 8953-8954

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/main/jsonschema/test_main_jsonschema.py` around lines 816 - 821, Tests
repeat manual construction of mock responses (setting status_code, headers,
text) which is error-prone; introduce a small factory function (e.g.,
make_mock_response or build_response_mock) used by tests in
tests/main/jsonschema/test_main_jsonschema.py to create configured mock
responses. The helper should accept common params (status_code, headers, text,
json/data) and return a mock object with those attributes so you can replace the
repeated blocks that set root_id_response and person_response (and the other
occurrences listed) with calls to make_mock_response(status_code=200,
headers={}, text="dummy") to centralize scaffolding and simplify future changes.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@src/datamodel_code_generator/http.py`:
- Around line 40-57: Network/transport errors from httpx.get can leak raw httpx
exceptions and Content-Type checks are case-sensitive; wrap the HTTP call in a
try/except that catches httpx.RequestError (and other httpx transport
exceptions) and re-raise a SchemaFetchError with the underlying error message,
and after receiving the response normalize the content type (e.g., content_type
= response.headers.get("content-type", "").lower()) before checking for
"text/html" (or matching JSON/YAML substrings) so comparisons are
case-insensitive; reference the httpx.get call, the response variable, the
content_type variable, and SchemaFetchError when making these changes.

In `@tests/main/jsonschema/test_main_jsonschema.py`:
- Around line 1010-1031: Update the assertion in
test_main_missing_local_ref_error_message so the stderr check verifies the
actual attempted ref path is included (e.g., "nonexistent.json" or the resolved
tmp_path string) instead of only the generic prefix; modify the call to
run_main_and_assert (or its expected_stderr_contains argument) to assert the
error message contains the specific ref filename "nonexistent.json" (or tmp_path
/ "nonexistent.json") so the test ensures the code surfaces which local $ref
failed.
- Around line 984-1007: The test test_main_remote_ref_blocked_by_default should
explicitly assert that no HTTP fetch occurs: patch httpx.get (or the HTTP client
used) at the start of the test, run run_main_and_assert with the same inputs,
and after the call assert the patched httpx.get was not called; update the test
function test_main_remote_ref_blocked_by_default to use a mock/monkeypatch for
httpx.get and assert no calls to it so the test guarantees no network request is
made even if stderr checking changes.

---

Nitpick comments:
In `@tests/main/jsonschema/test_main_jsonschema.py`:
- Around line 816-821: Tests repeat manual construction of mock responses
(setting status_code, headers, text) which is error-prone; introduce a small
factory function (e.g., make_mock_response or build_response_mock) used by tests
in tests/main/jsonschema/test_main_jsonschema.py to create configured mock
responses. The helper should accept common params (status_code, headers, text,
json/data) and return a mock object with those attributes so you can replace the
repeated blocks that set root_id_response and person_response (and the other
occurrences listed) with calls to make_mock_response(status_code=200,
headers={}, text="dummy") to centralize scaffolding and simplify future changes.

In `@tests/parser/test_jsonschema.py`:
- Around line 58-59: The mocked HTTP responses in tests use
mock_get.return_value.headers = {} so they don't assert the parser's
Content-Type validation; update the tests to set concrete Content-Type header
values on mock_get.return_value.headers (e.g., 'Content-Type':
'application/json' for JSON-schema success cases and 'Content-Type':
'application/x-yaml' or 'text/yaml' for YAML cases) at the occurrences where
mock_get.return_value.status_code and headers are set (the
mock_get.return_value.headers assignments around the blocks at lines referenced
in the review: 58-59, 101-102, 141-142, and 174-175) so the success-path tests
exercise the media-type checks used by the parser.

In `@tests/test_http.py`:
- Around line 17-60: Add a regression test that mocks httpx.get to raise a
transport error and verifies get_body raises SchemaFetchError: create a new test
(e.g., test_get_body_raises_on_transport_error) using MockerFixture to patch
"httpx.get" to raise an httpx.TransportError (or httpx.ConnectTimeout) when
called, then call get_body("https://example.com/schema.json") and assert it
raises SchemaFetchError (matching an appropriate message); reference the
existing tests and the get_body and SchemaFetchError symbols so the new test
follows the same style and assertions as test_get_body_raises_on_http_error and
test_get_body_succeeds_with_json_response.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 406afe18-7bde-422f-9922-8e134b30404a

📥 Commits

Reviewing files that changed from the base of the PR and between 5bd8eab and 3dbf428.

📒 Files selected for processing (19)
  • src/datamodel_code_generator/__init__.py
  • src/datamodel_code_generator/__main__.py
  • src/datamodel_code_generator/_types/generate_config_dict.py
  • src/datamodel_code_generator/_types/parser_config_dicts.py
  • src/datamodel_code_generator/arguments.py
  • src/datamodel_code_generator/cli_options.py
  • src/datamodel_code_generator/config.py
  • src/datamodel_code_generator/http.py
  • src/datamodel_code_generator/parser/base.py
  • src/datamodel_code_generator/parser/jsonschema.py
  • tests/data/expected/main/input_model/config_class.py
  • tests/main/jsonschema/test_main_jsonschema.py
  • tests/main/openapi/test_main_openapi.py
  • tests/main/test_main_json.py
  • tests/main/test_public_api_signature_baseline.py
  • tests/parser/test_jsonschema.py
  • tests/parser/test_openapi.py
  • tests/test_http.py
  • tests/test_main_kr.py

Comment thread src/datamodel_code_generator/http.py Outdated
Comment thread tests/main/jsonschema/test_main_jsonschema.py Outdated
Comment thread tests/main/jsonschema/test_main_jsonschema.py
Catch httpx transport exceptions (DNS, timeout, connection errors) and
wrap them in SchemaFetchError. Normalize Content-Type to lowercase
before checking for text/html.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (1)
src/datamodel_code_generator/http.py (1)

49-51: ⚠️ Potential issue | 🟠 Major

Narrow the exception handler to httpx.RequestError only.

Line 49 currently catches Exception, which can hide programming errors and unexpected failures. Since only httpx.get() is called in the try block, catch httpx.RequestError instead to handle transport/network errors specifically while allowing other exceptions to surface.

Proposed change
-    except Exception as e:
+    except httpx.RequestError as e:
         msg = f"Failed to fetch {url}: {e}"
         raise SchemaFetchError(msg) from e
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/datamodel_code_generator/http.py` around lines 49 - 51, Narrow the broad
exception handler around the httpx.get() call so it only catches
transport/network errors: replace "except Exception as e" with "except
httpx.RequestError as e" (the block that builds msg = f'Failed to fetch {url}:
{e}' and raises SchemaFetchError(msg) from e) so other exceptions bubble up
while network/request errors are wrapped in SchemaFetchError.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Duplicate comments:
In `@src/datamodel_code_generator/http.py`:
- Around line 49-51: Narrow the broad exception handler around the httpx.get()
call so it only catches transport/network errors: replace "except Exception as
e" with "except httpx.RequestError as e" (the block that builds msg = f'Failed
to fetch {url}: {e}' and raises SchemaFetchError(msg) from e) so other
exceptions bubble up while network/request errors are wrapped in
SchemaFetchError.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 5be70c6d-1fd6-464f-a7a0-345e4a4ca699

📥 Commits

Reviewing files that changed from the base of the PR and between 3dbf428 and 5341a91.

📒 Files selected for processing (2)
  • src/datamodel_code_generator/http.py
  • tests/test_http.py
🚧 Files skipped from review as they are similar to previous changes (1)
  • tests/test_http.py

Assert httpx.get is not called when remote refs are blocked, preventing
real HTTP leaks if the gate regresses. Assert the specific file path
in the missing local ref error, not just the generic prefix.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
tests/main/jsonschema/test_main_jsonschema.py (1)

813-831: Consider extracting a small HTTP mock-response factory to reduce repetition.

The repeated Mock() + status_code + headers + text setup appears many times; a helper would improve maintainability and reduce drift in future updates.

Also applies to: 847-875, 889-904, 908-927, 939-960

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/main/jsonschema/test_main_jsonschema.py` around lines 813 - 831,
Extract a small HTTP mock-response factory (e.g., make_http_response(text,
status_code=200, headers=None)) and replace the repeated mocker.Mock() +
status_code + headers + text setup in
test_main_root_id_jsonschema_with_local_file and the other JSON Schema tests in
the same file; update the tests to call the factory to create
person_response/root_id_response and keep the existing httpx.get
patching/assertions unchanged (reference the test function name
test_main_root_id_jsonschema_with_local_file and the other JSON Schema tests in
this module to locate all occurrences).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@tests/main/jsonschema/test_main_jsonschema.py`:
- Around line 813-831: Extract a small HTTP mock-response factory (e.g.,
make_http_response(text, status_code=200, headers=None)) and replace the
repeated mocker.Mock() + status_code + headers + text setup in
test_main_root_id_jsonschema_with_local_file and the other JSON Schema tests in
the same file; update the tests to call the factory to create
person_response/root_id_response and keep the existing httpx.get
patching/assertions unchanged (reference the test function name
test_main_root_id_jsonschema_with_local_file and the other JSON Schema tests in
this module to locate all occurrences).

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 04af1f13-5bd1-470c-8047-d56f66df7c5f

📥 Commits

Reviewing files that changed from the base of the PR and between 5341a91 and cdf4360.

📒 Files selected for processing (1)
  • tests/main/jsonschema/test_main_jsonschema.py

@koxudaxi

Copy link
Copy Markdown
Owner

@butvinm
Thanks for the PR! This is a real issue.
I've seen it reported several times and agree it should be fixed.

My concern is that changing the default to False would be a breaking change.
There are users who rely on implicit remote $ref resolution today, and their schemas would suddenly stop working after upgrading.

How about this instead? Introduce --allow-remote-refs but keep the default as True for now, and emit a deprecation warning when remote refs are fetched implicitly.
Then we flip the default to False in the next major version. That way users get a heads-up before anything breaks.

The error message improvements (clearer FileNotFoundError, HTML response validation) are great. I'd like to merge those regardless.

Per maintainer feedback, keep backward compatibility by allowing remote
$ref fetching by default but emit a FutureWarning when it happens
without explicit --allow-remote-refs. The flag becomes a three-state:
True (explicit opt-in, no warning), False (blocks), None (default,
allows with deprecation warning). The default will flip to False in
a future major version.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (3)
src/datamodel_code_generator/__main__.py (1)

1085-1088: Consider respecting explicit user setting when --url is provided.

Currently, allow_remote_refs is unconditionally set to True when config.url is provided. This would override an explicit --no-allow-remote-refs setting from the user. A user might want to fetch the main schema via --url while still blocking remote $ref resolution within that schema.

♻️ Proposed fix to only set when not explicitly configured
     # --url implies --allow-remote-refs since the user is explicitly fetching a remote schema
-    if config.url:
+    if config.url and config.allow_remote_refs is None:
         config.allow_remote_refs = True
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/datamodel_code_generator/__main__.py` around lines 1085 - 1088, The
current unconditional assignment in __main__.py sets config.allow_remote_refs =
True whenever config.url is provided, which overrides an explicit user choice;
change the logic so that when config.url is truthy you only set
config.allow_remote_refs to True if the option was not explicitly provided by
the user (e.g., check for a sentinel like None or a flag that indicates it was
unset). Update the block around the config.url handling to test the explicitness
of allow_remote_refs (e.g., if config.url and config.allow_remote_refs is None:
config.allow_remote_refs = True) so an explicit --no-allow-remote-refs or
--allow-remote-refs from the CLI is respected.
tests/main/jsonschema/test_main_jsonschema.py (1)

985-1015: Assert the warning-path test actually exercised remote fetch.

Line 994 patches the fetch function, but the test currently only checks for a FutureWarning. Add a call assertion so this test proves implicit remote resolution happened.

♻️ Suggested hardening
-    mocker.patch("httpx.get", return_value=person_response)
+    httpx_get_mock = mocker.patch("httpx.get", return_value=person_response)
@@
     with pytest.warns(FutureWarning, match="--allow-remote-refs"):
         run_main_and_assert(
             input_path=input_file,
             output_path=output_file,
             input_file_type="jsonschema",
         )
+    httpx_get_mock.assert_called()
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/main/jsonschema/test_main_jsonschema.py` around lines 985 - 1015, The
test test_main_remote_ref_emits_deprecation_warning patches httpx.get but never
asserts it was invoked, so add an assertion after run_main_and_assert to verify
the remote fetch actually happened; specifically, reference the patched
"httpx.get" mock (or the mock object returned, e.g., person_response) and assert
it was called (optionally with the expected URL
"https://example.com/other/schema.json" or at least assert_called_once()) so the
test proves implicit remote resolution occurred.
src/datamodel_code_generator/parser/jsonschema.py (1)

3821-3837: Scope the allow_remote_refs gate to HTTP/HTTPS explicitly.

Right now, Line 3821 only excludes file://; all other URL schemes are treated as “remote HTTP” in gating/warning text. Using explicit scheme checks keeps behavior aligned with the flag semantics and messaging.

Suggested patch
@@
-from urllib.parse import ParseResult, unquote
+from urllib.parse import ParseResult, unquote, urlparse
@@
     def _get_ref_body(self, resolved_ref: str) -> dict[str, YamlValue]:
         """Get the body of a reference from URL or remote file."""
         if is_url(resolved_ref):
-            if not resolved_ref.startswith("file://"):
+            scheme = urlparse(resolved_ref).scheme.lower()
+            is_http_ref = scheme in {"http", "https"}
+            if is_http_ref:
                 if self.allow_remote_refs is False:
                     msg = (
                         f"Fetching remote $ref is disabled: {resolved_ref}\n"
                         "Use --allow-remote-refs to enable HTTP fetching of remote references."
                     )
                     raise Error(msg)
                 if self.allow_remote_refs is None:
                     import warnings  # noqa: PLC0415
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/datamodel_code_generator/parser/jsonschema.py` around lines 3821 - 3837,
The current gate uses only resolved_ref.startswith("file://") so non-file but
non-HTTP schemes are incorrectly treated as remote HTTP; update the conditional
around allow_remote_refs to explicitly check for HTTP/HTTPS (e.g., if
resolved_ref.lower().startswith(("http://","https://")):) before applying the
allow_remote_refs False error and the None warning. Modify the branch that
references allow_remote_refs, resolved_ref and the warning/error text to run
only for HTTP(S) refs so other schemes (data:, ftp:, etc.) are not affected.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@src/datamodel_code_generator/__main__.py`:
- Around line 1085-1088: The current unconditional assignment in __main__.py
sets config.allow_remote_refs = True whenever config.url is provided, which
overrides an explicit user choice; change the logic so that when config.url is
truthy you only set config.allow_remote_refs to True if the option was not
explicitly provided by the user (e.g., check for a sentinel like None or a flag
that indicates it was unset). Update the block around the config.url handling to
test the explicitness of allow_remote_refs (e.g., if config.url and
config.allow_remote_refs is None: config.allow_remote_refs = True) so an
explicit --no-allow-remote-refs or --allow-remote-refs from the CLI is
respected.

In `@src/datamodel_code_generator/parser/jsonschema.py`:
- Around line 3821-3837: The current gate uses only
resolved_ref.startswith("file://") so non-file but non-HTTP schemes are
incorrectly treated as remote HTTP; update the conditional around
allow_remote_refs to explicitly check for HTTP/HTTPS (e.g., if
resolved_ref.lower().startswith(("http://","https://")):) before applying the
allow_remote_refs False error and the None warning. Modify the branch that
references allow_remote_refs, resolved_ref and the warning/error text to run
only for HTTP(S) refs so other schemes (data:, ftp:, etc.) are not affected.

In `@tests/main/jsonschema/test_main_jsonschema.py`:
- Around line 985-1015: The test test_main_remote_ref_emits_deprecation_warning
patches httpx.get but never asserts it was invoked, so add an assertion after
run_main_and_assert to verify the remote fetch actually happened; specifically,
reference the patched "httpx.get" mock (or the mock object returned, e.g.,
person_response) and assert it was called (optionally with the expected URL
"https://example.com/other/schema.json" or at least assert_called_once()) so the
test proves implicit remote resolution occurred.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: c66d847b-730b-4788-8d60-4fefb8607ded

📥 Commits

Reviewing files that changed from the base of the PR and between cdf4360 and 2c325e6.

📒 Files selected for processing (10)
  • src/datamodel_code_generator/__main__.py
  • src/datamodel_code_generator/_types/generate_config_dict.py
  • src/datamodel_code_generator/_types/parser_config_dicts.py
  • src/datamodel_code_generator/arguments.py
  • src/datamodel_code_generator/config.py
  • src/datamodel_code_generator/parser/base.py
  • src/datamodel_code_generator/parser/jsonschema.py
  • tests/data/expected/main/input_model/config_class.py
  • tests/main/jsonschema/test_main_jsonschema.py
  • tests/main/test_public_api_signature_baseline.py
🚧 Files skipped from review as they are similar to previous changes (5)
  • tests/main/test_public_api_signature_baseline.py
  • src/datamodel_code_generator/parser/base.py
  • src/datamodel_code_generator/_types/parser_config_dicts.py
  • src/datamodel_code_generator/arguments.py
  • src/datamodel_code_generator/_types/generate_config_dict.py

@butvinm

butvinm commented Mar 18, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the feedback! Updated the PR:

--allow-remote-refs is now a three-state flag:

  • Not set (default): remote refs still work but emit a FutureWarning pointing users to --allow-remote-refs
  • --allow-remote-refs: explicit opt-in, no warning
  • allow_remote_refs=False (library API): blocks remote fetching with a clear error, default behavior in future

Example output for the original issue (schema with $id + typo'd $ref):

FutureWarning: Fetching remote $ref without --allow-remote-refs: https://open-metadata.org/schema/auth/../incorrectPath/basic.json
In a future version, remote $ref fetching will be disabled by default.
Pass --allow-remote-refs explicitly to silence this warning.

HTTP 404 error fetching https://open-metadata.org/schema/auth/../incorrectPath/basic.json

koxudaxi
koxudaxi previously approved these changes Mar 19, 2026
@ilovelinux

ilovelinux commented Mar 19, 2026

Copy link
Copy Markdown
Collaborator

@koxudaxi @butvinm what if we allow --no-allow-remote-refs as it has been done for --use-annotated?

See: https://docs.python.org/3/library/argparse.html#argparse.BooleanOptionalAction

Use BooleanOptionalAction (like --use-annotated) so users can
explicitly opt out with --no-allow-remote-refs from the CLI.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
BooleanOptionalAction registers both --allow-remote-refs and
--no-allow-remote-refs in argparse; the sync test requires both
to be in CLI_OPTION_META.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@koxudaxi koxudaxi merged commit f6d4cbd into koxudaxi:main Apr 1, 2026
39 checks passed
@github-actions

github-actions Bot commented Apr 1, 2026

Copy link
Copy Markdown
Contributor

Breaking Change Analysis

Result: Breaking changes detected

Reasoning: The PR introduces three categories of breaking changes: (1) Missing local $ref files now raise Error instead of FileNotFoundError - a change in exception type that affects programmatic callers. (2) HTTP failures now raise SchemaFetchError early instead of letting bad content propagate to the YAML/JSON parser, changing what exceptions surface for these error cases. (3) A FutureWarning is emitted for implicit remote ref fetching, which could break CI/CD pipelines with strict warning filters. The core functionality is preserved (remote refs still work by default), but the error handling surface has changed in ways that could affect programmatic users.

Content for Release Notes

Error Handling Changes

  • Missing local $ref now raises Error instead of FileNotFoundError - Previously, when a $ref pointed to a non-existent local file, a raw FileNotFoundError propagated to callers. Now it raises datamodel_code_generator.Error with the message "$ref file not found: <path>". Programmatic users catching FileNotFoundError specifically will need to catch Error instead (Add --allow-remote-refs to disable HTTP fetching of $ref by default #3051)
  • HTTP fetch failures now raise SchemaFetchError instead of propagating raw exceptions - HTTP errors (4xx/5xx status codes), unexpected HTML responses, and transport errors (DNS, timeout, connection) that previously resulted in downstream YAML/JSON parse errors or raw httpx exceptions now raise SchemaFetchError (a subclass of Error) before parsing is attempted. Users catching specific parse errors or httpx exceptions for these scenarios will need to update their error handling (Add --allow-remote-refs to disable HTTP fetching of $ref by default #3051)

Default Behavior Changes

  • Implicit remote $ref fetching now emits FutureWarning - When a $ref resolves to an HTTP(S) URL and --allow-remote-refs is not explicitly passed, the tool still fetches the remote reference but emits a FutureWarning. This may cause failures in environments running with -W error (warnings as errors) or strict warning filters. Pass --allow-remote-refs explicitly to suppress the warning (Add --allow-remote-refs to disable HTTP fetching of $ref by default #3051)

This analysis was performed by Claude Code Action

@github-actions

github-actions Bot commented Apr 4, 2026

Copy link
Copy Markdown
Contributor

🎉 Released in 0.56.0

This PR is now available in the latest release. See the release notes for details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Local $ref to missing file triggers unexpected HTTP request and confusing error

4 participants