Cache parsed sources

[koxudaxi]

No description provided.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds an opt-in enable_parsed_source_cache flag (default False) to config, TypedDicts, and parser base classes. Implements a thread-safe, SHA-256 digest-keyed LRU in-memory cache for parsed source data in the core loading module and a separate two-tier LRU cache for XML Schema text and converted schema data with dependency-file tracking.

Changes

Parser Source Caching Feature

Layer / File(s)	Summary
Config contracts and TypedDict schemas src/datamodel_code_generator/_types/generate_config_dict.py, src/datamodel_code_generator/_types/parser_config_dicts.py, src/datamodel_code_generator/config.py, tests/data/expected/main/input_model/config_class.py	Adds enable_parsed_source_cache: NotRequired[bool] to GenerateConfigDict and ParserConfigDict, and enable_parsed_source_cache: bool = False to GenerateConfig and ParserConfig.
Core digest-keyed LRU cache src/datamodel_code_generator/__init__.py	Introduces thread-safe OrderedDict+RLock cache with max-size constant. Implements _load_parser_source_data_from_path() with SHA-256 digest keys and LRU eviction, _load_parser_source_data_from_bytes() for JSON/YAML byte decoding, and updates _clear_parser_source_data_cache().
Source.from_path and Parser.iter_source plumbing src/datamodel_code_generator/parser/base.py	Adds keyword-only enable_parsed_source_cache and keep_text parameters to Source.from_path; adds Parser._cache_parsed_sources_from_path class variable; computes self.enable_parsed_source_cache from config; threads both flags through all from_path call sites.
JSON Schema, OpenAPI, Avro, Protobuf cache flags src/datamodel_code_generator/parser/jsonschema.py, src/datamodel_code_generator/parser/openapi.py, src/datamodel_code_generator/parser/avro.py, src/datamodel_code_generator/parser/protobuf.py	Sets _cache_parsed_sources_from_path = True on JsonSchemaParser; introduces _load_source_dict() to enforce dict type validation; updates parse_raw() to catch TypeError and warn on invalid cached data; adjusts OpenAPIParser warning condition; opts AvroParser and ProtobufParser out with False.
XML Schema LRU caches with dependency tracking src/datamodel_code_generator/parser/xmlschema.py	Adds two LRU caches (decoded XML text; converted schema data with dependency freshness checks). Implements _read_xml_text, _load_xml_schema_data_from_path, and _source_from_xml_path for parsed-source-cache-aware loading; records all included/imported file paths in _XMLSchemaConverter.loaded_source_paths for invalidation; routes iter_source through _source_from_xml_path.
Test helpers and public API baseline tests/main/conftest.py, tests/main/test_public_api_signature_baseline.py	Adds generic cache assertion helpers (_value_at_path, assert_path_cache_reuses_value, assert_path_cache_invalidates_after_write); extends public API signature baseline with enable_parsed_source_cache: bool = False.
JSON Schema cache integration tests tests/main/jsonschema/test_main_jsonschema.py	Adds E2E test for generation with parsed-source caching enabled; adds unit tests for load_data_from_path() cache hit and invalidation; updates YAML-root caching test to enable parsed-source caching.
XML Schema cache integration tests tests/main/xmlschema/test_main_xmlschema.py	Adds E2E test for XML Schema generation with parsed-source caching enabled; adds unit tests for _read_xml_text() cache hit and invalidation behavior.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• koxudaxi/datamodel-code-generator#2832: Both PRs update the public API signature baseline in tests/main/test_public_api_signature_baseline.py with new keyword-only parameters for signature/config compatibility assertions.
• koxudaxi/datamodel-code-generator#3393: Both PRs modify src/datamodel_code_generator/parser/base.py to change source iteration and caching behavior around Source.from_path and Parser.iter_source.
• koxudaxi/datamodel-code-generator#2777: Main PR's new load_data_from_path() parser-source loader routes non-.json inputs through load_yaml(), and the retrieved PR modifies load_yaml() to use cached YAML parsing.

Suggested labels

breaking-change-analyzed, breaking-change

🐇 A cache of schemas, hashed with care,
SHA-256 keys floating through the air.
JSON and YAML, XML too—
Parsed only once, then stored for you!
LRU evicts the stale and old,
And thread-safe locks keep all controlled. 🗝️

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 49.12% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'Cache parsed sources' directly and clearly describes the main change: adding caching for parsed data sources throughout the codebase.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch parsed-source-cache

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

📚 Docs Preview: https://pr-3407.datamodel-code-generator.pages.dev

[codspeed-hq]

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 11 untouched benchmarks
⏩ 98 skipped benchmarks¹

---

_{Comparing parsed-source-cache (fe644e6) with main (93e2fe3)}

Footnotes

1. 98 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

src/datamodel_code_generator/parser/xmlschema.py (1)

332-335: ⚖️ Poor tradeoff

Consider moving freshness check outside the lock to reduce contention.

_xml_schema_cache_entry_is_fresh(entry) performs file I/O (path.is_file() and _digest_path()) for each dependency while holding _xml_schema_data_cache_lock. For schemas with many includes/imports, this could cause lock contention and block concurrent cache access.

Consider restructuring to:

1. Retrieve entry inside lock
2. Release lock
3. Check freshness outside lock
4. Reacquire lock only if cache update needed

The race condition this introduces (another thread updating cache between steps 2-4) is benign for a cache — worst case is redundant computation.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/datamodel_code_generator/parser/xmlschema.py` around lines 332 - 335, The
`_xml_schema_cache_entry_is_fresh()` function performs file I/O operations while
the `_xml_schema_data_cache_lock` is held, causing unnecessary lock contention.
Restructure the code to retrieve the cache entry and release the lock first,
then perform the freshness check outside the critical section. Only reacquire
the lock if the cache needs to be updated (i.e., if the entry is not fresh or
not found). This reduces lock duration and allows concurrent cache access
without blocking on expensive file I/O operations.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/datamodel_code_generator/parser/base.py`:
- Around line 966-970: The issue is that within the enable_parser_cache branch,
the file is read twice separately: once via path.read_text() for the text field
and once via _load_parser_source_data_from_path() for the raw_data field. A
concurrent file modification between these two reads can cause text and raw_data
to contain inconsistent content from different file versions. To fix this, read
the file once to obtain the raw bytes, then derive both the text field and
raw_data from that single read operation. Refactor the code to call
path.read_bytes() once and then either decode the bytes for text and pass the
bytes to _load_parser_source_data_from_path, or restructure
_load_parser_source_data_from_path to accept pre-read bytes to avoid a second
read operation.

---

Nitpick comments:
In `@src/datamodel_code_generator/parser/xmlschema.py`:
- Around line 332-335: The `_xml_schema_cache_entry_is_fresh()` function
performs file I/O operations while the `_xml_schema_data_cache_lock` is held,
causing unnecessary lock contention. Restructure the code to retrieve the cache
entry and release the lock first, then perform the freshness check outside the
critical section. Only reacquire the lock if the cache needs to be updated
(i.e., if the entry is not fresh or not found). This reduces lock duration and
allows concurrent cache access without blocking on expensive file I/O
operations.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: b15b3f6f-6657-4489-8e50-5a1c043ac7a4

📥 Commits

Reviewing files that changed from the base of the PR and between 93e2fe3 and 04be419.

📒 Files selected for processing (13)

• src/datamodel_code_generator/__init__.py
• src/datamodel_code_generator/_types/generate_config_dict.py
• src/datamodel_code_generator/_types/parser_config_dicts.py
• src/datamodel_code_generator/config.py
• src/datamodel_code_generator/parser/avro.py
• src/datamodel_code_generator/parser/base.py
• src/datamodel_code_generator/parser/jsonschema.py
• src/datamodel_code_generator/parser/openapi.py
• src/datamodel_code_generator/parser/protobuf.py
• src/datamodel_code_generator/parser/xmlschema.py
• tests/main/jsonschema/test_main_jsonschema.py
• tests/main/test_public_api_signature_baseline.py
• tests/main/xmlschema/test_main_xmlschema.py

[coderabbitai]

♻️ Duplicate comments (1)

src/datamodel_code_generator/parser/base.py (1)

966-970: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Read source bytes once when cache + text retention are both enabled.

Line 969 and Line 970 can observe different file revisions because this path performs two independent disk reads. Build both text and raw_data from the same byte buffer to keep Source internally consistent.

Suggested fix

     def from_path(
         cls,
         path: Path,
         base_path: Path,
         encoding: str,
         *,
         enable_parsed_source_cache: bool = False,
         keep_text: bool = False,
     ) -> Source:
         """Create a Source from a file path relative to base_path."""
         if enable_parsed_source_cache:
+            data = path.read_bytes()
             return cls(
                 path=path.relative_to(base_path),
-                text=path.read_text(encoding=encoding) if keep_text else "",
-                raw_data=_load_parser_source_data_from_path(path, encoding),
+                text=data.decode(encoding) if keep_text else "",
+                raw_data=_load_parser_source_data_from_bytes(path, data, encoding),
             )
         return cls(
             path=path.relative_to(base_path),
             text=path.read_text(encoding=encoding),
         )

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/datamodel_code_generator/parser/base.py` around lines 966 - 970, When
enable_parsed_source_cache is true, the code currently performs two independent
disk reads: one via path.read_text() on line 969 and another via
_load_parser_source_data_from_path() on line 970, which can observe different
file revisions if the file is modified between reads. Refactor this section to
read the source bytes once into a single buffer, then derive both the text and
raw_data parameters from that same buffer, passing it to
_load_parser_source_data_from_path() or decoding it directly for the text field
based on the keep_text condition, ensuring the Source instance remains
internally consistent.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Duplicate comments:
In `@src/datamodel_code_generator/parser/base.py`:
- Around line 966-970: When enable_parsed_source_cache is true, the code
currently performs two independent disk reads: one via path.read_text() on line
969 and another via _load_parser_source_data_from_path() on line 970, which can
observe different file revisions if the file is modified between reads. Refactor
this section to read the source bytes once into a single buffer, then derive
both the text and raw_data parameters from that same buffer, passing it to
_load_parser_source_data_from_path() or decoding it directly for the text field
based on the keep_text condition, ensuring the Source instance remains
internally consistent.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: f7ddc8f7-0eeb-4bcf-9239-ece36404c733

📥 Commits

Reviewing files that changed from the base of the PR and between 04be419 and 153e010.

📒 Files selected for processing (15)

• src/datamodel_code_generator/__init__.py
• src/datamodel_code_generator/_types/generate_config_dict.py
• src/datamodel_code_generator/_types/parser_config_dicts.py
• src/datamodel_code_generator/config.py
• src/datamodel_code_generator/parser/avro.py
• src/datamodel_code_generator/parser/base.py
• src/datamodel_code_generator/parser/jsonschema.py
• src/datamodel_code_generator/parser/openapi.py
• src/datamodel_code_generator/parser/protobuf.py
• src/datamodel_code_generator/parser/xmlschema.py
• tests/data/expected/main/input_model/config_class.py
• tests/main/conftest.py
• tests/main/jsonschema/test_main_jsonschema.py
• tests/main/test_public_api_signature_baseline.py
• tests/main/xmlschema/test_main_xmlschema.py

🚧 Files skipped from review as they are similar to previous changes (7)

• src/datamodel_code_generator/_types/parser_config_dicts.py
• src/datamodel_code_generator/parser/protobuf.py
• src/datamodel_code_generator/parser/avro.py
• src/datamodel_code_generator/parser/jsonschema.py
• src/datamodel_code_generator/parser/openapi.py
• src/datamodel_code_generator/init.py
• src/datamodel_code_generator/parser/xmlschema.py

[coderabbitai]

🧹 Nitpick comments (1)

src/datamodel_code_generator/parser/xmlschema.py (1)

347-350: 💤 Low value

File I/O occurs while holding the cache lock.

_xml_schema_cache_entry_is_fresh(entry) calls _digest_path() for each dependency, performing file reads while _xml_schema_data_cache_lock is held. Under concurrent access this can cause contention.

Consider extracting the entry under lock, releasing the lock, then performing the freshness check outside:

♻️ Suggested refactor

     with _xml_schema_data_cache_lock:
-        if (entry := _xml_schema_data_cache.get(cache_key)) is not None and _xml_schema_cache_entry_is_fresh(entry):
+        entry = _xml_schema_data_cache.get(cache_key)
+
+    if entry is not None and _xml_schema_cache_entry_is_fresh(entry):
+        with _xml_schema_data_cache_lock:
+            # Re-check entry is still present before returning
+            if _xml_schema_data_cache.get(cache_key) is entry:
+                _xml_schema_data_cache.move_to_end(cache_key)
-            _xml_schema_data_cache.move_to_end(cache_key)
-            return _copy_schema(entry.data)
+                return _copy_schema(entry.data)

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/datamodel_code_generator/parser/xmlschema.py` around lines 347 - 350, The
code performs file I/O through _xml_schema_cache_entry_is_fresh() while holding
_xml_schema_data_cache_lock, causing potential contention under concurrent
access. Extract the cache entry under the lock, then release the lock
immediately before checking freshness. Move the
_xml_schema_cache_entry_is_fresh(entry) check and the subsequent
_xml_schema_data_cache.move_to_end() and _copy_schema() calls outside the lock
context to avoid blocking other threads during file I/O operations.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/datamodel_code_generator/parser/xmlschema.py`:
- Around line 347-350: The code performs file I/O through
_xml_schema_cache_entry_is_fresh() while holding _xml_schema_data_cache_lock,
causing potential contention under concurrent access. Extract the cache entry
under the lock, then release the lock immediately before checking freshness.
Move the _xml_schema_cache_entry_is_fresh(entry) check and the subsequent
_xml_schema_data_cache.move_to_end() and _copy_schema() calls outside the lock
context to avoid blocking other threads during file I/O operations.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: eabf5675-8c8a-444b-9be4-7c568f27ce38

📥 Commits

Reviewing files that changed from the base of the PR and between 153e010 and b749b49.

📒 Files selected for processing (15)

• src/datamodel_code_generator/__init__.py
• src/datamodel_code_generator/_types/generate_config_dict.py
• src/datamodel_code_generator/_types/parser_config_dicts.py
• src/datamodel_code_generator/config.py
• src/datamodel_code_generator/parser/avro.py
• src/datamodel_code_generator/parser/base.py
• src/datamodel_code_generator/parser/jsonschema.py
• src/datamodel_code_generator/parser/openapi.py
• src/datamodel_code_generator/parser/protobuf.py
• src/datamodel_code_generator/parser/xmlschema.py
• tests/data/expected/main/input_model/config_class.py
• tests/main/conftest.py
• tests/main/jsonschema/test_main_jsonschema.py
• tests/main/test_public_api_signature_baseline.py
• tests/main/xmlschema/test_main_xmlschema.py

🚧 Files skipped from review as they are similar to previous changes (12)

• tests/data/expected/main/input_model/config_class.py
• src/datamodel_code_generator/_types/parser_config_dicts.py
• src/datamodel_code_generator/_types/generate_config_dict.py
• src/datamodel_code_generator/config.py
• src/datamodel_code_generator/parser/avro.py
• tests/main/test_public_api_signature_baseline.py
• src/datamodel_code_generator/parser/openapi.py
• src/datamodel_code_generator/parser/protobuf.py
• tests/main/xmlschema/test_main_xmlschema.py
• src/datamodel_code_generator/parser/jsonschema.py
• src/datamodel_code_generator/parser/base.py
• src/datamodel_code_generator/init.py

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/datamodel_code_generator/parser/jsonschema.py`:
- Around line 5356-5361: The `_load_source_dict` function returns the result of
`load_data(source.text)` without validating its type, while it properly
validates `source.raw_data` to ensure it is a dict. Since callers assume the
return value is a dict and call `.get(...)` on it, non-dict payloads loaded from
text will cause crashes. After calling `load_data(source.text)` when
`source.raw_data is None`, add a type check using `isinstance()` to validate
that the result is a dict. If the result is not a dict, raise a TypeError with a
message indicating the unexpected type (following the same pattern as the
validation for `source.raw_data`), before returning the data.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 3f4ffa1b-0a8d-41d6-a776-e1166a108d2c

📥 Commits

Reviewing files that changed from the base of the PR and between b749b49 and a9ae3c4.

📒 Files selected for processing (15)

• src/datamodel_code_generator/__init__.py
• src/datamodel_code_generator/_types/generate_config_dict.py
• src/datamodel_code_generator/_types/parser_config_dicts.py
• src/datamodel_code_generator/config.py
• src/datamodel_code_generator/parser/avro.py
• src/datamodel_code_generator/parser/base.py
• src/datamodel_code_generator/parser/jsonschema.py
• src/datamodel_code_generator/parser/openapi.py
• src/datamodel_code_generator/parser/protobuf.py
• src/datamodel_code_generator/parser/xmlschema.py
• tests/data/expected/main/input_model/config_class.py
• tests/main/conftest.py
• tests/main/jsonschema/test_main_jsonschema.py
• tests/main/test_public_api_signature_baseline.py
• tests/main/xmlschema/test_main_xmlschema.py

🚧 Files skipped from review as they are similar to previous changes (13)

• src/datamodel_code_generator/_types/generate_config_dict.py
• src/datamodel_code_generator/parser/avro.py
• tests/main/test_public_api_signature_baseline.py
• src/datamodel_code_generator/config.py
• src/datamodel_code_generator/parser/protobuf.py
• src/datamodel_code_generator/parser/openapi.py
• tests/data/expected/main/input_model/config_class.py
• tests/main/jsonschema/test_main_jsonschema.py
• tests/main/conftest.py
• tests/main/xmlschema/test_main_xmlschema.py
• src/datamodel_code_generator/parser/base.py
• src/datamodel_code_generator/init.py
• src/datamodel_code_generator/parser/xmlschema.py

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (93e2fe3) to head (fe644e6).

Additional details and impacted files

@@            Coverage Diff             @@
##              main     #3407    +/-   ##
==========================================
  Coverage   100.00%   100.00%            
==========================================
  Files          138       138            
  Lines        29734     30097   +363     
  Branches      3525      3556    +31     
==========================================
+ Hits         29734     30097   +363     

Flag	Coverage Δ
unittests	100.00% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

Breaking Change Analysis

Result: No breaking changes detected

Reasoning: PR #3407 "Cache parsed sources" is an internal performance optimization. It adds a new public function enable_parsed_source_cache() (added to __all__), but the parsed-source cache is disabled by default (_enable_parsed_source_cache = False), so existing behavior is preserved unless a user explicitly opts in — and the function returns a restore callback, making it reversible. The other caching (_load_yaml_dict_from_path_cached keyed by path+mtime+encoding, cached_path_exists) consists of internal helpers with correct invalidation keys, so they return identical parse results and do not change generated output. No public symbols, CLI options, default values, or error-handling behavior appear to be removed or changed. Per the guidance, newly added opt-in APIs with behavior-preserving defaults are not breaking changes. Note: the prepared diff files were inaccessible from the sandbox (path-restricted, shallow clone with no parent, network denied), so this determination is based on analysis of the merged repository state for commit 29dd6d7, which is exactly this PR.

---

This analysis was performed by Claude Code Action

[github-actions]

🎉 Released in 0.64.0

This PR is now available in the latest release. See the release notes for details.