Guard builtin formatter tokenization#3580
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
📝 WalkthroughWalkthroughAdds fast paths in the built-in formatter to skip tokenization when comment or multi-line-string checks are unnecessary. New regression tests cover the guards, and benchmark tests cover large Pydantic v2 model generation. ChangesTokenization Fast-Path Optimization
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
📚 Docs Preview: https://pr-3580.datamodel-code-generator.pages.dev |
Merging this PR will improve performance by 15.96%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | WallTime | test_perf_aws_style_openapi_pydantic_v2 |
2.3 s | 1.9 s | +19.27% |
| ⚡ | WallTime | test_perf_stripe_style_pydantic_v2 |
2.3 s | 1.9 s | +18.2% |
| ⚡ | WallTime | test_perf_openapi_large |
3.5 s | 2.9 s | +17.87% |
| ⚡ | WallTime | test_perf_kubernetes_style_pydantic_v2 |
3 s | 2.6 s | +17.52% |
| ⚡ | WallTime | test_perf_all_options_enabled |
6.2 s | 5.4 s | +16.15% |
| ⚡ | WallTime | test_perf_graphql_style_pydantic_v2 |
907.4 ms | 782.2 ms | +16.01% |
| ⚡ | WallTime | test_perf_large_models_pydantic_v2 |
4.1 s | 3.6 s | +15.76% |
| ⚡ | WallTime | test_perf_complex_refs |
2.3 s | 2 s | +14.93% |
| ⚡ | WallTime | test_perf_duplicate_names |
1.2 s | 1 s | +14.18% |
| ⚡ | WallTime | test_perf_deep_nested |
5.8 s | 5.2 s | +13.15% |
| ⚡ | WallTime | test_perf_multiple_files_input |
3.9 s | 3.4 s | +12.71% |
| 🆕 | WallTime | test_perf_large_models_pydantic_v2_builtin |
N/A | 928.4 ms | N/A |
| 🆕 | WallTime | test_perf_large_models_pydantic_v2_noformat |
N/A | 899.9 ms | N/A |
Tip
Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.
Comparing perf/builtin-formatter-guards (33dce84) with main (aa7d87f)
Footnotes
-
98 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
There was a problem hiding this comment.
🧹 Nitpick comments (2)
src/datamodel_code_generator/_builtin_formatter.py (1)
1718-1725: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low valueGuard logic is sound but only accounts for
\n-terminated backslash continuations.Per Python's lexical analysis, multi-line
STRINGtokens can only arise from triple-quoted literals or backslash line continuations, so the general approach here is correct. However, the check"\\\n" in codewill miss a backslash continuation that uses CRLF line endings (i.e.,"\\\r\n"), since the substring\immediately followed by\nwouldn't be present when a\rsits between them. Ifcodecould ever contain CRLF line endings alongside a backslash-continued single-quoted string, the guard would incorrectly skip tokenization andstring_lineswould stay empty, causing blank-line normalization logic to misclassify lines inside that string as top-level code.This is likely a non-issue in practice since generated code from this formatter is presumably always
\n-terminated, but worth confirming.🛡️ Optional defensive fix
- if ('"""' in code) or ("'''" in code) or ("\\\n" in code): + if ('"""' in code) or ("'''" in code) or ("\\\n" in code) or ("\\\r\n" in code):🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/datamodel_code_generator/_builtin_formatter.py` around lines 1718 - 1725, The guard in _normalize_top_level_blank_lines only detects backslash continuations with LF endings, so update the multi-line STRING precheck to also recognize CRLF backslash continuations before deciding whether to tokenize. Keep the existing tokenization-based approach in _normalize_top_level_blank_lines, but broaden the continuation detection so string_lines is populated correctly when code contains "\r\n" line endings.tests/main/test_performance.py (1)
128-158: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDuplicate test bodies — consider parametrizing.
The two tests are identical except for the
formattersargument and naming. Consolidate withpytest.mark.parametrize.♻️ Suggested consolidation
-@pytest.mark.perf -@pytest.mark.benchmark -def test_perf_large_models_pydantic_v2_builtin(tmp_path: Path) -> None: - """Performance test: Generate 500 Pydantic v2 models with the built-in formatter.""" - output_file = tmp_path / "output.py" - generate( - input_=PERFORMANCE_DATA_PATH / "large_models.json", - input_file_type=InputFileType.JsonSchema, - output=output_file, - output_model_type=DataModelType.PydanticV2BaseModel, - formatters=[Formatter.BUILTIN], - ) - content = output_file.read_text() - assert content.count("class Model") >= 500 - - -@pytest.mark.perf -@pytest.mark.benchmark -def test_perf_large_models_pydantic_v2_noformat(tmp_path: Path) -> None: - """Performance test: Generate 500 Pydantic v2 models without formatting.""" - output_file = tmp_path / "output.py" - generate( - input_=PERFORMANCE_DATA_PATH / "large_models.json", - input_file_type=InputFileType.JsonSchema, - output=output_file, - output_model_type=DataModelType.PydanticV2BaseModel, - formatters=[], - ) - content = output_file.read_text() - assert content.count("class Model") >= 500 +@pytest.mark.perf +@pytest.mark.benchmark +@pytest.mark.parametrize( + ("formatters", "test_id"), + [([Formatter.BUILTIN], "builtin"), ([], "noformat")], + ids=["builtin", "noformat"], +) +def test_perf_large_models_pydantic_v2_formatting(tmp_path: Path, formatters: list, test_id: str) -> None: + """Performance test: Generate 500 Pydantic v2 models with varying formatting.""" + output_file = tmp_path / "output.py" + generate( + input_=PERFORMANCE_DATA_PATH / "large_models.json", + input_file_type=InputFileType.JsonSchema, + output=output_file, + output_model_type=DataModelType.PydanticV2BaseModel, + formatters=formatters, + ) + content = output_file.read_text() + assert content.count("class Model") >= 500🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/main/test_performance.py` around lines 128 - 158, These two performance tests duplicate the same generate/assert flow and only differ in the formatters argument, so consolidate them into one parametrized pytest test. Refactor the duplicated logic in test_perf_large_models_pydantic_v2_builtin and test_perf_large_models_pydantic_v2_noformat using pytest.mark.parametrize over the formatter set while keeping the existing generate call and class count assertion intact.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@src/datamodel_code_generator/_builtin_formatter.py`:
- Around line 1718-1725: The guard in _normalize_top_level_blank_lines only
detects backslash continuations with LF endings, so update the multi-line STRING
precheck to also recognize CRLF backslash continuations before deciding whether
to tokenize. Keep the existing tokenization-based approach in
_normalize_top_level_blank_lines, but broaden the continuation detection so
string_lines is populated correctly when code contains "\r\n" line endings.
In `@tests/main/test_performance.py`:
- Around line 128-158: These two performance tests duplicate the same
generate/assert flow and only differ in the formatters argument, so consolidate
them into one parametrized pytest test. Refactor the duplicated logic in
test_perf_large_models_pydantic_v2_builtin and
test_perf_large_models_pydantic_v2_noformat using pytest.mark.parametrize over
the formatter set while keeping the existing generate call and class count
assertion intact.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 3f33572a-d0ad-4f7a-8598-5fc78fcc8fa4
📒 Files selected for processing (3)
src/datamodel_code_generator/_builtin_formatter.pytests/main/test_performance.pytests/test_format.py
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3580 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 163 163
Lines 35010 35033 +23
Branches 4047 4049 +2
=========================================
+ Hits 35010 35033 +23
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
🎉 Released in 0.68.1 This PR is now available in the latest release. See the release notes for details. |
Summary by CodeRabbit
Performance Improvements
Bug Fixes
Tests
#comments and multi-line string detection (including newline style variations).