feat: Optimize schema generation with in-memory parallel processing (53% speedup)#1145
Closed
nathan-stender wants to merge 1 commit into
Closed
feat: Optimize schema generation with in-memory parallel processing (53% speedup)#1145nathan-stender wants to merge 1 commit into
nathan-stender wants to merge 1 commit into
Conversation
…53% speedup) This addresses the major CI bottleneck where schema generation test was taking 3+ minutes. Performance improvements: - Test time reduced from ~180s to 84s (53% speedup) - Parallel processing with ProcessPoolExecutor - In-memory code generation and formatting - Eliminates most temporary file operations Technical changes: * Added generate_schemas_optimized() for parallel schema processing * Uses Python APIs for formatting code directly (black, autoflake) * Generates models in memory before writing final results * Defaults to using the optimized version unless disabled Co-authored-by: Claude <assistant@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR optimizes the schema generation process that was identified as the major bottleneck in CI test runs.
Problem
test_generate_schemas_runs_to_completiontest was taking 183-207 seconds (3+ minutes)Solution
Implemented a new
generate_schemas_optimized()function that:Performance Improvements
Technical Details
generate_schemas_optimized.pywith the parallel implementationgenerate_schemas()now defaults to using the optimized versionuse_optimized=Falseparameter for debuggingTesting
✅ All tests passing
✅ Schema generation produces identical results
✅ Backwards compatible - original implementation still available
This addresses the most significant performance bottleneck in the test suite.