perf: Optimize unstructure and validation for large arrays#1196
Merged
Conversation
…arrays
Data cube arrays (e.g., Biacore T200 sensorgram data) can contain millions of
floats. Two bottlenecks made serialization of such files impractical:
1. unstructure() recursed into every list element individually, making ~35M
function calls (each with 5 isinstance checks) just to return primitives
unchanged. A fast-path now detects lists of primitives by checking the first
element and returns the list directly. (13s → 0.2s)
2. jsonschema's items validator called validator.descend() per array element for
simple type schemas like {"type": "number"} and
{"anyOf": [{"type": "number"}, {"type": "null"}]}. A custom
FastDraft202012Validator bulk-checks via isinstance, falling back to the
original validator for complex schemas. (147s → 12.8s)
Total pipeline for a 135MB .bme file: ~176s → ~38s (4.7x speedup).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ajcariaga16
approved these changes
Apr 29, 2026
4 tasks
nathan-stender
added a commit
that referenced
this pull request
Apr 30, 2026
## Summary Follow-up to #1196 targeting the remaining validation bottlenecks, identified via cProfile on a 135MB Biacore T200 `.bme` file (917 measurements, 35M data points). - **Fast `oneOf` for typed arrays** — `tDimensionArray` and `tMeasureArray` are `oneOf` over 3 typed array refs (number/boolean/string). The default `oneOf` validator validates against the first matching branch, then re-validates against ALL remaining branches to confirm uniqueness. We short-circuit by inspecting the first element's type to select the correct branch directly. **(~6s saved)** - **Fast `oneOf` for array-vs-object** — `oneOf[tDimensionArray, tFunction]` patterns: if the instance is a list, it must be the array branch (tFunction is an object). Skip the redundant validation. **(917 calls saved)** - **Schema + validator caching** — Cache schema JSON, `RefResolver`, and `FastDraft202012Validator` instances per schema path to avoid redundant disk I/O and object construction on repeated validations. **(~1.7s saved on first call)** - **Eliminate redundant `fields()` call** — `unstructure()` called `fields()` twice per dataclass (once for iteration, once to build a set for custom_information_document check). Reuse the tuple. ### Benchmark (135MB .bme file) | Phase | Before #1196 | After #1196 | After this PR | |---|---|---|---| | unstructure | 13s | 0.2s | 0.2s | | validate | 147s | 12.8s | **5.0s** | | **Total** | **~176s** | **~38s** | **~25s** | ## Test plan - [x] Full test suite passes (1,119 tests) - [x] Lint + mypy clean - [x] Verified oneOf fast validator correctly handles: number/string/boolean arrays, nullable arrays, mixed-type rejection, non-array oneOf fallback, array-vs-object disambiguation - [x] Benchmarked end-to-end against 135MB `.bme` file 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
unstructure()fast-path — Lists whose first element is a primitive (int,float,str,bool,None) are returned directly without per-element recursion. For data cube float arrays with millions of elements, this eliminates ~35M unnecessary function calls. (13s → 0.2s)FastDraft202012Validator— Custom jsonschema validator that bulk-checks array items viaisinstancefor simple type-only schemas (e.g.{"type": "number"},{"anyOf": [{"type": "number"}, {"type": "null"}]}), falling back to the original per-element validation for complex schemas. (147s → 12.8s)Total pipeline for a 135MB Biacore T200 Evaluation
.bmefile (917 measurements, 35M data points, 548MB JSON output): ~176s → ~38s (4.7x speedup).Test plan
.bmefile end-to-end🤖 Generated with Claude Code