Commit d7fc281
authored
Use
## Summary
This specializes the `.jsonl` and `.ndjson` file extensions so they're
preprocessed like JSON instead of by the standard scanner. This prevents
them from creating thousands of sub machines and reduces scanning time
(see #17125 where this was done for `.json` files).
It seems reasonable to handle new-line delimited JSON files as well
otherwise scanning these files can take quite a long time.
It's quite unlikely that these will contain classes so, alternatively,
these *could* go in the binary extensions list so they get ignored
entirely.
## Test plan
I ran manual tests inside the `oxide` crate against some large-ish JSONL
files (5MB–15MB). These changes bring down scanning time from 2s–3s on
my M3 Max (via `cargo test --release …`) to less than 20ms.
I also ran tests through a full CLI build pipeline on a low-spec linux
box. This change brought scanning time down from ~90s to ~300ms for a
single ~15MB file.json preprocessor for new-line delimited JSON files (#19862)1 parent df6209a commit d7fc281
2 files changed
+2
-1
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
| 35 | + | |
35 | 36 | | |
36 | 37 | | |
37 | 38 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
474 | 474 | | |
475 | 475 | | |
476 | 476 | | |
477 | | - | |
| 477 | + | |
478 | 478 | | |
479 | 479 | | |
480 | 480 | | |
| |||
0 commit comments