turbo-tasks: replace TaskStorage TinyVec lazy area with bitmap + byte-packed tail in one allocation#93835
Draft
lukesandberg wants to merge 27 commits into
Draft
turbo-tasks: replace TaskStorage TinyVec lazy area with bitmap + byte-packed tail in one allocation#93835lukesandberg wants to merge 27 commits into
lukesandberg wants to merge 27 commits into
Conversation
Contributor
Author
|
Warning This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
This stack of pull requests is managed by Graphite. Learn more about stacking. |
lukesandberg
commented
May 13, 2026
lukesandberg
commented
May 13, 2026
lukesandberg
commented
May 13, 2026
lukesandberg
commented
May 13, 2026
lukesandberg
commented
May 13, 2026
lukesandberg
commented
May 13, 2026
lukesandberg
commented
May 13, 2026
lukesandberg
commented
May 13, 2026
lukesandberg
commented
May 13, 2026
Contributor
Tests PassedCommit: 7dd295e |
lukesandberg
commented
May 14, 2026
lukesandberg
added a commit
that referenced
this pull request
May 14, 2026
- Introduce `Tag(NonZeroU8)` newtype in `lazy_tail.rs` to carry the "1..=LAZY_N" invariant at the type level and centralize the bit/mask helpers (`bit`, `mask_below`, `mask_above`, `table_index`). Macro emits `LAZY_TAG_<NAME>: Tag` constants and dispatch fns take `Tag`. `Option<Tag>` is 1 byte via the niche. - Add a Lemire reference to `sum_padded_sizes`'s loop and note the stdlib has no built-in set-bit iterator. - Add unit tests for `sum_padded_sizes`, `offset_of`, the mask- partition invariant, and the `Option<Tag>` niche. - Rename `LazyTail::install_unchecked` -> `insert_unchecked` and `TaskStorage::lazy_install` -> `lazy_insert`. Cross-reference `replace_in_place` from `insert_unchecked`'s safety doc. - Add a `const _` assert pinning `size_of::<Option<TaskStorage>>() == size_of::<TaskStorage>()`. - Inline imports for `TaskStorage` in `backend::mod` and `backend::operation` (drop the fully-qualified spellings). - Fix a stale doc reference (`TaskStorageInner::evictability` -> `TaskStorage::evictability`) and drop a stale `CachedDataItem` mention from the schema module docs. - Rename `TurboMalloc::good_size` -> `get_bucket_size`. All `backend::lazy_tail::tests` (8) and `backend::storage_schema::tests` (16) pass under cargo test and Miri's default Stacked Borrows mode.
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
Contributor
Stats from current PR✅ No significant changes detected📊 All Metrics📖 Metrics GlossaryDev Server Metrics:
Build Metrics:
Change Thresholds:
⚡ Dev Server
📦 Dev Server (Webpack) (Legacy)📦 Dev Server (Webpack)
⚡ Production Builds
📦 Production Builds (Webpack) (Legacy)📦 Production Builds (Webpack)
📦 Bundle SizesBundle Sizes⚡ TurbopackClient Main Bundles
Server Middleware
Build DetailsBuild Manifests
📦 WebpackClient Main Bundles
Polyfills
Pages
Server Edge SSR
Middleware
Build DetailsBuild Manifests
Build Cache
🔄 Shared (bundler-independent)Runtimes
📎 Tarball URLCommit: 7dd295e |
lukesandberg
changed the title
lukesandberg
added a commit
that referenced
this pull request
May 17, 2026
- Introduce `Tag(NonZeroU8)` newtype in `lazy_tail.rs` to carry the "1..=LAZY_N" invariant at the type level and centralize the bit/mask helpers (`bit`, `mask_below`, `mask_above`, `table_index`). Macro emits `LAZY_TAG_<NAME>: Tag` constants and dispatch fns take `Tag`. `Option<Tag>` is 1 byte via the niche. - Add a Lemire reference to `sum_padded_sizes`'s loop and note the stdlib has no built-in set-bit iterator. - Add unit tests for `sum_padded_sizes`, `offset_of`, the mask- partition invariant, and the `Option<Tag>` niche. - Rename `LazyTail::install_unchecked` -> `insert_unchecked` and `TaskStorage::lazy_install` -> `lazy_insert`. Cross-reference `replace_in_place` from `insert_unchecked`'s safety doc. - Add a `const _` assert pinning `size_of::<Option<TaskStorage>>() == size_of::<TaskStorage>()`. - Inline imports for `TaskStorage` in `backend::mod` and `backend::operation` (drop the fully-qualified spellings). - Fix a stale doc reference (`TaskStorageInner::evictability` -> `TaskStorage::evictability`) and drop a stale `CachedDataItem` mention from the schema module docs. - Rename `TurboMalloc::good_size` -> `get_bucket_size`. All `backend::lazy_tail::tests` (8) and `backend::storage_schema::tests` (16) pass under cargo test and Miri's default Stacked Borrows mode.
lukesandberg
force-pushed
the
flattened_storage
branch
from
2dc466a to
1bb366a
Compare
lukesandberg
force-pushed
the
05-09-turbo-tasks_replace_taskstorage_lazy_vec_with_a_16_b_lazyvec
branch
2 times, most recently
from
42ece5a to
7221af0
Compare
lukesandberg
added a commit
that referenced
this pull request
May 19, 2026
- Introduce `Tag(NonZeroU8)` newtype in `lazy_tail.rs` to carry the "1..=LAZY_N" invariant at the type level and centralize the bit/mask helpers (`bit`, `mask_below`, `mask_above`, `table_index`). Macro emits `LAZY_TAG_<NAME>: Tag` constants and dispatch fns take `Tag`. `Option<Tag>` is 1 byte via the niche. - Add a Lemire reference to `sum_padded_sizes`'s loop and note the stdlib has no built-in set-bit iterator. - Add unit tests for `sum_padded_sizes`, `offset_of`, the mask- partition invariant, and the `Option<Tag>` niche. - Rename `LazyTail::install_unchecked` -> `insert_unchecked` and `TaskStorage::lazy_install` -> `lazy_insert`. Cross-reference `replace_in_place` from `insert_unchecked`'s safety doc. - Add a `const _` assert pinning `size_of::<Option<TaskStorage>>() == size_of::<TaskStorage>()`. - Inline imports for `TaskStorage` in `backend::mod` and `backend::operation` (drop the fully-qualified spellings). - Fix a stale doc reference (`TaskStorageInner::evictability` -> `TaskStorage::evictability`) and drop a stale `CachedDataItem` mention from the schema module docs. - Rename `TurboMalloc::good_size` -> `get_bucket_size`. All `backend::lazy_tail::tests` (8) and `backend::storage_schema::tests` (16) pass under cargo test and Miri's default Stacked Borrows mode.
lukesandberg
force-pushed
the
flattened_storage
branch
from
1bb366a to
25da638
Compare
lukesandberg
added a commit
that referenced
this pull request
May 19, 2026
- Introduce `Tag(NonZeroU8)` newtype in `lazy_tail.rs` to carry the "1..=LAZY_N" invariant at the type level and centralize the bit/mask helpers (`bit`, `mask_below`, `mask_above`, `table_index`). Macro emits `LAZY_TAG_<NAME>: Tag` constants and dispatch fns take `Tag`. `Option<Tag>` is 1 byte via the niche. - Add a Lemire reference to `sum_padded_sizes`'s loop and note the stdlib has no built-in set-bit iterator. - Add unit tests for `sum_padded_sizes`, `offset_of`, the mask- partition invariant, and the `Option<Tag>` niche. - Rename `LazyTail::install_unchecked` -> `insert_unchecked` and `TaskStorage::lazy_install` -> `lazy_insert`. Cross-reference `replace_in_place` from `insert_unchecked`'s safety doc. - Add a `const _` assert pinning `size_of::<Option<TaskStorage>>() == size_of::<TaskStorage>()`. - Inline imports for `TaskStorage` in `backend::mod` and `backend::operation` (drop the fully-qualified spellings). - Fix a stale doc reference (`TaskStorageInner::evictability` -> `TaskStorage::evictability`) and drop a stale `CachedDataItem` mention from the schema module docs. - Rename `TurboMalloc::good_size` -> `get_bucket_size`. All `backend::lazy_tail::tests` (8) and `backend::storage_schema::tests` (16) pass under cargo test and Miri's default Stacked Borrows mode.
lukesandberg
force-pushed
the
flattened_storage
branch
from
25da638 to
ba52926
Compare
lukesandberg
force-pushed
the
05-09-turbo-tasks_replace_taskstorage_lazy_vec_with_a_16_b_lazyvec
branch
from
7221af0 to
0938c4c
Compare
lukesandberg
added a commit
that referenced
this pull request
May 20, 2026
- Introduce `Tag(NonZeroU8)` newtype in `lazy_tail.rs` to carry the "1..=LAZY_N" invariant at the type level and centralize the bit/mask helpers (`bit`, `mask_below`, `mask_above`, `table_index`). Macro emits `LAZY_TAG_<NAME>: Tag` constants and dispatch fns take `Tag`. `Option<Tag>` is 1 byte via the niche. - Add a Lemire reference to `sum_padded_sizes`'s loop and note the stdlib has no built-in set-bit iterator. - Add unit tests for `sum_padded_sizes`, `offset_of`, the mask- partition invariant, and the `Option<Tag>` niche. - Rename `LazyTail::install_unchecked` -> `insert_unchecked` and `TaskStorage::lazy_install` -> `lazy_insert`. Cross-reference `replace_in_place` from `insert_unchecked`'s safety doc. - Add a `const _` assert pinning `size_of::<Option<TaskStorage>>() == size_of::<TaskStorage>()`. - Inline imports for `TaskStorage` in `backend::mod` and `backend::operation` (drop the fully-qualified spellings). - Fix a stale doc reference (`TaskStorageInner::evictability` -> `TaskStorage::evictability`) and drop a stale `CachedDataItem` mention from the schema module docs. - Rename `TurboMalloc::good_size` -> `get_bucket_size`. All `backend::lazy_tail::tests` (8) and `backend::storage_schema::tests` (16) pass under cargo test and Miri's default Stacked Borrows mode.
lukesandberg
force-pushed
the
flattened_storage
branch
from
ba52926 to
b1247f4
Compare
Migrate Arc<CachedTaskType> to triomphe::Arc via CachedTaskTypeArc newtype. Saves one usize per allocation (no weak count) and avoids the weak-count CAS in drop_slow compared to std::sync::Arc. We never need Weak<CachedTaskType>, so the trade-off is favorable.
Replace the `(CellRef, Option<u64>)` and `(CellId, Option<u64>, TaskId)`
tuples used for cell-edge tracking with a `CellDependency` enum:
enum CellDependency {
All(CellRef),
Hash(CellRef, u64),
}
`Option<u64>` previously cost a full 16 B (8 B discriminant + 8 B value,
aligned). With an explicit enum the layout algorithm reuses the niche on
`ValueTypeId` (`NonZero<u16>`) inside `CellRef.cell.type_id` for the
variant tag, dropping the element from 32 B to 24 B. That in turn shrinks
`LazyField` from 56 B to 48 B.
Also adds `CellDependency::into_parts()` and uses it in
`iter_cell_dependents` / `iter_cell_dependencies` hot loops to avoid
checking the enum discriminant twice via back-to-back
`cell_ref()` + `key()` calls.
Replace `TaskStorage::lazy: Vec<LazyField>` with a custom 16 B `TinyVec` (u8 len + u8 cap, the schema's max field count is well under 255). Drops `size_of::<TaskStorage>()` from 136 B to 128 B. Included micro-benchmarks show `TinyVec` push is 11-32% faster than `Vec` across realistic sizes and iter is neutral. The `TinyVec` type: * Carries a `const MAX: u8` generic parameter that strictly caps push count and tightens the growth schedule (doubles until it would exceed MAX, then caps exactly at MAX). The schema macro emits `TinyVec<LazyField, N>` where `N` is the exact lazy-field count, so the cap matches the actual schema size (e.g. with 24 variants we end at cap=24 instead of cap=32, saving a few slots per fully-populated task). A compile-time static assert rejects `MAX = 0` at monomorphization. * Tightens visibility: `new`, `capacity`, `as_slice`, `as_mut_slice`, `reserve` are private; `len` / `is_empty` stay pub as a clippy-preferred pair. * Delegates `retain_mut` to `Vec::retain_mut` via a `Vec::from_raw_parts` round-trip — `retain_mut` is cold relative to push so the round-trip cost is irrelevant, and this drops ~7 unsafe blocks with a panic partial-shift guard. * Delegates owned `IntoIter` to `std::vec::IntoIter` via the same Vec round-trip, dropping ~50 lines of unsafe. * Drops `Extend` and `FromIterator` trait impls; the only caller path uses an inherent `extend_exact` which requires an `ExactSizeIterator` and reserves exactly once. * Drops `iter`, `iter_mut`, `last_mut`, `Index`, `IndexMut` — all reachable through `Deref<Target = [T]>`. `for x in &tv` / `for x in &mut tv` still need `IntoIterator` impls for refs because the `for`-loop desugar doesn't apply `Deref` coercion across the reference boundary. Net unsafe count in `TinyVec` is 5 in the hot path plus 1 in `retain_mut` and 1 in `IntoIter` — each upholding a single local invariant or just round-tripping through Vec.
Each schema field's `I` (inline capacity) was previously fixed at 1. With `SmallVec`'s `union` feature on, the heap variant occupies 16 bytes (`NonNull<T> + usize`), so the SmallVec body is always `max(16, N * sizeof(T))` aligned to `max(align(T), 8)`. Net: growing `N` is free until `N * sizeof(T)` exceeds 16, after which each step adds `align_up(sizeof(T), 8)` bytes. Two opportunities follow: * For lazy fields, the `LazyField` enum already pays a 40-byte payload budget (the largest variants — `cell_data`, `cell_data_hash`, `AutoSet<CellDependency>`, `CounterMap<CollectibleRef, i32>` — saturate it at I=1). Smaller-element fields sat at 32 B with 8 B of unused padding. Bumping them to fill 40 B is zero-cost: TaskStorage and LazyField sizes don't change. * For inline fields on TaskStorage, raising `I` is free up to the SmallVec 16-byte body limit (e.g. `AutoSet<TaskId>` to I=4, `CounterMap<TaskId, u32>` to I=2). To allow per-field tuning, parameterize: * `CounterMap` over `const I: usize`, propagating to its `AutoMap` inner. * The `AutoSet` / `AutoMap` schema aliases drop their hardcoded `1`; each field declares its own `I`. Per-field choices: * `output_dependent` (inline) `AutoSet<TaskId>` → 4 (32 B) * `upper` (inline) `CounterMap<TaskId, u32>` → 2 (32 B) * `children`, `output_dependencies`, `outdated_output_dependencies` `AutoSet<TaskId>` → 6 (40 B) * `collectibles_dependencies`, `outdated_collectibles_dependencies` `AutoSet<CollectiblesRef>` → 3 (40 B) * `collectibles_dependents` `AutoSet<(TraitTypeId, TaskId)>` → 3 (40 B) * `followers`, `aggregated_dirty_containers`, `aggregated_current_session_clean_containers` `CounterMap<TaskId, _>` → 3 (40 B) * `cell_type_max_index` `AutoMap<ValueTypeId, u32>` → 3 (40 B) Variants already at 40 B (cell_data, cell_data_hash, AutoSet<CellDependency>, CounterMap<CollectibleRef, i32>) stay at I=1. `in_progress_cells` stays at I=1 to avoid overflowing LazyField under the `hanging_detection` feature (which inflates `Event` from 8 B to 16 B). `TaskStorage` stays at 128 B, `LazyField` at 48 B; only inline capacities change.
…change
Prefactor for replacing the TinyVec<LazyField> lazy area with a packed
byte-tail layout. Two structural changes that are pure refactors today:
1. New TaskStorageBox (thin Box<TaskStorage> newtype) plumbed through
the dashmap value types and StorageWriteGuard so callers no longer
bind to Box<TaskStorage> directly.
2. Every direct read/write of TaskStorage.lazy in the macro-generated
code and the hand-written helpers now funnels through new methods:
lazy_push, lazy_iter[_mut], lazy_len, lazy_swap_remove,
lazy_replace_at, lazy_slot_mut, lazy_retain_mut, lazy_cleanup,
lazy_shrink_to_fit, take_lazy_contents/lazy_extend_owned (opaque
LazyContents type), all_lazy_{data,meta}_empty_or_absent, and
all_transient_lazy_empty. build_lazy_index moved from a static fn
over &[LazyField] to a &self method.
No behavior change. All 58 backend tests pass.
Replace every `for field in lazy_iter()` / `match field { LazyField::V(_) =>
... }` pattern in the macro-generated code with an unrolled per-variant
sequence that uses the typed accessors (`get_<name>`, `set_<name>`,
`take_<name>`, `<name>_mut`, etc.). After this, codegen for:
* `encode_meta` / `encode_data`
* `clone_snapshot`
* `restore_meta_from` / `restore_data_from`
* `cleanup_after_execution`
* `drop_partial`
no longer iterates `&LazyField` references, so the underlying lazy storage
container can be swapped (next: TinyVec<LazyField> → packed byte tail with a
u32 presence bitmap) without touching any of those patterns again. The
closures-over-&LazyField in `find_lazy` / `set_lazy` / `take_lazy` / etc.
remain — they're still used by the per-variant accessor bodies and will be
addressed in the layout swap.
Drops the now-unused `LazyContents`, `take_lazy_contents`,
`lazy_extend_owned`, `lazy_retain_mut`, `lazy_cleanup`,
`build_lazy_index`, `restore_lazy_field`, and `gen_lazy_match_arms` /
`gen_drop_lazy_match_arm` helpers in their place.
All 50 backend tests pass; whole workspace builds.
Continues the prefactor for swapping the TaskStorage lazy area away from
TinyVec<LazyField>. The generic closure-taking helpers (find_lazy,
find_lazy_mut, find_lazy_ref, lazy_at_mut, lazy_take_at, get_or_create_lazy,
take_lazy, set_lazy) were the last places where a closure parameter forced
the rest of the codegen to know about the LazyField enum shape — every
caller was passing `|f| match f { LazyField::Variant(...) => ... }`
extractors.
Each lazy direct/collection accessor (`get_<name>`, `set_<name>`,
`take_<name>`, `<name>`, `<name>_mut`, `get_<name>_mut`) now inlines a
single per-variant `match` over `&LazyField` / `&mut LazyField` in its body
and goes directly to the encapsulated primitives (`lazy_iter`,
`lazy_iter_mut`, `lazy_push`, `lazy_slot_mut`, `lazy_swap_remove`,
`lazy_replace_at`). The trait-level set/take/extend/add/remove/update_with
bodies for direct and collection lazy fields collapse to the same shape as
the inline shape — they just route through the typed `<name>` /
`<name>_mut` / `take_<name>` accessors and let those do the per-variant
scan internally.
Once the byte-tail layout lands, the per-variant `match`es become unsafe
pointer casts; the inline shape and the trait-level shape don't change.
Drops `lazy_extractor_closure`, `lazy_matches_closure`,
`lazy_unwrap_closure`, `lazy_constructor`, and the eight closure-taking
helper methods. All 50 backend tests pass; whole workspace builds.
Adds the schema-level constants that the upcoming byte-tail layout will consume: - `LAZY_TAG_<NAME>: u8` per variant (1-based; tag 0 stays the bincode encode sentinel). - `LAZY_N: u8` and a compile-time `assert!(LAZY_N <= 32)` guarding the presence bitmap's `u32` width. - `LAZY_SIZE[tag]`, `LAZY_ALIGN[tag]`, and `LAZY_PADDED_SIZE[tag]` for computing byte offsets when walking the tail. - `LAZY_PERSISTENT_MASK`, `LAZY_META_MASK`, `LAZY_DATA_MASK` for category filtering at the bitmap level. No runtime behavior change. The `lazy: TinyVec<LazyField>` storage is untouched; nothing consumes the new constants yet. 50 backend tests pass.
Replaces the `TinyVec<LazyField, _>` lazy area with `LazyTail`, a 16 B header (presence bitmap + len + cap + ptr) backed by a heap buffer that holds payloads packed in tag order. Per-tag layout is driven by the schema-emitted `LAZY_SIZE` / `LAZY_ALIGN` / `LAZY_PADDED_SIZE` tables and `LAZY_TAG_<NAME>` constants from the previous commit. Implementation: - `LazyTail` (new file): `present: u32, len: u16, cap: u16, ptr: NonNull<u8>` with `find::<T>`, `find_mut::<T>`, `install::<T>`, `take::<T>`, `replace::<T>` primitives. All unsafe; callers pass the schema's tag for the matching payload type. Geometric grow policy (64 → power of two, capped at u16::MAX). - Macro-emitted accessors (`<name>`, `<name>_mut`, `get_<name>`, `set_<name>`, `take_<name>`, `get_<name>_mut`) now call directly into `LazyTail`'s typed primitives via the per-variant `LAZY_TAG_*` constant. The previous inline `match LazyField::V(v) => ...` patterns are gone. - `encode_*` / `decode_*` keep their on-disk format (per-category 1-based tag + payload + sentinel) but route decoded payloads to the global `LAZY_TAG_<NAME>` for tail installation, so the bitmap position is correct. - `clone_snapshot` and `restore_*_from` use `LazyTail::install` / `std::mem::take(&mut source.<field>)` so the `Drop` impl below doesn't block partial moves. - `TaskStorage` now has a hand-written `Drop` that walks `present`, invokes the macro-emitted `lazy_drop_dispatch(tag, ptr)` per payload, then lets `LazyTail`'s field-level Drop free the buffer. - Padded sizes round up to `LAZY_MAX_ALIGN` so every payload lands at an offset valid for its own alignment regardless of variant order — a smaller-aligned variant followed by a stricter-aligned one would otherwise misalign reads. `LazyField` is still around (encode/decode arms use it as a stable on- disk tag mapping) and `test_schema_size` still asserts `size_of::<TaskStorage>() == 128`. The `TinyVec<LazyField, _>` field is gone. All 50 backend tests pass; whole workspace builds.
The byte-tail layout doesn't store `LazyField` values anywhere — payloads live in the `LazyTail` buffer addressed by tag. Drop the enum, its `is_empty` / `is_persistent` / `is_meta` / `is_data` / `index_and_category` methods, and the `NUM_VARIANTS` constant from the macro emission. Update `test_schema_size` to assert `size_of::<LazyTail>() == 16` (the slot it replaced). `TaskStorage` is still 128 B; whole workspace builds; 50 backend tests pass.
TaskStorage's lazy area no longer uses `TinyVec<LazyField, _>` — `LazyTail` manages a presence bitmap plus a byte-packed payload buffer directly. Nothing else in the workspace references `TinyVec`, so drop: - `turbo-tasks/src/tiny_vec.rs` - `turbo-tasks/benches/tiny_vec.rs` and its `[[bench]]` entry - `tiny_vec` mod declaration and the `TinyVec` re-export from `turbo-tasks/src/lib.rs` Whole workspace builds; 50 backend tests pass.
mimalloc rounds allocation requests up to bin boundaries — asking for 65 B reserves the 80 B bin, so the next push up to 80 B is free. Two new behaviors: - `TurboMalloc::good_size(n)`: thin wrapper over `mi_good_size` (falls back to `next_power_of_two().max(16)` when `custom_allocator` is off). - `LazyTail::grow_to` allocates `good_size(min_bytes)` instead of the previous power-of-two doubling. Each grow now reserves exactly the bin it would have ended up in anyway, eliminating the second realloc that used to fire when an earlier push left the buffer just below the bin boundary. - `LazyTail::shrink_to_fit` is bin-aware: it only reallocates when `good_size(len) < good_size(cap)`. Within a bin the realloc is free but pointless, so we skip it. 50 backend tests pass; whole workspace builds.
Preparation for inlining the lazy-field byte buffer into TaskStorage's heap allocation: the inline (non-lazy, non-flag) fields declared by the schema now live in a generated `TaskStorageHead` struct embedded as `task.head`. The macro emits inline-field accessors as `self.head.<field>` accesses. `flags` and `lazy_tail` stay at the top level of `TaskStorage` because backend code reads them as bare field accesses; folding them in is a separate change. Touches every macro-emission site that wrote `self.<inline_field>` or `source.<inline_field>` — accessor generators, clone_inline, restore merges, inline cleanup, drop_partial inline reset, encode/decode inline arms, and the `category_inline_check` predicate. Two hand-written storage_schema.rs references (`init_transient_task`'s `aggregation_number` write, `evictability`'s `persistent_task_type` read) move to `self.head.<field>` too. Temporary regression: `size_of::<TaskStorage>()` grows from 128 B to 136 B because Rust can no longer pack the small `flags` / `lazy_tail` fields into the alignment gaps of the `Arc` / `Box` / `AutoMap` inline fields that now live inside `head`. Step 3 of the refactor (inlining `LazyTail`'s byte buffer into the same allocation, dropping `LazyTail::ptr`) recovers this 8 B regression and goes further. 50 backend tests pass; whole workspace builds.
`typed()` / `typed_mut()` on the `TaskStorageAccessors` trait now return `&TaskStorageBox` / `&mut TaskStorageBox` rather than the inner `&TaskStorage` / `&mut TaskStorage`. `TaskStorageBox` implements `Deref<Target = TaskStorage>`, so every existing macro-emitted callsite of the form `self.typed().<field>()` keeps working unchanged — the inline-field and lazy-field accessors are still defined on `TaskStorage` and reached via deref coercion. Why: step 3 of the refactor will inline `LazyTail`'s byte buffer into `TaskStorageBox`'s heap allocation, and the grow-capable lazy mutations (currently `LazyTail::install` on `&mut LazyTail`) need access to the owning pointer to relocate the buffer. Returning the box from `typed_mut()` is the seam those grow paths will hook into. `StorageWriteGuard::Deref::Target` flips from `TaskStorage` to `TaskStorageBox` for the same reason. The deref chain via `RefMut` collapses to a single explicit step. 50 backend tests pass; whole workspace builds.
Preparation for step 3 (inlining the lazy-tail byte buffer into
TaskStorageBox's allocation): after step 3, stack-allocated TaskStorage
values can't grow their tail because the allocation context belongs to
TaskStorageBox. Migrate every TaskStorage::new() / TaskStorage::default()
call site to TaskStorageBox::new() so no test or production path ends up
trying to mutate a stack-allocated TaskStorage.
Specific changes:
* All 20+ test sites in storage_schema.rs switch to TaskStorageBox::new().
Access to inline fields goes through Deref/DerefMut.
* Macro-emitted `clone_snapshot()` now returns TaskStorageBox.
* Macro-emitted `restore_from` / `restore_meta_from` / `restore_data_from`
take TaskStorageBox by value (formerly TaskStorage).
* `restore_meta_from` / `restore_data_from` made `pub` (previously
`pub(crate)`) for the tests that drive them directly.
* `BackingStorage::lookup_data` / `batch_lookup_data` (and the `Either`
forwarding impl) switch their TaskStorage parameter / return types to
TaskStorageBox. `kv_backing_storage` and `restore_task_data` /
`restore_task_data_batch` follow.
* `TaskRestoreEntry::{data,meta}_restore_result` and the
`apply_restore_result` parameter shape switch to TaskStorageBox.
* The verify_serialization round-trip in `encode_task_data_inner` now
builds via TaskStorageBox::new().
* Storage snapshot path drops the wrapper that previously did
`TaskStorageBox::from_boxed(Box::new(clone_snapshot()))` — clone_snapshot
returns the right type now.
The macro still emits a public `TaskStorage::new()` because the body of
TaskStorageBox::new() needs to construct an underlying TaskStorage; that
function is left in place for now and will be redesigned in step 3 when
the underlying allocation switches to one unified head+tail buffer.
50 backend tests pass; whole workspace builds.
`LazyTail` shrinks from 16 B to 8 B by dropping its `NonNull<u8>` field
and the explicit unsafe `Send`/`Sync` impls. The byte buffer it described
now lives in the same heap allocation as the surrounding
`TaskStorageBox`, immediately after the `TaskStorage` head:
offset 0: TaskStorage { inline fields,
flags, LazyTail (8 B) }
offset size_of::<TaskStorage>(): [u8; tail_cap] (payloads in tag order)
`TaskStorageBox` switches from a Box<TaskStorage> wrapper to a custom
NonNull<TaskStorage> that manages the unified allocation: alloc(head +
tail_cap), drop walks the bitmap dispatching per-tag drops then runs the
head Drop then deallocs. Realloc on grow updates the box's pointer
in-place.
Grow-capable lazy mutations move to `impl TaskStorageBox`:
- Macro-emitted `set_<name>` and `<name>_mut` for lazy fields. Read /
take / get-mut-if-present stay on `TaskStorage`.
- `drop_partial`, `decode_meta`, `decode_data`, `clone_snapshot`,
`restore_from`, `restore_meta_from`, `restore_data_from`.
- Hand-written `init_transient_task` and `decode` wrapper.
Read-only / non-grow methods stay on `TaskStorage` and reach the tail via
`(self as *const TaskStorage).add(size_of::<TaskStorage>())` arithmetic.
`Deref<Target = TaskStorage>` on `TaskStorageBox` lets all callsites
keep their `self.typed().<field>()` / `self.typed_mut().<field>()`
shape — the auto-deref picks whichever impl the method lives on.
Per-tag drop dispatch runs in `TaskStorageBox::drop`; the previous
hand-written `Drop for TaskStorage` is gone since stack-constructed
`TaskStorage` values now always have `tail_cap == 0` and no live
payloads.
`size_of::<TaskStorage>()` returns to 128 B (down from 136 B in step 1)
because `LazyTail` shrank by 8 B. `size_of::<LazyTail>()` is 8 B.
50 backend tests pass; whole workspace builds.
…geInner
Final cleanup rename. The thin owning pointer (formerly TaskStorageBox)
takes the cleaner public name `TaskStorage`; the macro-emitted struct
that holds the actual head fields (formerly `TaskStorage`) becomes
`TaskStorageInner`. The schema-organized inline-fields sub-struct
(`TaskStorageHead`) keeps its name; it's still embedded as
`TaskStorageInner.head`.
Public-facing types after the rename:
- `TaskStorage` — thin owning pointer (8 B, like Box<_>).
All grow-capable lazy mutations live here.
Deref<Target = TaskStorageInner>.
- `TaskStorageInner` — the macro-emitted struct that holds the
inline `head: TaskStorageHead`, `flags`,
and `lazy_tail` metadata. Read-only and
non-grow accessors live here.
- `TaskStorageHead` — the inline-fields-only sub-struct of
`TaskStorageInner`.
Also renames the module `task_storage_box.rs` → `task_storage.rs`.
50 unit tests + the full backend integration suite pass; whole workspace
builds.
….inline) The sub-struct that groups TaskStorage's inline fields is no longer "the head" in any meaningful sense — `TaskStorageInner` itself is the head of the unified head+tail allocation, and the sub-struct is just an organizational grouping of fields marked `inline` in the schema. `TaskStorageHead` → `TaskStorageInline`. The corresponding field `TaskStorageInner.head: TaskStorageHead` becomes `TaskStorageInner.inline: TaskStorageInline`. The macro-generated inline-field accessors switch from `self.head.<field>` to `self.inline.<field>` access patterns. The `TaskStorage::head()` helper method (returning `&TaskStorageInner`) keeps its name — `head` is correct there: it points at the head bytes of the unified allocation, as distinct from `tail_ptr`. 50 unit tests pass; integration test suite passes; whole workspace builds.
… safety `&TaskStorageInner` only carries provenance over the head bytes, but the lazy-payload accessors read past `size_of::<TaskStorageInner>()` into the tail. Miri's Stacked Borrows check flagged the retag as UB. Move all tail-reading accessors plus their callers (`evictability`, `meta_counts`, `encode_meta`, `encode_data`, `encode`, `is_empty_*`, `all_lazy_*`) from `impl TaskStorageInner` to `impl TaskStorage`, where the receiver derives the tail pointer from the owning `NonNull<TaskStorageInner>` (full- allocation provenance). Updated callsite signatures (`take_snapshot`'s `P` bound, `encode_task_data`, `add_counts`, `task_name`) from `&TaskStorageInner` to `&TaskStorage`. All 16 storage_schema tests pass under Miri's default Stacked Borrows mode.
- Introduce `Tag(NonZeroU8)` newtype in `lazy_tail.rs` to carry the "1..=LAZY_N" invariant at the type level and centralize the bit/mask helpers (`bit`, `mask_below`, `mask_above`, `table_index`). Macro emits `LAZY_TAG_<NAME>: Tag` constants and dispatch fns take `Tag`. `Option<Tag>` is 1 byte via the niche. - Add a Lemire reference to `sum_padded_sizes`'s loop and note the stdlib has no built-in set-bit iterator. - Add unit tests for `sum_padded_sizes`, `offset_of`, the mask- partition invariant, and the `Option<Tag>` niche. - Rename `LazyTail::install_unchecked` -> `insert_unchecked` and `TaskStorage::lazy_install` -> `lazy_insert`. Cross-reference `replace_in_place` from `insert_unchecked`'s safety doc. - Add a `const _` assert pinning `size_of::<Option<TaskStorage>>() == size_of::<TaskStorage>()`. - Inline imports for `TaskStorage` in `backend::mod` and `backend::operation` (drop the fully-qualified spellings). - Fix a stale doc reference (`TaskStorageInner::evictability` -> `TaskStorage::evictability`) and drop a stale `CachedDataItem` mention from the schema module docs. - Rename `TurboMalloc::good_size` -> `get_bucket_size`. All `backend::lazy_tail::tests` (8) and `backend::storage_schema::tests` (16) pass under cargo test and Miri's default Stacked Borrows mode.
…d sanity check Debug: - TaskStorage::fmt walks `lazy_tail.present` and routes each set bit through a new schema-emitted `lazy_debug_dispatch(tag, ptr, &mut DebugMap)`, rendering each present variant by its real field name and value. - Dropped Debug from LazyTail and TaskStorageInner: neither can format the tail without a pointer to it, so the owning TaskStorage's hand- written impl is the only one that makes sense. - Added `test_debug_impl_renders_lazy_variant_names` to pin the output contains the inline head plus the lazy variant names and payloads. Type-id sanity check: - Schema now emits `LAZY_TYPE_IDS: [TypeId; LAZY_N + 1]` alongside LAZY_SIZE/LAZY_ALIGN, indexed by tag. - `Tag::debug_assert_type::<T>()` consults the table and `debug_assert_eq!`s against `TypeId::of::<T>()`. Compiles to nothing in release. - Every typed `LazyTail` and `TaskStorage` method (`find`, `find_mut`, `take`, `insert_unchecked`, `replace_in_place`, `lazy_*`) now bounds `T: 'static` and calls `tag.debug_assert_type::<T>()` before the unsafe pointer cast — catches wrong-type-cast bugs at the call site rather than producing UB downstream. Clippy fixes (caught by `cargo clippy --tests`): - `LazyTail::find_mut(&self, ...) -> &mut T` → take `&mut self` (`clippy::mut_from_ref`). - `head_size % LAZY_MAX_ALIGN == 0` → `head_size.is_multiple_of(...)` (`clippy::manual_is_multiple_of`). - Two test setups switched from `LazyTail::default()` + field reassignment to struct-literal init (`clippy::field_reassign_with_default`). All `backend::lazy_tail::tests` (8) and `backend::storage_schema::tests` (17) pass under cargo test, clippy, and Miri's default Stacked Borrows mode.
…ize tail shifts The byte tail packs payloads in ascending tag order. Inserts and takes only memmove bytes belonging to variants packed AFTER the target tag — so churning a variant at offset 0 shifts the entire tail, while churning a variant at the end shifts nothing. Previously the iteration order was Transient -> Meta -> Data, placing the highest-churn category (transient variants like InProgress and Activeness that are installed during a task run and dropped at completion) at the head of the tail. Every transient install/remove was paying for shifting every persistent payload. Flip the order so Meta variants get the lowest tags (mostly stable, restored once on first access), Data variants get the middle tags (bulk, mostly written-once during execution), and Transient variants get the highest tags. Transient-only churn now leaves persistent payloads untouched. No on-disk format change: bincode encode/decode use per-category indices rather than the global tag, so the wire layout is unaffected. Mask constants (LAZY_PERSISTENT_MASK, LAZY_META_MASK, LAZY_DATA_MASK) and dispatch fns are all recomputed from the same iteration order, so they stay consistent. All 26 tests (18 storage_schema + 8 lazy_tail) pass under cargo test and Miri.
- Promote head alignment / size / TAIL_BUFFER_ALIGN checks from `debug_assert!` in `layout()` to compile-time `const _: () = assert!` so misalignment fails the build instead of being missed in release. - Replace `from_size_align_unchecked` with the checked variant + a proof-of-validity message; size+align bounds (u16 cap, power-of-two align) make the check trivially satisfied. - Snapshot `cap` before `drop_in_place` in `TaskStorage::drop` so the dealloc layout never has to read through a dropped value; drop the redundant `present`/`len` reset (auto-derived `LazyTail` drop is a no-op so nothing re-interprets the bytes). - Add explicit no-op `Drop` impl on `LazyTail` documenting that payload + buffer lifecycle is owned by `TaskStorage`. - Add `LazyTail::try_for_each_present` / `for_each_present` as the single iteration primitive for the bitmap walk. Encapsulates `sum_padded_sizes` (now private) and accumulates offsets across iterations, dropping the walk from O(popcount²) to O(popcount). Three call sites updated: `Drop`, `Debug::fmt`, `all_lazy_in_mask_empty`. - `insert_unchecked` returns `&mut T`; `lazy_get_or_create` uses it to skip the second `lazy_find_mut` lookup. - Compute `above_bytes` in `insert_unchecked` and `take` as `self.len - offset[ - payload_size]` rather than a second `sum_padded_sizes` walk over `mask_above`. - Factor `offset_for(tag)` out of `offset_of` so the unconditional offset math has one home. - Add `lazy_insert_shifts_existing_payloads` test exercising insert-side and take-side byte shifts across three direct lazy fields with distinct types.
…leanup encode_meta/encode_data used to ask every variant in the category "are you present?" via the per-variant accessor, paying an offset recomputation (popcount over `mask_below()`) on each hit. Replace the per-variant `if let Some(data) = self.get_<v>()` cascade with a single `LazyTail::try_for_each_present` walk: the iterator maintains the running byte-tail offset, the closure filters by the category mask, and per-tag match arms cast the ptr to the variant type and run the same encode body as before. Wire format is unchanged — the per-category index assignment still follows declaration order within the category, which matches the decode side. While here, drop `Tag::mask_above` and its test: the runtime "above-bytes" math in take/insert is computed from `self.len - offset - payload_size`, so the mask itself is unused. Also adds a comment on TaskStorage's `Drop` explaining the panic-mid-drop semantics (payload destructors are expected to be panic-free). All 7 lazy_tail tests and 18 storage_schema tests pass under cargo test, clippy, and Miri.
lukesandberg
force-pushed
the
flattened_storage
branch
from
b1247f4 to
7dd295e
Compare
lukesandberg
force-pushed
the
05-09-turbo-tasks_replace_taskstorage_lazy_vec_with_a_16_b_lazyvec
branch
from
ed75068 to
3192aa5
Compare
Base automatically changed from
05-09-turbo-tasks_replace_taskstorage_lazy_vec_with_a_16_b_lazyvec
to
canary
May 20, 2026 17:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What?
Replaces TaskStorage's
TinyVec<LazyField>lazy area with a presence bitmap plus a byte-packed payload tail that lives in the same heap allocation as the inline head. The result is one allocation per task instead of two, no per-entry discriminant tax, and the same 8-byte machine-word footprint for an owningTaskStorage(still a thin pointer).Why?
TinyVec<LazyField>was paying:TinyVecbuffer, with its own allocator metadata and fragmentation cost.We can't afford to make
Box<TaskStorage>a fat pointer (the DashMap stores 6.5M of them; fat pointers would add ~70MB of map overhead and erode the savings), so the new layout keeps the head sized and lays the tail bytes out immediately after the head in the same allocation, accessed via raw pointer arithmetic from(self.ptr).add(size_of::<TaskStorageInner>()).TaskStorageis a 1-wordNonNull<TaskStorageInner>andOption<TaskStorage>still niches.How?
The PR is intentionally many small commits — each a green build — so the review can follow the migration:
lazy.push/lazy.itercall sites behind typed accessors so the lazy storage representation is opaque to the rest of the crate.LAZY_TAG_<NAME>: Tag,LAZY_SIZE,LAZY_ALIGN,LAZY_PADDED_SIZE,LAZY_*_MASK, pluslazy_drop_dispatch/lazy_is_empty_dispatch/lazy_debug_dispatchper-tag match arms.LazyField/TinyVec, introduceLazyTail { present: u32, len: u16, cap: u16 }(8B head metadata) addressing a tag-ordered byte buffer.TaskStorage's own heap block.TaskStorageBox(nowTaskStorage) owns alloc / dealloc / grow / shrink with mimalloc-bin awareness (mi_good_size).Tag(NonZeroU8)newtype withbit/mask_belowhelpers, schema-emittedLAZY_TYPE_IDStable soTag::debug_assert_type::<T>()catches wrong-type casts in debug, customDebugimpl that pretty-prints each present lazy variant by name. Reorder lazy variantsMeta → Data → Transientso transient churn shifts the fewest bytes. Switchencode_meta/datato a single bitmap walk viaLazyTail::try_for_each_present.Miri-clean under default Stacked Borrows mode for the 25-test
backend::lazy_tail+backend::storage_schemasuites. Clippy-clean.Benchmarks
5 runs of a large Next.js production build (
pnpm next build --experimental-build-mode=compile,TURBOPACK_PERSISTENT_CACHE=0,nice -n -20, Spotlight + low-power mode disabled).05-09-...-16-b-lazyvec)The flattened-storage wall-time mean is dragged up by a single 44.07s outlier in run 3; the other four runs average 39.02s ± 0.27s.
Read of the numbers:
reallocinstead ofTinyVec's amortizedVecpush, but that cost is unavoidable for the layout to work.More samples would tighten the wall/RSS confidence intervals, but the CPU signal is already clean.
Follow-ups not in this PR
The real upside from this layout flexibility is letting
AutoSet/AutoMapkeep more of their content inline now that the lazy variant size isn't forced into a fixedLazyFieldenum slot. This PR doesn't change those types' inline sizes — it just removes the constraint that was preventing it.