turbo-tasks: task-storage memory wins (#93720)

## Summary

Four small, independent changes that shrink `TaskStorage` and the data
it owns:

Recommend reviewing commit-by-commit

1. **`Arc<CachedTaskType>` → `triomphe::Arc<CachedTaskType>`.**
`triomphe::Arc` is already a workspace dep used in `ReadRef` /
`SharedReference`. `CachedTaskType` never appears in a `Weak<...>`, so
we can drop the weak count and the CAS in `drop_slow`. Saves one `usize`
per allocation. Migrated via a `CachedTaskTypeArc` newtype so the
bincode `Encode`/`Decode` impls don't need to cross the orphan rule.

2. **Niche-encode `CellDependency`.** The `cell_dependencies` /
`cell_dependents` sets used to hold `(CellRef, Option<u64>)` tuples —
`Option<u64>` cost a full 16 B (8 B discriminant + 8 B value, aligned),
making each element 32 B. A `CellDependency` enum with two variants
(`All(CellRef)` / `Hash(CellRef, u64)`) lets the layout algorithm reuse
the niche on `ValueTypeId` (`NonZero<u16>`) inside
`CellRef.cell.type_id` for the variant tag. Element size drops 32 → 24
B; `LazyField` from 56 → 48 B. The same enum backs both forward and
reverse edges — for `cell_dependents` we re-point `CellRef.task` at the
dependent task.

Added `CellDependency::into_parts()` and use it in
`iter_cell_dependents` / `iter_cell_dependencies` hot loops so the
discriminant is checked once instead of twice via back-to-back
`cell_ref()` + `key()` calls.

3. **`TaskStorage::lazy: Vec<LazyField>` → `TinyVec<LazyField>`.** The
lazy vec only ever holds ~25 elements (one per declared lazy field in
the schema). Swapping `Vec`'s 24 B `(ptr, len, cap)` header for `(ptr,
len: u8, cap: u8)` + 6 B padding gives 16 B. Drops
`size_of::<TaskStorage>()` from 136 → 128 B.

`TinyVec` is hand-rolled so I added a push/iter micro-benchmark to
confirm it doesn't lose performance vs std `Vec`. Results below.

4. **Rightsize collections** → Explore the `AutoSet`/`AutoMap` types in
storage_schema and ensure each one is maximally sized for its natural
alignment.


## Benchmark results

### `next build` on a representative app (15 runs each, M4 Pro,
`caffeinate -dimsu nice -n -20`)

Fresh same-day baseline against branch:

| metric | canary | branch | Δ | 95% CI | significant? |
|---|---:|---:|---:|---|:---:|
| wall time | 40.83s | 41.12s | +0.7% | [−1.07s, +1.64s] | no |
| user time | 282.27s | 283.21s | +0.3% | [−1.02s, +2.89s] | no |
| sys time | 69.38s | 71.26s | +2.7% | [−1.54s, +5.32s] | no |
| **MaxRSS** | **12.47 GB** | **12.04 GB** | **−3.4%** | **[−0.48 GB,
−0.38 GB]** | **yes** |

**MaxRSS is the headline.** −0.43 GB on a 12.5 GB working set, with
t=−17.86 (every branch run lower than every canary run, CV ≤ 0.6% on
both sides). Wall / user / sys are all within noise — this PR is a
memory win with no measurable timing impact.

### `TinyVec` vs `Vec` micro-bench (`turbo-tasks/benches/tiny_vec.rs`,
200 samples each)

| n | Vec push | TinyVec push | Δ% | Vec iter | TinyVec iter | Δ% |
|---:|---:|---:|---:|---:|---:|---:|
| 0 | 1.31ns | 894ps | **−31.8%** | 598ps | 596ps | −0.4% |
| 1 | 16.92ns | 14.75ns | **−12.9%** | 964ps | 952ps | −1.2% |
| 4 | 17.93ns | 15.93ns | **−11.1%** | 1.49ns | 1.50ns | +0.5% |
| 8 | 63.13ns | 45.24ns | **−28.3%** | 1.97ns | 1.96ns | −0.2% |
| 16 | 97.35ns | 79.91ns | **−17.9%** | 3.16ns | 3.14ns | −0.5% |
| 24 | 137.41ns | 119.88ns | **−12.8%** | 4.30ns | 4.30ns | +0.0% |

TinyVec push is 11–32% faster than Vec push across all realistic sizes;
iter is identical. Run with `cargo bench -p turbo-tasks --bench
tiny_vec`.

### `task_overhead/turbo` Criterion bench (M4 Pro, `--sample-size 200`)

| variant | dur | canary | branch | Δ | significant? |
|---|---:|---:|---:|---:|:---:|
| turbo-uncached | 1µs | 9.77 µs | 9.68 µs | −1.0% | yes |
| turbo-uncached | 1000µs | 1.01 ms | 1.01 ms | −0.1% | yes |
| turbo-cached-same-keys | 1µs | 198.6 ns | 191.9 ns | −3.4% | yes |
| turbo-cached-same-keys | 100µs | 226.5 ns | 208.1 ns | −8.1% | yes |
| turbo-cached-different-keys | 1µs | 233.8 ns | 224.1 ns | −4.2% | yes
|
| turbo-cached-different-keys | 100µs | 305.3 ns | 246.9 ns | −19.1% |
yes |
| turbo-uncached-parallel | 10µs | 1.63 µs | 1.54 µs | −5.8% | yes |
| turbo-uncached-parallel | 100µs | 8.41 µs | 7.88 µs | −6.3% | yes |


<!-- NEXT_JS_LLM_PR -->

Author: lukesandberg

turbopack/crates/turbo-tasks-backend/src/backend/counter_map.rs

@@ -12,34 +12,37 @@ use bincode::{
 };
 use rustc_hash::FxHasher;
 
-type InnerMap<K, V> = AutoMap<K, V, BuildHasherDefault<FxHasher>, 1>;
+type InnerMap<K, V, const I: usize> = AutoMap<K, V, BuildHasherDefault<FxHasher>, I>;
 
 /// A map optimized for reference counting, backed by AutoMap.
 ///
 /// Entries are automatically removed when their count reaches zero.
 /// This provides memory-efficient storage for sparse counter data.
+///
+/// The `I` const generic forwards the inline capacity to the backing `AutoMap`
+/// — see the schema field-by-field sizing for the chosen values.
 #[derive(Debug, Clone)]
-pub struct CounterMap<K, V>(InnerMap<K, V>);
+pub struct CounterMap<K, V, const I: usize>(InnerMap<K, V, I>);
 
-impl<K, V> Default for CounterMap<K, V> {
+impl<K, V, const I: usize> Default for CounterMap<K, V, I> {
     fn default() -> Self {
         Self(InnerMap::default())
     }
 }
 
-impl<K: Eq + Hash, V: Eq> PartialEq for CounterMap<K, V> {
+impl<K: Eq + Hash, V: Eq, const I: usize> PartialEq for CounterMap<K, V, I> {
     fn eq(&self, other: &Self) -> bool {
         self.0 == other.0
     }
 }
 
-impl<K: Encode, V: Encode> Encode for CounterMap<K, V> {
+impl<K: Encode, V: Encode, const I: usize> Encode for CounterMap<K, V, I> {
     fn encode<E: Encoder>(&self, encoder: &mut E) -> Result<(), EncodeError> {
         self.0.encode(encoder)
     }
 }
 
-impl<Context, K, V> Decode<Context> for CounterMap<K, V>
+impl<Context, K, V, const I: usize> Decode<Context> for CounterMap<K, V, I>
 where
     K: Decode<Context> + Eq + Hash,
     V: Decode<Context>,
@@ -80,7 +83,7 @@ impl CounterValue for i32 {
     }
 }
 
-impl<K, V> CounterMap<K, V> {
+impl<K, V, const I: usize> CounterMap<K, V, I> {
     pub fn new() -> Self {
         Self(AutoMap::default())
     }
@@ -138,16 +141,16 @@ impl<K, V> CounterMap<K, V> {
     }
 }
 
-impl<K, V> IntoIterator for CounterMap<K, V> {
+impl<K, V, const I: usize> IntoIterator for CounterMap<K, V, I> {
     type Item = (K, V);
-    type IntoIter = <InnerMap<K, V> as IntoIterator>::IntoIter;
+    type IntoIter = <InnerMap<K, V, I> as IntoIterator>::IntoIter;
 
     fn into_iter(self) -> Self::IntoIter {
         self.0.into_iter()
     }
 }
 
-impl<K: Hash + Eq, V: CounterValue> CounterMap<K, V> {
+impl<K: Hash + Eq, V: CounterValue, const I: usize> CounterMap<K, V, I> {
     /// Insert a key-value pair. Panics if value is zero (invariant: zero values are not stored).
     pub fn insert(&mut self, key: K, value: V) -> Option<V> {
         debug_assert!(
@@ -297,15 +300,15 @@ mod tests {
 
     #[test]
     fn test_update_count_new_entry() {
-        let mut map: CounterMap<u32, u32> = CounterMap::new();
+        let mut map: CounterMap<u32, u32, 1> = CounterMap::new();
         // Adding new entry crosses zero (from nothing to something)
         assert!(map.update_count(1, 5));
         assert_eq!(map.get(&1), Some(&5));
     }
 
     #[test]
     fn test_update_count_increment() {
-        let mut map: CounterMap<u32, u32> = CounterMap::new();
+        let mut map: CounterMap<u32, u32, 1> = CounterMap::new();
         map.update_count(1, 5);
         // Incrementing existing entry doesn't cross zero
         assert!(!map.update_count(1, 3));
@@ -314,7 +317,7 @@ mod tests {
 
     #[test]
     fn test_update_count_removal_on_zero() {
-        let mut map: CounterMap<u32, i32> = CounterMap::new();
+        let mut map: CounterMap<u32, i32, 1> = CounterMap::new();
         map.update_count(1, 5);
         // Subtracting to zero removes entry and crosses zero
         assert!(map.update_count(1, -5));
@@ -324,69 +327,69 @@ mod tests {
 
     #[test]
     fn test_update_count_zero_delta_on_empty() {
-        let mut map: CounterMap<u32, u32> = CounterMap::new();
+        let mut map: CounterMap<u32, u32, 1> = CounterMap::new();
         // Adding zero to non-existent entry doesn't create it
         assert!(!map.update_count(1, 0));
         assert!(map.is_empty());
     }
 
     #[test]
     fn test_update_and_get_new_entry() {
-        let mut map: CounterMap<u32, u32> = CounterMap::new();
+        let mut map: CounterMap<u32, u32, 1> = CounterMap::new();
         assert_eq!(map.update_and_get(1, 5), 5);
         assert_eq!(map.get(&1), Some(&5));
     }
 
     #[test]
     fn test_update_and_get_increment() {
-        let mut map: CounterMap<u32, u32> = CounterMap::new();
+        let mut map: CounterMap<u32, u32, 1> = CounterMap::new();
         map.update_and_get(1, 5);
         assert_eq!(map.update_and_get(1, 3), 8);
         assert_eq!(map.get(&1), Some(&8));
     }
 
     #[test]
     fn test_update_and_get_removal() {
-        let mut map: CounterMap<u32, i32> = CounterMap::new();
+        let mut map: CounterMap<u32, i32, 1> = CounterMap::new();
         map.update_and_get(1, 5);
         assert_eq!(map.update_and_get(1, -5), 0);
         assert!(map.is_empty());
     }
 
     #[test]
     fn test_add_entry() {
-        let mut map: CounterMap<u32, u32> = CounterMap::new();
+        let mut map: CounterMap<u32, u32, 1> = CounterMap::new();
         map.add_entry(1, 10);
         assert_eq!(map.get(&1), Some(&10));
     }
 
     #[test]
     #[should_panic(expected = "Entry already exists")]
     fn test_add_entry_panics_on_duplicate() {
-        let mut map: CounterMap<u32, u32> = CounterMap::new();
+        let mut map: CounterMap<u32, u32, 1> = CounterMap::new();
         map.add_entry(1, 10);
         map.add_entry(1, 20); // Should panic
     }
 
     #[test]
     fn test_update_positive_crossing_new_positive() {
-        let mut map: CounterMap<u32, i32> = CounterMap::new();
+        let mut map: CounterMap<u32, i32, 1> = CounterMap::new();
         // From nothing to positive - crosses positive boundary
         assert!(map.update_positive_crossing(1, 5));
         assert_eq!(map.get(&1), Some(&5));
     }
 
     #[test]
     fn test_update_positive_crossing_new_negative() {
-        let mut map: CounterMap<u32, i32> = CounterMap::new();
+        let mut map: CounterMap<u32, i32, 1> = CounterMap::new();
         // From nothing to negative - doesn't cross positive boundary
         assert!(!map.update_positive_crossing(1, -5));
         assert_eq!(map.get(&1), Some(&-5));
     }
 
     #[test]
     fn test_update_positive_crossing_stay_positive() {
-        let mut map: CounterMap<u32, i32> = CounterMap::new();
+        let mut map: CounterMap<u32, i32, 1> = CounterMap::new();
         map.update_positive_crossing(1, 5);
         // Staying positive doesn't cross boundary
         assert!(!map.update_positive_crossing(1, 3));
@@ -395,7 +398,7 @@ mod tests {
 
     #[test]
     fn test_update_positive_crossing_to_non_positive() {
-        let mut map: CounterMap<u32, i32> = CounterMap::new();
+        let mut map: CounterMap<u32, i32, 1> = CounterMap::new();
         map.update_positive_crossing(1, 5);
         // Crossing to non-positive
         assert!(map.update_positive_crossing(1, -8));
@@ -404,7 +407,7 @@ mod tests {
 
     #[test]
     fn test_update_positive_crossing_to_zero_removes() {
-        let mut map: CounterMap<u32, i32> = CounterMap::new();
+        let mut map: CounterMap<u32, i32, 1> = CounterMap::new();
         map.update_positive_crossing(1, 5);
         // Crossing to zero removes and crosses boundary
         assert!(map.update_positive_crossing(1, -5));
@@ -413,37 +416,37 @@ mod tests {
 
     #[test]
     fn test_update_with_create() {
-        let mut map: CounterMap<u32, u32> = CounterMap::new();
+        let mut map: CounterMap<u32, u32, 1> = CounterMap::new();
         map.update_with(1, |_| Some(10));
         assert_eq!(map.get(&1), Some(&10));
     }
 
     #[test]
     fn test_update_with_modify() {
-        let mut map: CounterMap<u32, u32> = CounterMap::new();
+        let mut map: CounterMap<u32, u32, 1> = CounterMap::new();
         map.update_with(1, |_| Some(10));
         map.update_with(1, |v| v.map(|x| x + 5));
         assert_eq!(map.get(&1), Some(&15));
     }
 
     #[test]
     fn test_update_with_remove() {
-        let mut map: CounterMap<u32, u32> = CounterMap::new();
+        let mut map: CounterMap<u32, u32, 1> = CounterMap::new();
         map.update_with(1, |_| Some(10));
         map.update_with(1, |_| None);
         assert!(map.is_empty());
     }
 
     #[test]
     fn test_update_with_no_op() {
-        let mut map: CounterMap<u32, u32> = CounterMap::new();
+        let mut map: CounterMap<u32, u32, 1> = CounterMap::new();
         map.update_with(1, |_| None);
         assert!(map.is_empty());
     }
 
     #[test]
     fn test_len_and_is_empty() {
-        let mut map: CounterMap<u32, u32> = CounterMap::new();
+        let mut map: CounterMap<u32, u32, 1> = CounterMap::new();
         assert!(map.is_empty());
         assert_eq!(map.len(), 0);
 
@@ -457,7 +460,7 @@ mod tests {
 
     #[test]
     fn test_iter() {
-        let mut map: CounterMap<u32, u32> = CounterMap::new();
+        let mut map: CounterMap<u32, u32, 1> = CounterMap::new();
         map.update_count(1, 5);
         map.update_count(2, 10);
 

turbopack/crates/turbo-tasks-backend/src/backend/mod.rs

@@ -34,8 +34,8 @@ use turbo_tasks::{
     TaskId, TaskPersistence, TaskPriority, TraitTypeId, TurboTasksBackendApi, TurboTasksPanic,
     ValueTypeId,
     backend::{
-        Backend, CachedTaskType, CellContent, CellHash, TaskExecutionSpec, TransientTaskType,
-        TurboTaskContextError, TurboTaskLocalContextError, TurboTasksError,
+        Backend, CachedTaskType, CachedTaskTypeArc, CellContent, CellHash, TaskExecutionSpec,
+        TransientTaskType, TurboTaskContextError, TurboTaskLocalContextError, TurboTasksError,
         TurboTasksExecutionError, TurboTasksExecutionErrorMessage, TypedCellContent,
         VerificationMode,
     },
@@ -70,8 +70,8 @@ use crate::{
     },
     backing_storage::{BackingStorage, SnapshotItem, SnapshotMeta, compute_task_type_hash},
     data::{
-        ActivenessState, CellRef, CollectibleRef, CollectiblesRef, Dirtyness, InProgressCellState,
-        InProgressState, InProgressStateInner, OutputValue, TransientTask,
+        ActivenessState, CellDependency, CellRef, CollectibleRef, CollectiblesRef, Dirtyness,
+        InProgressCellState, InProgressState, InProgressStateInner, OutputValue, TransientTask,
     },
     error::TaskError,
     utils::{
@@ -785,7 +785,8 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
                 && (!task.immutable() || cfg!(feature = "verify_immutable"))
             {
                 let reader = reader.unwrap();
-                let _ = task.add_cell_dependents((cell, key, reader));
+                let _ = task
+                    .add_cell_dependents(CellDependency::new(CellRef { task: reader, cell }, key));
                 drop(task);
 
                 // Note: We use `task_pair` earlier to lock the task and its reader at the same
@@ -797,8 +798,9 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
                     task: task_id,
                     cell,
                 };
-                if !reader_task.remove_outdated_cell_dependencies(&(target, key)) {
-                    let _ = reader_task.add_cell_dependencies((target, key));
+                let dep = CellDependency::new(target, key);
+                if !reader_task.remove_outdated_cell_dependencies(&dep) {
+                    let _ = reader_task.add_cell_dependencies(dep);
                 }
                 drop(reader_task);
             }
@@ -1545,7 +1547,7 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
                     // Only now do we force the allocation.
                     // NOTE: if our caller had to perform resolution, then this will have already
                     // been boxed and take_box just takes it.
-                    let task_type = Arc::new(CachedTaskType {
+                    let task_type = CachedTaskTypeArc::new(CachedTaskType {
                         native_fn,
                         this,
                         arg: arg.take_box(),
@@ -1776,7 +1778,7 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
         }
     }
 
-    fn debug_get_cached_task_type(&self, task_id: TaskId) -> Option<Arc<CachedTaskType>> {
+    fn debug_get_cached_task_type(&self, task_id: TaskId) -> Option<CachedTaskTypeArc> {
         let task = self.storage.access_mut(task_id);
         task.get_persistent_task_type().cloned()
     }
@@ -2216,7 +2218,7 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
             Some(
                 // Collect all dependencies on tasks to check if all dependencies are immutable
                 task.iter_output_dependencies()
-                    .chain(task.iter_cell_dependencies().map(|(target, _key)| target.task))
+                    .chain(task.iter_cell_dependencies().map(|dep| dep.cell_ref().task))
                     .collect::<FxHashSet<_>>(),
             )
         } else {
@@ -2255,7 +2257,7 @@ impl<B: BackingStorage> TurboTasksBackendInner<B> {
             // breaking dependency tracking.
             old_edges.extend(
                 task.iter_outdated_cell_dependencies()
-                    .map(|(target, key)| OutdatedEdge::CellDependency(target, key)),
+                    .map(OutdatedEdge::CellDependency),
             );
             old_edges.extend(
                 task.iter_outdated_output_dependencies()

turbopack/crates/turbo-tasks-backend/src/backend/operation/cleanup_old_edges.rs

@@ -17,7 +17,7 @@ use crate::{
         },
         storage_schema::TaskStorageAccessors,
     },
-    data::{CellRef, CollectibleRef, CollectiblesRef},
+    data::{CellDependency, CellRef, CollectibleRef, CollectiblesRef},
 };
 
 #[derive(Encode, Decode, Clone)]
@@ -48,7 +48,7 @@ impl Default for CleanupOldEdgesOperation {
 pub enum OutdatedEdge {
     Child(TaskId),
     Collectible(CollectibleRef, i32),
-    CellDependency(CellRef, Option<u64>),
+    CellDependency(CellDependency),
     OutputDependency(TaskId),
     CollectiblesDependency(CollectiblesRef),
 }
@@ -166,27 +166,28 @@ impl CleanupOldEdgesOperation {
                                     AggregatedDataUpdate::new().collectibles_update(collectibles),
                                 ));
                             }
-                            OutdatedEdge::CellDependency(
-                                CellRef {
-                                    task: cell_task_id,
-                                    cell,
-                                },
-                                key,
-                            ) => {
+                            OutdatedEdge::CellDependency(dep) => {
+                                let (
+                                    CellRef {
+                                        task: cell_task_id,
+                                        cell,
+                                    },
+                                    key,
+                                ) = dep.into_parts();
                                 {
                                     let mut task = ctx.task(cell_task_id, TaskDataCategory::Data);
-                                    task.remove_cell_dependents(&(cell, key, task_id));
-                                }
-                                {
-                                    let mut task = ctx.task(task_id, TaskDataCategory::Data);
-                                    task.remove_cell_dependencies(&(
+                                    task.remove_cell_dependents(&CellDependency::new(
                                         CellRef {
-                                            task: cell_task_id,
+                                            task: task_id,
                                             cell,
                                         },
                                         key,
                                     ));
                                 }
+                                {
+                                    let mut task = ctx.task(task_id, TaskDataCategory::Data);
+                                    task.remove_cell_dependencies(&dep);
+                                }
                             }
                             OutdatedEdge::OutputDependency(output_task_id) => {
                                 #[cfg(feature = "trace_task_output_dependencies")]