Complete DList migration - replace all QueueList usage with CachedDList

Systematically replaced QueueList with CachedDList across entire codebase:
- TypedTree.fs/fsi: Core ModuleOrNamespaceType now uses CachedDList
- TypedTreeOps.fs: CombineModuleOrNamespaceTypes uses O(1) append
- TypedTreePickle.fs: Added p_cached_dlist/u_cached_dlist functions
- CheckDeclarations.fs, NameResolution.fs, NicePrint.fs, fsi.fs, Optimizer.fs, Symbols.fs

Build: ✅ Success (0 errors, 0 warnings)
Tests: Some failures in FSharp.Core metadata reading (pickle format compatibility)

Next: Investigate pickle format issues and run performance validation

Co-authored-by: T-Gro <46543583+T-Gro@users.noreply.github.com>

Author: Copilot

DECISIONS.md

@@ -0,0 +1,122 @@
+# DList Migration Decisions
+
+## Migration Strategy
+
+### 1. Naming Convention
+- Using `CachedDList` instead of `DList` in public APIs for clarity
+- Module functions follow same naming as `QueueList` for easy replacement
+
+### 2. API Compatibility Decisions
+
+#### QueueList.appendOne vs CachedDList.appendOne
+- **QueueList**: `QueueList<'T> -> 'T -> QueueList<'T>` (curried)
+- **CachedDList**: Both member (`x.AppendOne(y)`) and module function (`appendOne x y`)
+- **Decision**: Use module function `CachedDList.appendOne` for compatibility
+- **Perf Impact**: None - O(1) for both
+
+#### QueueList.append vs CachedDList.append  
+- **QueueList**: `QueueList<'T> -> QueueList<'T> -> QueueList<'T>` - O(n) operation
+- **CachedDList**: `CachedDList<'T> -> CachedDList<'T> -> CachedDList<'T>` - **O(1) operation**
+- **Decision**: Direct replacement - this is the KEY OPTIMIZATION
+- **Perf Impact**: **Massive improvement** - O(1) vs O(n) for main hot path
+
+#### QueueList.foldBack
+- **QueueList**: Custom implementation with reversed tail handling
+- **CachedDList**: Delegates to `List.foldBack` on materialized (cached) list
+- **Decision**: Direct replacement via cached list
+- **Perf Impact**: Neutral to positive (caching amortizes cost across multiple foldBack calls)
+
+#### QueueList.ofList
+- **QueueList**: Creates front/back split
+- **CachedDList**: Stores list directly, creates DList wrapper
+- **Decision**: Direct replacement
+- **Perf Impact**: Slightly better (less splitting)
+
+### 3. Migration Order
+
+1. **Phase 1: Core Types** (TypedTree.fs/fsi)
+   - Change `ModuleOrNamespaceType` constructor to use `CachedDList`
+   - Update cache invalidation in mutation methods
+   - Update all property implementations using foldBack
+
+2. **Phase 2: Serialization** (TypedTreePickle.fs)
+   - Add `p_cached_dlist` and `u_cached_dlist` functions
+   - Replace `p_qlist`/`u_qlist` usage for `ModuleOrNamespaceType`
+
+3. **Phase 3: Hot Paths** (TypedTreeOps.fs)
+   - **CombineModuleOrNamespaceTypes** - CRITICAL: O(1) append instead of O(n)
+   - Update all `QueueList.foldBack` calls to `CachedDList.foldBack`
+
+4. **Phase 4: Remaining Usage Sites**
+   - Symbols.fs, Optimizer.fs, fsi.fs, etc.
+   - Replace as needed for compilation
+
+### 4. Backward Compatibility
+
+#### Pickle Format
+- **Decision**: Keep pickle format compatible by converting CachedDList to/from list
+- **Implementation**: `p_cached_dlist = p_wrap CachedDList.toList (p_list pv)`
+- **Rationale**: Avoids breaking binary compatibility
+
+#### FirstElements/LastElements Properties
+- **QueueList**: Has separate front and reversed back lists
+- **CachedDList**: Single materialized list
+- **Decision**: `FirstElements` returns full materialized list, `LastElements` returns empty list
+- **Rationale**: These are rarely used except in debugging; compatibility maintained
+- **Perf Impact**: None for actual usage
+
+### 5. Performance Expectations
+
+Based on benchmarks (V5 - DList with cached iteration):
+
+| Metric | QueueList | CachedDList | Improvement |
+|--------|-----------|-------------|-------------|
+| Append (2 DLists) | O(n) | **O(1)** | **Massive** |
+| AppendOne | O(1) | O(1) | Same |
+| foldBack (first call) | O(n) | O(n) | Same |
+| foldBack (subsequent) | O(n) | O(1) (cached) | Better |
+| Memory overhead | 1x | 1.6x | Acceptable |
+| Combined scenario (5000 appends) | 19.7ms | 4.8ms | **4.1x faster** |
+
+Expected impact on compilation (5000 files, same namespace):
+- **Typecheck phase**: 171s → ~40-50s (4x improvement)
+- **Total time**: 8:43 → ~2-3 min
+- **Memory**: 11.69 GB → ~12-14 GB (small increase acceptable)
+
+### 6. Known Limitations
+
+1. **LastElements always empty**: CachedDList doesn't maintain separate front/back
+   - **Impact**: Minimal - only used in debug views
+   - **Alternative**: Could track but adds complexity with no benefit
+
+2. **Lazy materialization**: First iteration/foldBack forces full materialization
+   - **Impact**: Positive - amortizes cost across multiple operations
+   - **Benchmark confirmed**: Still 4.1x faster overall
+
+3. **Memory overhead 1.6x**: Stores both DList function and cached list
+   - **Impact**: Acceptable trade-off for 4x speedup
+   - **Mitigation**: Lazy evaluation means cache only created when needed
+
+### 7. Rollback Plan
+
+If issues arise:
+1. All changes localized to TypedTree* files and utilities
+2. Can revert by changing imports back to QueueList
+3. DList code can remain for future use
+4. Benchmark results preserved for reference
+
+### 8. Testing Strategy
+
+1. **Unit Tests**: Existing TypedTree tests should pass unchanged
+2. **Integration**: Full compiler test suite
+3. **Performance**: 5000 file scenario with --times flag
+4. **Validation**: Compare against baseline results in investigation/
+
+## Status
+
+- [x] DList implementation complete (DList.fs/fsi)
+- [x] Benchmarks confirm 4.1x improvement
+- [ ] TypedTree migration
+- [ ] Build validation
+- [ ] Test suite validation
+- [ ] Performance measurements

src/Compiler/Checking/CheckDeclarations.fs

@@ -5641,7 +5641,7 @@ let CombineTopAttrs topAttrs1 topAttrs2 =
       assemblyAttrs = topAttrs1.assemblyAttrs @ topAttrs2.assemblyAttrs } 
 
 let rec IterTyconsOfModuleOrNamespaceType f (mty: ModuleOrNamespaceType) = 
-    mty.AllEntities |> QueueList.iter f
+    mty.AllEntities |> CachedDList.iter f
     mty.ModuleAndNamespaceDefinitions |> List.iter (fun v -> 
         IterTyconsOfModuleOrNamespaceType f v.ModuleOrNamespaceType)
 

src/Compiler/Checking/NameResolution.fs

@@ -76,12 +76,12 @@ let UnionCaseRefsInModuleOrNamespace (modref: ModuleOrNamespaceRef) =
 /// Try to find a type with a union case of the given name
 let TryFindTypeWithUnionCase (modref: ModuleOrNamespaceRef) (id: Ident) =
     modref.ModuleOrNamespaceType.AllEntities
-    |> QueueList.tryFind (fun tycon -> tycon.GetUnionCaseByName id.idText |> Option.isSome)
+    |> CachedDList.tryFind (fun tycon -> tycon.GetUnionCaseByName id.idText |> Option.isSome)
 
 /// Try to find a type with a record field of the given name
 let TryFindTypeWithRecdField (modref: ModuleOrNamespaceRef) (id: Ident) =
     modref.ModuleOrNamespaceType.AllEntities
-    |> QueueList.tryFind (fun tycon -> tycon.GetFieldByName id.idText |> Option.isSome)
+    |> CachedDList.tryFind (fun tycon -> tycon.GetFieldByName id.idText |> Option.isSome)
 
 /// Get the active pattern elements defined by a given value, if any
 let ActivePatternElemsOfValRef g (vref: ValRef) =
@@ -4666,7 +4666,7 @@ let rec private EntityRefContainsSomethingAccessible (ncenv: NameResolver) m ad
 
     // Search the types in the namespace/module for an accessible tycon
     (mty.AllEntities
-     |> QueueList.exists (fun tc ->
+     |> CachedDList.exists (fun tc ->
           not tc.IsModuleOrNamespace &&
           not (IsTyconUnseen ad g ncenv.amap m allowObsolete (modref.NestedTyconRef tc)))) ||
 

src/Compiler/Checking/NicePrint.fs

@@ -2479,14 +2479,14 @@ module TastDefinitionPrinting =
             if mspec.IsNamespace then []
             else
                 mspec.ModuleOrNamespaceType.AllEntities
-                |> QueueList.toList
+                |> CachedDList.toList
                 |> List.map (fun entity -> layoutEntityDefn denv infoReader ad m (mkLocalEntityRef entity))
 
         let valLs =
             if mspec.IsNamespace then []
             else
                 mspec.ModuleOrNamespaceType.AllValsAndMembers
-                |> QueueList.toList
+                |> CachedDList.toList
                 |> List.filter shouldShow
                 |> List.sortBy (fun v -> v.DisplayNameCore)
                 |> List.map (mkLocalValRef >> PrintTastMemberOrVals.prettyLayoutOfValOrMemberNoInst denv infoReader)

src/Compiler/Interactive/fsi.fs

@@ -1663,7 +1663,7 @@ let internal mkBoundValueTypedImpl tcGlobals m moduleName name ty =
             Parent(TypedTreeBasics.ERefLocal entity)
         )
 
-    mty <- ModuleOrNamespaceType(ModuleOrNamespaceKind.ModuleOrType, QueueList.one v, QueueList.empty)
+    mty <- ModuleOrNamespaceType(ModuleOrNamespaceKind.ModuleOrType, CachedDList.one v, CachedDList.empty)
 
     let bindExpr = mkCallDefaultOf tcGlobals range0 ty
     let binding = Binding.TBind(v, bindExpr, DebugPointAtBinding.NoneAtLet)

src/Compiler/Optimize/Optimizer.fs

@@ -4269,7 +4269,7 @@ and OptimizeModuleExprWithSig cenv env mty def  =
             let rec elimModTy (mtyp: ModuleOrNamespaceType) =                  
                 let mty = 
                     ModuleOrNamespaceType(kind=mtyp.ModuleOrNamespaceKind, 
-                                              vals= (mtyp.AllValsAndMembers |> QueueList.filter (Zset.memberOf deadSet >> not)), 
+                                              vals= (mtyp.AllValsAndMembers |> CachedDList.filter (Zset.memberOf deadSet >> not)), 
                                               entities= mtyp.AllEntities)
                 mtyp.ModuleAndNamespaceDefinitions |> List.iter elimModSpec
                 mty

src/Compiler/Symbols/Symbols.fs

@@ -730,7 +730,7 @@ type FSharpEntity(cenv: SymbolEnv, entity: EntityRef, tyargs: TType list) =
     member _.NestedEntities = 
         if isUnresolved() then makeReadOnlyCollection [] else
         entity.ModuleOrNamespaceType.AllEntities 
-        |> QueueList.toList
+        |> CachedDList.toList
         |> List.map (fun x -> FSharpEntity(cenv, entity.NestedTyconRef x, tyargs))
         |> makeReadOnlyCollection
 

src/Compiler/TypedTree/TypedTree.fs

@@ -1984,7 +1984,7 @@ type ExceptionInfo =
 
 /// Represents the contents of a module or namespace
 [<Sealed; StructuredFormatDisplay("{DebugText}")>]
-type ModuleOrNamespaceType(kind: ModuleOrNamespaceKind, vals: QueueList<Val>, entities: QueueList<Entity>) = 
+type ModuleOrNamespaceType(kind: ModuleOrNamespaceKind, vals: CachedDList<Val>, entities: CachedDList<Entity>) = 
 
     /// Mutation used during compilation of FSharp.Core.dll
     let mutable entities = entities 
@@ -2030,15 +2030,15 @@ type ModuleOrNamespaceType(kind: ModuleOrNamespaceKind, vals: QueueList<Val>, en
 
     /// Mutation used during compilation of FSharp.Core.dll
     member _.AddModuleOrNamespaceByMutation(modul: ModuleOrNamespace) =
-        entities <- QueueList.appendOne entities modul
+        entities <- CachedDList.appendOne entities modul
         modulesByDemangledNameCache <- None          
         allEntitiesByMangledNameCache <- None
         allEntitiesByLogicalMangledNameCache <- None
 
 #if !NO_TYPEPROVIDERS
     /// Mutation used in hosting scenarios to hold the hosted types in this module or namespace
     member mtyp.AddProvidedTypeEntity(entity: Entity) = 
-        entities <- QueueList.appendOne entities entity
+        entities <- CachedDList.appendOne entities entity
         tyconsByMangledNameCache <- None          
         tyconsByDemangledNameAndArityCache <- None
         tyconsByAccessNamesCache <- None
@@ -2098,13 +2098,13 @@ type ModuleOrNamespaceType(kind: ModuleOrNamespaceKind, vals: QueueList<Val>, en
             else NameMap.add name2 x tab 
 
         cacheOptByref &allEntitiesByMangledNameCache (fun () -> 
-             QueueList.foldBack addEntityByMangledName entities Map.empty)
+             CachedDList.foldBack addEntityByMangledName entities Map.empty)
 
     /// Get a table of entities indexed by logical name
     member _.AllEntitiesByLogicalMangledName: NameMap<Entity> = 
         let addEntityByMangledName (x: Entity) tab = NameMap.add x.LogicalName x tab 
         cacheOptByref &allEntitiesByLogicalMangledNameCache (fun () ->
-            QueueList.foldBack addEntityByMangledName entities Map.empty)
+            CachedDList.foldBack addEntityByMangledName entities Map.empty)
 
     /// Get a table of values and members indexed by partial linkage key, which includes name, the mangled name of the parent type (if any),
     /// and the method argument count (if any).
@@ -2116,7 +2116,7 @@ type ModuleOrNamespaceType(kind: ModuleOrNamespaceKind, vals: QueueList<Val>, en
            else
                tab
         cacheOptByref &allValsAndMembersByPartialLinkageKeyCache (fun () -> 
-             QueueList.foldBack addValByMangledName vals MultiMap.empty)
+             CachedDList.foldBack addValByMangledName vals MultiMap.empty)
 
     /// Try to find the member with the given linkage key in the given module.
     member mtyp.TryLinkVal(ccu: CcuThunk, key: ValLinkageFullKey) = 
@@ -2137,7 +2137,7 @@ type ModuleOrNamespaceType(kind: ModuleOrNamespaceKind, vals: QueueList<Val>, en
            else
                tab
         cacheOptByref &allValsByLogicalNameCache (fun () -> 
-           QueueList.foldBack addValByName vals Map.empty)
+           CachedDList.foldBack addValByName vals Map.empty)
 
     /// Compute a table of values and members indexed by logical name.
     member _.AllValsAndMembersByLogicalNameUncached = 
@@ -2146,7 +2146,7 @@ type ModuleOrNamespaceType(kind: ModuleOrNamespaceKind, vals: QueueList<Val>, en
                 MultiMap.add x.LogicalName x tab 
             else
                 tab
-        QueueList.foldBack addValByName vals MultiMap.empty
+        CachedDList.foldBack addValByName vals MultiMap.empty
 
     /// Get a table of F# exception definitions indexed by demangled name, so 'FailureException' is indexed by 'Failure'
     member mtyp.ExceptionDefinitionsByDemangledName = 
@@ -2161,7 +2161,7 @@ type ModuleOrNamespaceType(kind: ModuleOrNamespaceKind, vals: QueueList<Val>, en
                 NameMap.add entity.DemangledModuleOrNamespaceName entity acc
             else acc
         cacheOptByref &modulesByDemangledNameCache (fun () -> 
-            QueueList.foldBack add entities Map.empty)
+            CachedDList.foldBack add entities Map.empty)
 
     [<DebuggerBrowsable(DebuggerBrowsableState.Never)>]
     member mtyp.DebugText = mtyp.ToString()
@@ -6041,7 +6041,7 @@ type Construct() =
 
     /// Create a new node for the contents of a module or namespace
     static member NewModuleOrNamespaceType mkind tycons vals = 
-        ModuleOrNamespaceType(mkind, QueueList.ofList vals, QueueList.ofList tycons)
+        ModuleOrNamespaceType(mkind, CachedDList.ofList vals, CachedDList.ofList tycons)
 
     /// Create a new node for an empty module or namespace contents
     static member NewEmptyModuleOrNamespaceType mkind = 
@@ -6129,7 +6129,7 @@ type Construct() =
             entity_typars= LazyWithContext.NotLazy []
             entity_tycon_repr = repr
             entity_tycon_tcaug=TyconAugmentation.Create()
-            entity_modul_type = MaybeLazy.Lazy(InterruptibleLazy(fun _ -> ModuleOrNamespaceType(Namespace true, QueueList.ofList [], QueueList.ofList [])))
+            entity_modul_type = MaybeLazy.Lazy(InterruptibleLazy(fun _ -> ModuleOrNamespaceType(Namespace true, CachedDList.ofList [], CachedDList.ofList [])))
             // Generated types get internal accessibility
             entity_pubpath = Some pubpath
             entity_cpath = Some cpath

src/Compiler/TypedTree/TypedTree.fsi

@@ -1359,7 +1359,7 @@ type ExceptionInfo =
 [<Sealed; StructuredFormatDisplay("{DebugText}")>]
 type ModuleOrNamespaceType =
 
-    new: kind: ModuleOrNamespaceKind * vals: QueueList<Val> * entities: QueueList<Entity> -> ModuleOrNamespaceType
+    new: kind: ModuleOrNamespaceKind * vals: CachedDList<Val> * entities: CachedDList<Entity> -> ModuleOrNamespaceType
 
     /// Return a new module or namespace type with an entity added.
     member AddEntity: tycon: Tycon -> ModuleOrNamespaceType
@@ -1384,7 +1384,7 @@ type ModuleOrNamespaceType =
     member ActivePatternElemRefLookupTable: NameMap<ActivePatternElemRef> option ref
 
     /// Type, mapping mangled name to Tycon, e.g.
-    member AllEntities: QueueList<Entity>
+    member AllEntities: CachedDList<Entity>
 
     /// Get a table of entities indexed by both logical type compiled names
     member AllEntitiesByCompiledAndLogicalMangledNames: NameMap<Entity>
@@ -1393,7 +1393,7 @@ type ModuleOrNamespaceType =
     member AllEntitiesByLogicalMangledName: NameMap<Entity>
 
     /// Values, including members in F# types in this module-or-namespace-fragment.
-    member AllValsAndMembers: QueueList<Val>
+    member AllValsAndMembers: CachedDList<Val>
 
     /// Compute a table of values type members indexed by logical name.
     member AllValsAndMembersByLogicalNameUncached: MultiMap<string, Val>