Memory leak: module-level seen cache in experimental mutators pins every mutated program's type-graph for the process lifetime

[xirzec]

Summary

The experimental mutator engine (packages/compiler/src/experimental/mutators.ts) holds a module-level, never-cleared, strongly-referenced cache named seen. Every call to mutateSubgraph / mutateSubgraphWithNamespace inserts Type values (both originals and clones) into this cache, and the cache is never cleared. As a result, the entire type-graph of every program that is ever mutated stays pinned in memory for the lifetime of the process.

When a host process compiles + emits many programs in the same process (e.g. running an emitter's unit test suite, or any tool that drives compile() in a loop), heap usage grows without bound (~32 MB per versioned program in my repro) and eventually OOMs.

This is the upstream root cause of Azure/typespec-azure#4536 ("[typespec-ts] Emitter is leaking memory between calls"). It affects all emitters that trigger versioning mutation via TCGC's createSdkContext (e.g. @azure-tools/typespec-ts, @azure-tools/typespec-autorest), but the bug itself is entirely in the core compiler and is independent of any Azure code.

Root cause

packages/compiler/src/experimental/mutators.ts:

const typeId = CustomKeyMap.objectKeyer();
const mutatorId = CustomKeyMap.objectKeyer();
const seen = new CustomKeyMap<[MutableTypeWithNamespace, Set<Mutator> | Mutator[]], Type>(
  ([type, mutators]) => {
    const key = `${typeId.getKey(type)}-${[...mutators.values()]
      .map((v) => mutatorId.getKey(v))
      .join("-")}`;
    return key;
  },
);

Three properties combine to make this a permanent leak:

1. seen is module scope and never cleared. grep for seen.clear / seen.delete / seen = finds only the declaration. Entries live as long as the module (i.e. the process).
2. CustomKeyMap is backed by a strong Map. From packages/compiler/src/utils/custom-key-map.ts:

   export class CustomKeyMap<K extends readonly any[], V> {
     #items = new Map<string, V>();   // strong refs to V (= Type) forever
     ...
   }

   The stored values are Type objects — clones in the general case, and the original input type in the "no mutation needed" branch (seen.set([type, activeMutators], type)). Holding any Type strongly pins its whole program graph (namespace links, SymbolTable, checker TypeInstantiationMap, etc.).
3. Keys never collide or overwrite across programs. CustomKeyMap.objectKeyer() assigns ids from a let count = 0 that increments per process and never resets. Types from a second program get fresh ids, so their keys never match the first program's keys — entries purely accumulate, they are never replaced.

Because the realm/clone identity is per top-level mutate call (each createMutatorEngine makes a fresh Realm), sharing seen across calls isn't just a leak — returning a clone that belongs to a previous realm would also be a latent correctness hazard.

Why it only reproduces on some paths

The leak is only exercised when something actually drives the mutator engine. The most common trigger is versioning: @azure-tools/typespec-client-generator-core's createSdkContext calls handleVersioningMutationForGlobalNamespace, which calls getVersioningMutators + unsafe_mutateSubgraphWithNamespace. So:

• Plain compile() with noEmit → mutator engine never runs → flat.
• createSdkContext on a non-versioned spec → no versioning mutation → flat.
• createSdkContext on a @versioned spec → versioning mutation runs → ~32 MB pinned per program.

This also explains a confusing observation downstream: an emitter's modular unit path looked flat because those fixtures were mostly non-versioned, while the production/integration path (versioned service specs) leaked.

Reproduction

Minimal in-process loop (Node with --expose-gc): compile + createSdkContext (or run any emitter that does) over a @versioned service spec N times, calling global.gc() twice between iterations and recording process.memoryUsage().heapUsed. The post-GC floor is the signal (RSS sawtooth is not).

A pure-compiler repro without any Azure packages: call unsafe_mutateSubgraphWithNamespace(program, versioningMutators, globalNamespace) in a loop over freshly compiled versioned programs and watch the post-GC floor climb.

Empirical evidence (A/B on identical input)

Same @versioned service spec, 8 in-process emit iterations, forced GC each iteration, post-GC heapUsed floor:

Build	iter 0 floor	iter 7 floor	per-iter growth
Unpatched	53.7 MB	280.1 MB	~32 MB/iter, linear, unbounded
seen cleared after each top-level mutate	53.4 MB	88.4 MB	flat after warmup (~0.4 MB/iter)

Heap-snapshot diffs on the unpatched build show the growth dominated by retained object::SymbolTable, object::RekeyableMapImpl, object::TypeInstantiationMap and closure::getMappedType (checker.js) — i.e. whole prior programs pinned, consistent with the seen analysis.

Recommended fix

Make seen (and the typeId / mutatorId keyers it depends on) per-engine instead of module-level — move them inside createMutatorEngine, so each top-level mutateSubgraph / mutateSubgraphWithNamespace call gets its own cache that becomes collectable as soon as the call's realm is released. This:

• bounds retention to the lifetime of a single mutation (no cross-program accumulation), and
• removes the cross-realm clone-identity hazard.

A minimal/低-risk alternative is to seen.clear() at the end of mutateSubgraph and mutateSubgraphWithNamespace (verified to flatten the floor in the table above), but per-engine scoping is the cleaner, more correct change. If any caller actually depends on cross-call clone identity, that contract should be made explicit rather than relying on an unbounded global cache.

(Whatever the fix, please add a regression test that loops the mutator over multiple programs and asserts the post-GC heap floor stays bounded.)

Environment

• @typespec/compiler 1.13.0
• core commit 37199629393b92015668f107b3ac2502ca734f53
• Node v22

Downstream reference

• [typespec-ts] Emitter is leaking memory between calls Azure/typespec-azure#4536 — the OOM symptom that led here.

[github-actions]

🎯 Issue Triage

The seen cache in packages/compiler/src/experimental/mutators.ts is declared at module scope and never cleared, causing every mutated program's full type-graph to be retained for the process lifetime — growing ~32 MB per iteration when emitting @versioned specs in a loop.

Category: Bug
Area: compiler:core

⚠️ No TypeSpec playground reproduction applies here (this is a runtime memory/process-lifecycle issue, not a type-checking or emit correctness issue). The issue includes empirical heap measurements and a clear in-process reproduction procedure.

🔍 Analysis

Root cause: Three properties combine:

1. seen, typeId, and mutatorId are module-level — they outlive any single mutateSubgraph call.
2. CustomKeyMap holds strong references to Type values, pinning entire program graphs (namespace links, SymbolTable, TypeInstantiationMap).
3. CustomKeyMap.objectKeyer() uses a monotonically incrementing counter per process — keys from different programs never collide, so entries only accumulate.

Affected file: packages/compiler/src/experimental/mutators.ts

Trigger path: Any caller of unsafe_mutateSubgraphWithNamespace with versioning mutators (most commonly TCGC's createSdkContext on a @versioned spec). Downstream impact covers all Azure client emitters (typespec-ts, typespec-autorest, etc.) via TCGC.

Recommended fix: Move seen, typeId, and mutatorId inside createMutatorEngine so the cache is scoped to a single mutation call and becomes collectable when the realm is released. A lower-risk alternative (already empirically validated) is to call seen.clear() at the end of mutateSubgraph / mutateSubgraphWithNamespace.

Also noted: Sharing seen across realm boundaries is a latent correctness hazard — a clone from a previous realm could be returned to a caller expecting a clone in the current realm.

Downstream reference: Azure/typespec-azure#4536

📝 Suggested next steps

• Move seen, typeId, mutatorId inside createMutatorEngine (preferred fix) — or add seen.clear() calls as a quick mitigation.
• Add a regression test: loop unsafe_mutateSubgraphWithNamespace over multiple freshly compiled @versioned programs with --expose-gc, and assert the post-GC heap floor stays bounded.

Generated by Agentic Triage