Stop NoncommutativeConvergence from retaining 270 GiB of Lévy area coefficients

[ChrisRackauckas-Claude]

Please ignore this PR until it has been reviewed by @ChrisRackauckas.

Fixes #4036.

What was wrong

NoncommutativeConvergence grew linearly without plateau (>51 GiB observed, stopped there, no known ceiling) and could not pass on any runner in the fleet. The cause is sim6's Lévy-area truncation parameter, not a solver leak.

RKMilGeneral(p = true, dt = h) resolves the truncation as

p == true && (p = Int(floor(c * dt^(1 // 1 - 2 // 1 * gamma)) + 1))   # γ = 1  ⇒  p = 1/h

which is the Kloeden–Platen Fourier rule (convorder = 1/2). RKMilGeneral approximates the Lévy area with MronRoe (convorder = 1), so the terms actually needed for the requested accuracy ε = h^(3/2) are far fewer:

terms_needed(m = 4, h = 1/2^12, ε = h^1.5, MronRoe(), MaxL2()) = 12
norv(4,   12) =   106
norv(4, 4097) = 32786     # p = true, dt = 1/2^12

_z_prototype sizes the auxiliary noise increment to hold norv(m, p) coefficients, so p = 4097 makes dZ 32786 floats = 262 KB per step. The reference solve is 2049 steps → 0.5 GiB of saved Z.

That 0.5 GiB is then retained per trajectory. analyticless_test_convergence with use_noise_grid = false stores appxtrue(sol, true_sol) for every (trajectory, dt), and appxtrue ends in build_solution(sol, …), which carries sol.W — a NoiseWrapper whose .source is the reference NoiseProcess. Nothing is released until all 500 trajectories finish.

Measured retention, replicating the inner loop verbatim (Julia 1.12.4, single thread):

	retained / trajectory	× 500
RKMilGeneral(p = true, dt = 1/2^12)	0.5398 GiB	269.9 GiB
RKMilGeneral()	0.0019 GiB	0.9 GiB

Confirming it is p and nothing else: save_noise = false on the coarse solves changes retention by 0.0% (a NoiseWrapper-driven solve never stored its own W/Z), and the other sims in the file retain 0.063 / 0.065 / 0.222 GiB.

The change

Drop p = true, dt = test_dt_mil and use the automatic truncation. Reference dt and trajectory count are unchanged, so the study itself is the same; only the Lévy-area term count changes, from 341× over-provisioned to what the accuracy target asks for.

Verification

Strong order across 7 seeds at 500 trajectories, reference dt = 1/2^12:

seed	100	1	42	2718	31415	7	999
𝒪est[:final]	0.9447	0.9416	0.9519	0.9249	0.9185	0.9456	0.9513
|𝒪 - 1|	0.055	0.058	0.048	0.075	0.082	0.054	0.049

All pass the existing < 0.2 tolerance with ≥2.4× margin; the tolerance is untouched. (The seed matters because sim1–sim5 consume draws before sim6, so the effective seed there is not pinned by the file's Random.seed!(100).)

Full group run, ODEDIFFEQ_TEST_GROUP=NoncommutativeConvergence, self-hosted 8vcpu/16gb:

Test Summary:              | Pass  Total     Time
Noncommutative Noise Tests |    7      7  3m34.6s
     Testing StochasticDiffEq tests passed

	Elapsed (wall clock) time (h:mm:ss or m:ss): 12:49.77
	Maximum resident set size (kbytes): 6230276      # 5.94 GiB, incl. test-env precompile
	Exit status: 0

For contrast, the current p = true form needs 145 s and peaks at 18.3 GiB for only 25 trajectories — which is why job 89959748664 lost the runner at 28 min on the 16 GB pool.

CI config

The group was pinned to a high-memory label that no runner in the fleet carries, so it never scheduled and held the whole Sublibrary CI run open (#4030). With the memory bounded at ~6 GiB it can go on the standard self-hosted pool, and the 240-minute timeout drops to 60 (≈17× headroom over the measured 3m35s test body, ≈4.7× over the 12m50s including precompilation).

Deliberately not in this PR

• The p == true formula itself. It hands you the Fourier truncation for an algorithm that no longer uses the Fourier expansion. Deriving it from terms_needed for the algorithm in use would be a solver behavior change and wants its own PR and tests.
• _z_prototype's p === nothing branch guesses max(10, m) rather than calling terms_needed, because dt is not in scope there. At h = 1/2^12 that gives 10 where 12 is wanted — a ~18% overshoot of the error target, visible as the order landing near 0.92–0.95 rather than 1.0. Fixing it properly means threading dt into _z_prototype, which is exported from StochasticDiffEqCore.
• The DiffEqDevTools retention. build_solution in appxtrue carries prob/W that calculate_ensemble_errors never reads. Fixing that would stop any caller from pinning reference noise, not just this one. This PR only removes the 300× multiplier in front of it.
• OOPWeakConvergence / IIPWeakConvergence stay pinned to high-memory. They fail for a different reason — test_convergence retaining all 5e5 × 5 solution objects, measured 2.3 KiB each ≈ 5.4 GiB per study, with two studies live at once at global scope. Bounded, unlike this one, but past 16 GB. Separate change.

---

Links

• Issue: NoncommutativeConvergence test group grows to >51 GiB with no plateau #4036
• Diagnosis with the full measurement set: NoncommutativeConvergence test group grows to >51 GiB with no plateau #4036 (comment)
• Blocked-CI issue: Route CI jobs to runner pools that exist and can hold them #4030
• Commit that introduced the parameters: 581148d

🤖 Generated with Claude Code

https://claude.ai/code/session_013WW2jGeoWZVZu7AiscUNSz

[ChrisRackauckas-Claude]

Superseded by #4038, which landed the same sim6 change (RKMilGeneral(p = true, dt = test_dt_mil) → RKMilGeneral()) plus the deeper solver fix I had deferred: deriving the truncation from MronRoe's error bound via a new four-argument _z_prototype(alg, rand_prototype, iip, dt), and making the automatic path honour the accuracy target instead of the max(10, m) guess. #4036 is fixed there.

Rebasing this branch would now do more harm than good. Relative to current master it would:

• delete the RKMilGeneral dZ truncation sizing testset that Size RKMilGeneral's Lévy area truncation from MronRoe's error bound #4038 added;
• add a comment warning against p = true that is no longer true — Size RKMilGeneral's Lévy area truncation from MronRoe's error bound #4038 redefined p = true to mean p = nothing, so it is now harmless;
• revert ODEDIFFEQ_TEST_GROUP back to GROUP, undoing Use GROUP consistently for sublibrary tests #3909;
• re-add a runner key that master deliberately dropped.

Closing as superseded. The measurements from the investigation are in #4036 (comment) and remain valid for the weak-convergence groups, which #4038 does not touch.