fix(#1380): set training mode to false for validation/test forward passes in Optimize loop (#1412)

* fix(#1380 part 1): set training mode to false for validation/test forward passes in Optimize loop

root cause (4-of-6-test bisection in HE Phase_PAPER_A_BuildAsyncDiagnostic_1380Bisect.cs):
- AdamOptimizer.Optimize (line 188) sets training mode TRUE for the entire
  optimize loop, then never flips it off for the per-epoch evaluation
  forward passes.
- EvaluateModelDirectly runs Predict on validation + test datasets.
- With training mode still TRUE, those forward passes update LayerNorm/
  BatchNorm running statistics using VALIDATION-batch stats — typically
  a much smaller dataset than training (e.g. 171 samples val vs 800
  train in the DataSplitter.Split 70/15/15 default).
- The corrupted running stats are then read on the next training forward
  pass, producing NaN gradients → NaN params under BuildAsync's
  IInputOutputDataLoader code path.

empirical evidence:
- HE Test 4 (XTrain=XVal=XTest=1144 full): finite L2 = 164.7 — works
- HE Test 6 (XTrain=800, XVal=XTest=1144 full): finite L2 = 1.91 — works
- HE Test 5 (XTrain=800, XVal=171, XTest=173 — DataSplitter.Split sizes):
  NaN params before this fix; 5.7e-6 params after this fix.

the fix:
- wrap EvaluateSolution body in SetTrainingMode(false) try/finally so
  the validation/test forward passes don't mutate running stats.
- also wrap PrepareAndEvaluateSolution (called directly by
  Adam.Optimize line 192 for the pre-epoch baseline eval, bypassing
  EvaluateSolution) — split body into PrepareAndEvaluateSolutionCore so
  the public-API wrapper can apply the same training-mode guard
  without changing the protected method signature.

status: NaN → finite is fixed. there is a separate sub-bug (params
collapse to ~0 instead of training to a meaningful state) that
manifests on the small-split-data configuration; tracked separately
as #1380 part 2.

test coverage: tests/Learning/Phase_PAPER_A_BuildAsyncDiagnostic_1380Bisect.cs
in cheatcountry/HarmonicEngine provides the regression bar — 6
[Fact] methods that progressively narrow the bug location.

co-authored-by: claude opus 4.7 (1m context) <noreply@anthropic.com>

* fix(#1380 part 2): port Step's anomaly guard + gradient clipping into UpdateSolution; default EnableGradientClipping=true

after part 1 (training-mode fix), Optimize still mode-collapsed or
diverged depending on configuration:
  - plain Adam + default L2:     params → 5.7e-6 (collapse)
  - plain Adam + NoReg:           params → 1.4e9 (explode)
  - AMSGrad + default L2:         params → 0 (collapse)

root cause (per AiDotNet#1413): AdamOptimizer has TWO update
implementations that drifted:
  - Step(TapeStepContext): used by nn.Train → TrainWithTape (bypass).
    has anomaly guard + gradient clipping check.
  - UpdateSolution(solution, gradient): used by Optimize → BuildAsync.
    LACKED anomaly guard and clipping check.

the bypass works because Step's safeguards prevent runaway gradients
from poisoning Adam's m/v moments. the Optimize path lacked them.

this PR ports BOTH safeguards into UpdateSolution:
1. AnyGradientIsAnomalous(Vector<T>) overload added — scans flat
   gradient for NaN/Inf and returns true on first sighting.
2. UpdateSolution checks ShouldRunAnomalyGuard() + the vector
   overload at entry; on positive, skips the entire update
   (matches Step's line 651-654 behavior — no params, no _m/_v,
   no _t change).
3. UpdateSolution calls ApplyGradientClipping(gradient) before
   the Adam math (matches Step's line 700-703).
4. EnableGradientClipping default flipped to TRUE — matches
   PyTorch's transformer-training canonical recipe
   (torch.nn.utils.clip_grad_norm_, max_norm=1.0). Callers
   wanting old behavior set false explicitly.

verification (cheatcountry/HarmonicEngine Test 5 — split data
800/171/173):
  shipped 0.206.0:           NaN
  + part 1 (training mode):  5.7e-6 (collapse)
  + part 2 (this PR):        81926 (finite, growing but not exploding)

the remaining gap from per-sample bypass (L2=42.5) is a regime
difference — Optimize runs 50 batched updates per 2 epochs while
per-sample runs 1600 per-sample updates. that's hyperparameter
tuning, not a bug.

failure modes now eliminated: NaN, exact-zero collapse, explode-to-inf.
Optimize produces meaningful training that the caller can tune.

filing on top of PR #1412 (part 1) commit.

co-authored-by: claude opus 4.7 (1m context) <noreply@anthropic.com>

* fix(#1380 #1413): consolidate flat-vector UpdateSolution into Step delegation across all 19 gradient-based optimizers

closes #1413 architectural split. PyTorch / TensorFlow / JAX-Optax all
have ONE update implementation per optimizer; AiDotNet had TWO
(Step(TapeStepContext) for tape path, UpdateSolution(flat) for Optimize
path) that drifted feature-wise. each subclass's UpdateSolution missed
fixes that landed in Step. this PR collapses to ONE.

architecture (GradientBasedOptimizerBase.UpdateSolution virtual):
  if (solution is INeuralNetwork<T>) {
    ctx = SynthesizeTapeStepContext(solution, flatGradient);
    Step(ctx);   // one impl per optimizer
    return solution;
  }
  return legacy UpdateParameters path  // non-NN models (regression etc.)

SynthesizeTapeStepContext helper builds a first-order TapeStepContext
from the model's live parameter chunks + slices the flat gradient
to match each chunk's shape. mutations to chunk tensors via Step
update the model in-place (zero-copy tape semantics).

all 19 optimizer subclasses (AdaDelta, Adagrad, Adam, Adam8Bit, AdaMax,
AdamW, AMSGrad, FTRL, GradientDescent, LAMB, LARS, Lion, MiniBatch,
Momentum, Nadam, NesterovAcceleratedGradient, NewtonMethod,
ProximalGradientDescent, RootMeanSquarePropagation, StochasticGradient
Descent) now have a `if (solution is INeuralNetwork<T>) return
base.UpdateSolution(...)` guard at the top of their UpdateSolution
override. NN solutions route through base → Step. non-NN solutions
keep the legacy flat-vector path for backward compat.

verified on HE Test 5 (split data 800/171/173 — the original #1380
reproducer):
  shipped 0.206.0:                NaN
  + part 1 (eval training mode):  5.7e-6 (collapse)
  + part 2 (safeguards):          81926 (finite, growing)
  + this PR (consolidation):      36.87 (matches per-sample bypass 42.5 within 14%)

industry standard: ✓ matches PyTorch/TF/JAX single-update-impl pattern
exceeds industry: parity contract test follow-up (no other ML lib has
the structural guarantee).

backward compat: subclasses retain their UpdateSolution overrides for
non-NN solutions, so callers using these optimizers on regression /
clustering / classical models see zero behavior change.

co-authored-by: claude opus 4.7 (1m context) <noreply@anthropic.com>

* test(#1413): UpdateSolution-on-Transformer consolidation contract — 7 optimizers

regression bar locking the #1413 consolidation in place. for each
named optimizer (Adam, AdamW, AdaMax, AMSGrad, SGD, Momentum, Nadam),
constructs a tiny Transformer, builds a deterministic flat gradient,
calls optimizer.UpdateSolution(model, flatGrad) via reflection, and
asserts the resulting parameter L2 norm is:
  - not NaN (consolidation's anomaly guard must engage)
  - not infinity (clipping must engage)
  - within 10× of init L2 (no runaway divergence)

any optimizer subclass that shortcircuits the base.UpdateSolution path
(bypassing the synthesize-and-delegate-to-Step machinery) would
produce divergent/NaN params and fail this test.

co-authored-by: claude opus 4.7 (1m context) <noreply@anthropic.com>

* fix(pr#1412): address 2 unresolved review comments

OptimizerBase.cs (EvaluateSolution + PrepareAndEvaluateSolution):
Capture a single NeuralNetworkBase<T> reference for BOTH the read and
write training-mode paths so IsTrainingMode and SetTrainingMode always
target identical runtime types. Today NeuralNetworkBase<T> is the only
concrete INeuralNetwork<T> implementer, so the prior split cast
(INeuralNetwork<T> for write, NeuralNetworkBase<T> for read) hit the
same instance -- but the future-proof form keeps the read/write paths
syntactically aligned regardless of what other implementers ship. Also
swap conditional "if wasTraining then SetTrainingMode true" for the
unconditional "SetTrainingMode wasTraining" so the restore always
mirrors the captured pre-call mode.

AdamOptimizer.cs comment-stale finding: auto-resolved. The reviewer
flagged "Defaults to false -- opt-in for backward compatibility" on
the ApplyGradientClipping call site, but that whole code block was
already removed in c84ea91 (the #1413 architectural consolidation
that delegates all NN solutions through Step / TapeStepContext). No
code changes required.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(pr#1412): address 11 CodeRabbit comments on #1413 consolidation

Tape-state reset (4 optimizers): AdaDelta, AdaMax, Adam, AMSGrad now
clear their NN tape-side accumulators (_tapeM/_tapeV/_tapeVHat/etc.)
and reset their _tapeStep bias-correction counter inside Optimize().
The flat-vector path got fresh Vectors per Optimize call; the tape
path used parameter-tensor-keyed ConcurrentDictionaries that PERSIST
across Optimize calls on the same optimizer instance. Without the
reset, reusing one optimizer for a second NN training would carry
prior moments (and AMSGrad's v_max running maximum) into the new run,
biasing every per-parameter step from iteration 1.

NN routing wired through Optimize() (3 optimizers): LAMB, LARS,
Nesterov, Momentum -- their Optimize loops called
UpdateSolutionWithLAMB / UpdateSolutionWithLARS / velocity-based
UpdateSolution directly, bypassing the #1413 NN-routing override.
Routed Optimize through UpdateSolution so the INeuralNetwork<T>
branch engages. For momentum-family optimizers (Momentum + NAG), the
base tape path expects RAW gradients (Step's SGD-with-momentum
kernel does its own per-parameter momentum bookkeeping); forwarding
the already-accumulated velocity would double-apply momentum. Routed
NN solutions to base.UpdateSolution(currentSolution, gradient) and
kept the legacy velocity-based path for non-NN solvers.

Newton-method exemption: NewtonMethodOptimizer must NOT route NN
through base.UpdateSolution. The base path's Step treats its second
argument as a gradient (params -= lr * gradient), but Newton's
direction is already -H^(-1) * gradient (sign-flipped + curvature-
scaled). Passing direction as gradient would flip the sign back,
lose curvature scaling, and silently degrade Newton to plain SGD on
the raw direction values. Both NN and non-NN now use the flat-vector
direction-based update -- that's the only path with correct Newton
semantics.

Partial-step safety in GradientBasedOptimizerBase.SynthesizeTapeStepContext:
return null on a short flat gradient instead of break (which would
have produced a TapeStepContext that only covered a prefix of
parameters, causing Step to mutate a prefix and leave the rest
un-updated -- a worse outcome than the legacy fallback's all-or-
nothing update). Also reject any leftover bytes in the flat gradient
(chunks out of sync with what produced the gradient).

Test assertion strengthened (Issue1413_UpdateSolutionConsolidationTests):
snapshot pre-step parameters and assert at least one parameter
changed. The prior finite-L2 / ΔL2 bounds passed trivially for a
silent-no-op UpdateSolution, defeating the entire consolidation-
regression guard.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(tests): two CI regressions on #1412 after #1413 consolidation

AdamOptimizerAnomalyGuardTests:
The #1413 consolidation added a SECOND AnyGradientIsAnomalous overload
on AdamOptimizer -- the original `(TapeStepContext<T>)` variant plus a
new flat-vector `(Vector<T>)` variant. The test's reflection lookup
was `GetMethod("AnyGradientIsAnomalous", BindingFlags...)` with no
parameter signature, which throws AmbiguousMatchException when more
than one overload shares a name. Disambiguate by passing the exact
parameter type tuple so the test always targets the tape-context
overload it was written for.

MultiVectorRetrieverTests:
Port the StubQueryEmbedder fix from fix/ci-failures-systematic. The
#1408 work made IQueryEmbedder<T> a mandatory dependency at retrieval
time (the source file throws NotSupportedException when the optional
ctor parameter is null), but this branch's tests still used the
zero-embedder ctor signature. Replace the entire file with the
fixed version: StubQueryEmbedder class + 30+ updated ctor call sites
that pass `queryEmbedder: new StubQueryEmbedder()`.

All 7 AnomalyGuard tests + all 43 MVR tests now pass locally.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: claude opus 4.7 (1m context) <noreply@anthropic.com>
Co-authored-by: franklinic <franklin@ivorycloud.com>

Author: 3 people

src/Models/Options/GradientBasedOptimizerOptions.cs

@@ -193,7 +193,17 @@ internal void SetLossFunctionFromAutoSync(ILossFunction<T> lossFunction)
     /// putting a speed limit on these updates to keep training stable.
     /// </para>
     /// </remarks>
-    public bool EnableGradientClipping { get; set; } = false;
+    // #1380 part 2: default ON. PyTorch's transformer training default is
+    // torch.nn.utils.clip_grad_norm_(params, max_norm=1.0) and this matches
+    // the canonical industry recipe. The previous default of `false`
+    // produced runaway parameter growth on the Optimize / BuildAsync code
+    // path when paired with the default DataSplitter.Split 70/15/15 small-
+    // validation sets, since the small-batch evaluation forward passes
+    // generate large gradient spikes that need clipping to stay stable.
+    // Callers who want the previous behavior can still opt out by setting
+    // this to false explicitly. See AiDotNet#1413 for the bisection
+    // evidence and AiDotNet#1380 for the headline issue.
+    public bool EnableGradientClipping { get; set; } = true;
 
     /// <summary>
     /// Gets or sets the gradient clipping method to use.

src/Optimizers/AMSGradOptimizer.cs

@@ -116,6 +116,20 @@ public override OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInpu
         _m = new Vector<T>(parameters.Length);
         _v = new Vector<T>(parameters.Length);
         _vHat = new Vector<T>(parameters.Length);
+        // Reset the NN tape-side state. The flat-vector path got fresh
+        // Vectors above; the tape path uses parameter-tensor-keyed
+        // dictionaries (_tapeM/_tapeV/_tapeVHat) plus a separate
+        // _tapeStep counter that PERSIST across Optimize calls on the
+        // same optimizer instance. Without this clear, a second Optimize
+        // call on the same optimizer would carry the prior run's first/
+        // second moments AND its v̂_max running maximum (the AMSGrad
+        // bound that defines this optimizer), plus a pre-advanced bias-
+        // correction counter — biasing every per-parameter step from
+        // iteration 1.
+        _tapeM.Clear();
+        _tapeV.Clear();
+        _tapeVHat.Clear();
+        _tapeStep = 0;
         InitializeAdaptiveParameters();
 
         for (int epoch = 0; epoch < _options.MaxIterations; epoch++)
@@ -171,6 +185,15 @@ public override OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInpu
     /// </remarks>
     protected override IFullModel<T, TInput, TOutput> UpdateSolution(IFullModel<T, TInput, TOutput> currentSolution, Vector<T> gradient)
     {
+        // #1413 CONSOLIDATION: NN solutions go through base.UpdateSolution
+        // which synthesizes a TapeStepContext and delegates to Step
+        // (one source of truth, matches PyTorch/TF/JAX). Non-NN solutions
+        // (regression, clustering, classical models) keep the legacy
+        // flat-vector path below for backward compatibility.
+        if (currentSolution is AiDotNet.Interfaces.INeuralNetwork<T>)
+        {
+            return base.UpdateSolution(currentSolution, gradient);
+        }
         var parameters = InterfaceGuard.Parameterizable(currentSolution).GetParameters();
 
         // Use shared UpdateParameters method to eliminate duplication

src/Optimizers/AdaDeltaOptimizer.cs

@@ -194,6 +194,16 @@ public override OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInpu
 
         _accumulatedSquaredGradients = new Vector<T>(parameters.Length);
         _accumulatedSquaredUpdates = new Vector<T>(parameters.Length);
+        // Reset the NN tape-side accumulators too. The flat-vector path
+        // gets a fresh Vector<T> per Optimize call (lines above); the
+        // tape-tracked path uses parameter-tensor-keyed dictionaries
+        // (_tapeAccSqGrad / _tapeAccSqUpd) that PERSIST across Optimize
+        // calls on the same optimizer instance. Without this clear, a
+        // second Optimize call on the same optimizer would carry the
+        // prior run's AdaDelta history into the new model, biasing
+        // every per-parameter step size from iteration 1.
+        _tapeAccSqGrad.Clear();
+        _tapeAccSqUpd.Clear();
         InitializeAdaptiveParameters();
 
         for (int epoch = 0; epoch < _options.MaxIterations; epoch++)
@@ -260,6 +270,15 @@ public override OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInpu
     /// </remarks>
     protected override IFullModel<T, TInput, TOutput> UpdateSolution(IFullModel<T, TInput, TOutput> currentSolution, Vector<T> gradient)
     {
+        // #1413 CONSOLIDATION: NN solutions go through base.UpdateSolution
+        // which synthesizes a TapeStepContext and delegates to Step
+        // (one source of truth, matches PyTorch/TF/JAX). Non-NN solutions
+        // (regression, clustering, classical models) keep the legacy
+        // flat-vector path below for backward compatibility.
+        if (currentSolution is AiDotNet.Interfaces.INeuralNetwork<T>)
+        {
+            return base.UpdateSolution(currentSolution, gradient);
+        }
         var parameters = InterfaceGuard.Parameterizable(currentSolution).GetParameters();
 
         // Initialize state vectors if needed

src/Optimizers/AdaMaxOptimizer.cs

@@ -202,6 +202,17 @@ public override OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInpu
 
         _m = new Vector<T>(parameters.Length);
         _u = new Vector<T>(parameters.Length);
+        // Reset the NN tape-side accumulators + bias-correction step count.
+        // The flat-vector path gets fresh Vectors above; the tape path
+        // uses parameter-tensor-keyed dictionaries (_tapeM/_tapeU) and a
+        // separate _tapeStep counter that PERSIST across Optimize calls
+        // on the same optimizer instance. Without this clear, a second
+        // Optimize call on the same optimizer would carry the prior run's
+        // first/inf moments AND a pre-advanced bias-correction counter,
+        // biasing every per-parameter step from iteration 1.
+        _tapeM.Clear();
+        _tapeU.Clear();
+        _tapeStep = 0;
         InitializeAdaptiveParameters();
 
         for (int epoch = 0; epoch < _options.MaxIterations; epoch++)
@@ -269,6 +280,15 @@ public override OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInpu
     /// </remarks>
     protected override IFullModel<T, TInput, TOutput> UpdateSolution(IFullModel<T, TInput, TOutput> currentSolution, Vector<T> gradient)
     {
+        // #1413 CONSOLIDATION: NN solutions go through base.UpdateSolution
+        // which synthesizes a TapeStepContext and delegates to Step
+        // (one source of truth, matches PyTorch/TF/JAX). Non-NN solutions
+        // (regression, clustering, classical models) keep the legacy
+        // flat-vector path below for backward compatibility.
+        if (currentSolution is AiDotNet.Interfaces.INeuralNetwork<T>)
+        {
+            return base.UpdateSolution(currentSolution, gradient);
+        }
         var parameters = InterfaceGuard.Parameterizable(currentSolution).GetParameters();
 
         // Initialize state vectors if needed

src/Optimizers/AdagradOptimizer.cs

@@ -250,6 +250,15 @@ private void UpdateAccumulatedSquaredGradients(Vector<T> gradient)
     /// </remarks>
     protected override IFullModel<T, TInput, TOutput> UpdateSolution(IFullModel<T, TInput, TOutput> currentSolution, Vector<T> gradient)
     {
+        // #1413 CONSOLIDATION: NN solutions go through base.UpdateSolution
+        // which synthesizes a TapeStepContext and delegates to Step
+        // (one source of truth, matches PyTorch/TF/JAX). Non-NN solutions
+        // (regression, clustering, classical models) keep the legacy
+        // flat-vector path below for backward compatibility.
+        if (currentSolution is AiDotNet.Interfaces.INeuralNetwork<T>)
+        {
+            return base.UpdateSolution(currentSolution, gradient);
+        }
         var parameters = InterfaceGuard.Parameterizable(currentSolution).GetParameters();
 
         // === Vectorized Adagrad Update using IEngine (Phase B: US-GPU-015) ===

src/Optimizers/Adam8BitOptimizer.cs

@@ -758,6 +758,15 @@ protected override void UpdateAdaptiveParameters(OptimizationStepData<T, TInput,
     /// </summary>
     protected override IFullModel<T, TInput, TOutput> UpdateSolution(IFullModel<T, TInput, TOutput> currentSolution, Vector<T> gradient)
     {
+        // #1413 CONSOLIDATION: NN solutions go through base.UpdateSolution
+        // which synthesizes a TapeStepContext and delegates to Step
+        // (one source of truth, matches PyTorch/TF/JAX). Non-NN solutions
+        // (regression, clustering, classical models) keep the legacy
+        // flat-vector path below for backward compatibility.
+        if (currentSolution is AiDotNet.Interfaces.INeuralNetwork<T>)
+        {
+            return base.UpdateSolution(currentSolution, gradient);
+        }
         var parameters = InterfaceGuard.Parameterizable(currentSolution).GetParameters();
 
         if (_mQuantized == null && _mFullPrecision == null)

src/Optimizers/AdamOptimizer.cs

@@ -157,6 +157,19 @@ public override OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInpu
         // previous run's running maximum as a lower bound and suppress
         // early updates in the new run. (PR #1350 round-2 review.)
         _vMaxVector = null;
+        // Reset the NN tape-side state. The flat-vector path got reset
+        // above; the tape path uses parameter-tensor-keyed dictionaries
+        // (_tapeM, _tapeV, _tapeVMax) and a separate _tapeStep counter
+        // that PERSIST across Optimize calls on the same optimizer
+        // instance. Without this clear, a second Optimize call on the
+        // same optimizer would carry the prior run's first/second moments
+        // (and AMSGrad's running maximum) plus a pre-advanced bias-
+        // correction counter, biasing every per-parameter step from
+        // iteration 1.
+        _tapeM.Clear();
+        _tapeV.Clear();
+        _tapeVMax.Clear();
+        _tapeStep = 0;
 
         // Initialize parameters
         InitializeAdaptiveParameters();
@@ -314,7 +327,15 @@ protected override void UpdateAdaptiveParameters(OptimizationStepData<T, TInput,
     }
 
     /// <summary>
-    /// Updates the current solution using the Adam update rule.
+    /// Updates the current solution using the Adam update rule. Kept for the
+    /// non-NN code path (regression, clustering, classical models where the
+    /// solution does NOT implement <see cref="AiDotNet.Interfaces.INeuralNetwork{T}"/>);
+    /// the base-class <see cref="GradientBasedOptimizerBase{T,TInput,TOutput}.UpdateSolution"/>
+    /// intercepts NN solutions and delegates to <see cref="Step(TapeStepContext{T})"/>
+    /// via <see cref="GradientBasedOptimizerBase{T,TInput,TOutput}.SynthesizeTapeStepContext"/>,
+    /// so the legacy flat-vector path here only runs for non-NN models — eliminating
+    /// the historical two-Adam-implementations split (#1413). All NN training
+    /// goes through Step, which has the anomaly guard + gradient clipping safeguards.
     /// </summary>
     /// <param name="currentSolution">The current solution being optimized.</param>
     /// <param name="gradient">The calculated gradient for the current solution.</param>
@@ -324,75 +345,16 @@ protected override void UpdateAdaptiveParameters(OptimizationStepData<T, TInput,
     /// It uses the current gradient and past information to decide how to change each parameter.
     /// </para>
     /// </remarks>
-    protected override IFullModel<T, TInput, TOutput> UpdateSolution(IFullModel<T, TInput, TOutput> currentSolution, Vector<T> gradient)
-    {
-        var parameters = InterfaceGuard.Parameterizable(currentSolution).GetParameters();
-
-        // Right-size _m/_v to gradient on first call or after lazy-layer expansion.
-        if (_m.Length != gradient.Length)
-        {
-            var newM = new Vector<T>(gradient.Length);
-            var newV = new Vector<T>(gradient.Length);
-            int copyLen = Math.Min(_m.Length, gradient.Length);
-            for (int i = 0; i < copyLen; i++) { newM[i] = _m[i]; newV[i] = _v[i]; }
-            _m = newM;
-            _v = newV;
-        }
-
-        // === Vectorized Adam Update using IEngine ===
-        // Phase B: US-GPU-015 - GPU-accelerated gradient updates
-
-        T oneMinusBeta1 = NumOps.Subtract(NumOps.One, _currentBeta1);
-        T oneMinusBeta2 = NumOps.Subtract(NumOps.One, _currentBeta2);
-        T biasCorrection1 = NumOps.Subtract(NumOps.One, NumOps.Power(_currentBeta1, NumOps.FromDouble(_t)));
-        T biasCorrection2 = NumOps.Subtract(NumOps.One, NumOps.Power(_currentBeta2, NumOps.FromDouble(_t)));
-        T epsilon = NumOps.FromDouble(_options.Epsilon);
-
-        // Update biased first moment: m = beta1 * m + (1 - beta1) * gradient
-        var mScaled = (Vector<T>)Engine.Multiply(_m, _currentBeta1);
-        var gradScaled = (Vector<T>)Engine.Multiply(gradient, oneMinusBeta1);
-        _m = (Vector<T>)Engine.Add(mScaled, gradScaled);
-
-        // Update biased second moment: v = beta2 * v + (1 - beta2) * gradient^2
-        var gradSquared = (Vector<T>)Engine.Multiply(gradient, gradient);
-        var vScaled = (Vector<T>)Engine.Multiply(_v, _currentBeta2);
-        var gradSquaredScaled = (Vector<T>)Engine.Multiply(gradSquared, oneMinusBeta2);
-        _v = (Vector<T>)Engine.Add(vScaled, gradSquaredScaled);
-
-        // Compute bias-corrected first moment: mHat = m / (1 - beta1^t)
-        var mHat = (Vector<T>)Engine.Divide(_m, biasCorrection1);
-
-        // Compute bias-corrected second moment: vHat = v / (1 - beta2^t)
-        var vHat = (Vector<T>)Engine.Divide(_v, biasCorrection2);
-
-        // AMSGrad: when enabled, divide by sqrt(running max of v̂) instead
-        // of sqrt(v̂) — the same correction the vector UpdateParameters
-        // path applies. Without this branch the UpdateSolution path
-        // silently ran plain Adam even with UseAMSGrad=true, defeating
-        // the purpose of the AMSGrad option on the BuildAsync/Optimize
-        // call path. PR #1350 review.
-        var vHatForDenominator = vHat;
-        if (_options.UseAMSGrad)
-        {
-            if (_vMaxVector is null || _vMaxVector.Length != vHat.Length)
-                _vMaxVector = new Vector<T>(vHat.Length);
-            _vMaxVector = (Vector<T>)Engine.Max(_vMaxVector, vHat);
-            vHatForDenominator = _vMaxVector;
-        }
-
-        // Compute update: update = learningRate * mHat / (sqrt(vHat_used) + epsilon)
-        var vHatSqrt = (Vector<T>)Engine.Sqrt(vHatForDenominator);
-        // Create epsilon vector for addition
-        var epsilonVec = Vector<T>.CreateDefault(vHatSqrt.Length, epsilon);
-        var denominator = (Vector<T>)Engine.Add(vHatSqrt, epsilonVec);
-        var updateDiv = (Vector<T>)Engine.Divide(mHat, denominator);
-        var update = (Vector<T>)Engine.Multiply(updateDiv, CurrentLearningRate);
-
-        // Apply update: parameters = parameters - update
-        var updatedParams = (Vector<T>)Engine.Subtract(parameters, update);
-
-        return InterfaceGuard.Parameterizable(currentSolution).WithParameters(updatedParams);
-    }
+    // #1413 ARCHITECTURAL CONSOLIDATION: AdamOptimizer's flat-vector
+    // UpdateSolution override is REMOVED. NN solutions go through the base
+    // class's UpdateSolution which synthesizes a TapeStepContext from the
+    // flat gradient and delegates to Step(TapeStepContext) — the SAME code
+    // path the per-sample nn.Train bypass uses, with the SAME anomaly
+    // guard, gradient clipping, AMSGrad, and float-loop fast path. Non-NN
+    // solutions fall through to the base's UpdateParameters dispatch which
+    // resolves to AdamOptimizer.UpdateParameters (still present below).
+    // This is the elimination of the two-Adam-implementations split that
+    // caused #1380.
 
     /// <summary>
     /// Updates a vector of parameters using the Adam optimization algorithm.
@@ -1450,6 +1412,22 @@ private bool AnyGradientIsAnomalous(TapeStepContext<T> context)
         return false;
     }
 
+    /// <summary>
+    /// Flat-vector overload of <see cref="AnyGradientIsAnomalous(TapeStepContext{T})"/>
+    /// for the Optimize / UpdateSolution path (#1380 part 2). Iterates the
+    /// gradient Vector directly since UpdateSolution doesn't have a
+    /// TapeStepContext to walk.
+    /// </summary>
+    private bool AnyGradientIsAnomalous(Vector<T> gradient)
+    {
+        for (int i = 0; i < gradient.Length; i++)
+        {
+            double v = NumOps.ToDouble(gradient[i]);
+            if (double.IsNaN(v) || double.IsInfinity(v)) return true;
+        }
+        return false;
+    }
+
     private static void ApplyGlobalNormGradientClipping(
         TapeStepContext<T> context,
         double maxNorm)

src/Optimizers/AdamWOptimizer.cs

@@ -255,6 +255,15 @@ protected override void UpdateAdaptiveParameters(OptimizationStepData<T, TInput,
     /// <returns>A new solution with updated parameters.</returns>
     protected override IFullModel<T, TInput, TOutput> UpdateSolution(IFullModel<T, TInput, TOutput> currentSolution, Vector<T> gradient)
     {
+        // #1413 CONSOLIDATION: NN solutions go through base.UpdateSolution
+        // which synthesizes a TapeStepContext and delegates to Step
+        // (one source of truth, matches PyTorch/TF/JAX). Non-NN solutions
+        // (regression, clustering, classical models) keep the legacy
+        // flat-vector path below for backward compatibility.
+        if (currentSolution is AiDotNet.Interfaces.INeuralNetwork<T>)
+        {
+            return base.UpdateSolution(currentSolution, gradient);
+        }
         var parameters = InterfaceGuard.Parameterizable(currentSolution).GetParameters();
 
         // Right-size _m/_v/_vMax to gradient on first call or after lazy-layer

src/Optimizers/FTRLOptimizer.cs

@@ -344,6 +344,15 @@ public override OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInpu
     /// <returns>The updated solution.</returns>
     protected override IFullModel<T, TInput, TOutput> UpdateSolution(IFullModel<T, TInput, TOutput> currentSolution, Vector<T> gradient)
     {
+        // #1413 CONSOLIDATION: NN solutions go through base.UpdateSolution
+        // which synthesizes a TapeStepContext and delegates to Step
+        // (one source of truth, matches PyTorch/TF/JAX). Non-NN solutions
+        // (regression, clustering, classical models) keep the legacy
+        // flat-vector path below for backward compatibility.
+        if (currentSolution is AiDotNet.Interfaces.INeuralNetwork<T>)
+        {
+            return base.UpdateSolution(currentSolution, gradient);
+        }
         // === Partially Vectorized FTRL Update using IEngine (Phase B: US-GPU-015) ===
         // FTRL uses L1 thresholding which requires conditional logic per-element
         // Vectorized: gradient operations, sigma calculation, state updates