fix(#1395): surface caught exception in CompiledTapeTrainingStep fallback (#1402)

When CompiledTapeTrainingStep.TryStepWithFusedOptimizer caught an
exception (anything thrown by plan.Step or ConfigureOptimizer), it
logged to Trace and returned false. The caller (NeuralNetworkBase) saw
ran=false plus _fusedTrainingCommitted=true and threw a generic
InvalidOperationException listing 'common causes' but NOT the actual
exception text. Tests failing on this path got an opaque error;
debugging required reproducing the failure locally just to see the
Trace output.

## Fix

CompiledTapeTrainingStep now stashes the caught exception in a
[ThreadStatic] field on the catch path, cleared on entry to each call.
NeuralNetworkBase's fused-committed throw at line 6240 reads it back
via GetLastFallbackException() and quotes the type+message inline,
plus attaches the underlying exception as innerException so callers
that introspect ex.InnerException get the full stack.

Three concrete exception types observed under parallel test load
(traced during AiDotNet#1395 investigation) that now surface inline
instead of being hidden:

- InvalidOperationException 'Parameter N has a layout that does not
  expose a live CPU backing array' (ConfigureOptimizerDouble)
- ArgumentException 'gradOutput shape [...] must be [...]' (kernel
  backward shape mismatch)
- InvalidOperationException 'Lazy tensor produced NaN at index 0'
  (DifferentiableNeuralComputer numerical guard)

Paired with the AiDotNet.Tensors-side
fix(AiDotNet#1395): drop IsDeterministicMode from CompiledModelCache
shape-key, which removes the cross-test cache-key drift that caused
the throw in the first place.

Build clean on net10.0 + net471.

Co-authored-by: franklinic <franklin@ivorycloud.com>

Author: ooples

src/NeuralNetworks/NeuralNetworkBase.cs

@@ -6262,20 +6262,39 @@ private bool TryTrainWithFusedOptimizer(
             //     plan (strict single-plan policy refused to configure it)
             //   - mutated optimizer hyperparameters between steps
             //     (attached LR scheduler, changed betas, etc.)
+            //   - a kernel-level exception caught and swallowed
+            //     (CompiledTrainingPlan.Step / ConfigureOptimizer threw)
             // Resolution: call ResetState() or InvalidateParameterCountCache()
             // to fully reset training state, then retrain with stable
             // shapes + fixed hyperparameters. Or disable compilation via
             // AllowNondeterminism / Configure(JitCompilationConfig.Disabled)
             // so training runs entirely on the eager path from the start.
+            //
+            // AiDotNet#1395: quote the swallowed exception (if any) inline so
+            // failing tests don't have to chase Trace output to learn the
+            // root cause (Parameter N non-contiguous CPU layout, shape mismatch
+            // in a backward kernel, NaN guard trip, etc.). The inner exception
+            // is also attached for catch (InvalidOperationException ex) callers
+            // that introspect ex.InnerException.
+            var fallbackEx = Training.CompiledTapeTrainingStep<T>.GetLastFallbackException();
+            var rootCauseSuffix = fallbackEx is not null
+                ? $" Root-cause exception (caught in CompiledTapeTrainingStep): " +
+                  $"{fallbackEx.GetType().FullName}: {fallbackEx.Message}"
+                : " (No exception was caught — fused path returned false from one of the explicit " +
+                  "refuse paths: plan reference changed, optimizer hyperparameters drifted, " +
+                  "TensorCodecOptions.EnableCompilation=false, or numeric/optimizer type unsupported.)";
             throw new InvalidOperationException(
                 "Fused compiled training has already run successfully, but the current step cannot " +
                 "engage the fused path. The plan-embedded Adam/AdamW/SGD state cannot be transferred " +
                 "to the eager optimizer, so falling back silently would produce a trajectory that " +
                 "diverges from the previous fused steps. Common causes: variable input/target shape " +
-                "(new compiled plan), LR scheduler or adaptive-rate changes, attached AMSGrad. " +
+                "(new compiled plan), LR scheduler or adaptive-rate changes, attached AMSGrad, or a " +
+                "kernel-level exception in plan.Step/ConfigureOptimizer that was caught and swallowed. " +
                 "Resolution: keep shapes and optimizer hyperparameters stable across steps, OR call " +
                 "ResetState() / InvalidateParameterCountCache() to explicitly reset training state, " +
-                "OR disable compilation (AiModelBuilder.ConfigureJitCompilation(JitCompilationConfig.Disabled)).");
+                "OR disable compilation (AiModelBuilder.ConfigureJitCompilation(JitCompilationConfig.Disabled))." +
+                rootCauseSuffix,
+                innerException: fallbackEx);
         }
         else
         {

src/Training/CompiledTapeTrainingStep.cs

@@ -102,6 +102,30 @@ public static class CompiledTapeTrainingStep<T>
     /// <summary>Resets the fused-step counter on the calling thread to zero.</summary>
     public static void ResetFusedStepCount() { _fusedStepCount = 0; }
 
+    /// <summary>
+    /// AiDotNet#1395: when <see cref="TryStepWithFusedOptimizer"/> falls back via
+    /// the catch path, the underlying exception is stored here so the caller
+    /// (NeuralNetworkBase) can surface it in the "fused has committed but step N
+    /// can't engage" InvalidOperationException. Previously the catch's exception
+    /// was logged to Trace only — users debugging from a failing test never saw
+    /// the actual root cause (e.g. "Parameter N non-contiguous CPU layout" from
+    /// <see cref="Tensors.Engines.Compilation.CompiledTrainingPlan{T}.ConfigureOptimizer"/>,
+    /// a shape mismatch from a backward kernel, a NaN guard trip). Now the
+    /// caller can quote the original exception's type + message + stack so the
+    /// error is self-diagnosing.
+    /// </summary>
+    [ThreadStatic]
+    private static System.Exception? _lastFallbackException;
+
+    /// <summary>
+    /// AiDotNet#1395: read the last exception that caused
+    /// <see cref="TryStepWithFusedOptimizer"/> to fall back, or <c>null</c> if
+    /// the most recent fallback was due to one of the explicit return-false
+    /// paths (plan switch, config drift, EnableCompilation=false, etc.) rather
+    /// than a swallowed exception.
+    /// </summary>
+    public static System.Exception? GetLastFallbackException() => _lastFallbackException;
+
     // Reflection-cached lookup of ICompiledTrainingPlan<T>.SetMaxGradNorm(double).
     // Populated lazily on first call per process and reused on every subsequent
     // step. Returns null when the underlying Tensors assembly pre-dates the
@@ -282,6 +306,11 @@ internal static bool TryStepWithFusedOptimizer(
         AiDotNet.Tensors.Engines.Compilation.LrSchedule? lrSchedule = null)
     {
         lossValue = MathHelper.GetNumericOperations<T>().Zero;
+        // AiDotNet#1395: clear the previous-call's exception buffer so the
+        // caller's GetLastFallbackException reflects only the outcome of THIS
+        // call. (Cleared on entry, not on success, so a successful step doesn't
+        // leak a stale exception from earlier.)
+        _lastFallbackException = null;
 
         if (!TensorCodecOptions.Current.EnableCompilation) return false;
         // Fused optimizer kernels support float and double on the Tensors
@@ -505,6 +534,13 @@ or AiDotNet.Tensors.Engines.Compilation.OptimizerType.Adam
             // a fused-path regression from logs requires reproducing the
             // failure locally. Clear the single-slot config state so any
             // next attempt reconfigures fresh.
+            //
+            // AiDotNet#1395: also stash the exception so the caller's
+            // "fused has committed but step cannot engage" InvalidOperationException
+            // can quote the underlying cause (Parameter N non-contiguous CPU
+            // layout, shape mismatch, NaN guard, etc.). Trace alone wasn't
+            // enough — failing tests don't surface Trace output by default.
+            _lastFallbackException = ex;
             System.Diagnostics.Trace.TraceWarning(
                 $"CompiledTapeTrainingStep.TryStepWithFusedOptimizer failed, falling back to eager: " +
                 $"{ex}");