Fix flaky Enzyme test_forward/test_reverse: StableRNG + accuracy-matched tolerance (#58)

The Enzyme `@easy_rule` returns the *exact* `^` derivative, but EnzymeTestUtils
`test_forward`/`test_reverse` compare it against finite differences of the
*approximate* `fastpower` primal. Because `fastpower` routes through a Float32
`fastlog2` polynomial, the *slope* of its primal differs from the exact slope by
~1e-2 relative near x=1 (measured: exact d/dx = 0.5 vs FD-of-fastpower = 0.5066,
i.e. 1.3e-2 relative), even at points like (1.0, 0.5) where the primal *value* is
exact. So the FD reference is off from the exact rule by `fastpower`'s inherent
approximation error, not by any rule bug. The old atol=1e-4, rtol=1e-3 sat below
that gap, so whether the lane passed depended on the random tangents drawn from
the global RNG and it went red intermittently (~4% of draws).

Two-part fix:

1. Determinism via StableRNG, not Xoshiro. Seeding the global RNG / `Xoshiro`
   does not actually pin the test, because those streams can change across Julia
   versions, so the flake could reappear on a new Julia. `StableRNGs.StableRNG`
   yields a stream guaranteed identical across Julia versions, passed as the
   `rng=` keyword that EnzymeTestUtils accepts.

2. Tolerance matched to fastpower's documented accuracy (atol=1e-3, rtol=1e-2),
   not reverted to the tight 1e-4/1e-3. Empirically the inherent gap is real: with
   the tight tolerance, 8/10 candidate StableRNG seeds pass the forward grid 52/52
   but seeds 123 and 31415 fail, and the all-seeds failure boundary is rtol~2e-3.
   Reverting to rtol=1e-3 would only "pass" by cherry-picking a lucky seed, which
   would hide the genuine (benign, expected) primal-approximation error. The chosen
   rtol=1e-2 sits ~5x above the measured worst-case relative discrepancy yet far
   below the O(1) relative error a genuinely wrong derivative rule would produce,
   so real regressions are still caught. Verified deterministic forward 52/52 +
   reverse 36/36 across 3 repeats on both Julia 1 and lts, and 52/52 for all 10
   candidate seeds on lts (so the tolerance is seed-independent, not seed-luck).

Swap the test dep Random -> StableRNGs in [extras]/[targets].test/[compat].

Co-authored-by: ChrisRackauckas-Claude <accounts@chrisrackauckas.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: 3 people

Project.toml

@@ -33,6 +33,7 @@ Mooncake = "0.4, 0.5"
 ReverseDiff = "1.14"
 SafeTestsets = "0.1, 1"
 SciMLTesting = "1"
+StableRNGs = "1"
 Test = "1"
 Tracker = "0.2"
 julia = "1.10"
@@ -45,8 +46,9 @@ Mooncake = "da2b9cff-9c12-43a0-ae48-6db2b0edb7d6"
 ReverseDiff = "37e2e3b7-166d-5795-8a7a-e32c996b4267"
 SafeTestsets = "1bc83da4-3b8d-516f-aca4-4fe02f6d838f"
 SciMLTesting = "09d9d899-5365-40a9-917a-5f67fddea283"
+StableRNGs = "860ef19b-820b-49d6-a774-d7a799459cd3"
 Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
 Tracker = "9f7883ad-71c0-57eb-9f7f-b5c9e6d3789c"
 
 [targets]
-test = ["Enzyme", "EnzymeTestUtils", "ForwardDiff", "Mooncake", "ReverseDiff", "SafeTestsets", "SciMLTesting", "Test", "Tracker"]
+test = ["Enzyme", "EnzymeTestUtils", "ForwardDiff", "Mooncake", "ReverseDiff", "SafeTestsets", "SciMLTesting", "StableRNGs", "Test", "Tracker"]

test/Enzyme/enzyme_forward_tests.jl

@@ -1,11 +1,28 @@
 using FastPower: fastpower
 using Enzyme, EnzymeTestUtils
+using StableRNGs
 using Test
 
+# `test_forward` compares the rule (which returns the *exact* `^` derivative) against finite
+# differences of the *approximate* `fastpower` primal. Because `fastpower` routes through a
+# Float32 `fastlog2` polynomial, the *slope* of its primal differs from the exact slope by
+# ~1e-2 relative near x=1 (even where the primal value itself is exact), so the FD reference
+# is off from the exact rule by that inherent approximation error rather than by any rule bug.
+# The previous atol=1e-4, rtol=1e-3 sat below that gap, so whether the lane passed depended on
+# the random tangents `test_forward` drew from the global RNG (it went red ~4% of the time).
+#
+# Fix: draw the tangents from a StableRNG (a fixed seed gives the *same* stream on every Julia
+# version, unlike the global RNG / Xoshiro whose stream can change across versions) so the test
+# is genuinely deterministic, and use atol=1e-3, rtol=1e-2 matched to `fastpower`'s documented
+# accuracy envelope (see test/fast_pow_tests.jl). That tolerance is ~5x above the measured
+# worst-case relative discrepancy in this grid (~2e-3) yet far below the O(1) relative error a
+# genuinely wrong derivative rule would produce, so real regressions are still caught. Reverting
+# to rtol=1e-3 is not possible without cherry-picking a lucky seed to hide the inherent gap.
+rng = StableRNG(123)
 @testset for RT in (Duplicated, DuplicatedNoNeed),
         Tx in (Const, Duplicated),
         Ty in (Const, Duplicated)
     x = 1.0
     y = 0.5
-    test_forward(fastpower, RT, (x, Tx), (y, Ty), atol = 1.0e-4, rtol = 1.0e-3)
+    test_forward(fastpower, RT, (x, Tx), (y, Ty), rng = rng, atol = 1.0e-3, rtol = 1.0e-2)
 end

test/Enzyme/enzyme_reverse_tests.jl

@@ -1,9 +1,15 @@
 using FastPower: fastpower
 using Enzyme, EnzymeTestUtils
+using StableRNGs
 using Test
 
+# See test/Enzyme/enzyme_forward_tests.jl: the finite-difference reference is taken on the
+# approximate `fastpower` primal, so the tolerance must cover its inherent approximation error.
+# Draw the cotangents from a StableRNG (stream is identical across Julia versions, so the test
+# is deterministic) and match `fastpower`'s documented accuracy with atol=1e-3, rtol=1e-2.
+rng = StableRNG(123)
 @testset for RT in (Active,), Tx in (Active, Const), Ty in (Active, Const)
     x = 1.0
     y = 0.5
-    test_reverse(fastpower, RT, (x, Tx), (y, Ty), atol = 1.0e-4, rtol = 1.0e-3)
+    test_reverse(fastpower, RT, (x, Tx), (y, Ty), rng = rng, atol = 1.0e-3, rtol = 1.0e-2)
 end