Skip to content

Fix downgrade CI: split the AD test group out of the downgrade lane#59

Merged
ChrisRackauckas merged 3 commits into
SciML:mainfrom
ChrisRackauckas-Claude:fix-downgrade-compat
Jun 21, 2026
Merged

Fix downgrade CI: split the AD test group out of the downgrade lane#59
ChrisRackauckas merged 3 commits into
SciML:mainfrom
ChrisRackauckas-Claude:fix-downgrade-compat

Conversation

@ChrisRackauckas-Claude

@ChrisRackauckas-Claude ChrisRackauckas-Claude commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Problem

The Downgrade (Core) job was red. The root cause is that the AD test extras (Enzyme, EnzymeCore, Mooncake) were part of the root test target ([targets].test), so the downgrade resolver had to place them in the minimum-version test env.

julia-downgrade-compat (Resolver.jl) minimizes Julia to the floor of its --julia range; on lts that admits 1.10.0. But every Mooncake 0.5.x requires julia >= 1.10.8, and the Enzyme 0.13 + EnzymeCore 0.8 + Mooncake 0.5 minimum-version set has no solution on the 1.10 LTS stdlib line, so the merged test env is Unsatisfiable. This is the Resolver.jl <-> Mooncake wall (StefanKarpinski/Resolver.jl#24). The AD deps should not gate the downgrade lane.

Fix: split the AD group out of downgrade (instead of pinning Julia)

  • Move the AD test deps into their own per-group sub-environments: test/Enzyme/Project.toml and test/Mooncake/Project.toml, each with [sources] FunctionWrappersWrappers = {path = "../.."}. Drop Enzyme/EnzymeCore/Mooncake from the root [extras]/[targets].test. (They stay as [weakdeps] + [compat] for the extensions.)
  • test/runtests.jl: the Enzyme/Mooncake groups now use SciMLTesting's per-group env = ... form (the same mechanism the nopre group already uses), so each runs in its own env.
  • test/test_groups.toml: add Enzyme and Mooncake groups (["lts", "1"]). They were declared in runtests.jl but had no matrix entry, so they were not running in CI at all — now they do.
  • The Downgrade job runs GROUP=Core, which resolves only the root test target — now AD-free — so it is satisfiable on lts (Julia 1.10) again.

Reverted #59 accommodations (they masked the AD-resolution problem)

  • .github/workflows/Downgrade.yml: julia-version back to "lts" (no longer pinned to 1.11).
  • PrecompileTools compat floor back to "1": the 1.0.0 GPUCompiler precompile failure only arose because Enzyme pulled GPUCompiler into the downgrade env; with AD out of that env it no longer applies.

Kept — genuine minimum-version bugs, independent of AD (verified on Julia 1.10)

  • TruncatedStacktraces = "1.1": src uses @truncate_stacktrace, which does not exist before v1.1.0 (1.0.0 is a 25-line stub).
  • FunctionWrappers = "1.1.2": 1.0.0 / 1.1.0 / 1.1.1 fail to build the FunctionWrapper cfunction (Module IR does not contain specified entry function) on Julia 1.10; 1.1.2 is the lowest that builds and runs. This is exercised by the package's own PrecompileTools @compile_workload, so it is a real floor regardless of the AD split. (This vindicates the earlier FunctionWrappers floor raise — the floor is real; only the exact version was off, 1.1.2 vs 1.1.3.)

Local verification (Julia 1.10 / lts — the actual downgrade leg)

Ran the real julia-actions/julia-downgrade-compat@v2 downgrade.jl (mode deps, projects ., effective skip = Julia stdlibs):

  • Resolution: exit 0, Successfully resolved minimal versions for . (with extras).
  • Downgraded manifest contains no Enzyme / Mooncake / GPUCompiler. Floors: FunctionWrappers 1.1.2, PrecompileTools 1.0.0, TruncatedStacktraces 1.1.0, SafeTestsets 0.1.0, SciMLTesting 1.0.0.
  • Pkg.test GROUP=Core with allow_reresolve=false, force_latest_compatible_version=false (the julia-runtest CI settings): precompiles cleanly and passesFunctionWrappersWrappers.jl 48/48, BigFloat + UnionAll 7/7.

AD groups still run via their sub-envs (Julia 1.11): GROUP=Enzyme 65/65, GROUP=Mooncake 13/13, with FunctionWrappersWrappersEnzymeExt / FunctionWrappersWrappersMooncakeExt precompiling in-group. The grouped-tests matrix script now emits 9 jobs (Core lts/1/pre, Enzyme lts/1, Mooncake lts/1, nopre lts/1).

Please ignore until reviewed by @ChrisRackauckas

The Downgrade (Core) job was failing with an Unsatisfiable resolution on
Julia lts (1.10). When julia-downgrade-compat pins every dep to its lowest
allowed version, FunctionWrappers was pinned to 1.0.0 (floor of "1"), but
Mooncake (a tested weakdep/extension target) requires FunctionWrappers
>= 1.1.3. That made the merged test environment unsatisfiable.

Raising the FunctionWrappers compat floor from "1" to "1.1.3" matches the
version actually exercised by the test extras and makes the minimum-version
resolution satisfiable. Verified locally on Julia 1.10: the merged test
project now resolves at minimum versions, the package loads, and basictests
pass with FunctionWrappers 1.1.3.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@ChrisRackauckas-Claude ChrisRackauckas-Claude changed the title Raise FunctionWrappers compat floor to 1.1.3 (fix downgrade CI) Fix downgrade CI: run downgrade leg on Julia 1.11 (Mooncake 0.5 floor needs >=1.10.8) Jun 20, 2026
…ools/TruncatedStacktraces floors

The Downgrade (Core) job was red on lts (Julia 1.10). Three distinct, genuine
problems, fixed at their causes:

1. Unsatisfiable resolution. The resolver (julia-downgrade-compat -> Resolver.jl)
   minimizes Julia to the floor of its --julia range; on lts that range (1.10)
   admits 1.10.0. But every Mooncake 0.5.x test extra requires julia >= 1.10.8,
   so Mooncake can never be placed -> Unsatisfiable. Even at 1.10.8 the
   Enzyme 0.13 + EnzymeCore 0.8 + Mooncake 0.5 minimum-version set has no
   solution on the 1.10 LTS stdlib line; it resolves only once Julia reaches
   1.11.6 (Mooncake 0.5's 1.11 floor). Resolver.jl ignores the project's
   [compat] julia for the Julia version itself, so no Project.toml change can
   force julia >= 1.10.8 -- the caller must request it. Fix: pin the Downgrade
   caller to julia-version "1.11" (resolver picks 1.11.6). Core 1.10 support is
   still exercised by the regular Tests workflow.

2. PrecompileTools floor too low. With resolution fixed, the minimum
   PrecompileTools 1.0.0 fails to precompile GPUCompiler (pulled transitively by
   Enzyme) on Julia 1.11: "UndefVarError: PrecompileTools not defined in
   GPUCompiler". GPUCompiler's compat says PrecompileTools "1", but 1.0.0 does
   not actually provide the API GPUCompiler uses; 1.1.0 does. Raise the floor
   to "1.1".

3. TruncatedStacktraces floor too low. src uses
   TruncatedStacktraces.@truncate_stacktrace, which does not exist before v1.1.0
   ("UndefVarError: @truncate_stacktrace not defined in TruncatedStacktraces" at
   the v1.0.0 floor). Raise the floor to "1.1".

Also reverts the earlier misdiagnosed FunctionWrappers floor raise (1 -> 1.1.3):
no registered test dep constrains FunctionWrappers and the source only uses the
FunctionWrapper{R,A} constructor available since 1.0.0. Version bumped to 1.9.2
for the compat-floor changes.

Local verification on Julia 1.11 (the new downgrade leg): the actual
julia-downgrade-compat downgrade.jl resolves the merged test project (exit 0,
"Successfully resolved minimal versions"); the downgraded manifest (Enzyme
0.13.0 / GPUCompiler 0.27.5 / PrecompileTools 1.1.0 / TruncatedStacktraces 1.1.0
/ Mooncake 0.5.0) precompiles cleanly; and Pkg.test GROUP=Core passes
(FunctionWrappersWrappers.jl 48/48, BigFloat + UnionAll 7/7).

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@ChrisRackauckas-Claude ChrisRackauckas-Claude changed the title Fix downgrade CI: run downgrade leg on Julia 1.11 (Mooncake 0.5 floor needs >=1.10.8) Fix downgrade CI: run downgrade leg on Julia 1.11 + raise PrecompileTools/TruncatedStacktraces floors Jun 20, 2026
The Downgrade (Core) red was caused by the AD test extras (Enzyme,
EnzymeCore, Mooncake) being part of the root test target. The downgrade
resolver (julia-downgrade-compat -> Resolver.jl) minimizes Julia to the
floor of its --julia range, but every Mooncake 0.5.x requires
julia >= 1.10.8 and the Enzyme 0.13 + EnzymeCore 0.8 + Mooncake 0.5 floor
set has no minimum-version solution on the 1.10 LTS line, so the merged
test env was Unsatisfiable (the Resolver.jl <-> Mooncake wall,
StefanKarpinski/Resolver.jl#24). The AD deps shouldn't gate the downgrade
lane.

Fix: move the AD test deps into their own per-group sub-environments
(test/Enzyme/Project.toml, test/Mooncake/Project.toml) and drop them from
the root [extras]/[targets].test. SciMLTesting's run_tests already supports
per-group sub-envs (the `env = ...` group form, as the nopre group uses);
the Enzyme/Mooncake groups now declare their own env. The downgrade job
runs GROUP=Core, which resolves only the root test target -- now AD-free --
so it is satisfiable on lts (Julia 1.10) again. The AD groups are added to
test/test_groups.toml so they run as their own jobs in normal CI (lts, 1);
previously they were declared in runtests.jl but had no matrix entry and so
never ran.

Reverts the SciML#59 accommodations that masked the AD-resolution problem:
- .github/workflows/Downgrade.yml: julia-version back to "lts" (no longer
  needs to be pinned to 1.11 to dodge the AD floor set).
- PrecompileTools compat floor back to "1": the 1.0.0 GPUCompiler
  precompile failure only arose because Enzyme pulled GPUCompiler into the
  downgrade env; with AD out of that env it no longer applies.

Kept, because they are genuine minimum-version bugs independent of AD
(both verified locally on Julia 1.10):
- TruncatedStacktraces = "1.1": src uses @truncate_stacktrace, which does
  not exist before v1.1.0 (1.0.0 is a 25-line stub).
- FunctionWrappers = "1.1.2": 1.0.0 / 1.1.0 / 1.1.1 fail to build the
  FunctionWrapper cfunction ("Module IR does not contain specified entry
  function") on Julia 1.10; 1.1.2 is the lowest that builds and runs. This
  is exercised by the package's own PrecompileTools @compile_workload, so
  it is a real floor regardless of the AD split.

Local verification (Julia 1.10/lts, the actual downgrade leg):
- julia-downgrade-compat downgrade.jl: exit 0, "Successfully resolved
  minimal versions for . (with extras)"; downgraded manifest contains no
  Enzyme/Mooncake/GPUCompiler (FunctionWrappers 1.1.2, PrecompileTools
  1.0.0, TruncatedStacktraces 1.1.0, SafeTestsets 0.1.0, SciMLTesting
  1.0.0).
- Pkg.test GROUP=Core with allow_reresolve=false,
  force_latest_compatible_version=false (the julia-runtest CI settings):
  precompiles cleanly and passes (FunctionWrappersWrappers.jl 48/48,
  BigFloat + UnionAll 7/7).
- The AD groups still run via their sub-envs: GROUP=Enzyme 65/65,
  GROUP=Mooncake 13/13 (Julia 1.11), with the Enzyme/Mooncake extensions
  precompiling in-group.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@ChrisRackauckas-Claude ChrisRackauckas-Claude changed the title Fix downgrade CI: run downgrade leg on Julia 1.11 + raise PrecompileTools/TruncatedStacktraces floors Fix downgrade CI: split the AD test group out of the downgrade lane Jun 21, 2026
@ChrisRackauckas ChrisRackauckas marked this pull request as ready for review June 21, 2026 16:17
@ChrisRackauckas ChrisRackauckas merged commit 9c11a85 into SciML:main Jun 21, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants