Add more test groups, and TEMPORARILY disable some tests on Julia nightly (the tests that are causing CI to time out)

[DilumAluthge]

Fixes #61

Once these tests are working on Julia nightly, we should reenable them.

I've opened #73 so that we don't forget to reenable these tests later.

---

Please note: we only skip these tests on CI. When you run the tests locally, these tests will not be skipped.

[DilumAluthge]

@mcabbott Looks like it's not just the KernelAbstractions tests that are timing out on Julia nightly. Group 3 is also timing out.

[mcabbott]

Yes, I saw that LoopVectorization tests in #71 timed out, on nightly, with lots of "WARNING: Wrapping Vararg directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping Vararg directly in UnionAll is deprecated (wrap the tuple instead)." I presume this is some interaction with JuliaLang/julia#38136? Cc @chriselrod

[DilumAluthge]

@mcabbott Any idea what is causing the test failure in https://github.com/mcabbott/Tullio.jl/pull/72/checks?check_run_id=1581043675 (Julia 1.4, one thread, test group 2)?

┌ Info: Beginning to run test file
│   i = 2
└   file_i = "/home/runner/work/Tullio.jl/Tullio.jl/test/group-2.jl"
KernelAbstractions + gradients: Test Failed at /home/runner/work/Tullio.jl/Tullio.jl/test/gradients.jl:79
  Expression: g3 ≈ (_gradient((x->(sum(#= /home/runner/work/Tullio.jl/Tullio.jl/test/gradients.jl:79 =# @tullio($(Expr(:(:=), :(y[ind3[i]]), :(i ^ 2 * x[i])))));)), ones(size(ind3))))[1]
   Evaluated: [1.0, 4.0, 9.0, 16.0, 25.0, 36.0, 0.0, 64.0, 81.0, 100.0, 121.0] ≈ [1.0, 4.0, 9.0, 16.0, 25.0, 36.0, 49.0, 64.0, 81.0, 100.0, 121.0] (tracked)
Stacktrace:
 [1] top-level scope at /home/runner/work/Tullio.jl/Tullio.jl/test/gradients.jl:79
 [2] include(::String) at ./client.jl:439
 [3] top-level scope at /home/runner/work/Tullio.jl/Tullio.jl/test/group-2.jl:18
 [4] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Test/src/Test.jl:1113
 [5] top-level scope at /home/runner/work/Tullio.jl/Tullio.jl/test/group-2.jl:12
Test Summary:                  | Pass  Fail  Broken  Total
KernelAbstractions + gradients |   77     1       1     79
  @inferred                    |    4                    4
  from TensorTrace             |   16                   16
  min/max                      |   18                   18
  mod, clamp, pad              |    3                    3
  finalisers                   |    4                    4
ERROR: LoadError: LoadError: Some tests did not pass: 77 passed, 1 failed, 0 errored, 1 broken.
in expression starting at /home/runner/work/Tullio.jl/Tullio.jl/test/group-2.jl:11
in expression starting at /home/runner/work/Tullio.jl/Tullio.jl/test/runtests.jl:35
ERROR: Package Tullio errored during testing
Stacktrace:
 [1] pkgerror(::String, ::Vararg{String,N} where N) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/Types.jl:53
 [2] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/Operations.jl:1510
 [3] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:316
 [4] #test#68 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:297 [inlined]
 [5] #test#65 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:294 [inlined]
 [6] top-level scope at none:1

[DilumAluthge]

Seems like, for some reason, the ForwardDiff gradient (g3) has a 0.0 where the Tracker gradient has a 49.0.

[DilumAluthge]

Also, all of the group 5 tests are failing (1.4 and 1.5, one thread and two threads). E.g. https://github.com/mcabbott/Tullio.jl/pull/72/checks?check_run_id=1581043810:

┌ Info: Beginning to run test file
│   i = 5
└   file_i = "/home/runner/work/Tullio.jl/Tullio.jl/test/group-5.jl"
Test Summary:                         | Pass  Broken  Total
gradients: Tracker + TensorOperations |   62       1     63
Test Summary:                        | Pass  Broken  Total
gradients: Zygote + TensorOperations |   64       2     66
analytic: Error During Test at /home/runner/work/Tullio.jl/Tullio.jl/test/group-5.jl:30
  Test threw exception
  Expression: g1 ≈ (_gradient((x->(real(sum(#= /home/runner/work/Tullio.jl/Tullio.jl/test/group-5.jl:30 =# @tullio($(Expr(:(:=), :(y[i, j]), :(x[i, k] * x[k, j]))))));)), x0))[1]
  MethodError: no method matching Strided.StridedView(::FillArrays.Fill{Int64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}})
  Closest candidates are:
    Strided.StridedView(!Matched::Array{S,N} where N) where S at /home/runner/.julia/packages/Strided/BFLNk/src/stridedview.jl:9
    Strided.StridedView(!Matched::Array{S,N} where N, !Matched::Tuple{Vararg{Int64,N}}) where {S, N} at /home/runner/.julia/packages/Strided/BFLNk/src/stridedview.jl:9
    Strided.StridedView(!Matched::Array{S,N} where N, !Matched::Tuple{Vararg{Int64,N}}, !Matched::Tuple{Vararg{Int64,N}}) where {S, N} at /home/runner/.julia/packages/Strided/BFLNk/src/stridedview.jl:9
    ...
  Stacktrace:
   [1] _native_contract!(::Bool, ::FillArrays.Fill{Int64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}, ::Symbol, ::Array{Complex{Int64},2}, ::Symbol, ::Bool, ::Array{Complex{Int64},2}, ::Tuple{Int64}, ::Tuple{Int64}, ::Tuple{Int64}, ::Tuple{Int64}, ::Tuple{Int64,Int64}, ::Tuple{Int64}, ::Tuple{Int64}, ::Tuple{Int64}, ::Tuple{Int64}) at /home/runner/.julia/packages/TensorOperations/Ydu0m/src/implementation/stridedarray.jl:426
   [2] contract!(::Bool, ::FillArrays.Fill{Int64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}, ::Symbol, ::Array{Complex{Int64},2}, ::Symbol, ::Bool, ::Array{Complex{Int64},2}, ::Tuple{Int64}, ::Tuple{Int64}, ::Tuple{Int64}, ::Tuple{Int64}, ::Tuple{Int64,Int64}, ::Tuple{Symbol,Symbol,Symbol}) at /home/runner/.julia/packages/TensorOperations/Ydu0m/src/implementation/stridedarray.jl:322
   [3] contract!(::Bool, ::FillArrays.Fill{Int64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}, ::Symbol, ::Array{Complex{Int64},2}, ::Symbol, ::Bool, ::Array{Complex{Int64},2}, ::Tuple{Int64}, ::Tuple{Int64}, ::Tuple{Int64}, ::Tuple{Int64}, ::Tuple{Int64,Int64}, ::Tuple{}, ::Tuple{Symbol,Symbol,Symbol}) at /home/runner/.julia/packages/TensorOperations/Ydu0m/src/implementation/stridedarray.jl:89
   [4] (::var"#∇ℳ𝒶𝓀ℯ#332")(::FillArrays.Fill{Int64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}, ::Array{Complex{Int64},2}, ::Array{Complex{Int64},2}) at /home/runner/work/Tullio.jl/Tullio.jl/src/tensor.jl:157
   [5] (::Tullio.var"#233#234"{Tullio.Eval{var"#ℳ𝒶𝓀ℯ#331",var"#∇ℳ𝒶𝓀ℯ#332"},Tuple{Array{Complex{Int64},2}},Array{Complex{Int64},2}})(::FillArrays.Fill{Int64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) at /home/runner/work/Tullio.jl/Tullio.jl/src/grad/zygote.jl:8
   [6] (::Tullio.var"#2462#back#235"{Tullio.var"#233#234"{Tullio.Eval{var"#ℳ𝒶𝓀ℯ#331",var"#∇ℳ𝒶𝓀ℯ#332"},Tuple{Array{Complex{Int64},2}},Array{Complex{Int64},2}}})(::FillArrays.Fill{Int64,2,Tuple{Base.OneTo{Int64},Base.OneTo{Int64}}}) at /home/runner/.julia/packages/ZygoteRules/OjfTt/src/adjoint.jl:59
   [7] #322 at /home/runner/work/Tullio.jl/Tullio.jl/src/tensor.jl:88 [inlined]
   [8] (::typeof(∂(#322)))(::Int64) at /home/runner/.julia/packages/Zygote/ggM8Z/src/compiler/interface2.jl:0
   [9] (::Zygote.var"#41#42"{typeof(∂(#322))})(::Int64) at /home/runner/.julia/packages/Zygote/ggM8Z/src/compiler/interface.jl:40
   [10] gradient(::Function, ::Array{Complex{Int64},2}) at /home/runner/.julia/packages/Zygote/ggM8Z/src/compiler/interface.jl:49
   [11] _gradient(::Function, ::Vararg{Any,N} where N) at /home/runner/work/Tullio.jl/Tullio.jl/test/group-5.jl:17
   [12] top-level scope at /home/runner/work/Tullio.jl/Tullio.jl/test/group-5.jl:30
   [13] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115
   [14] top-level scope at /home/runner/work/Tullio.jl/Tullio.jl/test/group-5.jl:28
   [15] top-level scope at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115
   [16] top-level scope at /home/runner/work/Tullio.jl/Tullio.jl/test/group-5.jl:24

[mcabbott]

The bug in "/test/gradients.jl:79" is that vcat(unique(rand(1:1024, 10)), 1) sometimes has 1 twice, I think. Finally fixed here:

https://github.com/mcabbott/Tullio.jl/pull/57/files#diff-aa2ea92ca835ecf86ced3e9f51716c9957d30921f425b2c7e711e8b03e2c0c43R100

[DilumAluthge]

The bug in "/test/gradients.jl:79" is that vcat(unique(rand(1:1024, 10)), 1) sometimes has 1 twice, I think. Finally fixed here:

https://github.com/mcabbott/Tullio.jl/pull/57/files#diff-aa2ea92ca835ecf86ced3e9f51716c9957d30921f425b2c7e711e8b03e2c0c43R100

Thanks! I have applied that fix to this PR.

[DilumAluthge]

Alright, so the only remaining test failures are in group 5 on Julia 1.4 and 1.5 (but not Julia nightly).

Any idea what the cause is?

[chriselrod]

Yes, I saw that LoopVectorization tests in #71 timed out, on nightly, with lots of "WARNING: Wrapping Vararg directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping Vararg directly in UnionAll is deprecated (wrap the tuple instead)." I presume this is some interaction with JuliaLang/julia#38136? Cc @chriselrod

Didn't see that PR. Thanks for the heads up.
My code made extensive use of Varargss and of .parameters.
I'll build the latest Julia master and try to fix all of it in a 1.5-compatible way.

[mcabbott]

The error with TensorOperations might be a real bug, brought to light by re-ordering the tests. (Zygote's gradient for sum makes a FillArray, which @tensor doesn't like, but ought to be caught by a spacial case here, but perhaps that fails...) Will have a look.

However I think I'd rather do something simpler to temporarily skip tests on nightly, rather than making more groups, like so:
ebf5c70

I see you're also adding a test on nightly to .buildkite/pipeline.yml. That's not a bad idea although perhaps it may as well wait until KernelAbstractions has some hope of running there?

[chriselrod]

Any idea how to find the source of those warnings?
When running LoopVectorization's tests:

     Testing Running tests...
WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).WARNING: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).Julia Version 1.7.0-DEV.122
Commit 676ccf4eaa (2020-12-18 19:53 UTC)

Would be helpful to have line numbers.
Loading Julia with --depwarn=yes or --depwarn=error and using LoopVectorization from the REPL don't show anything.

[mcabbott]

For me --depwarn=error gives the following when I run tests:

     Testing Running tests...
ERROR: LoadError: LoadError: Wrapping `Vararg` directly in UnionAll is deprecated (wrap the tuple instead).
Stacktrace:
  [1] UnionAll(v::TypeVar, t::Any)
    @ Core ./boot.jl:254
  [2] top-level scope
    @ ~/.julia/packages/LoopVectorization/N0Zry/src/map.jl:2
  [3] include(mod::Module, _path::String)
    @ Base ./Base.jl:386
  [4] include(x::String)
    @ LoopVectorization ~/.julia/packages/LoopVectorization/N0Zry/src/LoopVectorization.jl:1
  [5] top-level scope
    @ ~/.julia/packages/LoopVectorization/N0Zry/src/LoopVectorization.jl:57
  [6] include
    @ ./Base.jl:386 [inlined]
  [7] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String)
    @ Base ./loading.jl:1208
  [8] top-level scope
    @ none:1
  [9] eval
    @ ./boot.jl:369 [inlined]
 [10] eval(x::Expr)
    @ Base.MainInclude ./client.jl:446
 [11] top-level scope
    @ none:1
in expression starting at /Users/me/.julia/packages/LoopVectorization/N0Zry/src/map.jl:2
in expression starting at /Users/me/.julia/packages/LoopVectorization/N0Zry/src/LoopVectorization.jl:1

[chriselrod]

It turns out the warnings/errors only come up during precompilation.
I should have them fixed soon.
EDIT: Should be fixed on master. I'll probably tag a release in the next few hours.