std::bad_alloc which might related to OhMyThreads

[kockahonza]

Hi there. First of all thanks to all the developers for the wonderful package, it really strikes the perfect balance of features and simplicity for me!

I am now however dealing with a (to me) very cryptic error message that refers to a lot of very low level code and at the end also OhMyThreads and given how tricky threading can at times be I am wondering if that is causing it. I sadly do not have a MWE as this is all a part of a large codebase, I run it on a slurm cluster with 100 threads and I reliably get this always near the same time (note that the core code itself is very very random so I am finding this suspicious). This is the error

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

[84958] signal (6.-6): Aborted
in expression starting at none:1
gsignal at /lib64/libc.so.6 (unknown line)
abort at /lib64/libc.so.6 (unknown line)
unknown function (ip: 0x7f7d8ef909ba)
unknown function (ip: 0x7f7d8efa0369)
_ZSt9terminatev at /home/xucapjko/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/bin/../lib/julia/libstdc++.so.6 (unknown line)
__cxa_throw at /home/xucapjko/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/bin/../lib/julia/libstdc++.so.6 (unknown line)
unknown function (ip: 0x7f7d8ef905c5)
unknown function (ip: 0x7f7d897e98ee)
_ZN4llvm17PMTopLevelManager11setLastUserENS_8ArrayRefIPNS_4PassEEES3_ at /home/xucapjko/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/bin/../lib/julia/libLLVM-15jl.so (unknown line)
_ZN4llvm13PMDataManager3addEPNS_4PassEb at /home/xucapjko/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/bin/../lib/julia/libLLVM-15jl.so (unknown line)
_ZN4llvm17PMTopLevelManager12schedulePassEPNS_4PassE at /home/xucapjko/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/bin/../lib/julia/libLLVM-15jl.so (unknown line)
_ZN4llvm16TargetPassConfig7addPassEPNS_4PassE at /home/xucapjko/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/bin/../lib/julia/libLLVM-15jl.so (unknown line)
unknown function (ip: 0x7f7d8c01f38c)
_ZN4llvm16TargetPassConfig16addMachinePassesEv at /home/xucapjko/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/bin/../lib/julia/libLLVM-15jl.so (unknown line)
_ZN4llvm17LLVMTargetMachine17addPassesToEmitMCERNS_6legacy15PassManagerBaseERPNS_9MCContextERNS_17raw_pwrite_streamEb at /home/xucapjko/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/bin/../lib/julia/libLLVM-15jl.so (unknown line)
_ZN4llvm3orc14SimpleCompilerclERNS_6ModuleE at /home/xucapjko/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/bin/../lib/julia/libLLVM-15jl.so (unknown line)
unknown function (ip: 0x7f7d8dea5ace)
_ZN4llvm3orc14IRCompileLayer4emitESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EENS0_16ThreadSafeModuleE at /home/xucapjko/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/bin/../lib/julia/libLLVM-15jl.so (unknown line)
_ZN4llvm3orc16IRTransformLayer4emitESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EENS0_16ThreadSafeModuleE at /home/xucapjko/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/bin/../lib/julia/libLLVM-15jl.so (unknown line)
unknown function (ip: 0x7f7d8dea7dec)
_ZN4llvm3orc31BasicIRLayerMaterializationUnit11materializeESt10unique_ptrINS0_29MaterializationResponsibilityESt14default_deleteIS3_EE at /home/xucapjko/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/bin/../lib/julia/libLLVM-15jl.so (unknown line)
_ZN4llvm3orc19MaterializationTask3runEv at /home/xucapjko/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/bin/../lib/julia/libLLVM-15jl.so (unknown line)
unknown function (ip: 0x7f7d8b685d1c)
_ZN4llvm3orc16ExecutionSession22dispatchOutstandingMUsEv at /home/xucapjko/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/bin/../lib/julia/libLLVM-15jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession17OL_completeLookupESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EESt10shared_ptrINS0_23AsynchronousSymbolQueryEESt8functionIFvRKNS_8DenseMapIPNS0_8JITDylibENS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISF_vEEEENSG_ISD_vEENS_6detail12DenseMapPairISD_SI_EEEEEE at /home/xucapjko/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/bin/../lib/julia/libLLVM-15jl.so (unknown line)
unknown function (ip: 0x7f7d8b6a5a0c)
_ZN4llvm3orc16ExecutionSession19OL_applyQueryPhase1ESt10unique_ptrINS0_21InProgressLookupStateESt14default_deleteIS3_EENS_5ErrorE at /home/xucapjko/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/bin/../lib/julia/libLLVM-15jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupENS0_10LookupKindERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS8_EENS0_15SymbolLookupSetENS0_11SymbolStateENS_15unique_functionIFvNS_8ExpectedINS_8DenseMapINS0_15SymbolStringPtrENS_18JITEvaluatedSymbolENS_12DenseMapInfoISI_vEENS_6detail12DenseMapPairISI_SJ_EEEEEEEEESt8functionIFvRKNSH_IS6_NS_8DenseSetISI_SL_EENSK_IS6_vEENSN_IS6_SV_EEEEEE at /home/xucapjko/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/bin/../lib/julia/libLLVM-15jl.so (unknown line)
_ZN4llvm3orc16ExecutionSession6lookupERKSt6vectorISt4pairIPNS0_8JITDylibENS0_19JITDylibLookupFlagsEESaIS7_EENS0_15SymbolLookupSetENS0_10LookupKindENS0_11SymbolStateESt8functionIFvRKNS_8DenseMapIS5_NS_8DenseSetINS0_15SymbolStringPtrENS_12DenseMapInfoISI_vEEEENSJ_IS5_vEENS_6detail12DenseMapPairIS5_SL_EEEEEE at /home/xucapjko/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/bin/../lib/julia/libLLVM-15jl.so (unknown line)
unknown function (ip: 0x7f7d8deade8e)
unknown function (ip: 0x7f7d8deb29fd)
unknown function (ip: 0x7f7d8deb3c7e)
jl_generate_fptr_impl at /home/xucapjko/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/bin/../lib/julia/libjulia-codegen.so.1.10 (unknown line)
jl_compile_method_internal at /home/xucapjko/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/bin/../lib/julia/libjulia-internal.so.1.10 (unknown line)
ijl_apply_generic at /home/xucapjko/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/bin/../lib/julia/libjulia-internal.so.1.10 (unknown line)
wait at ./task.jl:359 [inlined]
fetch at ./task.jl:379 [inlined]
fetch at /home/xucapjko/.julia/packages/StableTasks/YtV0L/src/internals.jl:23 [inlined]
macro expansion at ./reduce.jl:265 [inlined]
macro expansion at ./simdloop.jl:77 [inlined]
mapreduce_impl at ./reduce.jl:263
mapreduce_impl at ./reduce.jl:277 [inlined]
_mapreduce at ./reduce.jl:447
ijl_apply_generic at /home/xucapjko/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/bin/../lib/julia/libjulia-internal.so.1.10 (unknown line)
_mapreduce_dim at ./reducedim.jl:367
#mapreduce#821 at ./reducedim.jl:359 [inlined]
mapreduce at ./reducedim.jl:359
ijl_apply_generic at /home/xucapjko/.julia/juliaup/julia-1.10.9+0.x64.linux.gnu/bin/../lib/julia/libjulia-internal.so.1.10 (unknown line)
_tmapreduce at /home/xucapjko/.julia/packages/OhMyThreads/P7uYA/src/implementation.jl:131
#tmapreduce#22 at /home/xucapjko/.julia/packages/OhMyThreads/P7uYA/src/implementation.jl:100 [inlined]
tmapreduce at /home/xucapjko/.julia/packages/OhMyThreads/P7uYA/src/implementation.jl:84 [inlined]
#tforeach#102 at /home/xucapjko/.julia/packages/OhMyThreads/P7uYA/src/implementation.jl:391
terminate called recursively
/var/spool/slurmd/job33316/slurm_script: line 7: 84958 Aborted

Any ideas and/or tips? Is this realted to OhMyThreads or something completely different?

[MasonProtter]

Interesting! Without more info this'd be pretty hard to track down the source of it. Do you know what scheduler you're using from OhMyThreads? Is it the default dynamic scheduler?

If it's the StaticScheduler, then it might be OhMyThreads' fault, but otherwise I suspect it orginated from julia itself. Can't really say more withouth any more info though.

[kockahonza]

Yes, it uses the default. The threading is done as follows

    @localize solver_kwargs @tasks for i in 1:num_repeats
        # Prealloc variables in each thread (task)
        @local begin
            M1 = Matrix{Float64}(undef, N, N)
            M = Matrix{Float64}(undef, N, N)
            mrls = Vector{Float64}(undef, length(ks))
        end
    ...

and that's it to do with threading. I should have really included this with the issue.

I have since then come across this possibly being an issue with memory. The code should not be particularly memory intensive but I have had issues with julia before (though totally different code) where adding a GC.gc() to the loop fixed it. I am running a test now and it may be that this is confirmed as unrelated!

[kockahonza]

Ok, I can now quite confidently confirm that this is a memory issue likely unrelated to OhMyThreads. After adding a GC.gc() call to a loop one step above the threaded loop I no longer get this error which was really quite reliable before (I had done ~5 runs ending the same way). I find this quite concerning and don't know how to debug it further but it's unrelated and so I am closing the issue. Sorry for the disturbance.

[MasonProtter]

Hi @kockahonza, no problem and no need to apologize for the the disturbance. My recommendation would be to open an issue upstream in the https://github.com/JuliaLang/julia repo, bug reports like this can be quite useful for tracking down problems in the language.

---

As a sidenote,

I run it on a slurm cluster with 100 threads

Julia often does not scale very well to 100 threads, especially if allocations are happening in all of those threads. I would recommend that you try some tests to see if e.g. you see much of a performance difference between e.g. 30 threads and 100 threads.

One big problem is that since all threads share one memory space, if the garbage collector has to run anywhere, it has to stop every single thread while it runs. This can mean that at a certain point if you have too many threads, they'll constantly get paused and resumed over and over again while GC runs.

There's also other things that might have it hard to scale to so many threads that I understand less well. So you might find more performance by splitting your computation into a distributed part using Distributed.jl, and a multithreaded part. e.g. maybe 5 distributed processes each with 20 threads will work better (and may even avoid this bug!).

Good luck, and thanks for the kind words about the package.