Issue: Unstructured Variogram Estimate Slower with OpenMP [v1.3.5]?

[mathijsvdven]

Hi All, Thanks for your efforts to create and maintain GSTools. I'm wanting to create an omnidirectional variogram for a dataset of ~45k unstructured points (x-y-z). Running "gstools.variogram.vario_estimate" in my default installation [1.3.4] takes about 34 seconds to complete.

310 function calls (301 primitive calls) in 34.927 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
...
        1   34.917   34.917   34.917   34.917 {gstools.variogram.estimator.unstructured}
...

To speed up the code, I installed v 1.3.5 from source with OpenMP enabled following the instructions in the documentation. I installed this compiler which includes OpenMP support (I am on Windows 10) and ran:

set GSTOOLS_BUILD_PARALLEL=1
pip install --no-binary=gstools gstools

I then ran the variogram function on the same dataset but found that runtime has increased by almost a factor 10:

310 function calls (301 primitive calls) in 295.988 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
...
        1  295.977  295.977  295.977  295.977 {gstools.variogram.estimator.unstructured}
...

Is this a known issue or does this look like an issue with my installation?

[LSchueler]

Hi mathijsvdven,

thank you so much for reporting this!
Have you confirmed that multiple CPU cores are loaded during the execution? This would of course not explain the difference, but it would still be nice to get a confirmation on that.
My best guess is that your compiler uses some default settings, like including debug information and not including optimisations, which can slow down performance significantly.

The easiest thing for you to try, if you mainly want faster executions, is to use the new computational backend GSTools Core.
In the installation instructions, scroll down to the paragraph "Using experimental GSTools-Core for even more speed". Although it says "experimental", it is well tested.

I hope, this somehow helps.
I have to see, if I can get my hands on a Windows PC to try to replicate your slowdown with MinGW.

[mathijsvdven]

Hi LSchueler, thanks for getting back. I have not confirmed that multiple cores are loaded. How would I go about that? While I installed MinGW, I am not entirely convinced that pip actually uses g++. During the installation, pip complained about Visual C++ 14.0 being missing on the system, which may suggest that it defaults to a different compiler. As far as I am aware, pip does not allow you to specify the compiler...

I will give the Rust implementation a try!

[LSchueler]

I think you can open your task manager and look at something like "performance" or "resources", where you see a timeline of how much each core is working. If multiple curves peak during the execution of GSTools, you can be pretty sure, that it is being computed in parallel.
Apart from that, unfortunately, I don't have much experience compiling code under Windows.
It would be great if you could give us feedback about using the Rust implementation!

[MuellerSeb]

Hey @mathijsvdven. The pre-built wheels use the Visual Studio Compilers. I don't know if there is comes a bottleneck using MinGW. So maybe you could get a speed up with installing VS C and C++:
https://docs.microsoft.com/en-us/cpp/build/vscpp-step-0-installation?view=msvc-170

AFAIU this should also provide OpenMP. Then you can use the commands you already used.
Beside that, I think the compilers can be selected by setting the environment variables CC=<path to c compiler> and CXX=<path to c++ compiler>.

As @LSchueler already said, installing gstools-core should be the best option for now to try for a speedup.

And another option would be to use the conda-package of gstools, which uses openmp by default.

Cheers,
Sebastian