Set up Fedora and Debian container for GH actions runner

[azonenberg]

Doesn't seem like there is native support for hat-based distros but we can probably do something with docker?

[Gherkin]

Got it building in fedora both with repo vulkan and standalone, also debian with repo, just have to verify with standalone.

I'm guessing that we don't care about building the docs? tested building them in the fedora container, but im having issues.

[azonenberg]

Docs in the container are a lower priority, but it would be nice to have eventually. We'd like to be able to use the same workflow to create official binary package releases, and it will be easier to create packages for each distro in the appropriate container (meaning the doc package for fedora needs to build in the fedora container).

[Gherkin]

Debian bookworm builds fully (sdk repo/standalone and docs), fedora 42 has problems building docs.

ended up hitting the same issue as this ngscopeclient/scopehal-docs#97. So I'm guessing we need to build tex4ht from source? i was gonna try it out, but the svn seems down currently, I think this is good enough for now. removed the build for docs in fedora since the fail-fast strategy is default (if one job fails they are all cancelled), but left all the steps.

For some reason the tests quite often fails in fedora? no idea why, this fails the entire job. I left it since I deemed it outside the scope for this.

We could definitely add more versions if we want, should not be very hard. Also if we want to build in different OS:s (like arch).

Created PR #859

[d235j]

Do we have CTAN in these containers? If so it might be possible to install tex4ht from there.

[Gherkin]

not natively, but i guess it can be installed, they are not the most minimal containers. so that might be a possibility, from the arch issue linked in that ngscopeclient/scopehal-docs#97 it seems like it needs a specific build flag, so that would also need to be included in the ctan-supplied version

[azonenberg]

Arch would be nice to have building as well.

Which tests are failing? I've seen I think the de-embed test crashing intermittently on some github CI runners and working on every physical machine I've tested. Have not yet root caused it, but it's nothing to do with your build setup.

[Gherkin]

Ill look into building Arch also :)

looking a bit at the test, it seems like it always is the Filter_EyePattern test that fails occasionally, weirdly enough by segfault. (here with repo)

Test project /__w/scopehal-apps/scopehal-apps/build
      Start  1: Buffers_CpuOnly
 1/16 Test  #1: Buffers_CpuOnly ..................   Passed    0.07 sec
      Start  2: Buffers_CpuGpu
 2/16 Test  #2: Buffers_CpuGpu ...................   Passed    0.03 sec
      Start  3: Filter_Add
 3/16 Test  #3: Filter_Add .......................   Passed    0.10 sec
      Start  4: Filter_ACRMS
 4/16 Test  #4: Filter_ACRMS .....................   Passed    0.70 sec
      Start  5: Filter_DeEmbed
 5/16 Test  #5: Filter_DeEmbed ...................   Passed    2.00 sec
      Start  6: Filter_EyePattern
 6/16 Test  #6: Filter_EyePattern ................Subprocess aborted***Exception:   0.62 sec
Filters: "Filter_EyePattern"
Randomness seeded to: 3495703740
Baseline (expecting no mask hits)

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Filters is a Catch2 v3.8.0 host application.
Run with -? for options

-------------------------------------------------------------------------------
Filter_EyePattern
  Baseline
-------------------------------------------------------------------------------
/__w/scopehal-apps/scopehal-apps/tests/Filters/Filter_EyePattern.cpp:119
...............................................................................

/__w/scopehal-apps/scopehal-apps/tests/Filters/Filter_EyePattern.cpp:119: FAILED:
  {Unknown expression after the reported line}
due to a fatal error condition:
  SIGSEGV - Segmentation violation signal

double free or corruption (out)

      Start  7: Filter_FIR
 7/16 Test  #7: Filter_FIR .......................   Passed    0.34 sec
      Start  8: Filter_FFT
 8/16 Test  #8: Filter_FFT .......................   Passed    1.08 sec
      Start  9: Filter_Subtract
 9/16 Test  #9: Filter_Subtract ..................   Passed    0.12 sec
      Start 10: Filter_Upsample
10/16 Test #10: Filter_Upsample ..................   Passed    0.40 sec
      Start 11: Filter_FrequencyMeasurement
11/16 Test #11: Filter_FrequencyMeasurement ......   Passed    0.28 sec
      Start 12: Primitive_Averager
12/16 Test #12: Primitive_Averager ...............   Passed    0.52 sec
      Start 13: Primitive_Convert8BitSamples
13/16 Test #13: Primitive_Convert8BitSamples .....   Passed    0.14 sec
      Start 14: Primitive_Convert16BitSamples
14/16 Test #14: Primitive_Convert16BitSamples ....   Passed    0.16 sec
      Start 15: Primitive_FindZeroCrossings
15/16 Test #15: Primitive_FindZeroCrossings ......   Passed    1.72 sec
      Start 16: Primitive_SampleOnRisingEdges
16/16 Test #16: Primitive_SampleOnRisingEdges ....   Passed    0.18 sec

94% tests passed, 1 tests failed out of 16

Total Test time (real) =   8.48 sec

The following tests FAILED:
	  6 - Filter_EyePattern (Subprocess aborted)
Errors while running CTest
Error: Process completed with exit code 8.

Also another weird issue. building fedora with standalone the tests refuse to run with the following error, everything builds fine though.

I should have looked a bit closer earlier!

Dont know what to do about it though.

Test project /root/scopehal-apps/build
      Start  1: Buffers_CpuOnly
 1/16 Test  #1: Buffers_CpuOnly ..................***Failed    0.01 sec
    ERROR: vk::SystemError: Context::createInstance: ErrorIncompatibleDriver

      Start  2: Buffers_CpuGpu
 2/16 Test  #2: Buffers_CpuGpu ...................***Failed    0.01 sec
    ERROR: vk::SystemError: Context::createInstance: ErrorIncompatibleDriver

      Start  3: Filter_Add
 3/16 Test  #3: Filter_Add .......................***Failed    0.01 sec
    ERROR: vk::SystemError: Context::createInstance: ErrorIncompatibleDriver

      Start  4: Filter_ACRMS
 4/16 Test  #4: Filter_ACRMS .....................***Failed    0.01 sec
    ERROR: vk::SystemError: Context::createInstance: ErrorIncompatibleDriver

      Start  5: Filter_DeEmbed
 5/16 Test  #5: Filter_DeEmbed ...................***Failed    0.02 sec
    ERROR: vk::SystemError: Context::createInstance: ErrorIncompatibleDriver

      Start  6: Filter_EyePattern
 6/16 Test  #6: Filter_EyePattern ................***Failed    0.02 sec
    ERROR: vk::SystemError: Context::createInstance: ErrorIncompatibleDriver

      Start  7: Filter_FIR
 7/16 Test  #7: Filter_FIR .......................***Failed    0.01 sec
    ERROR: vk::SystemError: Context::createInstance: ErrorIncompatibleDriver

      Start  8: Filter_FFT
 8/16 Test  #8: Filter_FFT .......................***Failed    0.00 sec
    ERROR: vk::SystemError: Context::createInstance: ErrorIncompatibleDriver

      Start  9: Filter_Subtract
 9/16 Test  #9: Filter_Subtract ..................***Failed    0.01 sec
    ERROR: vk::SystemError: Context::createInstance: ErrorIncompatibleDriver

      Start 10: Filter_Upsample
10/16 Test #10: Filter_Upsample ..................***Failed    0.02 sec
    ERROR: vk::SystemError: Context::createInstance: ErrorIncompatibleDriver

      Start 11: Filter_FrequencyMeasurement
11/16 Test #11: Filter_FrequencyMeasurement ......***Failed    0.02 sec
    ERROR: vk::SystemError: Context::createInstance: ErrorIncompatibleDriver

      Start 12: Primitive_Averager
12/16 Test #12: Primitive_Averager ...............***Failed    0.01 sec
    ERROR: vk::SystemError: Context::createInstance: ErrorIncompatibleDriver

      Start 13: Primitive_Convert8BitSamples
13/16 Test #13: Primitive_Convert8BitSamples .....***Failed    0.01 sec
    ERROR: vk::SystemError: Context::createInstance: ErrorIncompatibleDriver

      Start 14: Primitive_Convert16BitSamples
14/16 Test #14: Primitive_Convert16BitSamples ....***Failed    0.01 sec
    ERROR: vk::SystemError: Context::createInstance: ErrorIncompatibleDriver

      Start 15: Primitive_FindZeroCrossings
15/16 Test #15: Primitive_FindZeroCrossings ......***Failed    0.02 sec
    ERROR: vk::SystemError: Context::createInstance: ErrorIncompatibleDriver

      Start 16: Primitive_SampleOnRisingEdges
16/16 Test #16: Primitive_SampleOnRisingEdges ....***Failed    0.01 sec
    ERROR: vk::SystemError: Context::createInstance: ErrorIncompatibleDriver


0% tests passed, 16 tests failed out of 16

Total Test time (real) =   0.20 sec

The following tests FAILED:
	  1 - Buffers_CpuOnly (Failed)
	  2 - Buffers_CpuGpu (Failed)
	  3 - Filter_Add (Failed)
	  4 - Filter_ACRMS (Failed)
	  5 - Filter_DeEmbed (Failed)
	  6 - Filter_EyePattern (Failed)
	  7 - Filter_FIR (Failed)
	  8 - Filter_FFT (Failed)
	  9 - Filter_Subtract (Failed)
	 10 - Filter_Upsample (Failed)
	 11 - Filter_FrequencyMeasurement (Failed)
	 12 - Primitive_Averager (Failed)
	 13 - Primitive_Convert8BitSamples (Failed)
	 14 - Primitive_Convert16BitSamples (Failed)
	 15 - Primitive_FindZeroCrossings (Failed)
	 16 - Primitive_SampleOnRisingEdges (Failed)
Errors while running CTest

[azonenberg]

Guessing it has to do with the vulkan setup in the VM. Maybe a mismatch of the vulkan headers and binary libs or something? Or complete lack of a vulkan implementation (normally these vms have llvmpipe in them if there's no GPU)?

I think I may have seen the eye pattern test segfault in CI before. Will have to dig into that, havent reproduced on any of my hardware yet.

[azonenberg]

Aha, it's a bug in the AVX2 and AVX2+FMA implementations of the eye pattern filter that use an incorrect loop bound copied from the AVX512F version, which has a larger vector size. Fixed in ngscopeclient/scopehal@9dd3972.

Both of my main dev machines are Xeons with AVX512F so it slipped through the cracks until I ran under asan on my work laptop which does not have AVX512 (do we build CI unit tests builds with sanitizers? if not, we should, this would have caught it quickly if AVX512 wasn't supported... we should also make non-sanitizer release versions for end users to install though)

[azonenberg]

I guess that might tie into a broader question of if we should re-run some of the unit tests multiple times manually turning off e.g. AVX512 if available, so that we can stress each of the different kernel implementations. If the cloud servers all have AVX512 the buggy code in question would never have been executed