[Bug] Unable to run tests

[ksapchuk]

What are you really trying to do?

Something changed in the last few days and we are no longer able to run tests using the temporal test library.

Describe the bug

The tests fail to start the temporal test server with an error

The current machine does not support all of the following CPU features that are required by the image: [CX8, CMOV, FXSR, MMX, SSE, SSE2, SSE3, SSSE3, SSE4_1, SSE4_2, POPCNT, LZCNT, AVX, AVX2, BMI1, BMI2, FMA, F16C].

Minimal Reproduction

Install the time skipping test example https://github.com/temporalio/samples-typescript/tree/main/timer-examples#testing and run the tests

timer-examples % npm test

> temporal-timer@0.1.0 test
> mocha --require ts-node/register --require source-map-support/register src/test/*.test.ts


  countdownWorkflow
The current machine does not support all of the following CPU features that are required by the image: [CX8, CMOV, FXSR, MMX, SSE, SSE2, SSE3, SSSE3, SSE4_1, SSE4_2, POPCNT, LZCNT, AVX, AVX2, BMI1, BMI2, FMA, F16C].
Please rebuild the executable with an appropriate setting of the -march option.

Environment/Versions

• OS and processor: Apple M3 Pro
• Temporal Version: Typescript 1.11.8

Additional context

[EssentKoen]

+1
We are experiencing the same issue.

OS - mac - M1 through M4.
Something seems to be broken with https://temporal.download as well, but the error seems to indicate a version is available, but somehow broken.
I see that I have a new version which is 10MB smaller. You can actually use this (search for "temporal-test-server-sdk-typescript-1.11.8" in /private/var) to locate the broken executable.

Colleagues who still have the temporal-dev-server cached locally have no problems whatsoever, so as a workaround if you really need to, you can manually put that executable on your laptop on that location, then run again. Not sure where you can download a correct version at the moment, we are just lucky to still have the old one lying around.

If you download a correct test server, btw - you can also configure temporal to use a custom path so you don't have to finnick around as much as I previously indicated ;)

[vault-pay]

I'm experiencing the same problem. I'm on M2 Max.

[Seven4ME]

^ Same for me

[Seven4ME]

Who faced same issue - try to execute fd temporal /private/var/folders | xargs rm and then re-run tests.
Its helped to me.

[EssentKoen]

I think @Seven4ME 's comment suggests the download is no longer broken.
I verified this by removing the existing executable and it indeed works. The download size is still down by 10MB, so that's nice I guess.

I would love to hear a report out from someone from Temporal on how to prevent something like this - we're now quite dependent on the download actually working at all times.

Other than that, I think this issue can be closed.

[mjameswh]

TLDR

The upstream download file has been fixed. If you're still seeing the error message, you may need to remove the cached download file; just run rm $TMPDIR/temporal-test-server-sdk-typescript-* from a terminal window.

We're very sorry for that.

What happened?

The Temporal Test Server is developed in Java. It is then built into a standalone executable using GraalVM.

Until recently, due to limitations of our build system, it was not possible to use GraalVM to build standalone executables for ARM-based architectures. On MacOS, that was not a problem in most cases, because Apple's Rosetta would allow transparent execution the AMD64 build instead. But that wouldn't work on Linux, including in Docker running on Mac, which was a pain to many of our users.

Test Server 1.30.0 added native support for ARM64. To achieve that, we had to completely rewrite our native-build scripts. However, this new native-build script unintentionally turned on some compiler optimizations that depend on specific AMD64 CPU features. Some of those CPU features are not emulated by Apple's Rosetta, which means that using the Test Server 1.30.0 on an AMD-based Mac would not work, resulting in this error message:

The current machine does not support all of the following CPU features that are required by the image: [CX8, CMOV, FXSR, MMX, SSE, SSE2, SSE3, SSSE3, SSE4_1, SSE4_2, POPCNT, LZCNT, AVX, AVX2, BMI1, BMI2, FMA, F16C].

To be clear, we did not explicitly turn on those compiler optimizations: they are enabled by default by GraalVM's compiler, and need to be explicitly opted out. We didn't, and that was an oversight from our part.

The native-build process for our Test Server has now been fixed, so Test Server 1.30.1 will again run properly on MacOS.

Why are we still downloading the AMD64 build on MacOS?

Since we are now publishing an ARM64 build of the Test Server for MacOS, some may wonder why did the SDK downloaded the AMD64 1.30.0 build, which requires these CPU flags, rather than downloading and using the ARM build, which would completely avoid the need for Rosetta? After all, using the proper executable would have also avoided the observed issue.

That's because download of the Test Server executable happens indirectly through a download service we maintain (https://temporal.download/), which is there to provide a caching layer over downloading the executables directly from GitHub. That service also provides some basic routing rules, including the rule that says "If requesting Test Server for ARM64 on MacOS, then return the AMD64 build instead". As part of the work on publishing Test Server for ARM64, our download service has obviously been fixed is; that is: "If requesting Test Server prior to 1.30.0, for ARM64 on MacOS, then return the AMD64 build instead; otherwise, return the ARM64 build".

We had already fixed the download service code, but we intentionally delayed redeployment of the service, to be done after the release of Test Server 1.30.0. There was simply no reason to urge this redeployment, as the legacy behavior (i.e. sending AMD64 build instead of ARM64 to MacOS users) should have still been correct, simply not optimal. More importantly, doing things in that order would make it easier for us to run some final tests before turning this on by default for all of our users.

How can we protect ourself from similar incidents in the future?

The SDK's TestWorkflowEnvironment.createLocal() and TestWorkflowEnvironment.createTimeSkipping() functions both accept passing in the path to predownloaded executables, rather than let them be automatically downloaded at run time by the SDK. Simply download manually the Temporal CLI and the Temporal Test Server executables from their respective project's GitHub release page, then provide the file path of those executables when creating your TestWorkflowEnvironment.

Despite the extra work, this may actually be highly desirable depending on your specific needs.

Pre-downloading both executables have several benefits, including:

• Avoid test timeout flakes due to download taking too much time;
• Avoid being affected by ocasional outtages that may affect the download service;
• Guarantee that your tests are always running with the same version of the server.

[EssentKoen]

Thanks for the extensive update @mjameswh - much appreciated!