Extend differential tests with newyork optimizer#555
Conversation
| # The exact compiler modes to use for compiling and comparing hashes by retester. | ||
| COMPILE_MODES: "Y M0 S+, Y M3 S+, Y Mz S+, NY M3 S+, NY Mz S+" | ||
| # The compiler modes allowed to run during e2e test execution by retester. | ||
| # E.g. this can be updated to allow only `"Y, NY Mz S+"` (all Yul modes + newyork with -Oz and solc optimizer enabled). | ||
| E2E_ALLOWED_MODES: "Y, NY" |
There was a problem hiding this comment.
The E2E_ALLOWED_MODES allow all requested modes that use Yul and newyork IR pipelines, which is still fast so it's good to keep this. The COMPILE_MODES currently uses a subset.
A bytecode mismatch was detected on Windows for 1 contract using NY Mz S- (newyork, -Oz, solc optimizer disabled) which I'm currently investigating. This PR will exclude that mode from the comparison so that it can be merged with the current settings, and I'll continue investigating the mismatch in the mean time.
| # The compiler modes to use for compilation by retester. | ||
| MODES: "Y M0 S+, Y M3 S+, Y Mz S+" | ||
| # The exact compiler modes to use for compiling and comparing hashes by retester. | ||
| COMPILE_MODES: "Y M0 S+, Y M3 S+, Y Mz S+, NY M3 S+, NY Mz S+" |
There was a problem hiding this comment.
We first discussed initially adding newyork running only with Oz (Mz in retester's representation) due to CI duration (having all 3 counterparts would double the compilation time). I think for now we could use these for PRs:
Y M0 S+Y M3 S+Y Mz S+NY Mz S+
But for pushes to main we could additionally run:
NY M0 S+NY M3 S+
Since macOS intel is extra slow (and runs only on main) having it also run these extra will take time, but if we experience that as a problem on main we could later look into separating Y and NY compilations onto different runners.
I know you mentioned perhaps having a cron job, that could probably be something we could do in addition to the above if we'd like to test S- as well. Would then schedule it to run during the night. Running with S- would be beneficial with NY since we currently have a bytecode mismatch with that combination (for 1 contract) that I'll investigate.
P.S. The e2e differential tests are running with all optimization combinations (LLVM + solc optimizer) with both Yul and newyork pipelines.
2 participants
Description
Extends our differential tests such that the compile+compare bytecode hashes and the E2E tests also exercise the newyork IR pipeline.
The requested compiler modes that E2E test cases run with are still controlled by the metadata in the corpus, but the related PR listed below also enables support for an allow-list to optionally narrow the set of modes run, via an input to the GH action used.
The changes to the workflow in this PR allows all newyork retester compiler modes (all LLVM opt levels + solc optimizer on/off) to run for the E2E tests since they still complete fast.
Related
The PRs implemented as prerequisites alongside this PR are:
revive-differential-testsresolc-compiler-testsResolved issues
Closes #553