Test if non CUDA builds are not added to accelorator path with jax by laraPPr · Pull Request #917 · EESSI/software-layer

laraPPr · 2025-02-13T13:24:29Z

5 out of 86 required modules missing:



* absl-py/2.1.0-GCCcore-12.3.0 (absl-py-2.1.0-GCCcore-12.3.0.eb)

* pytest/7.4.2-GCCcore-12.3.0 (pytest-7.4.2-GCCcore-12.3.0.eb)

* pytest-xdist/3.3.1-GCCcore-12.3.0 (pytest-xdist-3.3.1-GCCcore-12.3.0.eb)

* ml_dtypes/0.3.2-gfbf-2023a (ml_dtypes-0.3.2-gfbf-2023a.eb)

* jax/0.4.25-gfbf-2023a-CUDA-12.1.1 (jax-0.4.25-gfbf-2023a-CUDA-12.1.1.eb)

eessi-bot · 2025-02-13T13:24:33Z

Instance eessi-bot-mc-aws is configured to build for:

architectures: x86_64/generic, x86_64/intel/haswell, x86_64/intel/sapphire_rapids, x86_64/intel/skylake_avx512, x86_64/amd/zen2, x86_64/amd/zen3, aarch64/generic, aarch64/neoverse_n1, aarch64/neoverse_v1
repositories: eessi.io-2023.06-software, eessi.io-2023.06-compat

eessi-bot · 2025-02-13T13:24:34Z

Instance eessi-bot-mc-azure is configured to build for:

architectures: x86_64/amd/zen4
repositories: eessi.io-2023.06-software, eessi.io-2023.06-compat

eessi-bot-casparvl-eessi · 2025-02-13T13:24:34Z

Instance eessi-bot-casparvl is configured to build for:

architectures: x86_64/amd/zen4, x86_64/amd/zen2
repositories: eessi.io-2023.06-software, eessi-hpc.org-2023.06-compat, eessi-hpc.org-2023.06-software, eessi.io-2023.06-compat

gpu-bot-ugent · 2025-02-13T13:24:34Z

Instance eessi-bot-vsc-ugent is configured to build for:

architectures: x86_64/amd/zen3
repositories: eessi-hpc.org-2023.06-compat, eessi.io-2023.06-software, eessi-hpc.org-2023.06-software, eessi.io-2023.06-compat

eessi-bot-trz42 · 2025-02-13T13:24:34Z

Instance trz42-GH200-jr is configured to build for:

architectures: aarch64/nvidia/grace
repositories: eessi.io-2023.06-software

laraPPr · 2025-02-13T13:25:30Z

bot: build instance:eessi-bot-vsc-ugent repo:eessi.io-2023.06-software accel:nvidia/cc80

eessi-bot · 2025-02-13T13:25:33Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build instance:eessi-bot-vsc-ugent repo:eessi.io-2023.06-software accel:nvidia/cc80 from laraPPr
- expanded format: build instance:eessi-bot-vsc-ugent repository:eessi.io-2023.06-software accelerator:nvidia/cc80
handling command build instance:eessi-bot-vsc-ugent repository:eessi.io-2023.06-software accelerator:nvidia/cc80 resulted in:
- no jobs were submitted

eessi-bot · 2025-02-13T13:25:34Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command build instance:eessi-bot-vsc-ugent repo:eessi.io-2023.06-software accel:nvidia/cc80 from laraPPr
- expanded format: build instance:eessi-bot-vsc-ugent repository:eessi.io-2023.06-software accelerator:nvidia/cc80
handling command build instance:eessi-bot-vsc-ugent repository:eessi.io-2023.06-software accelerator:nvidia/cc80 resulted in:
- no jobs were submitted

eessi-bot-casparvl-eessi · 2025-02-13T13:25:34Z

Updates by the bot instance eessi-bot-casparvl (click for details)

account laraPPr has NO permission to send commands to the bot

gpu-bot-ugent · 2025-02-13T13:25:34Z

Updates by the bot instance eessi-bot-vsc-ugent (click for details)

received bot command build instance:eessi-bot-vsc-ugent repo:eessi.io-2023.06-software accel:nvidia/cc80 from laraPPr
- expanded format: build instance:eessi-bot-vsc-ugent repository:eessi.io-2023.06-software accelerator:nvidia/cc80
handling command build instance:eessi-bot-vsc-ugent repository:eessi.io-2023.06-software accelerator:nvidia/cc80 resulted in:
- submitted job 15445297, for details & status see Test if non CUDA builds are not added to accelorator path with jax #917 (comment)

eessi-bot-trz42 · 2025-02-13T13:25:34Z

Updates by the bot instance trz42-GH200-jr (click for details)

account laraPPr has NO permission to send commands to the bot

gpu-bot-ugent · 2025-02-13T13:25:56Z

New job on instance eessi-bot-vsc-ugent for CPU micro-architecture x86_64-amd-zen3 and accelerator nvidia/cc80 for repository eessi.io-2023.06-software in job dir /scratch/gent/vo/002/gvo00211/SHARED/jobs/2025.02/pr_917/15445297

date	job status	comment
Feb 13 13:25:54 UTC 2025	submitted	job id `15445297` awaits release by job manager
Feb 13 13:26:58 UTC 2025	released	job awaits launch by Slurm scheduler
Feb 13 13:29:02 UTC 2025	running	job `15445297` is running
Feb 13 15:05:06 UTC 2025	finished	😢 FAILURE (click triangle for details) Details ✅ job output file `slurm-15445297.out` ✅ no message matching `FATAL:` ❌ found message matching `ERROR:` ❌ found message matching `FAILED:` ❌ found message matching `required modules missing:` ❌ no message matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen3-1739456837.tar.gz` size: 6 MiB (6667595 bytes) entries: 1191 modules under 2023.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80/modules/all `absl-py/2.1.0-GCCcore-12.3.0.lua` `ml_dtypes/0.3.2-gfbf-2023a.lua` `pytest/7.4.2-GCCcore-12.3.0.lua` `pytest-xdist/3.3.1-GCCcore-12.3.0.lua` software under 2023.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80/software `absl-py/2.1.0-GCCcore-12.3.0` `ml_dtypes/0.3.2-gfbf-2023a` `pytest/7.4.2-GCCcore-12.3.0` `pytest-xdist/3.3.1-GCCcore-12.3.0` other under 2023.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80 no other files in tarball
Feb 13 15:05:06 UTC 2025	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ OK ] (1/1) EESSI_LAMMPS_lj %device_type=gpu %module_name=LAMMPS/2Aug2023_update2-foss-2023a-kokkos-CUDA-12.1.1 %scale=1_4_node /497af4b1 @BotBuildTests:x86_64_amd_zen3_nvidia_cc80+default P: perf: 4447.069 timesteps/s (r:0, l:None, u:None) [ PASSED ] Ran 1/1 test case(s) from 1 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file `slurm-15445297.out` ❌ found message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

laraPPr · 2025-02-13T13:41:32Z

@trz42 @ocaisa this looks like it is not doing what we expect it to do because it seems to be installing pytest-xdist in the accelerator path.

/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/software/Python/3.11.3-GCCcore-12.3.0/bin/python -m pip install --prefix=/cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen3/accel/nvidia/cc80/software/pytest-xdist/3.3.1-GCCcore-12.3.0  --verbose  --no-deps  --ignore-installed  --no-index  --no-build-isolation  .

laraPPr · 2025-02-13T13:43:48Z

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80

eessi-bot · 2025-02-13T13:43:51Z

Updates by the bot instance eessi-bot-mc-aws (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80 from laraPPr
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80 resulted in:
- submitted job 45927, for details & status see Test if non CUDA builds are not added to accelorator path with jax #917 (comment)

eessi-bot · 2025-02-13T13:43:51Z

Updates by the bot instance eessi-bot-mc-azure (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80 from laraPPr
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80 resulted in:
- no jobs were submitted

eessi-bot-casparvl-eessi · 2025-02-13T13:43:51Z

Updates by the bot instance eessi-bot-casparvl (click for details)

account laraPPr has NO permission to send commands to the bot

gpu-bot-ugent · 2025-02-13T13:43:51Z

Updates by the bot instance eessi-bot-vsc-ugent (click for details)

received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 accel:nvidia/cc80 from laraPPr
- expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80
handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 accelerator:nvidia/cc80 resulted in:
- no jobs were submitted

eessi-bot-trz42 · 2025-02-13T13:43:51Z

Updates by the bot instance trz42-GH200-jr (click for details)

account laraPPr has NO permission to send commands to the bot

eessi-bot · 2025-02-13T13:43:55Z

New job on instance eessi-bot-mc-aws for CPU micro-architecture x86_64-amd-zen2 and accelerator nvidia/cc80 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2025.02/pr_917/45927

date	job status	comment
Feb 13 13:43:54 UTC 2025	submitted	job id `45927` awaits release by job manager
Feb 13 13:44:41 UTC 2025	released	job awaits launch by Slurm scheduler
Feb 13 13:53:28 UTC 2025	running	job `45927` is running
Feb 13 14:18:15 UTC 2025	finished	😢 FAILURE (click triangle for details) Details ✅ job output file `slurm-45927.out` ✅ no message matching `FATAL:` ❌ found message matching `ERROR:` ❌ found message matching `FAILED:` ❌ found message matching `required modules missing:` ❌ no message matching `No missing installations` ✅ found message matching `.tar.gz created!` Artefacts `eessi-2023.06-software-linux-x86_64-amd-zen2-1739455531.tar.gz` size: 6 MiB (6661749 bytes) entries: 1191 modules under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/modules/all `absl-py/2.1.0-GCCcore-12.3.0.lua` `ml_dtypes/0.3.2-gfbf-2023a.lua` `pytest/7.4.2-GCCcore-12.3.0.lua` `pytest-xdist/3.3.1-GCCcore-12.3.0.lua` software under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80/software `absl-py/2.1.0-GCCcore-12.3.0` `ml_dtypes/0.3.2-gfbf-2023a` `pytest/7.4.2-GCCcore-12.3.0` `pytest-xdist/3.3.1-GCCcore-12.3.0` other under 2023.06/software/linux/x86_64/amd/zen2/accel/nvidia/cc80 no other files in tarball
Feb 13 14:18:15 UTC 2025	test result	😢 FAILURE (click triangle for details) Reason EESSI test suite was not run, test step itself failed to execute. Details ✅ job output file `slurm-45927.out` ❌ found message matching `ERROR:` ✅ no message matching `[\sFAILED\s].Ran . test case`

laraPPr · 2025-02-13T15:15:35Z

It failed building jax with this error:

FAILED: Installation ended unsuccessfully (build directory: /tmp/vsc48506/easybuild/build/jax/0.4.25/gfbf-2023a-CUDA-12.1.1): build failed (first 300 chars): Failed to determine installation prefix for binutils (took 39 mins 48 secs

and as you can see in the artifacts the non enabled cuda builds were build in the accelerator path.

…think it should work

laraPPr · 2025-06-06T14:52:40Z

reopen to test replacement for dorny

laraPPr · 2025-06-06T15:17:20Z

Filtering works and check fail as expected

Test is non CUDA builds are not add to accelorator path with jax

3a7498d

laraPPr added the accel:nvidia label Feb 17, 2025

laraPPr added 7 commits February 25, 2025 12:33

Create test-check_easystacks.yml

b008485

add the filter for changed easystack files

fb7ceae

fix syntax of yml file

c4b39d3

run check_missing_installation.sh in workflow

5b5434a

debugging checking missing installation step

b875d62

debugging getting changed file

9a981a5

debugging getting changed file

e8acea1

laraPPr added 26 commits June 4, 2025 14:17

try to trigger run and don't set up EESSI in test

493e4bf

try action with eessi in the background

8f03b3b

try again

592d7db

try again

60acbd4

try again

9cb0dbd

see I can get head and base and cary environvarible to next step

97f39ad

try to cary on changed variable to next step

493b62d

try same diff as done in dorny

b83e7db

See what is wrong with git diff in environment of workflow because I …

48e79e2

…think it should work

fix typo

a48d6d6

check where the software-layer repo is

5c59701

test other diff command from dorny

2c55316

Try with fetch-depth 0

cf489ed

fix typo

ee1c8c6

try with origin/

0f77b73

See if exit code is not 1 after diff

8ca8075

add python script for filtering output of git diff

41521ef

debug python script

ef6cb8b

debug python script

2fc2f60

fix workflow

2aeca30

debug python script

202524c

test: check if environment variable exists

39e97ce

replace dorny with new filter

2aa0854

get the right fetch depth

e200201

use the set env_var instead of the dorny action

bf32c3f

remove test action no longer needed

8b3b0a2

laraPPr reopened this Jun 6, 2025

laraPPr closed this Jun 6, 2025

Conversation

laraPPr commented Feb 13, 2025

Uh oh!

eessi-bot Bot commented Feb 13, 2025

Uh oh!

eessi-bot Bot commented Feb 13, 2025

Uh oh!

eessi-bot-casparvl-eessi Bot commented Feb 13, 2025

Uh oh!

gpu-bot-ugent Bot commented Feb 13, 2025

Uh oh!

eessi-bot-trz42 Bot commented Feb 13, 2025

Uh oh!

laraPPr commented Feb 13, 2025

Uh oh!

eessi-bot Bot commented Feb 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot Bot commented Feb 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot-casparvl-eessi Bot commented Feb 13, 2025

Uh oh!

gpu-bot-ugent Bot commented Feb 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot-trz42 Bot commented Feb 13, 2025

Uh oh!

gpu-bot-ugent Bot commented Feb 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

laraPPr commented Feb 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

laraPPr commented Feb 13, 2025

Uh oh!

eessi-bot Bot commented Feb 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot Bot commented Feb 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot-casparvl-eessi Bot commented Feb 13, 2025

Uh oh!

gpu-bot-ugent Bot commented Feb 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eessi-bot-trz42 Bot commented Feb 13, 2025

Uh oh!

eessi-bot Bot commented Feb 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

laraPPr commented Feb 13, 2025

Uh oh!

laraPPr commented Jun 6, 2025

Uh oh!

laraPPr commented Jun 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

eessi-bot Bot commented Feb 13, 2025 •

edited

Loading

eessi-bot Bot commented Feb 13, 2025 •

edited

Loading

gpu-bot-ugent Bot commented Feb 13, 2025 •

edited

Loading

gpu-bot-ugent Bot commented Feb 13, 2025 •

edited

Loading

laraPPr commented Feb 13, 2025 •

edited

Loading

eessi-bot Bot commented Feb 13, 2025 •

edited

Loading

eessi-bot Bot commented Feb 13, 2025 •

edited

Loading

gpu-bot-ugent Bot commented Feb 13, 2025 •

edited

Loading

eessi-bot Bot commented Feb 13, 2025 •

edited

Loading