Skip to content

Bug: mon.csv write race causes flaky parallel ctest failures #390

Description

@lukelowry

Description how to reproduce the bug

Failure mode varies between runs: Failed, SEGFAULT, or Subprocess aborted. I got each at least once. Sequential (-j 1) always passes. Re-running the parallel command in a loop reproduces within a few iterations.

$ ctest --test-dir build -j10 -R "TwoBusBasic"
...
The following tests FAILED:
	 17 - TwoBusBasic_using_app (Subprocess aborted)
Details
$ ctest --test-dir build -j$(nproc) -R "TwoBusBasic" --output-on-failure
...
17: Test command: /home/lukel/GridKit/build/application/PhasorDynamics/PDSim "TwoBusBasic.solver.json"
17: Working Directory: /home/lukel/GridKit/build/examples/PhasorDynamics/Tiny/TwoBus/Basic
17: [ERROR] Files "mon.csv" and "TwoBusBasic.ref.csv" have different number of variables.
17: terminate called after throwing an instance of 'std::length_error'
17:   what():  cannot create std::vector larger than max_size()

The following tests FAILED:
	 17 - TwoBusBasic_using_app (Subprocess aborted)

GridKit™ version

develop @ aaf6f81 ("Add INSTALL.md file. (#383)")

System and environment details

  • OS: Ubuntu 24.04.3 LTS on WSL2 (kernel 6.6.87.2-microsoft-standard-WSL2)
  • CPU: 12th Gen Intel Core i9-12900K (24 logical cores)
  • Compiler: clang/clang++ 16.0.6 (Ubuntu 16.0.6-23ubuntu4)
  • CMake: 3.28.3
  • SUNDIALS: 7.4.0 with KLU sparse support
  • Enzyme: via Spack env
  • Build preset: enzyme (RelWithDebInfo, -march=haswell -mtune=native)

Additional information

Affected directories (any with multiple tests sharing one mon.csv in the same WORKING_DIRECTORY):

  • examples/PhasorDynamics/Tiny/TwoBus/Basic/ — tests 14–17
  • examples/PhasorDynamics/Tiny/ThreeBus/Basic/ — tests 24–27

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions