[BUG] Investigate non-determinism in evo2 recipe data preprocessing script and/or test

[jstjohn]

BioNeMo Framework Version

main

Bug Description

The recent PR #1492 addressed this issue by relaxing the constraints. However the intermittent CI failure that lead to the number of splits not being the same in every run highlights a bug where randomness is leaking into the process somewhere.

Steps to Reproduce

See #1492, reset the test to the previous strict equality and there are intermittent (rare) failures.

Error Messages and Logs

Docker Image

No response

System Information

Environment Details:

• OS: [e.g., Ubuntu 20.04]
• CPU: [e.g., Intel i9-12900K]
• RAM: [e.g., 64GB]

GPU Details:

• GPU Model: [e.g., NVIDIA RTX 4090]
• GPU Memory: [e.g., 24GB]
• CUDA Version: [e.g., 12.1]
• CUDA Driver: [e.g., 525.85.05]
• cuDNN Version: [e.g., 8.9.0]

Additional Context

No response

[trvachov]

Whoops -- closed accidentally.