fix(orchestrator): fix race condition in Cleanup causing missed cleanup functions

[AdaAibaby]

There was a TOCTOU race between Add()/AddPriority() and run():

1. Add() checks hasRun (false) outside the lock
2. Add() blocks waiting for mu.Lock()
3. run() acquires lock, sets hasRun=true, executes all cleanups, unlocks
4. Add() acquires lock, appends f — but run() already finished

Result: f is never executed, potentially leaking resources (network namespaces, cgroups, file descriptors, etc.).

Fix:

• Move hasRun.Store(true) inside the lock in run()
• Add double-checked locking in Add()/AddPriority(): re-check hasRun after acquiring the lock and execute f inline if cleanup already ran

Add race-condition tests that reliably reproduce the bug with -race.

/cc @jakubno @dobrac @ValentaTomas Would appreciate your review on this change, thanks!

[gemini-code-assist]

Code Review

Executing the cleanup and priority cleanup functions while holding the mutex lock can lead to deadlocks if the functions attempt to acquire the same lock or perform blocking operations. Releasing the lock before executing these functions avoids this risk.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.