Make @worker_task workers spawn-safe; default to spawn on all platforms

[bradyyie]

Problem

On macOS, any worker registered via @worker_task crashes with exitcode=-11 (SIGSEGV) at startup under the SDK-forced fork start method and is silently restarted by TaskHandler.monitor_processes every ~5s (the monitor interval), so no task ever completes. The documented escape hatch CONDUCTOR_MP_START_METHOD=spawn failed immediately with TypeError: cannot pickle '_thread.lock' object — and then hung the interpreter forever (the non-daemon logger subprocess never got its shutdown sentinel).

Fixes #264, fixes #271.

Root causes

Three distinct pickling defects blocked spawn (which pickles Process args), verified by attribute-level probing:

Defect	Error
Worker.__init__ eagerly built ApiClient() → httpx.Client + RESTClientObject._reset_lock	TypeError: cannot pickle '_thread.lock' object
Worker._pending_tasks_lock raw threading.Lock on the instance	same
@worker_task rebinds the module-level name to its wrapper while Worker holds the original function	PicklingError: it's not the same object as module.name

The eager ApiClient also performed TLS/auth in the parent before fork() — precisely the native state that makes forked children SIGSEGV on macOS (Apple frameworks; see CPython bpo-33725, which is why CPython itself defaults to spawn on macOS since 3.8). The restart path is worse: the monitor re-forks from a background thread every 5s.

Changes

Worker pickling

• Worker.api_client is now a lazy property (created in the process that uses it; setter preserved for injection)
• __getstate__/__setstate__ exclude process-local state (lock, background loop, pending async futures) and rebuild it in the child
• Decorated functions pickle as importable references (module, qualname, __wrapped__ depth) resolved in the child; plain module-level functions keep default pickling; nested functions/lambdas fail fast in the parent with actionable guidance

Start method

• Default is now spawn on all platforms (was fork everywhere except Windows). This reinstates the direction of efeabbfa, whose revert reason ("spawn requires all Process arguments to be picklable") is resolved by this PR. CONDUCTOR_MP_START_METHOD=fork remains for deployments relying on copy-on-write inheritance

Robustness/diagnostics

• start_processes() cleans up already-started subprocesses on failure and raises with guidance (fixes the hang)
• Signal deaths log the signal number + PYTHONFAULTHANDLER=1 hint instead of restarting silently
• docs/WORKER.md no longer tells macOS users to force set_start_method('fork') (that advice reproduced the bug); documents spawn requirements (__main__ guard, module-level workers, picklable args)

Test hygiene

• test_worker_config_integration.py replaced sys.modules['...task_handler'] with a Mock at import time, poisoning the whole pytest session (invisible under fork, fatal under spawn) — removed

Verification

• examples/spawn_safety_probe.py: before → 3 FAIL (both pickle errors + spawn child failure); after → ALL CHECKS PASSED on Python 3.11.14 and 3.13.7 (macOS 15.1, Apple Silicon)
• Live TaskHandler repro (worker + MetricsSettings(http_port=...), Orkes-style HTTPS auth config) on Python 3.11: worker pid stable, restart_count=0, metrics provider serving, clean shutdown
• New regression tests include executing a pickled @worker_task Worker inside a real spawn child process
• Full suite: unit 781 passed / 1 skipped; backward-compat + serdes + chaos + integration 1073 passed / 139 skipped; 0 failures

Behavior change / migration

Linux default changes fork → spawn. Requirements under spawn are the standard multiprocessing rules: if __name__ == '__main__': guard, module-level workers, picklable TaskHandler args. Anything relying on fork COW inheritance: set CONDUCTOR_MP_START_METHOD=fork.

Related work

Overlaps with #<branch fix/worker-task-spawn-safety> (@Kowser) — same diagnosis for the Worker pickling; this PR additionally removes the parent-side TLS-before-fork poison (lazy ApiClient), resolves functions via importlib (covers import_modules=[...] workers whose modules aren't pre-imported in the child), changes the default start method, fixes the start-failure hang, and adds diagnostics. Happy to reconcile the two.

[codecov]

Codecov Report

❌ Patch coverage is 88.75000% with 9 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/conductor/client/automator/task_handler.py	58.33%	5 Missing ⚠️
src/conductor/client/worker/worker.py	94.11%	4 Missing ⚠️

Files with missing lines	Coverage Δ
src/conductor/client/worker/worker.py	62.69% <94.11%> (+6.58%)	⬆️
src/conductor/client/automator/task_handler.py	78.23% <58.33%> (-1.04%)	⬇️

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.