Is there any way to capture the executorWorker's errors?

[matiaslin]

Hi,

I'm using the OrchestratorConfig along with the executorWorker executable.

If the executorWorker running in a separate MPI process encounters an error, the program will throw an exception. For example:

IRuntime::deserializeCudaEngine: Error Code 6: API Usage Error (The engine plan file is generated on an incompatible device, expecting...

As of today, it seems that these errors are simply ignored, see the following snippet:

       MPICHECK(MPI_Comm_spawn(workerExecPath.c_str(), MPI_ARGV_NULL, tp * pp * cp, mpiInfo, 0, MPI_COMM_SELF,
                 &intercomm, MPI_ERRCODES_IGNORE));

Is there a way to capture these exceptions happening in executorWorker?

Thank you.

[Advait251206]

At the moment, there isn't a built-in mechanism in OrchestratorConfig/executorWorker to propagate exceptions thrown inside the spawned executorWorker process back to the parent process.

The MPI_Comm_spawn call itself is using MPI_ERRCODES_IGNORE, but it's worth noting that this only ignores the spawn-time error codes returned by MPI. Runtime exceptions that occur later inside executorWorker (such as TensorRT deserialization failures like IRuntime::deserializeCudaEngine) happen after the worker has been successfully launched, so MPI does not automatically forward those exceptions to the parent process.

If you need visibility into these failures, your options are generally:

• Modify executorWorker to catch exceptions at the top level, log the error, and communicate the failure back to the orchestrator (for example, via MPI messages, an exit code, or another IPC mechanism).
• Inspect the worker's stderr/stdout logs, if your MPI launcher preserves them. Many MPI implementations (Open MPI, MPICH, Intel MPI, etc.) will print the worker's uncaught exception or stderr output, although this behavior depends on the launcher and configuration.
• Customize the orchestration layer if you're building from source, so that the worker reports structured error information before terminating.

If your goal is specifically to detect cases like an incompatible TensorRT engine, another approach is to perform validation before spawning the worker (where feasible), or ensure that the engine is built for the target GPU architecture to avoid the runtime deserialization error in the first place.

So, with the current implementation, there isn't an API that allows you to directly "capture" exceptions thrown inside executorWorker from the parent process—you would need to add explicit error-reporting in the worker or extend the orchestration code to propagate those failures.

---

If this answer helped or pointed you in the right direction, I'd appreciate it if you could mark it as the accepted answer so it's easier for others with the same issue to find.

Also, if you found my contribution useful, I'd appreciate it if you could check out my GitHub profile, follow me, and star any repositories you find interesting.

GitHub: https://github.com/Advait251206