get Traceback Information when use subprocesses

[bitranox]

• I'm submitting a ...
  • feature request

get Traceback information when use_signals=False, possibly by using tblib
(see https://pypi.python.org/pypi/tblib)

• What is the current behavior?
  there is no Traceback Information when use_signals=False, because that cant be pickled from the process
  just the Exception is raised

• What is the motivation / use case for changing the behavior?
  give correct traceback informaton for debugging

• Please tell us about your environment:
  ALL

[eponraj27392]

@bitranox
I am facing the same issue here.
logger information is not available when i try to run the wrapper function concurrently (sub-process)
How can i make the logger and print information available ?

[bitranox]

To set up Python logging for subprocesses using a listener approach, you can use a logging configuration that allows each subprocess to send its log records to a listener running in the parent process. This is useful when you have multiple subprocesses and want them all to log in a coordinated way.

Here’s how you can set up a multiprocessing-safe logging configuration:

Step-by-Step Guide
Set up a logging listener in the parent process: The parent process runs a logging server that listens for logging messages from subprocesses.
Configure logging in subprocesses to send logs to the listener: Each subprocess sends its logging messages to the listener via a QueueHandler.

Listener in Parent Process: A QueueListener is set up in the parent process. It listens for log records that are sent from subprocesses via the log_queue.

Subprocess Logging: In each subprocess, we configure the logger to use a QueueHandler that forwards log messages to the log_queue. These messages are handled by the listener in the parent process.

QueueHandler: Used in each subprocess to send log records to a queue.
QueueListener: Used in the parent process to listen for log records from the queue and process them using the configured handlers.
multiprocessing.Queue: A queue that allows safe communication between processes.
This setup ensures that logs from all subprocesses are handled in a central location, which is helpful for debugging and monitoring parallel execution.