During long runs, generate_stitch.py appears to be holding objects in memory after they've been processed.
Possible "band-aid" solution: close and recreate multiprocessing pool for every day of processing. This wouldn't solve the root cause but would at least limit the accumulated objects. For reference, 5 days of processing has left most processes in the pool with ~5GB of memory usage (each). With a pool size of 20, this is basically utilizing all available memory.
During long runs, generate_stitch.py appears to be holding objects in memory after they've been processed.
Possible "band-aid" solution: close and recreate multiprocessing pool for every day of processing. This wouldn't solve the root cause but would at least limit the accumulated objects. For reference, 5 days of processing has left most processes in the pool with ~5GB of memory usage (each). With a pool size of 20, this is basically utilizing all available memory.