@infotroph sorry for the late reply, this patch would make restart path ignore the control$run_parallel. I also noticed the existing non-restart foreach path already does that. Would you prefer I keep this PR narrow and fix only the restart path, or update both foreach config-writing paths so run_parallel is respected consistently?

existing non-restart foreach path already does that

Good catch. If it's easy to fix in this PR go for it, but not required.

But more importantly for this PR, how sure are you that this actually fixes the issue? I don't see either approach printing to the console:

> f <- function(msg) {
+   Sys.sleep(2)
+   PEcAn.logger::logger.warn(msg)
+   msg
+ }
> library(foreach)
> cl <- parallel::makeCluster(7)
> doSNOW::registerDoSNOW(cl)
> res_foreach <- foreach::foreach(x=LETTERS[1:10]) %dopar% {f(x)}
> parallel::stopCluster(cl)
> foreach::registerDoSEQ()
> 
> future::plan(future::multicore, workers = 7)
> res_furrr <- furrr::future_map(LETTERS[1:10], f)
>

Note that no logger output is shown in either case.

You mentioned using logger.setOutputFile() as part of your test -- can you show a reprex? Unless the SDA process does its own log redirection I doubt most users will be incorporating that into their runs.

@infotroph After digging a bit more, I think you’re right, switching from furrr to foreach/doSNOW alone doesn’t actually prove the issue is fixed. I don’t see the SDA/workflow path doing any explicit capture or replay of worker stderr, so even if the worker emits PEcAn.logger messages, that doesn’t guarantee they show up in the main console in normal runs.

A probable fix seems like it would be to stop relying on worker console output entirely and instead return log messages from each worker as part of the task result, then print them in the parent process after collection. In other words, have the worker return something like list(result = ..., logs = ...), and let the main process emit those logs. That should work regardless of whether the backend is furrr or foreach, and it would avoid depending on backend-specific stdout/stderr behavior.

It wouldn’t be truly live streaming, but it should make the messages reliably visible in normal SDA runs and is a lot easier to reason about and test.