feat: split console and file logging levels

[FabianHofmann]

Changes proposed in this Pull Request

Splits the logging level between the console and the rule log file so that parallel atlite-based rules stay readable on the console while the log file still captures full INFO output.

configure_logging now sets per-handler levels:

• logging.level — root log level (unchanged default: INFO)
• logging.console_level — stream handler (new, default: WARNING)
• logging.file_level — file handler (new, defaults to level)

Users who want the old behaviour set logging.console_level: INFO.

Follow-up on the discussion in #2144 (comment). The xarray MultiIndex fix has been dropped from this PR and is now tracked upstream in PyPSA/atlite#501.

Checklist

Required:

• Changes are tested locally and behave as expected.
• Code and workflow changes are documented.
• A release note entry is added to doc/release_notes.rst.

[coroa]

I disagree with removing the log messages. And i would prefer solving the multi index warning at atlite level.

[coroa]

Fixed in PyPSA/atlite#501

[FabianHofmann]

I disagree with removing the log messages.

The reason being? With the faster and parallel execution of the atlite-based rules, you can't see which log refers to which conversion. The main time benchmarking logs (which state the tech) we keep. Also snakemake rules have descriptive messages nowadays. I don't see a reason to keep such kind of descriptive logs. If it is about debugging we should change the logger mode at least.

[coroa]

I think instead of removing log messages we should change where they are displayed.

Ideally, we would create a logging sink that the snakemake logger plugin pulls up and then instead of showing the log messages on the console only forward them there.

The console would then be replaced with an interactive tui where you see which jobs are running and you can choose to see the logs of them.

As immediate solution, maybe we decide instead that info messages should only go into the log file and only warning messages are printed onto the console. I'd be fine with such a change happening globally in PyPSA-Eur (or locally for that matter)

The only two things missing then to make log files more helpful would be: A bridge that ensures that progressbars also write a minimal version into the log, and a default exception trap that also logs exceptions into the log.

[FabianHofmann]

@coroa pr is taking a different approach now, see updated summary above.