feat: fetch remote FirecREST logs into sessions

[SalimKayal]

Summary

Fetch remote Slurm session stdout/stderr logs via FirecREST and stream them into the controller pod's stdout, making them visible through kubectl logs.

Motivations and Context

Remote sessions running on Slurm clusters write their output to files on the cluster filesystem (e.g., slurm-<jobid>.out). Previously, these logs were inaccessible from the Kubernetes sidecar, making it hard for users and operators to debug running or completed remote sessions. This change polls the log files through FirecREST's filesystem view API and forwards any new lines to the pod's stdout.

Changes

• Added fetchSessionLogs — polls remote stdout and stderr files via GetViewFilesystemSystemNameOpsViewGet and writes complete lines to os.Stdout prefixed with [session/stdout] or [session/stderr].
• Added fetchLogStream — handles a single stream: buffered reading, byte-offset tracking, and line-by-line flushing so partial lines survive across polls.
• Extended savedState — now persists stdout_path, stderr_path, stdout_offset, and stderr_offset alongside the existing job_id.
• Renamed saveJobID → saveState and recoverJobID → recoverState — updated to serialize and deserialize the new fields.
• Updated periodicSessionStatus — calls fetchSessionLogs on each status tick and persists offsets to disk.
• Updated Start — constructs expected Slurm log file paths after job submission and calls saveState.

Notes

• Log fetch errors are logged at Warn level and do not fail the status loop; missing files early in the job lifecycle are silently ignored.
• The buffer/offset approach ensures we don't duplicate lines across polls or after a controller restart.

[leafty]

Why read from the file system when stdout and stderr are returned from GET /compute/{system_name}/jobs/{job_id}/metadata?

Get metadata of a job by {job_id}

[SalimKayal]

Why read from the file system when stdout and stderr are returned from GET /compute/{system_name}/jobs/{job_id}/metadata?

thanks I missed this one.

[leafty]

Why read from the file system when stdout and stderr are returned from GET /compute/{system_name}/jobs/{job_id}/metadata?

thanks I missed this one.

There may be a good case for reading from the file system, just making sure we are considering the simplest option first.

[SalimKayal]

I read the path for the stdout/err files from the metadata with a fallback for slurm defaults in 594448c.

[SalimKayal]

FYI this is what FirecREST runs to get the metadata. https://github.com/eth-cscs/firecrest-v2/blob/master/src/lib/scheduler_clients/slurm/cli_commands/sacct_job_metadata_command.py#L12