Keep last metrics for finished jobs

[un-def]

The retention window is 1800 seconds (last 30 minutes) by default, configurable via the DSTACK_SERVER_METRICS_WINDOW_SECONDS environment variable.

Closes: #2618

[peterschmidt85]

Anything required on the UI's side?

[r4victor]

So the last 30 minutes of logs are stored forever after the run is finished?

@peterschmidt85 is this what you proposed or having logs for some time after runs finish would be sufficient?

[peterschmidt85]

Ideally of course we could introduce a TTL but I assume we don't support it yet for logs too?

[r4victor]

@peterschmidt85, sorry I meant metrics of course not logs. We already had a ttl for metrics:

dstack/src/dstack/_internal/server/settings.py

Line 48 in c521a57

SERVER_METRICS_TTL_SECONDS = int(os.getenv("DSTACK_SERVER_METRICS_TTL_SECONDS", 3600))

And the ttl applied to finished runs as well. So you could see metrics of finished runs for this ttl.

So what this PR does I believe is keeps last metrics of finished runs forever.

[peterschmidt85]

That was TTL for metrics of a live run. For finished run, I guess it makes sense to set TTL as a week or so. We discussed that with @un-def no?

[r4victor]

That was TTL for metrics of a live run.

Well, it applied to all runs before. If the idea is to introduce different TTLs for active and finished runs, it's not what the PR does.

[un-def]

That was TTL for metrics of a live run. For finished run, I guess it makes sense to set TTL as a week or so. We discussed that with @un-def no?

I've just replaced the retention window with a simpler solution using two separate TTL settings for running and finished jobs.

By default:

• DSTACK_SERVER_METRICS_RUNNING_TTL_SECONDS — 1 hour
• DSTACK_SERVER_METRICS_FINISHED_TTL_SECONDS — 1 week