Add health metrics (Part 1)

[Nadine-H]

Part of #2736 - adding http metrics.

Chose to implement these metrics as custom rather than relying on standard fastapi metrics via instrumentation libraries. Reason being, we need to label project_name in order to get metrics at endpoint and project level. Standard libraries like prometheus-fastapi-instrumentator and opentelemetry-instrumentation-fastapi don't allow adding custom dynamic labels to their default metrics.

Note:

• When adding these metrics, they were overwriting the existing custom metrics since they're both emitted on the same /metrics endpoint.
• I had to fetch the http metrics in the metrics router endpoint, and combine them with the custom metrics.
• The current approach is emitting the metrics as plain text on the /metrics endpoint, however the more standard Prometheus way is to register all the metrics using prometheus_client which uses a global registry to collect the metrics from all the app and takes care of emitting them on the /metrics endpoint.
• I did this implementation as a quick solution, however I will look into using prometheus_client in part 2 of this ticket for the additional custom metrics, if it makes adding new metrics easier, in which case it might require some refactoring of the existing prometheus metrics code.

[Nadine-H]

Realized that the handler label in the metrics doesn't resolve the project_name in the endpoint path:
dstack_server_http_requests_total{handler="/api/projects/{project_name}/get",method="POST",status="2xx"} 1.0

I think it's because the endpoint path is defined as a pattern, trying to see how I can add a custom label for project_name using prometheus-fastapi-instrumentator

[Nadine-H]

Converting the PR to draft while I address the issue above, so no need for reviews in the meantime.

[dakota-shop]

Instead of manually parsing this out, I wonder if the parsing might already be done for us... anything interesting in the request object, e.g. in path_params?

[Nadine-H]

No path_params isn't available at the time the middleware is called. Middlewares are executed before FastAPI routes the request to an endpoint, so we only have access to the raw path.

A "middleware" is a function that works with every request before it is processed by any specific path operation. And also with every response before returning it.

[Nadine-H]

This is the request object when it gets to the middleware:
request.__dict__ {'scope': {'type': 'http', 'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'http_version': '1.1', 'server': ('127.0.0.1', 3000), 'client': ('127.0.0.1', 55612), 'scheme': 'http', 'method': 'POST', 'root_path': '', 'path': '/api/project/main/repos/get', 'raw_path': b'/api/project/main/repos/get', 'query_string': b'', 'headers': [(b'host', b'127.0.0.1:3000'), (b'user-agent', b'python-requests/2.32.3'), (b'accept-encoding', b'gzip, deflate, br'), (b'accept', b'*/*'), (b'connection', b'keep-alive'), (b'authorization', b'Bearer REDACTED'), (b'x-api-version', b'0.0.0'), (b'content-type', b'application/json'), (b'content-length', b'60')], 'state': {}, 'app': <fastapi.applications.FastAPI object at 0x1182e7680>}, '_receive': <function BaseHTTPMiddleware.__call__.<locals>.call_next.<locals>.receive_or_disconnect at 0x12b1494e0>, '_send': <function empty_send at 0x1186fbe20>, '_stream_consumed': False, '_is_disconnected': False, '_form': None, '_wrapped_rcv_disconnected': False, '_wrapped_rcv_consumed': False, '_wrapped_rc_stream': <async_generator object Request.stream at 0x12b151380>}

[un-def]

LGTM