pod creation / reconciliation takes a really long time

[rokroskar]

Two problems with delays in handling session state:

• When the user clicks "Launch", the AmaltheaSession object is created right away - but it can take ~1 minute for the pod to appear.
• When a user stops a session, the pod is removed immediately but the session is shown as "deleting" for a long time; if the page is refreshed, the session may still be shown as running for a while (several minutes)

[leafty]

We are not currently collecting prometheus metrics from the amalthea controller (deploy/renku-amalthea-sessions-controller-manager).
Maybe once we have them (next release, or we can maybe push this to earlier), we could see if the reconcile loops are getting swamped in production.

[leafty]

Context: the metrics are served up, but there is currently no scrape annotation, so the metrics are not collected. The #1094 PR adds annotations, so we will get metrics collected (new custom metrics but also work queue and reconcile loop metrics from the kubebuilder operator framework).