When there is autoscaling that takes a while (more than a minute or so) the session shows up as failed

[olevski]

When the session gets scheduled the failed status clears. But it is not clear what is going on to users.

Seen in data svc 0.76.0.

I think in amalthea we have a timeout for how long we wait for the session to be scheduled before we mark it as failed. This may have to be adjusted. I also thought that Amalthea tries to catch the pod events that indicate that autoscaling was started. But it does not or the event names have changed.

[leafty]

Wrong logic in amalthea:

amalthea/internal/controller/children.go

Line 579 in 1488d77

func EventsInferedState(ctx context.Context, cr *amaltheadevv1alpha1.AmaltheaSession, client client.Reader) (EventsInferedStateResult, error) {

Scheduled or TriggeredScaleUp events will not appear in any specific order compared to FailedScheduling events. If any of Scheduled or TriggeredScaleUp happens, the pod should get scheduled soon.

[leafty]

@eikek see above: events do not have a specific ordering, which means the current logic is incorrect.

[eikek]

@leafty What about a case, when there is a TriggeredScaleUp but then scaling fails due to resource issues (or whatever)?

[leafty]

@leafty What about a case, when there is a TriggeredScaleUp but then scaling fails due to resource issues (or whatever)?

I don't know exactly what are the timeout we have, but we can try to mark failed if the last relevant scheduling event is too old.

[leafty]

@eikek I think you may be confused about how k8s pod scheduling and node autoscaling works.

Both pod scheduling and node auto-scaling are handled by controllers which run their own reconcile loop (similar to amalthea).
In the logs posted above, this is what is happening (simplified):

1. The pod is created (from the StatefulSet)
2. The pod creation events triggers the reconcile loop of default-scheduler (the default pod scheduler -- note that specific k8s infra like knative may use another pod scheduler)
   a. The scheduler sees that no eligible node can take on the pod workload => it emits a FailedScheduling event
3. The default-scheduler listens to pod events, node events and other cluster events. This means that the pod will try to be scheduled again multiple times => we get more FailedScheduling events
4. The cluster-autoscaler controller (separate controller) also listens to pod events and node events
   a. The autoscaler sees that the pod cannot be scheduled and decides to scale up the relevant node pool => it emits a TriggeredScaleUp event
   b. The new node will start up and emit node events => it will trigger the default-scheduler reconcile loop
5. While the new node is being brought up the events trigger more reconcile loops from default-scheduler
   a. The new node is not immediately ready => the pod cannot be scheduled while the node is in NotReady condition => more FailedScheduling events are emitted
6. The new node is finally ready => the default-scheduler schedules the pod onto it => a PodScheduled event is emitted

As we can see here, the TriggeredScaleUp event WILL be followed by more FailedScheduling events, and this is completely expected.