Summary
When an actor's latest snapshot is a local checkpoint, AssignWorkerStep.findFreeWorker hard-restricts candidate workers to the node(s) listed in LatestSnapshotInfo.Local.NodeVmsWithLocalSnapshots (correctly — a local checkpoint only exists on the node that wrote it). But when that restriction is what eliminates all candidates, the returned error is the generic:
FailedPrecondition: no free workers available
This is misleading in both real failure modes:
- The snapshot's node has no free workers — free workers exist elsewhere in eligible pools, but they cannot restore the local checkpoint. The error reads as cluster-wide capacity exhaustion when it is actually a locality conflict.
- The snapshot's node is gone (scaled down, failed) — the actor's checkpointed state is unreachable, which is much more serious than "no free workers", and the operator gets no signal that this is the situation.
Proposed fix
When no worker is picked and a node restriction was in effect, report a diagnostic error that distinguishes the cases, e.g.:
actor's local snapshot is on node(s) [gke-node-abc] but all 4 eligible workers on those nodes are busy; 12 free workers on other nodes cannot restore the local snapshotactor's local snapshot is on node(s) [gke-node-abc] but no eligible workers exist on those nodes (the node(s) may have been removed); 12 free workers on other nodes cannot restore the local snapshot
The status code stays FailedPrecondition; only the message improves.
Non-goals
This does not make the actor schedulable in either case. Real remediation (falling back to an external snapshot when one exists, snapshot placement tracking with liveness, load-shedding via snapshot replication) requires the snapshot data model to record external and local snapshot state simultaneously — today's SnapshotInfo oneof erases the external URI on pause. That redesign is a separate proposal; this issue only stops the error from pointing operators at the wrong problem.
Summary
When an actor's latest snapshot is a local checkpoint,
AssignWorkerStep.findFreeWorkerhard-restricts candidate workers to the node(s) listed inLatestSnapshotInfo.Local.NodeVmsWithLocalSnapshots(correctly — a local checkpoint only exists on the node that wrote it). But when that restriction is what eliminates all candidates, the returned error is the generic:This is misleading in both real failure modes:
Proposed fix
When no worker is picked and a node restriction was in effect, report a diagnostic error that distinguishes the cases, e.g.:
actor's local snapshot is on node(s) [gke-node-abc] but all 4 eligible workers on those nodes are busy; 12 free workers on other nodes cannot restore the local snapshotactor's local snapshot is on node(s) [gke-node-abc] but no eligible workers exist on those nodes (the node(s) may have been removed); 12 free workers on other nodes cannot restore the local snapshotThe status code stays
FailedPrecondition; only the message improves.Non-goals
This does not make the actor schedulable in either case. Real remediation (falling back to an external snapshot when one exists, snapshot placement tracking with liveness, load-shedding via snapshot replication) requires the snapshot data model to record external and local snapshot state simultaneously — today's
SnapshotInfooneof erases the external URI on pause. That redesign is a separate proposal; this issue only stops the error from pointing operators at the wrong problem.