Skip to content

feat: gate Kubernetes workers until weight update#48

Draft
S1ro1 wants to merge 3 commits into
mainfrom
exp/mx-admin-fanout
Draft

feat: gate Kubernetes workers until weight update#48
S1ro1 wants to merge 3 commits into
mainfrom
exp/mx-admin-fanout

Conversation

@S1ro1

@S1ro1 S1ro1 commented Jul 23, 2026

Copy link
Copy Markdown

Summary

Router-side PoC for safe inference autoscaling during NIXL + ModelExpress RL runs.

  • fans /pause, /update_weights, and /resume out to backend admin servers
  • adds a separate usable worker state in addition to health
  • registers Kubernetes-discovered workers as healthy but unusable
  • excludes unusable workers from every routing policy, readiness, and health availability
  • marks a worker usable only after its first successful /update_weights
  • exposes is_usable from /workers for Prime-RL's transfer coordinator
  • reconciles static DNS endpoints with Kubernetes-discovered pod-IP endpoints
  • processes discovery events concurrently and retries cleanly after watcher errors

Stack

Current PoC limitations

  • usability is in-memory and is lost when the router restarts
  • replacement pods that reuse a static worker URL need a stronger identity/re-registration model
  • fanout is fail-stop: any backend admin failure fails the synchronization
  • this is readiness gating, not request-level retry or transparent recovery for requests already executing on a killed worker

Checks

  • cargo check
  • complete cargo test suite: all tests passed, one pre-existing ignored test
  • end-to-end staggered 2×TP8 → 3×TP8 H200 run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant