fix(cicd): sequence C-Node drain, P-Node, then C-Node redeploy with rehydration hold by abs2023 · Pull Request #714 · MorpheusAIs/Morpheus-Lumerin-Node

abs2023 · 2026-04-22T18:55:51Z

Summary

Promotes #713 from dev → test so we can exercise the new deployment sequencing against the dev infrastructure before cutting v7.0.0 to main.

What's in this PR

Only one commit ahead of test:

68e39db — fix(cicd): sequence C-Node drain, P-Node, then C-Node redeploy with rehydration hold (merged via fix(cicd): sequence C-Node drain, P-Node, then C-Node redeploy with rehydration hold #713)

Everything else that was previously in dev (TEE Phase 2, docs, version bump to 7, ECS wait-timing fix) is already live on test via #710.

What changes on merge to test

The test branch only runs the C-Node deploy path (no P-Node exists in dev infra). The reworked workflow will still exercise:

Drain-Morpheus-C-Node job — discovers test/dev-env TGs and deregisters current targets.
Deploy-to-Morpheus-P-Node is needs-skipped; dependency resolution treats it as success.
Deploy-to-Morpheus-C-Node:
- update-service with --health-check-grace-period-seconds 600
- poll new task ENI IP
- deregister IP from TGs
- sleep 90s (default cnode_rehydration_wait_secs)
- re-register + wait target-in-service
- public /healthcheck version-match (existing logic)

Note on dev env: the dev C-Node currently has switches.efs_storage = true (persistent BadgerDB on EFS). This means the 90s hold is technically overkill for dev — rehydration has no work to do since state is persistent. That's fine; the hold is a no-op and lets us verify the new workflow paths behave correctly in dev before exercising them on prd (where BadgerDB is ephemeral and the hold matters).

What to review

Workflow YAML structure (parallelism, needs, if gates)
The two shell blocks (Drain-Morpheus-C-Node + the controlled-redeploy section in Deploy-to-Morpheus-C-Node) for correctness and idempotency
Confirm the cnode_rehydration_wait_secs default and the guard-against-non-numeric logic
Verify IAM: dev IAM policy already includes ELBv2ReadOnly + ELBv2TargetRegistration (applied from Morpheus-Infra)

Test plan (after you merge to test)

Merge triggers the deploy workflow against dev env.
Confirm Drain-Morpheus-C-Node job lists the correct TGs for svc-dev-router and successfully deregisters.
Confirm Deploy-to-Morpheus-C-Node:
- finds the new task with a fresh task-definition ARN
- resolves an ENI IP
- deregister → 90s hold → re-register completes
- target-in-service confirms healthy
- public /healthcheck at router.dev.mor.org:8082 returns the new version
Observe the dev C-Node's BadgerDB during the hold — should be a no-op (EFS-persistent), confirming the workflow is benign when rehydration isn't needed.
If green, cut v7.0.0 PR from test → main to exercise the full sequence on prd.

…ehydration hold Prevents the mid-deploy "messy outage" we've seen when the C-Node and P-Node restart simultaneously and the C-Node BadgerDB ends up referencing sessions whose upstream providers have just cycled. Changes to .github/workflows/build.yml: - New job `Drain-Morpheus-C-Node`: discovers the C-Node service's target groups and deregisters all current targets before any node restarts. connection_termination=true on the router TGs closes live TCP sessions promptly so upstream failures arrive as clean 503s instead of half-dead mid-stream hangs. - `Deploy-to-Morpheus-P-Node` now depends on `Drain-Morpheus-C-Node` so the C-Node NLB is quiet before provider traffic is disrupted. - `Deploy-to-Morpheus-C-Node` now depends on both the drain job and the P-Node deploy (skipped on `test`, which is still treated as success by GitHub Actions dependency resolution). - The C-Node deploy step itself is rewritten to: 1. Register the new task def with `--health-check-grace-period-seconds 600` so the ECS deployment circuit breaker tolerates the deregister window. 2. Poll until a task on the NEW task definition is RUNNING and has an ENI IP. 3. Deregister that IP from every target group. 4. Sleep `cnode_rehydration_wait_secs` (default 90s) so the proxy-router's BadgerDB rehydration loop can catch up from on-chain state (ephemeral BadgerDB on prd, EFS-backed on dev). 5. Re-register the IP to every TG and wait for `target-in-service`. 6. Fall through to the existing public `/healthcheck` version-match loop. - New workflow_dispatch input `cnode_rehydration_wait_secs` (default "90") lets operators tune the hold window per run. Companion changes (separate PR in Morpheus-Infra): - GitHub Actions IAM policy now grants ELBv2 Describe + Register/Deregister on Target Groups so the drain/register steps have the permissions they need. - Planning doc CICD_HA_IMPROVEMENTS_PLAN.md captures the deferred items (P-Node HA, graceful shutdown, API GW retry tuning, persistent BadgerDB promotion) that this change does NOT address. Made-with: Cursor

…ehydration hold (#713) ## Summary Rewrites the C-Node / P-Node deployment sequencing in `build.yml` so the C-Node NLB is fully drained **before** any provider restart, and so the new C-Node task is held out of the load balancer for a configurable rehydration window after it comes up. Addresses the recurring "messy 5-minute" outage pattern where simultaneous C-Node and P-Node restarts leave the C-Node's BadgerDB with orphaned session state and require manual cleanup. ## Changes ### New job: `Drain-Morpheus-C-Node` - Discovers the C-Node service's target groups dynamically (no hard-coded ARNs). - Calls `elbv2 deregister-targets` on every current target. - Waits 45s for deregistration to propagate across AZs. The router TGs have `connection_termination=true` and `deregistration_delay=0`, so live TCP sessions close promptly. - Exports `tg_arns` + `drained_ips` as outputs for the C-Node deploy job. ### Reworked `Deploy-to-Morpheus-C-Node` Now depends on both the drain job and the P-Node deploy, and the `update-service` call is followed by a controlled-traffic sequence: 1. `aws ecs update-service --health-check-grace-period-seconds 600` so the ECS deployment circuit breaker tolerates the deregister window. 2. Poll (up to 15 min) until a task running the **new** task definition is `RUNNING` with an ENI IP. 3. Deregister that IP from every target group. 4. Sleep `cnode_rehydration_wait_secs` (default 90s) to let the proxy-router's BadgerDB rehydration loop catch up from on-chain state. Important for prd where BadgerDB is ephemeral; harmless for dev where it's EFS-backed. 5. Re-register the IP to every TG, then `aws elbv2 wait target-in-service` per TG. 6. Fall through to the existing public `/healthcheck` version-match loop. ### `Deploy-to-Morpheus-P-Node` Gated behind `Drain-Morpheus-C-Node` so the C-Node NLB is already quiet before the provider restarts and briefly takes hosted models offline. ### New workflow_dispatch input - `cnode_rehydration_wait_secs` (default `"90"`): operator-tunable hold window. ### Titan deploy Unchanged — still runs independently and in parallel with the P-Node after container build, as agreed. ## Expected behavior post-merge (prd) | Phase | Duration | What the user sees | |---|---|---| | Drain | ~45s | API GW upstream errors (503), clean TCP close | | P-Node restart | ~2-3 min | Hosted-model sessions briefly unavailable (single-provider SPOF, known) | | C-Node old task stop | ~2 min | Same — NLB already empty | | C-Node new task up | ~30s | Task running, still deregistered | | Rehydration hold | 90s | Task rehydrating BadgerDB from chain, still deregistered | | Re-register + healthy | ~90s | NLB targets come healthy, API GW begins forwarding again | Total: ~6-8 min of deliberate, bounded outage instead of the previous 5 min of messy failure followed by 30 min of manual recovery. ## Companion changes (separate repo) `Morpheus-Infra`: - `01_iam_role_gh_actions.tf`: grants `ELBv2ReadOnly` + `ELBv2TargetRegistration` so the new drain/register steps can authenticate. - `.ai-docs/CICD_HA_IMPROVEMENTS_PLAN.md`: captures deferred follow-ups (persistent BadgerDB promotion, regional P-Node HA, `/readyz` split, graceful shutdown, API GW retry tuning). Both IAM policies have already been applied to dev + prd via terraform. ## Test plan - [x] YAML syntax validated with `yaml.safe_load` - [x] Go build of proxy-router on current HEAD passes (cross-compiled linux/amd64) - [ ] Merge to `dev` → exercise `test` branch deploy (which runs only the C-Node path — P-Node job is `needs`-skipped for `test`) and confirm: - Drain job succeeds and lists target groups - New C-Node task registers a new task-def ENI IP - Deregister → 90s hold → re-register cycle completes without ECS rollback - `/healthcheck` version-match passes - [ ] On prd (next release), confirm P-Node job runs **after** drain, and that the C-Node completes the rehydration-hold sequence before returning traffic. - [ ] Observe v6.1.x → v7.0.x deploy cleanly with no manual BadgerDB intervention. Made with [Cursor](https://cursor.com)

…deploy needs (#715) ## Summary Hotfix for the first v7 `test`-branch deployment attempt ([Actions run #1447 / run id 24796856145](https://github.com/MorpheusAIs/Morpheus-Lumerin-Node/actions/runs/24796856145)). The previous PR (#714 → #713) introduced a new job sequence: ``` Drain-Morpheus-C-Node → Deploy-to-Morpheus-P-Node → Deploy-to-Morpheus-C-Node ``` On push to `test`, `Deploy-to-Morpheus-P-Node` is correctly **skipped** (its own `if` restricts it to `main` — no dev P-Node exists). That skip then propagated onto `Deploy-to-Morpheus-C-Node` through the implicit `success()` guard that every job has when no explicit `if` tolerates a skipped dependency. Net effect: - `Drain-Morpheus-C-Node` ✅ removed the running dev C-Node task from both NLB target groups. - `Deploy-to-Morpheus-C-Node` ❌ silently skipped — no new task registered, no re-register of the existing task. - Dev C-Node endpoints (`router.dev.mor.org:8082`/`:8545`) stopped serving traffic until we manually re-registered the old task to the TGs. ## What this PR changes `.github/workflows/build.yml`, `Deploy-to-Morpheus-C-Node` job only: ### Explicit `if` guard that tolerates the P-Node skip ```yaml if: | !cancelled() && github.repository == 'MorpheusAIs/Morpheus-Lumerin-Node' && needs.GHCR-Build-and-Push.result == 'success' && needs.Drain-Morpheus-C-Node.result == 'success' && (needs.Deploy-to-Morpheus-P-Node.result == 'success' || needs.Deploy-to-Morpheus-P-Node.result == 'skipped') && ( (github.event_name == 'push' && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/test')) || (github.event_name == 'workflow_dispatch' && github.event.inputs.build_all_os == 'true' && github.event.inputs.create_deployment == 'true') ) ``` - Still requires real success from the drain + GHCR jobs. - Explicitly accepts `success` OR `skipped` for the P-Node dependency. - `!cancelled()` short-circuits the job if someone cancels the run, so we don't try to redeploy a half-cancelled sequence. ### Comment rewrite The previous inline comment claimed skipped jobs are treated as successful for dependency resolution. That's the opposite of how GitHub Actions actually behaves — the note is replaced with an accurate explanation and a reference to this incident so future-us (or other maintainers) won't step on the same rake. ## What this PR does NOT change - No change to the drain job or the controlled-traffic C-Node deploy sequence (register→dereg→hold→rereg→wait). - No change to the P-Node job or its `if` gating. - No change to any other workflow or deploy script. This is a pure guard-correctness fix. ## Test plan - [x] YAML syntax validated (`yaml.safe_load`). - [x] Manually walked through `needs` outcomes: - On push to `test`: P-Node `skipped`, drain + GHCR `success` → C-Node **runs**. ✅ - On push to `main`: all three `success` → C-Node **runs**. ✅ - Drain fails → C-Node **skipped** (won't redeploy while the TG state is ambiguous). ✅ - GHCR fails → C-Node **skipped**. ✅ - P-Node fails on main (not skipped) → C-Node **skipped** (we want to bail rather than leave prd with a v-mismatch between providers and the consumer). ✅ - [ ] Merge to `dev`, promote to `test`, confirm `Deploy-to-Morpheus-Consumer` actually runs this time and completes the drain → hold → re-register → /healthcheck cycle end-to-end. - [ ] Once green on test, cut `test` → `main` for first prd exercise. ## Related - Previous PR: #714 (dev → test) carrying #713 (initial drain/sequence/hold). - Companion IAM + planning doc already applied in `Morpheus-Infra`. Made with [Cursor](https://cursor.com)

…deploy needs (#716) ## Summary Promotes the #715 hotfix from `dev` → `test` so we can retry the v7 test deploy and actually exercise the new drain / hold / re-register sequence end-to-end. ## What's in this PR Single-commit delta ahead of `test`: - **`2b14c58` — fix(cicd): allow skipped Deploy-to-Morpheus-P-Node to satisfy C-Node deploy needs** (merged via #715) Everything else in `dev` is already on `test` from #714. ## Background First v7 test run ([Actions run #1447 / run id 24796856145](https://github.com/MorpheusAIs/Morpheus-Lumerin-Node/actions/runs/24796856145)) revealed a correctness gap in the workflow from #713/#714: - `Drain-Morpheus-C-Node` ran and successfully drained the dev C-Node targets from both NLB target groups. - `Deploy-to-Morpheus-P-Node` was correctly skipped (main-only). - `Deploy-to-Morpheus-C-Node` was **silently skipped** because the implicit `success()` guard propagated the skip from the P-Node job — GitHub Actions treats skipped `needs` as non-success, contrary to the inline comment in the previous PR. - Dev C-Node remained functional in ECS but without LB membership until manually re-registered. ## Fix (this PR carries) Explicit `if` guard on `Deploy-to-Morpheus-C-Node` that: - Requires real success from `GHCR-Build-and-Push` and `Drain-Morpheus-C-Node`. - Accepts either `success` or `skipped` as the outcome for `Deploy-to-Morpheus-P-Node`. - Wraps everything in `!cancelled()` to short-circuit on manual cancel. The misleading comment is replaced with an accurate explanation referencing this incident. ## Expected behavior after merge to test On push to `test` the sequence becomes: 1. Build + test + GHCR push (~10–20 min). 2. `Drain-Morpheus-C-Node` ✅ deregisters the existing dev C-Node targets. 3. `Deploy-to-Morpheus-P-Node` skipped (expected). 4. `Deploy-to-Morpheus-C-Node` ✅ **runs this time** because the new `if` accepts the P-Node skip: - `update-service` with `--health-check-grace-period-seconds 600`. - Poll new task ENI IP. - Deregister IP from TGs. - Sleep 90s (`cnode_rehydration_wait_secs`). - Re-register IP, `wait target-in-service`. - Public `/healthcheck` version match on `router.dev.mor.org:8082`. 5. `Deploy-to-Titan` + `Deploy-TEE-SecretVM-test` run in parallel to the deploy sequence (unchanged). On dev the rehydration hold is effectively a no-op because EFS-backed BadgerDB is persistent there — perfect for a first validation of the workflow plumbing without depending on rehydration correctness. ## Review focus - The new `if` block on `Deploy-to-Morpheus-C-Node` (lines ~1298–1320 of `build.yml`). - Reasoning captured in the inline comment. - No changes to deploy logic, drain logic, or sequencing — only the guard. ## Test plan (after you merge to test) 1. Merge triggers the deploy workflow against dev. 2. Confirm `Deploy-to-Morpheus-Consumer` enters `in_progress` (not skipped) after the drain completes. 3. Watch the controlled-traffic sequence: - New task gets an ENI IP. - `deregister-targets` runs. - 90s hold. - `register-targets` + `wait target-in-service` succeed. - `/healthcheck` version-match passes on the new image tag. 4. If green, proceed with `test` → `main` for the first prd exercise (existing open PR). ## Related - Previous merges on the new CICD flow: #713, #714, #715. - Companion IAM + planning doc already applied in `Morpheus-Infra`. Made with [Cursor](https://cursor.com)

abs2023 and others added 2 commits April 22, 2026 13:53

nomadicrogue approved these changes Apr 22, 2026

View reviewed changes

nomadicrogue merged commit 1a3b07d into test Apr 22, 2026
9 checks passed

abs2023 mentioned this pull request Apr 22, 2026

fix(cicd): allow skipped Deploy-to-Morpheus-P-Node to satisfy C-Node deploy needs #715

Merged

4 tasks

abs2023 mentioned this pull request Apr 22, 2026

fix(cicd): allow skipped Deploy-to-Morpheus-P-Node to satisfy C-Node deploy needs #716

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(cicd): sequence C-Node drain, P-Node, then C-Node redeploy with rehydration hold#714

fix(cicd): sequence C-Node drain, P-Node, then C-Node redeploy with rehydration hold#714
nomadicrogue merged 2 commits into
testfrom
dev

abs2023 commented Apr 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

abs2023 commented Apr 22, 2026

Summary

What's in this PR

What changes on merge to test

What to review

Test plan (after you merge to test)

Related

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants