[TRTLLM-12728][feat] Add WideEP FT E2E MVP integration prototype (DO NOT REVIEW)

[chienchunhung]

@coderabbitai summary

Status: DRAFT — integration prototype, not ready to merge

This PR is the production-component integration branch for the WideEP fault-tolerance MVP. It replaces the stub-heavy seam-finding role of #14198 with a branch composed from the real implementation PRs, so we can exercise cross-component behavior on physical hardware and use the result to guide the remaining PRs and their reviews.

This draft does not yet claim an end-to-end recovery success or MVP completion. The missing vertical slices and physical acceptance gates are listed below.

Description

The branch stacks the current WideEP FT implementation units on main:

Plan item	PR	Role in this integration branch
1a.2	#13404 (merged)	Launch-time/next-launch NVLinkOneSided rank mask foundation.
1a.3 + 1a.4	#15524	Python mask wiring plus detection-only AlltoAll watchdog, bound to committed mask generations.
1b.1 + 1b.2	#15525 (merged)	Mask-only EPLB C++ entry point and Python wrapper.
1c.1	#15677	EP-specific error-classification patterns.
1c.3	#15785	Failure evidence on a distinct monotonic detected-rank state; no direct committed-membership mutation.
1a.7	#15789	Coordinator-driven NCCL abort/rebuild primitives and AllGatherReduceScatter wiring.
1d.3	#15788	Passive telemetry over coordinator-committed membership and generation.

The integration contract is intentionally strict:

detect → abort failed epoch → reconcile evidence → validate admission → quiesce → prepare EPLB → rebuild survivor control/NCCL → apply graph policy → commit mask + ActiveRankMap + generation → apply request disposition → resume

Detection is evidence, not authorization to mutate the data-plane mask. No output from the failed execution epoch may escape.

Remaining work before this becomes an E2E working MVP

• 1a.8: running-kernel-observable abort/generation primitive and recoverable return instead of the destructive timeout/trap path.
• 1b.2a: per-layer/per-expert survivor-placement invariant and startup/recovery admission.
• 1c.3a + 1c.4a: survivor control communicator, immutable ActiveRankMap, and survivor-aware attention-DP/PyExecutor collectives.
• 1c.4 + 1c.4b + 1c.4c: model-engine hook, sole atomic recovery coordinator, and explicit failed-epoch/request disposition.
• 1a.11: eager fallback plus generation-scoped CUDA-graph invalidation and post-commit recapture.
• 1d.0a + 1d.1: poisoned-MPI lifecycle/shutdown and tested launcher/runtime admission.
• 1d.4 + 1d.4a: realistic multi-GPU process-death testing, followed by NVL72 FABRIC/IMEX peer-memory-containment acceptance. Healthy-GPU process death alone cannot prove Q3 containment.

Until these land, unsupported routes and physical failure classes must fail closed rather than return partial or zero-filled logits.

Review guidance

• Use this draft to review cross-PR interfaces and build the no-mock vertical path.
• Continue reviewing each component in its source PR; this branch deliberately preserves their separation and dependency order.
• Do not merge this aggregate branch as a substitute for the component PRs.

Canonical design and execution tracking:

• Implementation plan
• MVP dependency graph
• Source-of-truth correction checklist

Test Coverage

Pre-publication integration checks:

• All 14 branch commits contain DCO sign-offs.
• git diff --check upstream/main...HEAD passes.
• All 39 changed Python files compile under the repository-supported Python syntax level.
• The branch composes cleanly with the latest fetched upstream/main in a merge-tree check.
• The 1c.3 focused detected-state suite passed 57 tests; the broader local harness previously passed 71 tests.
• Source-PR formatting, lint, DCO, and focused smoke checks pass where locally available.

Not yet run on this aggregate branch:

• Full repository CI.
• Real-model multi-GPU serving with process-death injection.
• Failed-epoch output suppression and continued-request correctness.
• NVL72 FABRIC/IMEX inaccessible-peer-memory containment.

PR Checklist

Please review the following before submitting this PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions).

• If PR introduces API changes, an appropriate PR label is added—either api-compatible or api-breaking.

• Any new dependencies have been scanned for license and vulnerabilities.

• CODEOWNERS is updated if ownership changes.

• Documentation is updated as needed.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, comment /bot help.