|
| 1 | +# Bootstrap `--test-mode`: resilient source builds with pre-built fallback |
| 2 | + |
| 3 | +- Author: Lalatendu Mohanty, Doug Hellmann |
| 4 | +- Created: 2026-06-01 (retrospective; feature merged 2025-12-19) |
| 5 | +- Status: Implemented |
| 6 | +- GitHub issue: [#713](https://github.com/python-wheel-build/fromager/issues/713) |
| 7 | +- Pull request: [#865](https://github.com/python-wheel-build/fromager/pull/865) |
| 8 | + |
| 9 | +## What |
| 10 | + |
| 11 | +Serial `fromager bootstrap --test-mode` continues after failures instead of |
| 12 | +aborting. On source-build failure, fromager downloads a pre-built wheel for |
| 13 | +the same package so traversal can proceed; dependents are still discovered and |
| 14 | +built. The run ends with `test-mode-failures-<timestamp>.json` and exit code 1 |
| 15 | +if any failure was recorded. |
| 16 | + |
| 17 | +Primary outputs of a degraded run are `graph.json`, `build-order.json`, and the |
| 18 | +failure report—not a full set of source-built wheels. |
| 19 | + |
| 20 | +## Why |
| 21 | + |
| 22 | +Large source bootstraps often fail one package at a time. Without fallback, |
| 23 | +dependents fail too, hiding which packages have real build problems. Operators |
| 24 | +work around this by manually marking failures as `pre_built` and re-running |
| 25 | +bootstrap repeatedly ([#713](https://github.com/python-wheel-build/fromager/issues/713)). |
| 26 | + |
| 27 | +Test mode automates that loop: continue traversal, substitute pre-built wheels |
| 28 | +on source failure, surface many build gaps in one run, and emit a structured |
| 29 | +failure list for automation. |
| 30 | + |
| 31 | +## Goals |
| 32 | + |
| 33 | +- Continue serial `bootstrap` after failures; fallback via |
| 34 | + `BootstrapRequirementResolver.resolve(..., pre_built=True)`. |
| 35 | +- Classify failures: `resolution`, `bootstrap` (fatal, error log) vs `hook`, |
| 36 | + `dependency_extraction` (non-fatal, warning log). All are recorded; any |
| 37 | + failure yields exit code 1. |
| 38 | +- Preserve graph traversal and record `source_url_type` in `build-order.json` |
| 39 | + (`"prebuilt"` marks fallback wheels). |
| 40 | + |
| 41 | +## Non-goals |
| 42 | + |
| 43 | +- `bootstrap-parallel` (serial only); incompatible with `--sdist-only`. |
| 44 | +- Replacing fail-fast default bootstrap or treating fallback wheels as the |
| 45 | + shipped product. |
| 46 | +- Recording source-build failures when fallback succeeds ([#1166](https://github.com/python-wheel-build/fromager/issues/1166)). |
| 47 | + |
| 48 | +## How |
| 49 | + |
| 50 | +`commands/bootstrap.py` enables the flag; logic lives in `Bootstrapper` |
| 51 | +([iterative-bootstrap.rst](iterative-bootstrap.rst)). |
| 52 | + |
| 53 | +### Architecture |
| 54 | + |
| 55 | +**Run flow** — serial `bootstrap` with `--test-mode` never aborts the whole |
| 56 | +command on a single package failure; it finishes the run and reports at the end. |
| 57 | + |
| 58 | +Resolve top-level requirements → Bootstrap dependency tree → finalize → |
| 59 | +graph.json + build-order.json + failures JSON |
| 60 | + |
| 61 | +**On error** — `_handle_phase_error()` in `bootstrapper.py` decides whether to |
| 62 | +skip the package, substitute a pre-built wheel, or continue with a warning. |
| 63 | + |
| 64 | +On phase error, branch by context: |
| 65 | + |
| 66 | +- **Resolution** → Record failure, skip package |
| 67 | +- **Source build** → Is a pre-built wheel available? |
| 68 | + - Yes → Use wheel, continue traversal |
| 69 | + - No → Record failure, skip package |
| 70 | +- **Hook or dep extraction** → Record warning, continue |
| 71 | + |
| 72 | +**Phase errors** (`_handle_phase_error()`): |
| 73 | + |
| 74 | +| Context | Behavior | |
| 75 | +| -- | -- | |
| 76 | +| `RESOLVE` | Record `resolution`; skip | |
| 77 | +| `PREPARE_SOURCE` / `PREPARE_BUILD` / `BUILD` | `_handle_test_mode_failure()`: resolve+download pre-built wheel; on success advance to `PROCESS_INSTALL_DEPS` with `SourceType.PREBUILT`; else record `bootstrap` | |
| 78 | +| Post-bootstrap hooks | Record `hook` (warning); continue | |
| 79 | +| Install dep extraction | Record `dependency_extraction` (warning); empty deps | |
| 80 | + |
| 81 | +Settings `pre_built` packages skip the fallback path. Failure records include |
| 82 | +package, version, exception type/message, and `failure_type`; see |
| 83 | +`FailureRecord` in `bootstrapper.py`. |
| 84 | + |
| 85 | +**Artifacts:** `build-order.json` `source_url_type` reflects runtime fallback; |
| 86 | +graph node `pre_built` reflects settings only (not updated after fallback). |
| 87 | + |
| 88 | +| | `--test-mode` | `--multiple-versions` | |
| 89 | +| -- | -- | -- | |
| 90 | +| Purpose | Source-build gap analysis | All matching versions | |
| 91 | +| On build failure | Pre-built fallback, keep traversing | Remove version from graph | |
| 92 | +| Output | `test-mode-failures-*.json` | Logs only | |
| 93 | + |
| 94 | +## Usage |
| 95 | + |
| 96 | +```bash |
| 97 | +fromager bootstrap --test-mode -r requirements.txt |
| 98 | +``` |
| 99 | + |
| 100 | +Review `test-mode-failures-*.json`, `build-order.json` (`source_url_type: "prebuilt"`), and `graph.json`. Tests: `tests/test_bootstrap_test_mode.py`, |
| 101 | +`e2e/test_mode_*.sh`. |
| 102 | + |
| 103 | +## Limitations |
| 104 | + |
| 105 | +1. Source-build failure not recorded when fallback succeeds ([#1166](https://github.com/python-wheel-build/fromager/issues/1166)). |
| 106 | +2. Audit fallback via `build-order.json`, not graph `pre_built`. |
| 107 | +3. Serial bootstrap only. |
| 108 | +4. Pre-built version may differ from requested source version (warning logged). |
| 109 | + |
| 110 | +## Key source files |
| 111 | + |
| 112 | +`commands/bootstrap.py`, `bootstrapper.py`, `bootstrap_requirement_resolver.py`, |
| 113 | +`dependency_graph.py` |
0 commit comments