Skip to content

Commit 47622cc

Browse files
LalatenduMohantyclaude
LalatenduMohanty
and
claude
committed
docs: add architecture overview docs
Also add an AGENTS.md rule to keep the docs in sync when the architecture changes. Closes: #1247 Co-Authored-By: Claude <claude@anthropic.com> Signed-off-by: Lalatendu Mohanty <lmohanty@redhat.com>
1 parent fdf55b3 commit 47622cc

7 files changed

Lines changed: 634 additions & 0 deletions

‎AGENTS.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,12 @@ Look at these before writing code:
102102

103103
- Use single backticks around function and class names in markdown (e.g. `req_ctxvar_context()`, `WorkContext`), double backticks in .rst (reStructuredText)
104104
- Use Sphinx `versionadded`, `versionremoved`, `versionchanged` directives for user-facing changes (get next version from last git tag)
105+
- When code changes affect the architecture described in `docs/concepts/`, update the relevant doc:
106+
- `architecture-overview.rst` — major subsystems, data flow, extension points, key data structures
107+
- `bootstrapper-architecture.rst` — phase pipeline, class hierarchy, bootstrapper-phase interaction
108+
- `resolver-architecture.rst` — resolution strategies, provider hierarchy, version filtering
109+
- `hooks-and-overrides.rst` — override hook points, global hooks, plugin discovery
110+
- `package-settings.rst` — settings loading flow, merge order, PackageBuildInfo facade
105111

106112
## Commit Messages
107113

‎docs/concepts/architecture-overview.rst‎

Lines changed: 141 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
Architecture Overview
2+
=====================
3+
4+
Fromager rebuilds complete dependency trees of Python wheels from
5+
source. This document maps the major subsystems and how they connect.
6+
7+
.. seealso::
8+
9+
:doc:`bootstrap-vs-build` explains the two operating modes.
10+
:doc:`bootstrapper-architecture` details the bootstrap engine.
11+
:doc:`resolver-architecture` covers version resolution.
12+
:doc:`hooks-and-overrides` describes the two plugin systems.
13+
:doc:`package-settings` explains the settings layering.
14+
15+
Major Subsystems
16+
----------------
17+
18+
.. code-block:: text
19+
20+
┌───────────────────────────────────────────────────────┐
21+
│ CLI & Context │
22+
│ command parsing, WorkContext, settings │
23+
└───────────────────────────┬───────────────────────────┘
24+
│
25+
▼
26+
┌───────────────────────────────────────────────────────┐
27+
│ Bootstrap Engine │
28+
│ iterative DFS loop over phase pipeline │
29+
│ │
30+
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
31+
│ │ Resolution │ │ Source │ │ Build │ │
32+
│ │ PyPI, graph │ │ Acquisition │ │ System │ │
33+
│ │ git URLs │ │ download, │ │ isolated │ │
34+
│ │ │ │ patch, Rust │ │ envs, │ │
35+
│ │ │ │ vendor │ │ wheels │ │
36+
│ └─────────────┘ └─────────────┘ └─────────────┘ │
37+
└───────────────────────────┬───────────────────────────┘
38+
│
39+
┌────────────────┴────────────────┐
40+
▼ ▼
41+
┌──────────────────┐ ┌──────────────────┐
42+
│ Per-Package │ │ Global Hooks │
43+
│ Overrides │ │ post_bootstrap, │
44+
│ (stevedore) │ │ post_build │
45+
└──────────────────┘ └──────────────────┘
46+
47+
The **bootstrap engine** orchestrates the other subsystems. For each
48+
package it resolves a version, downloads the source, builds a wheel,
49+
and recurses into dependencies. **Resolution**, **source acquisition**,
50+
and **build** are called as needed during each phase of the pipeline.
51+
52+
**Extension points** intercept the pipeline at specific steps, allowing
53+
per-package overrides (e.g. custom download logic) and global hooks
54+
(e.g. post-build notifications).
55+
56+
Data Flow
57+
---------
58+
59+
A ``fromager bootstrap`` invocation flows through the subsystems in
60+
this order:
61+
62+
.. code-block:: text
63+
64+
requirements.txt
65+
│
66+
▼
67+
┌─ CLI & Context ───────────────────────────────────────┐
68+
│ parse args, load settings, create WorkContext │
69+
└───────────────────────────┬───────────────────────────┘
70+
│
71+
▼
72+
┌─ Bootstrap Engine ────────────────────────────────────┐
73+
│ for each package (iterative DFS): │
74+
│ resolve version ──► download source │
75+
│ ──► build wheel ──► extract deps │
76+
│ ──► recurse into dependencies │
77+
└───────────────────────────┬───────────────────────────┘
78+
│
79+
▼
80+
┌── Outputs ──┐
81+
│ graph.json │
82+
│ build-order │
83+
│ wheels/ │
84+
│ constraints │
85+
└─────────────┘
86+
87+
The ``build`` command uses the same source acquisition and build
88+
subsystems but skips resolution and recursion -- it compiles a
89+
single package given a name, version, and source URL.
90+
91+
Extension Points
92+
----------------
93+
94+
Fromager has two plugin systems, both using stevedore:
95+
96+
.. list-table::
97+
:header-rows: 1
98+
:widths: 25 35 40
99+
100+
* - System
101+
- Scope
102+
- When it fires
103+
* - **Per-package overrides**
104+
- One package
105+
- At each pipeline step (download, build sdist, build wheel,
106+
dependency extraction, etc.)
107+
* - **Global hooks**
108+
- All packages
109+
- After specific events (``post_bootstrap``, ``post_build``,
110+
``prebuilt_wheel``)
111+
112+
Per-package overrides replace the default implementation of a step for
113+
a specific package. Global hooks run in addition to the default logic
114+
for every package. See :doc:`hooks-and-overrides` for the full
115+
breakdown.
116+
117+
Key Data Structures
118+
-------------------
119+
120+
Four data structures flow between subsystems:
121+
122+
.. list-table::
123+
:header-rows: 1
124+
:widths: 30 70
125+
126+
* - Structure
127+
- Role
128+
* - ``WorkContext``
129+
- Central configuration and state: directory paths, constraints,
130+
dependency graph, settings, and variant info. Passed to every
131+
subsystem.
132+
* - ``DependencyGraph``
133+
- In-memory directed graph of all resolved dependencies.
134+
Serialized to ``graph.json``. Thread-safe.
135+
* - ``PackageBuildInfo``
136+
- Per-package build configuration: environment variables, patches,
137+
resolver settings, and build options. Derived from settings
138+
files.
139+
* - ``BuildEnvironment``
140+
- Isolated virtual environment for building one package. Created
141+
per source build, cleaned up after completion.

‎docs/concepts/bootstrapper-architecture.rst‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
Bootstrapper Architecture
2+
=========================
3+
4+
The bootstrap command uses an iterative depth-first loop to resolve and
5+
build an entire dependency tree. This document describes the phase
6+
pipeline, class hierarchy, and interaction model at a high level.
7+
8+
.. seealso::
9+
10+
:doc:`architecture-overview` maps the major subsystems.
11+
:doc:`bootstrap-vs-build` covers the difference between bootstrap and
12+
build modes.
13+
14+
Bootstrap Phase Flow
15+
--------------------
16+
17+
Every package passes through a sequence of phases. Source packages
18+
traverse the full pipeline; prebuilt wheels skip the build phases.
19+
20+
.. code-block:: text
21+
22+
RESOLVE ──► START ──► PREPARE_SOURCE ─┬─► PREPARE_BUILD ──► BUILD ─┐
23+
│ │ │
24+
│ (prebuilt)└────────────────────────────┤
25+
│ │
26+
│ ┌────────────────────────────────┘
27+
│ ▼
28+
│ PROCESS_INSTALL_DEPS ──► COMPLETE
29+
│
30+
(already seen) ──► drop
31+
32+
Each phase returns a list of new items to push onto the work stack.
33+
Dependency-discovery phases (``PREPARE_SOURCE``, ``PREPARE_BUILD``,
34+
``PROCESS_INSTALL_DEPS``) also emit ``RESOLVE`` items for newly
35+
discovered dependencies, which drives the recursive traversal.
36+
37+
Phase Class Hierarchy
38+
---------------------
39+
40+
All phases inherit from the ``Phase`` abstract base class, which defines
41+
the contract every phase must satisfy:
42+
43+
- ``run(bt)`` — execute the phase logic; return new items for the stack.
44+
- ``background_work(bt)`` — optionally return a callable for background
45+
I/O (used by ``Resolve`` and ``PrepareSource``).
46+
- ``requires_exclusive_run`` — return ``True`` to drain the thread pool
47+
before running (used by ``Build`` for exclusive builds).
48+
49+
.. list-table::
50+
:header-rows: 1
51+
:widths: 25 75
52+
53+
* - Phase
54+
- Role
55+
* - ``Resolve``
56+
- Resolve available versions; fan out one ``Start`` per version
57+
* - ``Start``
58+
- Add to dependency graph; deduplicate already-seen packages
59+
* - ``PrepareSource``
60+
- Download source or prebuilt wheel; read build-system deps
61+
* - ``PrepareBuild``
62+
- Install build-system deps; discover backend and sdist deps
63+
* - ``Build``
64+
- Build sdist and/or wheel; update the local mirror
65+
* - ``ProcessInstallDeps``
66+
- Run post-bootstrap hooks; extract install deps; record build order
67+
* - ``Complete``
68+
- Clean up build directories (terminal phase)
69+
70+
Each phase wraps a ``WorkItem`` dataclass that accumulates per-package
71+
state (resolved version, build environment, build result, etc.) as it
72+
moves through the pipeline.
73+
74+
Bootstrapper and Phase Interaction
75+
----------------------------------
76+
77+
The ``Bootstrapper`` class owns the work stack and thread pool. It runs
78+
an iterative DFS loop:
79+
80+
.. code-block:: text
81+
82+
┌─────────────────────────────────────────────────────┐
83+
│ Bootstrapper Loop │
84+
│ │
85+
│ while stack: │
86+
│ item = stack.pop() │
87+
│ │
88+
│ if item.requires_exclusive_run: │
89+
│ drain thread pool │
90+
│ │
91+
│ new_items = item.run(self) │
92+
│ ↑ phase blocks on its own bg_future │
93+
│ inside run() if it has one │
94+
│ │
95+
│ for each new_item in new_items: │
96+
│ submit background_work() to thread pool │
97+
│ stack.push(new_item) │
98+
└─────────────────────────────────────────────────────┘
99+
100+
The ``Bootstrapper`` provides services that phases call back into during
101+
``run()``: dependency graph updates, seen-set tracking, build-order
102+
recording, and work-item creation for discovered dependencies.
103+
104+
Background I/O (version resolution, source downloads) runs in a thread
105+
pool. When new items are pushed onto the stack, the ``Bootstrapper``
106+
submits their ``background_work()`` immediately so I/O overlaps with
107+
the main thread processing other items. Each phase is responsible for
108+
blocking on its own ``bg_future`` inside ``run()`` when it needs the
109+
result.

‎docs/concepts/hooks-and-overrides.rst‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
Hooks and Overrides
2+
===================
3+
4+
Fromager has two stevedore-based plugin systems that serve different
5+
purposes: **per-package overrides** replace default behavior for a
6+
specific package, while **global hooks** broadcast notifications after
7+
events for every package.
8+
9+
.. seealso::
10+
11+
:doc:`architecture-overview` maps the major subsystems.
12+
:doc:`/reference/hooks` documents each override hook's arguments and
13+
return values. :doc:`/customization` covers global hooks with code
14+
examples.
15+
16+
Comparison
17+
----------
18+
19+
.. list-table::
20+
:header-rows: 1
21+
:widths: 20 40 40
22+
23+
* - Aspect
24+
- Per-Package Overrides
25+
- Global Hooks
26+
* - **Scope**
27+
- One named package
28+
- Every package
29+
* - **Cardinality**
30+
- At most one override module per package
31+
- Multiple plugins per hook
32+
* - **Semantics**
33+
- Replaces the default implementation
34+
- Runs in addition to the default
35+
* - **Return value**
36+
- Used by the caller
37+
- None (fire-and-forget)
38+
* - **Typical use**
39+
- Custom download, build, or resolution logic
40+
- Publishing wheels, validation, logging
41+
42+
Per-Package Overrides
43+
---------------------
44+
45+
An override module provides alternative implementations for specific
46+
build steps. Fromager looks up the module by the canonicalized package
47+
name. If the module defines the requested method, it is called instead
48+
of the default; otherwise the default runs.
49+
50+
Override hooks fire at these points in the pipeline:
51+
52+
.. list-table::
53+
:header-rows: 1
54+
:widths: 30 70
55+
56+
* - Category
57+
- Hook points
58+
* - **Resolution**
59+
- ``get_resolver_provider`` — provide a custom resolvelib provider
60+
* - **Source**
61+
- ``download_source`` — download the source archive
62+
63+
``prepare_source`` — unpack and patch source
64+
65+
``expected_source_archive_name`` — filename for re-discovery
66+
67+
``expected_source_directory_name`` — directory for re-discovery
68+
* - **Build**
69+
- ``build_sdist`` — build a source distribution
70+
71+
``build_wheel`` — build a wheel
72+
73+
``add_extra_metadata_to_wheels`` — inject extra metadata
74+
* - **Dependencies**
75+
- ``get_build_system_dependencies`` — PEP 517 build-system deps
76+
77+
``get_build_backend_dependencies`` — build-backend deps
78+
79+
``get_build_sdist_dependencies`` — sdist build deps
80+
81+
``get_install_dependencies_of_sdist`` — install deps from sdist
82+
* - **Environment**
83+
- ``update_extra_environ`` — customize build environment variables
84+
85+
Third-party packages register overrides via the
86+
``fromager.project_overrides`` entry-point group in their
87+
``pyproject.toml``, mapping a package name to a Python module.
88+
89+
See :doc:`/reference/hooks` for the full API of each hook.
90+
91+
Global Hooks
92+
------------
93+
94+
Global hooks are event callbacks that fire for every package. All
95+
registered plugins for a hook are called sequentially. They cannot
96+
alter the build result — they are purely side-effecting notifications.
97+
98+
.. list-table::
99+
:header-rows: 1
100+
:widths: 25 75
101+
102+
* - Hook
103+
- When it fires
104+
* - ``post_build``
105+
- After a wheel is built from source (not for prebuilt wheels)
106+
* - ``prebuilt_wheel``
107+
- After a prebuilt wheel is downloaded (not built from source)
108+
* - ``post_bootstrap``
109+
- After a package is bootstrapped, before its install dependencies
110+
are processed
111+
112+
Third-party packages register hooks via the ``fromager.hooks``
113+
entry-point group in their ``pyproject.toml``, mapping a hook name to
114+
a callable.
115+
116+
See :doc:`/customization` for examples and argument details.

0 commit comments

Comments
 (0)