Introduce a structured YAML partition source and migrate Glymur-CRD#134
Open
Igor Opaniuk (igoropaniuk) wants to merge 17 commits into
Open
Introduce a structured YAML partition source and migrate Glymur-CRD#134Igor Opaniuk (igoropaniuk) wants to merge 17 commits into
Igor Opaniuk (igoropaniuk) wants to merge 17 commits into
Conversation
The module ran argv parsing and XML emission as top-level statements at import time, so it could not be imported by pytest or any library caller. This blocks the follow-up work: unit-testing the parser and splitting it into a loaders package. Wrap the flow in main(argv=None) -> int gated by __name__ == "__main__", move the module-level accumulators into locals, and return status codes instead of sys.exit(). The runpy dispatcher in cli.py is unaffected because it sets __name__ to "__main__". Behaviour-preserving: every platforms/*/*/partitions.xml is unchanged. Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
The .conf format was hard-coded into gen_partition.py, with no boundary between where data comes from and how partition XML is emitted. Adding another format (YAML) would mean duplicating main() or scattering format branches through every helper. Add qcom_ptool/spec.py for the canonical internal shape (DiskParams, PartitionEntry, PartitionsByLun, LoadedSpec) and qcom_ptool/loaders/ with a load(path, image_map) dispatcher keyed on file extension: a new format becomes a new module plus a suffix registration. Move the conf parser into loaders/conf.py and apply the image-map override in one post-load pass, removing the last shared-global coupling. Behaviour-preserving: every platforms/*/*/partitions.xml is unchanged; ruff and mypy stay clean. Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
make integration only checks that referenced files exist; it does not cover size parsing, attribute-bit decoding, LUN grouping or the image-map override -- the contract any future loader must reproduce. Add tests/unit/ pinning those cases against conf.load() and the loaders dispatcher. Wire it in with a make unit-test target, add it to make check, install python3-pytest in CI, and keep pytest config in pyproject.toml so pytest runs from the repo root. Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
The YAML partition source introduced in [1] needs a YAML parser and a schema validator at runtime. Declare PyYAML and jsonschema as install dependencies so pip pulls them in automatically. No code imports these yet; this only prepares the ground for the YAML loader. Build, lint, unit tests and the pinned checksum manifest are unaffected. [1] qualcomm-linux#124 Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
The Dependencies section still claimed the tool ran on the standard library alone. Update it to note the two runtime dependencies now declared in pyproject.toml (PyYAML and jsonschema) and that pip installs them automatically. Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
Introduce the structured YAML source format from [1] for a single storage device. loaders/yaml.py loads a `disk` mapping and `partitions` list, validates it against a packaged JSON Schema, and normalises it into the same LoadedSpec the .conf loader produces, so both formats emit byte-identical XML. Normalisation mirrors loaders/conf.py field for field: defaults are copied in the same key order, partition_size_in_kb is reused and the attribute bits are decoded identically. YAML booleans are rendered as lowercase "true"/"false" so they match the legacy strings. The schema enforces the YAML footgun mitigations: GUIDs and sizes must be quoted strings, unknown keys are rejected, and the disk type is constrained to the known storage classes. Register .yaml/.yml in the loaders dispatcher behind a late import so the .conf path never pays for PyYAML or jsonschema, and ship the schema as package data. [1] qualcomm-linux#124 Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
Pin the load-bearing property that the YAML and .conf loaders produce the same LoadedSpec: an equivalence test against the real glymur-crd/nvme board, a synthetic multi-LUN spec exercising attribute bits, and an end-to-end assertion that both formats emit byte-identical XML through the shared emitter. Add schema-rejection cases for the documented footguns (unquoted all-digit GUID, unquoted size, unknown key, missing type-guid, malformed size, unknown disk type) and a non-mapping top-level document. Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
Introduce the board format from [1]: a board declares a platform and a list of storage devices, each with partitions. board.py resolves the three composition mechanisms into a ResolvedBoard - board-level `extends:` (inherit one base board, override by storage id / partition name), storage-level `includes:` (concatenate shared _common fragments, transitively), and variant overlays (--hlos / --boot-fw) applied last. Merges are deep and field-by-field, matched by stable identifier, with unmatched entries appended in file order; resolution order is base board -> derived board -> storage includes -> variant overlays. There is no partition deletion in v1. Cycles in extends or includes are rejected. Input files are validated structurally against board.schema.json (and include.schema.json for fragments), which stay permissive because overrides are partial. Each fully-resolved storage is then validated strictly against the existing partitions.schema.json and reduced to a LoadedSpec through the same normalisation helpers the YAML loader uses, so a resolved board emits XML byte-identical to an equivalent hand-written single-storage file. [1] qualcomm-linux#124 Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
Pin the composition contract: extends inheritance with field-by-field overrides and appends, storage includes with transitivity, variant overlays applied last (overriding an included partition in place and appending new ones), the deterministic partition ordering, cycle rejection for both extends and includes, strict validation of the resolved storage, and a byte-identical-XML check against the equivalent single-storage YAML. Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
Extend the CLI with --board / --hlos / --boot-fw (and --root) so a whole board resolves and emits one partitions.xml per storage, in declared order, from a single invocation. Outputs are given as one -o per storage; a count mismatch is reported with the storage ids. The board resolver is imported lazily so the legacy -i single-storage path never pulls in PyYAML or jsonschema, and that path is unchanged. Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
Derived boards and heavy includes make the effective layout impossible to read from one file. `qcom-ptool show --board <f> [--hlos ..] [--boot-fw ..]` resolves the full composition chain and prints the result as YAML, so reviewers can see exactly what will be emitted without chasing extends/includes by hand. Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
Assert that --board emits one XML per storage in order, that an output count mismatch fails, that exactly one source (-i or --board) is required, that the legacy single-storage path still works, and that show prints the resolved spec with variant overlays applied. Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
Author the first board definition [1] as a proof of concept: platforms/boards/glymur-crd.yaml declares both storages (NVMe HLOS + SPI-NOR boot / platform-config) with every partition inline in the exact order of the legacy .conf files, and variants/hlos/qcom-deb-images.yaml supplies the NVMe rootfs size and filename. Partition order is preserved verbatim because it determines GPT offsets; reordering changes the emitted layout. That rules out factoring the shared SPI-NOR blocks through _common includes here, since includes prepend and the shared groups are interleaved with board-specific partitions in the current layout - the only order-safe composition for an existing board is the in-place variant override used for rootfs. The .conf files are left in place and the Makefile still builds from them, so generated artifacts and the checksum manifest are unchanged. The YAML board emits byte-identical partitions.xml and GPT for both storages, verified against the .conf output. [1] qualcomm-linux#124 Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
Resolve the YAML board with the Debian HLOS variant and assert each storage reduces to exactly the LoadedSpec the corresponding .conf file produces, which guarantees byte-identical partitions.xml and GPT, and that storage order is preserved. Guards against drift while both source forms coexist during the migration. Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
Build Glymur-CRD from platforms/boards/glymur-crd.yaml instead of the legacy per-storage partitions.conf. A grouped rule resolves the board with the Debian HLOS variant and emits both storage XMLs in one call, and the board is excluded from the .conf glob. The rule creates the output directory because a migrated storage need not carry any tracked file. Generated artifacts and the pinned checksum manifest are unchanged. [1] qualcomm-linux#124 Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
The YAML loader parity tests referenced platforms/glymur-crd, which is migrating away from .conf. Add a self-contained .conf fixture under tests/unit/data/ and point the parity tests at it, and reduce the Glymur-CRD guard test to a structural check - byte fidelity is covered by the pinned checksum manifest. Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
Glymur-CRD is now built from its YAML board through the Makefile, so remove the two legacy partitions.conf files. Generated artifacts and the checksum manifest are unchanged. [1] qualcomm-linux#124 Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
Igor Opaniuk (igoropaniuk)
requested review from
Dmitry Baryshkov (lumag),
Nicolas Dechesne (ndechesne) and
Viswanath Kraleti (vkraleti)
as code owners
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This series replaces the legacy line-format
partitions.confwith a structured, schema-validated YAML source and migrates the first board, Glymur-CRD, onto it without changing a single generated artifact.contents.xml.infor glymur-crd stays intact, as it feeds a separate emitter (gen_contents), not the partition source. This work only replaced partitions.conf. Generating contents.xml from the board YAML (and thus removing the template) will be added in a next PR.It first decouples the parser behind a
LoadedSpecintermediate representation and a loaders package, then adds a YAML loader validated by a packaged JSON Schema that normalises field-for-field to the same LoadedSpec the.confloader produces, so both formats emit byte-identical XML.On top of that it adds a multi-storage board model with a composition resolver - board-level extends for single-base inheritance, storage-level includes for shared partition fragments, and
hlos/boot-fwvariant overlays, all deep-merged by stable identifier under a deterministic ordering rule - together with ashowsubcommand and agen_partition --board modethat emits onepartitions.xmlper storage.Finally it authors
platforms/boards/glymur-crd.yamlwith a Debian HLOS variant, switches the Makefile to build Glymur-CRD from that board, and drops the two now-redundantpartitions.conffiles; everypartitions.xml, GPT binary andcontents.xmlis verified byte-identical to the previous output through the pinned checksum manifest.PyYAMLandjsonschemaare the only new runtime dependencies. Partition order is preserved verbatim because it determines GPT offsets, which is why the shared SPI-NOR blocks stay inline rather than being factored through includes for this already-laid-out board. See [1] for the design discussion and rationale.[1] #124