Enable one-click crates.io release for mxc-sdk (ESRP) [in-repo, supersedes #647] - #689
Open
dhoehna wants to merge 9 commits into
Open
Enable one-click crates.io release for mxc-sdk (ESRP) [in-repo, supersedes #647]#689dhoehna wants to merge 9 commits into
dhoehna wants to merge 9 commits into
Conversation
Add crates.io-required metadata so mxc-sdk and its first-party dependency closure (10 crates) can be published: - workspace: add repository URL; give the 8 first-party path-deps in [workspace.dependencies] explicit versions (cargo publish requires a version on every path dependency). - wxc_common: version its optional `nanvix_common` path-dep. Although it is gated by the off-by-default `microvm` feature, cargo requires every dependency in a published manifest -- optional ones included -- to carry a version and exist on crates.io, so nanvix_common must ship too. It is a leaf (serde/serde_json only). - add description to wxc_common, mxc_pty, and the appcontainer / bubblewrap / lxc / seatbelt common crates. - add license + repository to mxc_telemetry; repository to nanvix_common. - inherit repository on each published crate. - make sandbox_spec publishable (remove publish = false). Validated with `cargo metadata` and per-crate `cargo package --list` (plus `cargo publish --dry-run --no-verify` for leaf crates) against real crates.io. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e5d2aa5b-7f04-4e4d-83d3-a02efe7020ab
New ADO pipeline publishes the mxc-sdk crate and its 10-crate first-party closure to crates.io from a single "Run pipeline" click, with a dryRun parameter (default true) that previews every crate before anything uploads. - .azure-pipelines/scripts/publish_crates_to_cratesio.py: leaf-first ordered publish (mxc_telemetry, nanvix_common, mxc_pty, sandbox_spec, wxc_common, lxc_common, seatbelt_common, appcontainer_common, bwrap_common, mxc-sdk), per-crate version resolved via `cargo metadata`, idempotent version-exists skip via the crates.io sparse index, `cargo publish --no-verify` with a post-publish index-propagation wait, and a --dry-run mode that runs `cargo package --list` (validates each manifest without needing first-party deps on the index yet). Modeled on scripts/seed_feed.py conventions. - .azure-pipelines/1ES.Crate.Release.yml: trigger:none, extends the 1ES Official template; DryRun stage always previews; Publish stage is compiled in only when dryRun=false and gates on manual approval via the MXC-CratesIo-Production environment. Reuses Rust.Toolchain.Public.yml and deliberately skips Cargo.Setup.* so cargo talks to the real crates.io. Secrets stay out of the repo: CARGO_REGISTRY_TOKEN comes from the MXC-CratesIo-Publish variable group and is mapped to env only in the publish step. One-time ADO/crates.io setup is documented in the pipeline header. Validated locally against real crates.io: the full --dry-run passes for all 10 crates; all 10 target names are currently unregistered on crates.io. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e5d2aa5b-7f04-4e4d-83d3-a02efe7020ab
Resolved Cargo.toml conflicts: - src/Cargo.toml [workspace.dependencies]: kept version pins on wxc_common and appcontainer_common; kept upstream's new mxc_engine and mxc-sdk path deps (left unversioned - not part of the current publish closure). - src/mxc_telemetry/Cargo.toml: kept repository.workspace = true. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e5d2aa5b-7f04-4e4d-83d3-a02efe7020ab
Rework the crates.io release per PR #647 review feedback (Brandon Bonaby): publish through the ESRP OSS release system instead of a bespoke CARGO_REGISTRY_TOKEN, folded into the existing 1ES.Release.yml so one pipeline releases npm, crates.io, or both (modeled on azure-sdk-for-rust). - scripts/crates_release.py: package/verify-order/stage/wait subcommands; cargo-packages the closure leaf-first, no token, no publish endpoint. - templates/Package.Crates.Job.yml: official-build job -> mxc-crates-package. - templates/Publish.CratesIo.Job.yml: releaseJob, EsrpRelease@12 contenttype Rust, one task per crate leaf-first, verify each on the crates.io index before the next. - 1ES.Release.yml: publishNpm/publishCrates params; gate NPM stage; add Publish_to_CratesIo stage; ESRP one-time-setup notes in the header. - 1ES.Build.Stages.yml: add Package_Crates stage. - Delete 1ES.Crate.Release.yml + publish_crates_to_cratesio.py (token approach). publishCrates defaults to false: enabling actual publishing still needs the crate closure metadata reconciled (7 path deps need versions; isolation_session_bindings is publish=false) plus OSPO OSS-release setup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 54ab05fd-1ef7-4fa4-b130-ea17a9ef8a47
cmd_package looped `cargo package -p <crate>` one crate at a time, which fails on the first non-leaf crate: packaging a crate alone strips the path from each dependency and cargo then looks for the sibling on the crates.io index, where it is not published yet (e.g. wxc_common -> mxc_telemetry -> "no matching package named `mxc_telemetry` found"). Package the entire closure in a single `cargo package` invocation with every crate passed via -p, so cargo resolves intra-closure deps from target/package/ instead of crates.io (the azure-sdk-for-rust Pack-Crates.ps1 approach). Verified locally: all 9 packageable crates now tar in one run; the only remaining failure is the known mxc-sdk -> mxc_engine missing-version blocker. Also corrected two docstring claims that wrongly implied per-crate `--no-verify` packaging works. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 54ab05fd-1ef7-4fa4-b130-ea17a9ef8a47
The 1ES official build runs network-isolated and cannot reach the public crates.io sparse index (https://index.crates.io), so crates_release.py's post-publish version check would fail at runtime. Query the private Mxc-Azure-Feed sparse index instead (reachable in-pipeline), authenticated with the build identity's $(System.AccessToken). - crates_release.py: replace CRATES_IO_SPARSE_INDEX with AZURE_FEED_SPARSE_INDEX (matches .azure-pipelines/.cargo/config.toml); add _feed_request() Bearer-auth helper reading SYSTEM_ACCESSTOKEN; _published_versions() now hits the feed; relabel poll/index wording to "Azure Artifacts feed". - Publish.CratesIo.Job.yml: map $(System.AccessToken) -> SYSTEM_ACCESSTOKEN on the wait step so the feed request can authenticate. Addresses Brandon Bonaby's review comment on PR #647. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 54ab05fd-1ef7-4fa4-b130-ea17a9ef8a47
Applies Brandon Bonaby's PR #647 review comments: - 1ES.Build.Stages.yml: gate the Package_Crates stage on isOfficialBuild so crate packaging only runs on the official signed build, not PR/CI. - 1ES.Release.yml: default publishNpm to false too, so the operator must explicitly select npm and/or crates in the run dialog. - 1ES.Release.yml: reduce the oversized header comment (and drop the stale index.crates.io egress note superseded by the private-feed fix). - Add a cratesDryRun parameter (1ES.Release.yml -> Publish.CratesIo.Job.yml) that runs stage + verify-order but skips the EsrpRelease publish and the feed wait, for testing the pipeline without publishing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 54ab05fd-1ef7-4fa4-b130-ea17a9ef8a47
Resolve src/Cargo.toml conflict in [workspace.dependencies]: keep main's roxmltree -> quick-xml migration and the new windows_sandbox_lifecycle dependency, alongside the version = "0.7.0" pins added for crate publishing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 54ab05fd-1ef7-4fa4-b130-ea17a9ef8a47
The pipeline's .cargo/config.toml replaces the crates-io source with the internal Mxc-Azure-Feed, so `cargo package` aborts with "crates-io is replaced with remote registry Mxc-Azure-Feed; include --registry Mxc-Azure-Feed or --registry crates-io". Package for crates-io -- the real publish target -- so intra-closure sibling deps are recorded as plain crates.io deps (packaging for the feed would embed a registry-index that crates.io rejects). The source replacement still serves index reads in the network-isolated build. Found via MXC-Official-Build run 153175941 (Package Crates stage failed at cargo package exit 101). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 54ab05fd-1ef7-4fa4-b130-ea17a9ef8a47
Contributor
Author
|
@bbonaby @MGudgin -- moving this to an in-repo branch (supersedes #647). Before I publish anything to crates.io, there's a dependency-ordering issue plus a fix I'd like your thumbs-up on. Issue -- the closure must be published in dependency orderPublishing an interdependent set of crates isn't atomic. When Solution -- leaf-first bootstrapPublish the closure one crate at a time, leaf-first: each crate reaches crates.io before anything that depends on it is packaged. Once a dependency is live (and mirrored into the feed), its dependents build cleanly. This is the standard pattern (cargo-release / release-plz work this way) and sidesteps the bug. Mechanically: a one-off leaf-first ESRP publish to seed the closure, after which the normal pipeline runs unblocked. Names that would land on crates.io (permanent)Full first-party closure of
(The pipeline is presently exercising an older 10-crate subset.) Since crates.io names are permanent, this is also the moment to confirm final naming -- e.g. whether we adopt an AskA thumbs-up on (1) the leaf-first bootstrap approach and (2) the crate names / closure above, before I publish anything to crates.io. Happy to rehearse the exact flow against the private feed (reversible) first to prove it end-to-end. |
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds ESRP-based crates.io publishing infrastructure for the mxc-sdk dependency closure.
Changes:
- Adds publishable crate metadata and versioned local dependencies.
- Packages crate artifacts during official builds.
- Adds selectable npm/crates.io ESRP release stages with ordering checks.
Reviewed changes
Copilot reviewed 15 out of 16 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
src/Cargo.toml |
Adds repository metadata and dependency versions. |
src/mxc_telemetry/Cargo.toml |
Adds repository metadata. |
src/core/wxc_common/Cargo.toml |
Adds publish metadata and dependency version. |
src/core/mxc-sdk/Cargo.toml |
Adds repository metadata. |
src/core/mxc_pty/Cargo.toml |
Adds publish metadata. |
src/core/generated/base_container_specification/Cargo.toml |
Makes sandbox_spec publishable. |
src/backends/seatbelt/common/Cargo.toml |
Adds publish metadata. |
src/backends/nanvix/common/Cargo.toml |
Adds repository metadata. |
src/backends/lxc/common/Cargo.toml |
Adds publish metadata. |
src/backends/bubblewrap/common/Cargo.toml |
Adds publish metadata. |
src/backends/appcontainer/common/Cargo.toml |
Adds publish metadata. |
.azure-pipelines/templates/Publish.CratesIo.Job.yml |
Defines per-crate ESRP publishing. |
.azure-pipelines/templates/Package.Crates.Job.yml |
Creates the crate release artifact. |
.azure-pipelines/templates/1ES.Build.Stages.yml |
Adds official-build crate packaging. |
.azure-pipelines/scripts/crates_release.py |
Implements packaging, staging, ordering, and propagation checks. |
.azure-pipelines/1ES.Release.yml |
Adds selectable npm and crates.io release stages. |
Comment on lines
+99
to
+100
| "mxc-sdk", | ||
| ] |
Comment on lines
+219
to
+224
| package_args = ["cargo", "package", "--no-verify", "--allow-dirty", | ||
| "--registry", "crates-io", | ||
| "--manifest-path", manifest] | ||
| for crate in CRATES: | ||
| package_args += ["-p", crate] | ||
| rc = _run(package_args) |
Comment on lines
+87
to
+88
| - ${{ if parameters.isOfficialBuild }}: | ||
| - stage: Package_Crates |
Comment on lines
+313
to
+319
| # ESRP ran with waitforreleasecompletion:true, so the publish itself already | ||
| # completed; only index propagation is unconfirmed. Warn (don't fail the | ||
| # release) and let the next crate proceed. | ||
| print(f"##vso[task.logissue type=warning]{crate} {version} not confirmed on the " | ||
| f"Azure Artifacts feed within {args.timeout}s; continuing (ESRP reported the " | ||
| f"publish complete).") | ||
| return 0 |
Comment on lines
+76
to
+77
| - task: EsrpRelease@12 | ||
| displayName: "Publish ${{ crate }} to crates.io" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #647 (fork PR could not exercise the ESRP release pipeline). Moved to this in-repo branch per @bbonaby's guidance; full history and review feedback preserved.
What this does
Adds a single ADO pipeline that publishes the
mxc-sdkcrate closure to public crates.io via ESRP (Microsoft OSS release path -- no registry tokens in-repo), while third-party build dependencies are pulled from the internalMxc-Azure-FeedAzure Artifacts feed..azure-pipelines/scripts/crates_release.py-- ESRP package/release helper.azure-pipelines/templates/Publish.CratesIo.Job.yml-- leaf-firstEsrpRelease@12per crate.azure-pipelines/.cargo/config.toml-- redirects crates.io -> internal feed for the buildShips with
publishCrates=falseby default; theverify-orderguard fails safe if the crate list and release order disagree.Status -- open design question (needs sign-off)
See the comment below. There is a dependency-ordering issue (cargo #17196) and a leaf-first fix I need @bbonaby and @MGudgin to approve before anything is published to crates.io. crates.io publishes are permanent, so I want a thumbs-up on the approach and the exact crate names first.
Microsoft Reviewers: Open in CodeFlow