Add NXF_DEV launcher support for automated dev builds#7219
Conversation
Adds an NXF_DEV environment variable to the launcher to run automated development builds published in the nextflow-io/nextflow-dev-builds repository: NXF_DEV=master for the latest master build, NXF_DEV=pr-1234 for the latest build of a pull request. Also adds a workflow that posts a comment on each pull request with ready-made commands for testing that PR's dev build, updated in place on each push. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Phil Ewels <phil.ewels@seqera.io>
✅ Deploy Preview for nextflow-docs-staging ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
When the DEV_BUILDS_DISPATCH_TOKEN secret is configured, PR pushes and master pushes dispatch an immediate build in nextflow-dev-builds instead of waiting for the scheduled run there. Without the secret, or if the dispatch fails, everything falls back to the scheduled build and the PR comment says which mode applied. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Phil Ewels <phil.ewels@seqera.io>
|
This would be useful for testing on remote systems, e.g. checking a fix works when launching a pipeline with Seqera Platform. |
|
How does it handle PRs that modify core plugins? Currently, you can either make a standard build (which won't include the core plugins because they have to be published) or you can make an all-in-one build (which doesn't support third-party plugins) The e2e CI actually creates a launch container with the nextflow build and the core plugins baked into the plugin cache, so that we can test nf-core pipelines (which use third party plugins) against a nextflow PR build That's the only way we have been able to get the "best of both worlds" when testing the core plugins. Another way might be to make the all-in-one build support third-party plugins... |
Why
Testing an unreleased Nextflow change today means waiting for an edge release, or building from source and hosting the result somewhere. That makes it slow to verify a fix on master, hard for issue reporters to confirm that a fix works for them, and effectively impossible for most users to try out functionality from an open PR.
With this change, anyone can do:
No build tools, no waiting for a release. Set inline as above, nothing persists in the environment, and the next plain
nextflowinvocation behaves exactly as before.How the system works
The builds come from nextflow-io/nextflow-dev-builds, which is already live:
masterand recently active PRs within ~30 minutes of a push, publishing each build as a pre-release (fat jar, standalone executable, checksums). PR code is compiled in a job with a read-only token; publishing happens in a separate job that never executes source code.latest/master,latest/pr-1234) track the newest build per channel. Old builds are pruned automatically, and builds for closed PRs are deleted.NXF_BASEplusNXF_VER, since the launcher's download URL layout is matched exactly.What this PR adds
NXF_DEVin the launcher: resolves the channel's latest build version from the pointer file and pointsNXF_BASEat the dev-builds repo. Zero overhead when unset (one bash conditional).self-updaterefuses to run whileNXF_DEVis set, so a user cannot accidentally replace their installed launcher with a dev build.dev-build-comment.ymlworkflow: posts a comment on each PR with copy-paste commands for testing that PR's build, updated in place on each push (never repeated). It usespull_request_targetso it also works for fork PRs; this is safe because it never checks out or executes PR code, it only reads PR metadata and theVERSIONfile, whose content is validated against a character allowlist before use.dev-build-dispatch.yml) dispatch a build in the dev-builds repo right away, instead of waiting for the scheduled run. This needs theDEV_BUILDS_DISPATCH_TOKENsecret (see follow-up below). Without it, or if the dispatch fails, everything falls back gracefully to the scheduled ~30 minute build, and the PR comment states which mode applied so a misconfiguration is visible without blocking anyone.NXF_DEVentry in the environment variable reference.Follow-up after merge
To enable immediate builds (a few minutes instead of up to 30), add a repository secret. Until then the workflows fall back to the scheduled builds automatically.
Setting up
DEV_BUILDS_DISPATCH_TOKENnextflow-ionextflow-io/nextflow-dev-buildsDEV_BUILDS_DISPATCH_TOKENBuild dev releasesrun appears in nextflow-dev-builds actions.Notes
NXF_BASE=https://github.com/nextflow-io/nextflow-dev-builds/releases/download NXF_VER=<build version> nextflow run <pipeline>. The PR comment shows both forms.Companion docs PR: nextflow-io/nextflow-dev-builds#1
🤖 Generated with Claude Code