Maintainers also can use Breeze for other purposes (those are commands that regular contributors likely do not need or have no access to run). Those are usually connected with releasing Airflow:
Those are all of the available release management commands:
Running airflow release commands is part of the release procedure performed by the release managers and it is described in detail in dev .
You can prepare airflow packages using Breeze:
breeze release-management prepare-airflow-packageThis prepares airflow .whl package in the dist folder.
Again, you can specify optional --package-format flag to build selected formats of airflow packages,
default is to build both type of packages sdist and wheel.
breeze release-management prepare-airflow-package --package-format=wheel
You can prepare airflow source tarball using Breeze:
breeze release-management prepare-airflow-tarballThis prepares airflow -source.tar.gz package in the dist folder.
You must specify --version flag which is a pre-release version of Airflow you are preparing the
tarball for.
breeze release-management prepare-airflow-tarball --version 2.8.0rc1
When we create a new minor branch of Airflow, we need to perform a few maintenance tasks. This command automates it.
breeze release-management create-minor-branch
When we prepare release candidate, we automate some of the steps we need to do.
breeze release-management start-rc-process
When we prepare final release, we automate some of the steps we need to do.
breeze release-management start-release
The Python client source code can be generated and Python client packages could be built. For that you need to have python client's repository checked out
breeze release-management prepare-python-client --python-client-repo ~/code/airflow-client-pythonYou can also generate python client with custom security schemes.
These are all of the available flags for the command:
The Production image can be released by release managers who have permissions to push the image. This happens only when there is an RC candidate or final version of Airflow released.
You release "regular" and "slim" images as separate steps.
Releasing "regular" images:
breeze release-management release-prod-images --airflow-version 2.4.0Or "slim" images:
breeze release-management release-prod-images --airflow-version 2.4.0 --slim-imagesBy default when you are releasing the "final" image, we also tag image with "latest" tags but this
step can be skipped if you pass the --skip-latest flag.
These are all of the available flags for the release-prod-images command:
You can prepare helm chart source tarball using Breeze:
breeze release-management prepare-helm-chart-tarballThis prepares helm chart -source.tar.gz package in the dist folder.
You must specify --version and --version-suffix flags that specify
which version of Helm Chart you are preparing the tarball for.
breeze release-management prepare-helm-chart-tarball --version 1.12.0 --version-suffix rc1
You can prepare helm chart package and optionally sign it using Breeze:
breeze release-management prepare-helm-chart-packageThis prepares helm chart .tar.gz package in the dist folder.
breeze release-management prepare-helm-chart-package --sign myemail@apache.org
Preparing provider release is part of the release procedure by the release managers and it is described in detail in dev .
You can use Breeze to prepare provider documentation.
The below example perform documentation preparation for provider packages.
breeze release-management prepare-provider-documentationYou can also add --answer yes to perform non-interactive build.
You can use Breeze to prepare provider packages.
The packages are prepared in dist folder. Note, that this command cleans up the dist folder
before running, so you should run it before generating airflow package below as it will be removed.
The below example builds provider packages in the wheel format.
breeze release-management prepare-provider-packagesIf you run this command without packages, you will prepare all packages, you can however specify
providers that you would like to build. By default both types of packages are prepared (
wheel and sdist, but you can change it providing optional --package-format flag.
breeze release-management prepare-provider-packages google amazonYou can see all providers available by running this command:
breeze release-management prepare-provider-packages --help
In some cases we want to just see if the provider packages generated can be installed with airflow without
verifying them. This happens automatically on CI for sdist pcackages but you can also run it manually if you
just prepared provider packages and they are present in dist folder.
breeze release-management install-provider-packagesYou can also run the verification with an earlier airflow version to check for compatibility.
breeze release-management install-provider-packages --use-airflow-version 2.4.0All the command parameters are here:
Breeze can also be used to verify if provider classes are importable and if they are following the
right naming conventions. This happens automatically on CI but you can also run it manually if you
just prepared provider packages and they are present in dist folder.
breeze release-management verify-provider-packagesYou can also run the verification with an earlier airflow version to check for compatibility.
breeze release-management verify-provider-packages --use-airflow-version 2.4.0All the command parameters are here:
The release manager can generate providers metadata per provider version - information about provider versions including the associated Airflow version for the provider version (i.e first airflow version released after the provider has been released) and date of the release of the provider version.
These are all of the available flags for the generate-providers-metadata command:
You can use Breeze to generate a provider issue when you release new providers.
During the provider releases, we need to clean up the older provider versions in the SVN release folder.
Earlier this was done using a script, but now it is being migrated to a breeze command to ease the life of
release managers for providers. This can be achieved using breeze release-management clean-old-provider-artifacts
command.
These are all available flags of clean-old-provider-artifacts command:
Whenever pyproject.toml gets modified, the CI main job will re-generate constraint files. Those constraint
files are stored in separated orphan branches: constraints-main, constraints-2-0.
Those are constraint files as described in detail in the CONTRIBUTING.rst#pinned-constraint-files contributing documentation.
You can use breeze release-management generate-constraints command to manually generate constraints for
all or selected python version and single constraint mode like this:
Warning
In order to generate constraints, you need to build all images with --upgrade-to-newer-dependencies
flag - for all python versions.
breeze release-management generate-constraints --airflow-constraints-mode constraintsConstraints are generated separately for each python version and there are separate constraints modes:
- 'constraints' - those are constraints generated by matching the current airflow version from sources
- and providers that are installed from PyPI. Those are constraints used by the users who want to install airflow with pip.
- "constraints-source-providers" - those are constraints generated by using providers installed from current sources. While adding new providers their dependencies might change, so this set of providers is the current set of the constraints for airflow and providers from the current main sources. Those providers are used by CI system to keep "stable" set of constraints.
- "constraints-no-providers" - those are constraints generated from only Apache Airflow, without any providers. If you want to manage airflow separately and then add providers individually, you can use those.
These are all available flags of generate-constraints command:
In case someone modifies pyproject.toml, the scheduled CI Tests automatically upgrades and
pushes changes to the constraint files, however you can also perform test run of this locally using
the procedure described in the
Manually generating image cache and constraints
which utilises multiple processors on your local machine to generate such constraints faster.
This bumps the constraint files to latest versions and stores hash of pyproject.toml. The generated constraint
and pyproject.toml hash files are stored in the files folder and while generating the constraints diff
of changes vs the previous constraint files is printed.
Sometimes (very rarely) we might want to update individual packages in constraints that we generated and
tagged already in the past. This can be done using breeze release-management update-constraints command.
These are all available flags of update-constraints command:
You can read more details about what happens when you update constraints in the Manually generating image cache and constraints
To publish the documentation generated by build-docs in Breeze to airflow-site,
use the release-management publish-docs command:
breeze release-management publish-docsThe publishing documentation consists steps:
- checking out the latest
mainof clonedairflow-site - copying the documentation to
airflow-site - running post-docs scripts on the docs to generate back referencing HTML for new versions of docs
breeze release-management publish-docs --package-filter apache-airflow-providers-amazonThe flag --package-filter can be used to selectively publish docs during a release. It can take
values such as apache-airflow, helm-chart, apache-airflow-providers, or any individual providers.
The documentation publication happens based on this flag.
breeze release-management publish-docs --override-versionedThe flag --override-versioned is a boolean flag that is used to override the versioned directories
while publishing the documentation.
breeze release-management publish-docs --airflow-site-directoryYou can also use shorthand names as arguments instead of using the full names for airflow providers. To find the short hand names, follow the instructions in :ref:`generating_short_form_names`.
The flag --airflow-site-directory takes the path of the cloned airflow-site. The command will
not proceed if this is an invalid path.
When you have multi-processor machine docs publishing can be vastly sped up by using --run-in-parallel option when
publishing docs for multiple providers.
These are all available flags of release-management publish-docs command:
To add back references to the documentation generated by build-docs in Breeze to airflow-site,
use the release-management add-back-references command. This is important to support backward compatibility
the airflow documentation.
You have to specify which packages you run it on. For example you can run it for all providers:
release-management add-back-references --airflow-site-directory DIRECTORY all-providersThe flag --airflow-site-directory takes the path of the cloned airflow-site. The command will
not proceed if this is an invalid path.
You can also run the command for apache-airflow (core documentation):
breeze release-management publish-docs --airflow-site-directory DIRECTORY apache-airflowAlso for helm-chart package:
breeze release-management publish-docs --airflow-site-directory DIRECTORY helm-chartYou can also manually specify (it's auto-completable) list of packages to run the command for including individual providers - you can mix apache-airflow, helm-chart and provider packages this way:
breeze release-management publish-docs --airflow-site-directory DIRECTORY apache.airflow apache.beam googleThese are all available flags of release-management add-back-references command:
Maintainers also can use Breeze for SBOM generation:
In order to generate SBOM information for providers, we need to generate requirements for them. This is
done by the generate-providers-requirements command. This command generates requirements for the
selected provider and python version, using the airflow version specified.
Thanks to our constraints captured for all versions of Airflow we can easily generate SBOM information for
Apache Airflow. SBOM information contains information about Airflow dependencies that are possible to consume
by our users and allow them to determine whether security issues in dependencies affect them. The SBOM
information is written directly to docs-archive in airflow-site repository.
These are all of the available flags for the update-sbom-information command:
In order to generate providers requirements, we need docker images with all airflow versions pre-installed,
such images are built with the build-all-airflow-images command.
This command will build one docker image per python version, with all the airflow versions >=2.0.0 compatible.
Next step: Follow the Advanced Breeze topics to learn more about Breeze internals.