Skip to content

feat: parallelize serial (--strategy build) image builds #647

Description

@ianpittwood

Summary

BakeryConfig.build_targets() (posit-bakery/posit_bakery/config/config.py) has an ImageBuildStrategy.BUILD branch that loops over self.targets and calls target.build(...) serially, wrapped in _retry_build. --strategy build exists specifically for generic container builders (Podman, nerdctl) that don't support Buildx Bake's native parallelism, so it currently gets none of the concurrency that --strategy bake gets for free.

Unlike hadolint/wizcli (see related issues), this path does not shell out via a plain cmd: list[str]target.build() (posit_bakery/image/image_target.py) calls the python_on_whales SDK directly. The existing ShellTask/ParallelShellExecutor primitives from #588 assume a raw subprocess command, so this isn't a drop-in port.

Scope (needs a design decision before implementation)

  • Investigate whether to:
    1. Extend posit_bakery/parallel/ with a callable-based job abstraction (run an arbitrary function per target on the executor, not just a shell command) — generalizing the direction ShellJob/CommandRunner took in feat: parallelize bakery ci publish across image targets #641/feat: retry-with-backoff for the parallel publish path #642 for oras/soci.
    2. Or shell out to docker build/podman build directly as a cmd: list[str], dropping down from the python_on_whales SDK for this strategy specifically.
  • Bound concurrency and preserve existing --fail-fast, retry, and BakeryBuildErrorGroup aggregation behavior.
  • Preserve _merge_sequential_build_metadata_files() metadata-file writing behavior when --metadata-file is set.

Out of scope

  • Changing --strategy bake behavior (already parallel via Buildx Bake).

Part of the investigation started in #584.

Metadata

Metadata

Assignees

No one assigned

    Labels

    cvp:0Necessary projects we are undertaking that don’t directly deliver value to the customerdockerRelated to container images we producetdp:1The individual working on it notices.

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions