Skip to content

Use oc adm release extract --tools to download OCP binaries instead of release-controller file-cache#15

Open
tusharjadhav3302 wants to merge 1 commit into
mainfrom
use-oc-adm-release-extract-for-binaries
Open

Use oc adm release extract --tools to download OCP binaries instead of release-controller file-cache#15
tusharjadhav3302 wants to merge 1 commit into
mainfrom
use-oc-adm-release-extract-for-binaries

Conversation

@tusharjadhav3302

Copy link
Copy Markdown

Summary

Replace the release-controller file-cache (openshift-release-artifacts) with
oc adm release extract --tools for downloading OCP installer and client
binaries. The file-cache has no SLA or guaranteed support and can get stuck
indefinitely during tool extraction, causing job failures.

This was triggered by a 4.21 nightly job failure (OSPRH-31439) on 17 June 2026
where the artifacts server got permanently stuck extracting tools for build
4.21.0-0.nightly-2026-06-16-000931 — the URL is still stuck 2 days later.

What changed

  • get_openshift_release_build_name.yml: Save the pullSpec from the
    release stream API response (already returned but previously discarded).

  • get_openshift_release_binaries.yml: Replace HTTP polling of the
    file-cache with oc adm release extract --tools, which pulls binaries
    directly from the container registry. The pull secret is extracted at
    runtime from the host cluster via rhoso_kubeconfig.

Why

Per release-controller maintainer Brad Williams:

"The release-controller's file-cache is not bound by any SLA or guaranteed
support and therefore probably shouldn't be used for automation."

"Much more reliable to call the command directly and work locally:
oc adm release extract --tools --to=<PATH> <pullSpec>"

Manual validation on serval70 (18 June 2026)

All steps tested inside the shiftstackclient-shiftstack pod on a live
RHOSO 18.0 deployment:

  1. oc available: /usr/local/bin/oc (4.22.1) present in the pod
  2. Host cluster pull secret accessible via rhoso_kubeconfig:
    registry.ci.openshift.org confirmed present in auths
  3. oc adm release extract --tools succeeded:
    Downloaded all tarballs for 4.21.0-0.nightly-2026-06-18-005110
  4. Binaries verified:
    • openshift-install version4.21.0-0.nightly-2026-06-18-005110
    • oc version --client4.21.0-0.nightly-2026-06-18-005110
  5. Two kubeconfigs confirmed:
    • Default oc in pod → guest cluster (api.ostest.shiftstack.local)
    • --kubeconfig=rhoso_kubeconfig → host cluster (api.ocp.openstack.lab)
    • Pull secret correctly extracted from host cluster at all times

Scope / limitations

  • Covers nightly (build_name: "") and 4-stable code paths — both go
    through the release stream API which returns pullSpec.
  • Channel-based paths (candidate/fast/stable/eus) do not use the release
    stream API and are unaffected by this change.

@tusharjadhav3302 tusharjadhav3302 added the ready-for-review PR is ready for code review label Jun 18, 2026
@dlaw4608

Copy link
Copy Markdown

LGTM, once E2E testing is done in a live environment and this passes this will be good to merge

@imatza-rh imatza-rh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main concern is build type coverage - openshift_release_pull_spec is only set for the nightly/4-stable path today, but we need it for all paths (candidate, fast, stable, eus, specific builds). The release stream API returns pullSpec for nightly and 4-stable. For channel builds, release.txt on mirror.openshift.com has a Pull From: field with a digest-pinned pull spec that could be parsed directly. See the inline comment for details.

Also worth removing openshift_download_url from defaults/main.yml since it's now unused, and the task name fix from "installer" to "client" at line 72 is a nice catch.

@tusharjadhav3302 tusharjadhav3302 force-pushed the use-oc-adm-release-extract-for-binaries branch from 853342b to bda356e Compare June 18, 2026 11:10
@tusharjadhav3302

Copy link
Copy Markdown
Author

Update: CI validation findings and revised approach

We ran the Zuul test on a clean environment and discovered that the shiftstackclient pod starts from a completely clean state — no oc binary is pre-installed. The previous approach assumed oc would be available in the pod to extract the pull secret and run oc adm release extract, but that creates a circular dependency since oc itself is what this role installs.

Root cause of the CI failure: rc=127 (command not found) on both pull secret extraction attempts — the pod has no oc in PATH.

Revised approach (validated manually on a clean pod):

  1. Pull secret extraction — Instead of oc get secret, we now use curl with the kubeconfig's client certificate to call the Kubernetes API directly. No external binaries needed beyond curl and python3, both of which are always present in the pod.

  2. Bootstrap oc — Download a stable oc client tarball from mirror.openshift.com using ansible.builtin.unarchive (HTTP download, no auth required, no file-cache involved). This gives us a working oc to run the extract.

  3. oc adm release extract --tools — Using the bootstrapped oc + the extracted pull secret, pull version-matched binaries directly from the release image. Works for both GA and nightly releases.

  4. Cleanup — Both the pull secret file and bootstrap oc directory are removed in an always: block.

Manual validation results (clean pod, no pre-existing binaries):

Step Result
Pull secret via curl + client certs 6 registries extracted
Bootstrap oc from mirror Downloaded and working
oc adm release extract --tools (4-stable → 4.22.2) Completed in ~63 seconds
Installer binary openshift-install 4.22.2 verified
Client binary oc 4.22.2 verified

New commit pushed — ready for re-review.

@tusharjadhav3302

Copy link
Copy Markdown
Author

@dlaw4608 dlaw4608 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice once getting it passing LGTM

@imatza-rh imatza-rh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good after the revisions - all prior review points addressed. One non-blocking suggestion inline. Consider squashing the 5 commits before merge.

@tusharjadhav3302 tusharjadhav3302 force-pushed the use-oc-adm-release-extract-for-binaries branch from 8548bec to 4e56ee5 Compare June 25, 2026 14:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-review PR is ready for code review

Development

Successfully merging this pull request may close these issues.

4 participants