Skip to content

fix(install-catalog-source): use the same logic for installing unreleased CI builds on OCP [RHDHBUGS-2489]#2126

Merged
rm3l merged 1 commit into
redhat-developer:mainfrom
rm3l:rhdhbugs-2489-the-ci-job-failed-during-the-dynamic-plugin-installation-step-due-to-an-inability-to-download-the-plugin-catalog-index-image-from-the-red-hat-registry
Jan 14, 2026
Merged

fix(install-catalog-source): use the same logic for installing unreleased CI builds on OCP [RHDHBUGS-2489]#2126
rm3l merged 1 commit into
redhat-developer:mainfrom
rm3l:rhdhbugs-2489-the-ci-job-failed-during-the-dynamic-plugin-installation-step-due-to-an-inability-to-download-the-plugin-catalog-index-image-from-the-red-hat-registry

Conversation

@rm3l

@rm3l rm3l commented Jan 14, 2026

Copy link
Copy Markdown
Member

Description

We used to use different methods depending on the type of OCP cluster:

  • regular OCP cluster: leveraging
    ImageContentSourcePolicy/ImageDigestMirrorSet to let OCP transparently mirror
    image pulls from internal registries to quay.io.
  • OCP cluster with hosted control plane: unpacking the operator bundles
    and manually replacing the internal image refs, because the ICSP/IDMS
    approach would not work there

The regular OCP cluster approach works for images that are pulled by
the underlying kubelet, but won't work for image pulls that are done
inside a container (as is the case now with the RHDH
install-dynamic-plugins init container that needs to pull the plugin
catalog index image).

So this simplifies the logic by always using the same method for
installing CI builds of the Operator on OCP.

Which issue(s) does this PR fix or relate to

PR acceptance criteria

  • Tests
  • Documentation

How to test changes / Special notes to the reviewer

On a regular OCP cluster, install the next downstream build of the Operator:

.rhdh/scripts/install-rhdh-catalog-source.sh --next --install-operator rhdh

Then create a Backstage CR. Without the changes here, the RHDH init container would fail to start because it would be trying to pull the catalog index image from registry.redhat.io
With the changes here, it should start properly, and the RHDH instance should be accessible.

image

…d builds of the Operator on OCP [RHDHBUGS-2489]

We used to use different methods depending on the type of OCP cluster:
- regular OCP cluster: leveraging
  ImageContentSourcePolicy/ImageDigestMirrorSet to let OCP transparently mirror
  image pulls from internal registries to quay.io.
- OCP cluster with hosted control plane: unpacking the operator bundles
  and manually replacing the internal image refs, because the ICSP/IDMS
  approach would not work there

The regular OCP cluster approach works for images that are pulled by
the underlying kubelet, but won't work for image pulls that are done
inside a container (as is the case now with the RHDH
install-dynamic-plugins init container that needs to pull the plugin
catalog index image).

So this simplifies the logic by always using the same method for
installing CI builds of the Operator on OCP.
@sonarqubecloud

Copy link
Copy Markdown

@rhdh-qodo-merge

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

RHDHBUGS-2489 - Partially compliant

Compliant requirements:

  • Fix CI failure during dynamic plugin installation caused by inability to download the plugin catalog index image from registry.redhat.io.

Non-compliant requirements:

  • Ensure a valid pull secret for registry.redhat.io exists in the showcase namespace.
  • Ensure the pod/service account used has the correct image pull secret attached.

Requires further human verification:

  • Address the authentication/authorization failure to Red Hat Registry in CI (missing/invalid pull secret).
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🔒 No security concerns identified
⚡ Recommended focus areas for review

Behavior Change

The script now unconditionally uses the bundle-rewrite/rebuild path on OpenShift (newIIBImage=$(ocp_install)), and unconditionally requires umoci and opm on any OCP cluster. Validate that this is intended for all OCP environments (including “regular” clusters) and that CI/dev environments running this script reliably have these tools available, otherwise installs may start failing even when registry mirroring would have worked.

function ocp_install() {
  # Clusters with an hosted control plane do not propagate ImageContentSourcePolicy/ImageDigestMirrorSet resources
  # to the underlying nodes, causing an issue mirroring internal images effectively.
  # Also, ImageContentSourcePolicy/ImageDigestMirrorSet only works when the images are pulled by the underlying Kubelet,
  # not if the images need to be pulled inside a container (as is the case with the RHDH init container that needs to
  # pull images like the catalog index image).
  # This function works around this by locally modifying the bundles (replacing all refs to the internal registries
  # with their mirrors on quay.io), rebuilding and pushing the images to the internal cluster registry.
  # TODO: this will work by replacing the catalog index image ref in the operator bundle, but we may need to do the same
  # type of replacement for the plugin references that are mentioned in the catalog index image itself (and rebuild it).

  set -euo pipefail

  render_iib >&2

  # 1. Expose the internal cluster registry if not done already
  debugf "Exposing cluster registry..." >&2
  internal_registry_url="image-registry.openshift-image-registry.svc:5000"
  oc patch configs.imageregistry.operator.openshift.io/cluster --patch '{"spec":{"defaultRoute":true}}' --type=merge >&2
  my_registry=$(oc get route default-route -n openshift-image-registry --template='{{ .spec.host }}')
  skopeo login -u kubeadmin -p "$(oc whoami -t)" --tls-verify=false "$my_registry" >&2
  if oc -n openshift-marketplace get secret internal-reg-auth-for-rhdh &> /dev/null; then
    oc -n openshift-marketplace delete secret internal-reg-auth-for-rhdh >&2
  fi
  if oc -n openshift-marketplace get secret internal-reg-ext-auth-for-rhdh &> /dev/null; then
    oc -n openshift-marketplace delete secret internal-reg-ext-auth-for-rhdh >&2
  fi
  oc -n openshift-marketplace create secret docker-registry internal-reg-ext-auth-for-rhdh \
    --docker-server="${my_registry}" \
    --docker-username=kubeadmin \
    --docker-password="$(oc whoami -t)" \
    --docker-email="admin@internal-registry-ext.example.com" >&2
  oc -n openshift-marketplace create secret docker-registry internal-reg-auth-for-rhdh \
    --docker-server="${internal_registry_url}" \
    --docker-username=kubeadmin \
    --docker-password="$(oc whoami -t)" \
    --docker-email="admin@internal-registry.example.com" >&2
  for ns in rhdh-operator rhdh; do
    # To be able to push images under this scope in the internal image registry
    if ! oc get namespace "$ns" > /dev/null; then
      oc create namespace "$ns" >&2
    fi
    oc adm policy add-cluster-role-to-user system:image-signer system:serviceaccount:${ns}:default >&2 || true
  done
  oc policy add-role-to-user system:image-puller system:serviceaccount:openshift-marketplace:default -n openshift-marketplace >&2 || true
  oc policy add-role-to-user system:image-puller system:serviceaccount:rhdh-operator:default -n rhdh-operator >&2 || true

  # 3. Regenerate the IIB image with the local changes to the render.yaml file and build and push it from within the cluster
  update_refs_in_iib_bundles "$internal_registry_url" "$my_registry" >&2

  debugf "Submitting in-cluster build request for the updated IIB..." >&2
  if ! oc -n rhdh get buildconfig.build.openshift.io/iib >& /dev/null; then
    oc -n rhdh new-build --strategy docker --binary --name iib >&2
  fi
  oc -n rhdh patch buildconfig.build.openshift.io/iib -p '{"spec": {"strategy": {"dockerStrategy": {"dockerfilePath": "rhdh.Dockerfile"}}}}' >&2
  oc -n rhdh start-build iib --wait --follow --from-dir=rhdh >&2
  local imageStreamWithTag="rhdh/iib:${IIB_TAG}"
  oc tag rhdh/iib:latest "${imageStreamWithTag}" >&2

  local result="${internal_registry_url}/${imageStreamWithTag}"
  debugf "IIB built and pushed to internal cluster registry: $result..." >&2
  printf "%s" "${result}"
}

function k8s_install() {
  set -euo pipefail

  local namespace="rhdh-operator"
  local image="registry:2"
  local registry_name="local-registry"
  local username="registryuser"
  local password
  password=$(echo "$RANDOM" | base64 | head -c 20)

  render_iib >&2

  if ! invoke_cluster_cli get namespace "${namespace}" &> /dev/null; then
    invoke_cluster_cli create namespace "${namespace}" >&2
  fi

  # We cannot use ICSP/IDMS resources here => deploy a registry where the updated IIBs will be pushed to
  if invoke_cluster_cli -n "${namespace}" get secret "${registry_name}-auth-creds" &> /dev/null; then
    username=$(invoke_cluster_cli -n "${namespace}" get secret "${registry_name}-auth-creds" -o json | jq -r '.data.username' | base64 -d)
    password=$(invoke_cluster_cli -n "${namespace}" get secret "${registry_name}-auth-creds" -o json | jq -r '.data.password' | base64 -d)
  else
    debugf "Generating auth secret for mirror registry. FYI, those creds will be stored in a secret named '${registry_name}-auth-creds' in ${namespace} ..." >&2
    cat <<EOF | invoke_cluster_cli apply -f - >&2
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: "${registry_name}-auth-creds"
  namespace: "${namespace}"
stringData:
  username: "${username}"
  password: "${password}"
EOF
  fi

  registry_htpasswd=$(htpasswd -Bbn "${username}" "${password}")
  cat <<EOF | invoke_cluster_cli apply -f - >&2
apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: "${registry_name}-auth"
  namespace: "${namespace}"
stringData:
  htpasswd: "${registry_htpasswd}"
EOF

  debugf "Creating the registry Deployment: deployment/${registry_name} ..." >&2
  cat <<EOF | invoke_cluster_cli apply -f - >&2
apiVersion: apps/v1
kind: Deployment
metadata:
  name: "${registry_name}"
  namespace: "${namespace}"
  labels:
    app: "${registry_name}"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: "${registry_name}"
  template:
    metadata:
      labels:
        app: "${registry_name}"
    spec:
      containers:
        - image: "${image}"
          name: "${registry_name}"
          imagePullPolicy: IfNotPresent
          env:
            - name: REGISTRY_AUTH
              value: "htpasswd"
            - name: REGISTRY_AUTH_HTPASSWD_REALM
              value: "RHDH Private Registry"
            - name: REGISTRY_AUTH_HTPASSWD_PATH
              value: "/auth/htpasswd"
            - name: REGISTRY_STORAGE_DELETE_ENABLED
              value: "true"
          ports:
            - containerPort: 5000
          volumeMounts:
            - name: auth-vol
              mountPath: "/auth"
              readOnly: true
      # -----------------------------------------------------------------------
      volumes:
        - name: auth-vol
          secret:
            secretName: "${registry_name}-auth"
EOF

  debugf "Creating the registry Service: service/${registry_name} ..." >&2
  # NOTE: We need a NodePort here, for the kubelet to be able to pull insecurely from localhost.
  # We cannot use the internal Service DNS name (ending with .svc.cluster.local) because the Kubelet is responsible for
  # pulling the images and does not seem able to resolve such internal DNS names.
  cat <<EOF | invoke_cluster_cli apply -f - >&2
apiVersion: v1
kind: Service
metadata:
  name: "${registry_name}"
  namespace: "${namespace}"
  labels:
    app: "${registry_name}"
spec:
  type: NodePort
  ports:
    - port: 5000
      protocol: TCP
      targetPort: 5000
  selector:
    app: "${registry_name}"
EOF
  debugf "Waiting for service $registry_name in namespace $namespace to become ready..." >&2
  sleep 7
  local registrySvcNodePort
  registrySvcNodePort=$(invoke_cluster_cli -n "${namespace}" get service "$registry_name" -o jsonpath='{.spec.ports[0].nodePort}')
  if [[ -z "$registrySvcNodePort" ]]; then
    errorf "NodePort not allocated for service/$registry_name or Service not found"
    return 1
  fi

  debugf "Waiting for deployment $registry_name in namespace $namespace to become ready..." >&2
  if ! invoke_cluster_cli rollout status deployment/$registry_name -n $namespace --timeout=5m &>/dev/null; then
    errorf "Timed out waiting for deployment $registry_name to be ready." >&2
    return 1
  fi

  local registry_port_fwd_out="${TMPDIR}/k8s.registry_port_fwd.out.txt"
  invoke_cluster_cli port-forward "service/$registry_name" -n "$namespace" :5000 &> "${registry_port_fwd_out}" &
  local port_fwd_pid
  port_fwd_pid=$!
  debugf "Port-forwarding process: $port_fwd_pid" >&2
  sleep 7
  # Check if the port-forward is running
  if ! kill -0 $port_fwd_pid &> /dev/null; then
      errorf "Port-forwarding to the cluster registry failed to start. Logs:" >&2
      cat "${registry_port_fwd_out}"
      return 1
  fi
  trap '[[ -n "${port_fwd_pid:-}" ]] && kill ${port_fwd_pid} || true' EXIT

  local portFwdLocalPort
  portFwdLocalPort=$(grep -oP '127\.0\.0\.1:\K[0-9]+' "${registry_port_fwd_out}")
  if [[ -z "$portFwdLocalPort" ]]; then
      errorf "Failed to determine the local port. Logs:" >&2
      cat "${registry_port_fwd_out}"
      return 1
  fi
  debugf "Port-forwarding from localhost:${portFwdLocalPort} to the cluster registry..." >&2
  local internal_registry_url
  internal_registry_url="localhost:${registrySvcNodePort}"
  skopeo login -u "${username}" -p "${password}" --tls-verify=false "localhost:$portFwdLocalPort" >&2

  local kaniko_internal_registry_url
  kaniko_internal_registry_url="${registry_name}.${namespace}.svc.cluster.local:5000"

  invoke_cluster_cli -n olm create secret docker-registry internal-reg-ext-auth-for-rhdh \
      --docker-server="${internal_registry_url}" \
      --docker-username="${username}" \
      --docker-password="${password}" \
      --docker-email="admin@internal-registry-ext.example.com" \
      --dry-run=client -o=yaml | \
      invoke_cluster_cli apply -f - >&2
  invoke_cluster_cli -n olm create secret docker-registry internal-reg-auth-for-rhdh \
      --docker-server="${internal_registry_url}" \
      --docker-username="${username}" \
      --docker-password="${password}" \
      --docker-email="admin@internal-registry.example.com" \
      --dry-run=client -o=yaml | \
      invoke_cluster_cli apply -f - >&2

  # 3. Regenerate the IIB image with the local changes to the render.yaml file and build and push it from within the cluster
  update_refs_in_iib_bundles "$internal_registry_url" "localhost:$portFwdLocalPort" >&2

  local imageStreamWithTag="rhdh/iib:${IIB_TAG}"
  local result="${internal_registry_url}/${imageStreamWithTag}"
  local kanikoResult="${kaniko_internal_registry_url}/${imageStreamWithTag}"

  # 4. Rebuild the IIB image in the cluster using Kaniko
  debugf "Rebuilding the IIB Image using Kaniko in the cluster..." >&2
  invoke_cluster_cli -n "${namespace}" create secret docker-registry kaniko-registry-secret \
      --docker-server="${kaniko_internal_registry_url}" \
      --docker-username="${username}" \
      --docker-password="${password}" \
      --docker-email="admin@internal-registry-ext.kaniko.example.com" \
      --dry-run=client -o=yaml | \
      invoke_cluster_cli apply -f - >&2
  local timestamp
  local kanikoJobName
  local kanikoPod
  local kanikoLogsPid
  local localContext
  timestamp=$(date +%s)
  kanikoJobName="kaniko-build-${timestamp}"
  cat <<EOF | invoke_cluster_cli apply -f - >&2
apiVersion: batch/v1
kind: Job
metadata:
  name: "${kanikoJobName}"
  namespace: "${namespace}"
spec:
  backoffLimit: 0
  ttlSecondsAfterFinished: 3600
  parallelism: 1
  completions: 1
  template:
    spec:
      containers:
      - name: kaniko
        image: gcr.io/kaniko-project/executor:debug
        command: ["/bin/sh", "-c"]
        args:
        - |
            while [ ! -f /workspace/context.tar.gz ]; do echo 'Waiting for the build context archive...'; sleep 2; done
            /kaniko/executor --context=tar:///workspace/context.tar.gz --dockerfile=rhdh.Dockerfile --destination=$kanikoResult
        #resources:
        #  requests:
        #    cpu: 250m
        #    memory: 512Mi
        #  limits:
        #    cpu: 500m
        #    memory: 1Gi
        volumeMounts:
        - name: build-context
          mountPath: /workspace
        - name: registry-secret
          mountPath: /kaniko/.docker/
          readOnly: true
      restartPolicy: Never
      volumes:
      - name: build-context
        emptyDir: {}
      - name: registry-secret
        secret:
          secretName: kaniko-registry-secret
          items:
          - key: .dockerconfigjson
            path: config.json
EOF

  # RHDHBUGS-2478: the job pod might not be scheduled right away after job creation
  if ! invoke_cluster_cli -n "${namespace}" wait "job/${kanikoJobName}" --for=jsonpath='{.status.active}'=1 --timeout=300s >&2; then
    errorf "unable to get Kaniko job active. Something might be preventing Jobs from being scheduled properly in this cluster"
    invoke_cluster_cli -n "${namespace}" describe job "${kanikoJobName}" >&2
    return 1
  fi
  kanikoPod=$(invoke_cluster_cli -n "${namespace}" get pods --selector=job-name="${kanikoJobName}" -o jsonpath='{.items[0].metadata.name}')
  if [ -z "$kanikoPod" ]; then
    errorf "unable to determine the Kaniko Pod"
    return 1
  fi
  debugf "Waiting for Kaniko pod $kanikoPod to be ready..." >&2
  invoke_cluster_cli -n "${namespace}" wait --for=condition=Ready "pod/$kanikoPod" --timeout=60s >&2
  invoke_cluster_cli -n "${namespace}" logs -f "${kanikoPod}" >&2 &
  kanikoLogsPid=$!
  trap '[[ -n "${kanikoLogsPid:-}" ]] && kill ${kanikoLogsPid} &>/dev/null || true' EXIT

  localContext=context.tar.gz
  tar -czf "${localContext}" -C rhdh . >&2
  invoke_cluster_cli -n "${namespace}" cp "${localContext}" "${kanikoPod}:/workspace/${localContext}" >&2

  invoke_cluster_cli -n "${namespace}" wait --for=condition=complete "job/${kanikoJobName}" --timeout=300s >&2

  debugf "IIB built and pushed to internal cluster registry: $result..." >&2
  printf "%s" "${result}"
}

##########################################################################################
# Script start
##########################################################################################
if [[ "$#" -lt 1 ]]; then
  usage
  exit 0
fi

# minimum requirements
if ! command -v jq &> /dev/null; then
  errorf "Please install jq 1.2+ from an RPM or https://pypi.org/project/jq/"
  exit 1
fi
if ! command -v skopeo &> /dev/null; then
  errorf "Please install skopeo 1.11+"
  exit 1
fi

TMPDIR=$(mktemp -d)
pushd "${TMPDIR}" > /dev/null
debugf ">>> WORKING DIR: $TMPDIR <<<"
# shellcheck disable=SC2064
trap "rm -fr $TMPDIR || true" EXIT

detect_ocp_and_set_env_var
if [[ "${IS_OPENSHIFT}" = "true" ]]; then
  debugf "Detected an OpenShift cluster"
  if ! command -v oc &> /dev/null; then
    errorf "Please install oc 4.10+ from an RPM or https://mirror.openshift.com/pub/openshift-v4/clients/ocp/"
    exit 1
  fi
  # Check we're logged into a cluster
  if ! oc whoami &> /dev/null; then
    errorf "Not logged into an OpenShift cluster"
    exit 1
  fi
else
  if ! command -v oc &> /dev/null && ! command -v kubectl &> /dev/null; then
    errorf "Please install kubectl or invoke_cluster_cli 4.10+ (from an RPM or https://mirror.openshift.com/pub/openshift-v4/clients/ocp/)"
    exit 1
  fi
  debugf "Falling back to a standard K8s cluster"
  # Check that OLM is installed
  if ! invoke_cluster_cli get crd catalogsources.operators.coreos.com &> /dev/null; then
    errorf "
OLM not installed (CatalogSource CRD not found) or you don't have enough permissions.
Check that you are correctly logged into the cluster and that OLM is installed.
See https://olm.operatorframework.io/docs/getting-started/#installing-olm-in-your-cluster to install OLM."
    exit 1
  fi
fi

OCP_VER="v4.21"
OCP_ARCH="x86_64"
if [[ "${IS_OPENSHIFT}" = "true" ]]; then
  # log into your OCP cluster before running this or you'll get null values for OCP vars!
  ocpVerJson=$(invoke_cluster_cli version -o json)
  OCP_VER="v$(echo "$ocpVerJson" | jq -r '.openshiftVersion' | sed -r -e "s#([0-9]+\.[0-9]+)\..+#\1#")"
  if [[ $OCP_VER == "vnull" ]]; then # try releaseClientVersion = 4.16.14
    OCP_VER="v$(echo "$ocpVerJson" | jq -r '.releaseClientVersion' | sed -r -e "s#([0-9]+\.[0-9]+)\..+#\1#")"
  fi
  OCP_ARCH="$(echo "$ocpVerJson" | jq -r '.serverVersion.platform' | sed -r -e "s#linux/##")"
  if [[ $OCP_ARCH == "amd64" ]]; then
    OCP_ARCH="x86_64"
  fi
fi

# if logged in, this should return something like latest-v4.12-x86_64
IIB_TAG="latest-${OCP_VER}-${OCP_ARCH}"
TO_INSTALL=""

while [[ "$#" -gt 0 ]]; do
  case $1 in
    '--install-operator')
      TO_INSTALL="$2"
      shift 1
      ;;
    '--next'|'--latest')
      # if logged in, this should return something like latest-v4.12-x86_64 or next-v4.12-x86_64
      IIB_TAG="${1/--/}-${OCP_VER}-$OCP_ARCH"
      ;;
    '-v')
      IIB_TAG="${2}-${OCP_VER}-$OCP_ARCH"
      OLM_CHANNEL="fast-${2}"
      shift 1
      ;;
    '--catalog-source')
      UPSTREAM_IIB_OVERRIDE="$2"
      shift 1
      ;;
    '--install-plan-approval')
      if [[ "$2" != "Manual" && "$2" != "Automatic" ]]; then
        errorf "Unknown parameter used: $2. Must be Manual or Automatic."
        usage
        exit 1
      fi
      INSTALL_PLAN_APPROVAL="$2"
      shift 1
      ;;
    '-h'|'--help')
      usage
      exit 0
      ;;
    *)
      errorf "Unknown parameter is used: $1."
      usage
      exit 1
      ;;
  esac
  shift 1
done

if [[ $UPSTREAM_IIB_OVERRIDE ]]; then
  UPSTREAM_IIB="$UPSTREAM_IIB_OVERRIDE"
else
  UPSTREAM_IIB="quay.io/rhdh/iib:${IIB_TAG}"
fi

# shellcheck disable=SC2086
UPSTREAM_IIB_MANIFEST="$(skopeo inspect docker://${UPSTREAM_IIB} --raw || exit 2)"
if [[ $UPSTREAM_IIB_MANIFEST == *"Error parsing image name "* ]] || [[ $UPSTREAM_IIB_MANIFEST == *"manifest unknown"* ]]; then
  errorf "Problem with image $UPSTREAM_IIB: $UPSTREAM_IIB_MANIFEST"
  exit 3
else
  infof "Using IIB from image $UPSTREAM_IIB"
  IIB_IMAGE="${UPSTREAM_IIB}"
fi

# Add CatalogSource for the IIB
IIB_NAME="${UPSTREAM_IIB##*:}"
IIB_NAME="${IIB_NAME//_/-}"
IIB_NAME="${IIB_NAME//./-}"
IIB_NAME="$(echo "$IIB_NAME" | tr '[:upper:]' '[:lower:]')"
OPERATOR_NAME_TO_INSTALL=${TO_INSTALL:-rhdh}
if [[ $UPSTREAM_IIB_OVERRIDE == "brew.registry.redhat.io/rh-osbs/iib-pub-pending"* ]]; then
  CATALOGSOURCE_NAME="brew-registry-stage"
  DISPLAY_NAME_SUFFIX="brew-registry-stage"
else
  CATALOGSOURCE_NAME="${OPERATOR_NAME_TO_INSTALL}-${OLM_CHANNEL}"
  DISPLAY_NAME_SUFFIX="${IIB_NAME}"
fi

OPERATOR_GROUP_NAME="${OPERATOR_NAME_TO_INSTALL}-operator-group"

if [ -n "$TO_INSTALL" ]; then
  # OLM allows a single OperatorGroup per namespace.
  # Err out early if there are existing OperatorGroups in the Operator namespace.
  existing_ogs=$(invoke_cluster_cli get operatorgroup -n "${NAMESPACE_SUBSCRIPTION}" --no-headers -o custom-columns=":metadata.name" || true)
  filtered=$(echo "$existing_ogs" | grep -v "^${OPERATOR_GROUP_NAME}$" || true)
  debugf "filtered=$filtered"
  if [[ -n "$filtered" ]]; then
    errorf "
Only one OperatorGroup is allowed per namespace. The following were found in '${NAMESPACE_SUBSCRIPTION}':
---
${filtered}
---
Please remove them so that I can create/update the one I am expecting: '${OPERATOR_GROUP_NAME}'"
    exit 1
  fi
fi

# Using the current working dir, otherwise tools like 'skopeo login' will attempt to write to /run, which
# might be restricted in CI environments.
export REGISTRY_AUTH_FILE="${TMPDIR}/.auth.json"

newIIBImage=${IIB_IMAGE}

if [[ "${IS_OPENSHIFT}" = "true" ]]; then
  # Defaulting to the hosted control plane behavior which has more chances to work
  CONTROL_PLANE_TECH=$(oc get infrastructure cluster -o jsonpath='{.status.controlPlaneTopology}' || \
    (warnf 'Could not determine the cluster type => defaulting to the hosted control plane behavior' >&2 && echo 'External'))
  IS_HOSTED_CONTROL_PLANE="false"
  if [[ "${CONTROL_PLANE_TECH}" == "External" ]]; then
    # 'External' indicates that the control plane is hosted externally to the cluster
    # and that its components are not visible within the cluster.
    IS_HOSTED_CONTROL_PLANE="true"
  fi

  if [[ "${IS_HOSTED_CONTROL_PLANE}" = "true" ]]; then
    infof "Detected an OpenShift cluster with a hosted control plane"
  fi
  if ! command -v umoci &> /dev/null; then
    errorf "Please install umoci 0.4+. See https://github.com/opencontainers/umoci?tab=readme-ov-file#install"
    exit 1
  fi
  if ! command -v opm &> /dev/null; then
    errorf "Please install opm v1.47+. See https://github.com/operator-framework/operator-registry/releases"
    exit 1
  fi
  newIIBImage=$(ocp_install)
else
Incomplete Fix Risk

The added note indicates only the operator bundle’s catalog index reference may be replaced, but plugin image references embedded in the catalog index image itself may still point at registry.redhat.io. Validate end-to-end that the init container pull(s) no longer hit registry.redhat.io (or that any remaining pulls have valid creds), otherwise the original failure mode may persist.

# Also, ImageContentSourcePolicy/ImageDigestMirrorSet only works when the images are pulled by the underlying Kubelet,
# not if the images need to be pulled inside a container (as is the case with the RHDH init container that needs to
# pull images like the catalog index image).
# This function works around this by locally modifying the bundles (replacing all refs to the internal registries
# with their mirrors on quay.io), rebuilding and pushing the images to the internal cluster registry.
# TODO: this will work by replacing the catalog index image ref in the operator bundle, but we may need to do the same
# type of replacement for the plugin references that are mentioned in the catalog index image itself (and rebuild it).
📚 Focus areas based on broader codebase context

Prerequisites

The script adds inline prerequisite checks for umoci and opm, but does not follow the existing pattern of a single prerequisite-check function and may miss validating other required CLIs used on this path (e.g., oc, kubectl, curl). Consider consolidating prerequisite validation into a dedicated function and calling it before installation logic to avoid partial failures mid-run. (Ref 2, Ref 5)

if ! command -v umoci &> /dev/null; then
  errorf "Please install umoci 0.4+. See https://github.com/opencontainers/umoci?tab=readme-ov-file#install"
  exit 1
fi
if ! command -v opm &> /dev/null; then
  errorf "Please install opm v1.47+. See https://github.com/operator-framework/operator-registry/releases"
  exit 1
fi
newIIBImage=$(ocp_install)

Reference reasoning: Other bash utilities in this codebase centralize CLI validation early (e.g., a checkPrerequisite function that verifies required commands like oc exist) and then run the main flow. Reusing that approach here would make requirements explicit and prevent runtime failures after the script has already mutated cluster state.

📄 References
  1. redhat-developer/rhdh-chart/hack/merge_secrets.sh [1-47]
  2. redhat-developer/rhdh-operator/config/profile/rhdh/plugin-infra/gitops-secret-setup.sh [133-182]
  3. redhat-developer/rhdh-chart/hack/orchestrator-templates-setup.sh [256-296]
  4. redhat-developer/rhdh-operator/config/profile/rhdh/plugin-infra/gitops-secret-setup.sh [199-222]
  5. redhat-developer/rhdh-operator/config/profile/rhdh/plugin-infra/plugin-infra.sh [1-49]
  6. redhat-developer/rhdh-operator/config/profile/rhdh/plugin-infra/plugin-infra.sh [50-75]
  7. redhat-developer/rhdh-operator/config/profile/rhdh/plugin-infra/gitops-secret-setup.sh [1-35]
  8. redhat-developer/rhdh-operator/config/profile/rhdh/plugin-infra/gitops-secret-setup.sh [87-130]

@rhdh-qodo-merge rhdh-qodo-merge Bot added enhancement New feature or request Bug fix labels Jan 14, 2026
@rhdh-qodo-merge

Copy link
Copy Markdown

PR Type

Bug fix, Enhancement


Description

  • Unify OCP cluster installation method for unreleased operator builds

  • Remove ImageContentSourcePolicy/ImageDigestMirrorSet approach that fails for container image pulls

  • Always use bundle unpacking and image reference replacement for all OCP clusters

  • Simplify conditional logic by eliminating separate regular vs hosted control plane paths


File Walkthrough

Relevant files
Bug fix
install-rhdh-catalog-source.sh
Consolidate OCP installation methods into single approach

.rhdh/scripts/install-rhdh-catalog-source.sh

  • Removed ocp_install_regular_cluster() function that used
    ImageContentSourcePolicy/ImageDigestMirrorSet
  • Renamed ocp_install_hosted_control_plane_cluster() to ocp_install() as
    the unified installation method
  • Moved tool dependency checks (umoci and opm) outside conditional block
    to always validate requirements
  • Updated comments to explain why bundle unpacking approach is needed
    for container image pulls
  • Simplified main logic to always call ocp_install() regardless of
    cluster type
+16/-86 

@rhdh-qodo-merge

Copy link
Copy Markdown

PR Code Suggestions ✨

Explore these optional code suggestions:

CategorySuggestion                                                                                                                                    Impact
Possible issue
Emit rebuilt image at end

The ocp_install function should output the rebuilt image reference to standard
output so the calling code can capture it. Add a printf or echo statement at the
end of the function to return the image name.

.rhdh/scripts/install-rhdh-catalog-source.sh [206-217]

 fi
 if ! command -v opm &> /dev/null; then
   errorf "Please install opm v1.47+. See https://github.com/operator-framework/operator-registry/releases"
   exit 1
 fi
+# Output the rebuilt IIB image for capture by the caller
+printf "%s" "${IIB_IMAGE}"

[To ensure code accuracy, apply this suggestion manually]

Suggestion importance[1-10]: 9

__

Why: The suggestion correctly identifies a critical bug where the ocp_install function fails to return the new image reference, causing newIIBImage to be empty. This breaks the script's core logic.

High
High-level
The fix is incomplete

The PR fixes the pull issue for the catalog index image but neglects to address
the plugin images referenced within that catalog. This omission, noted in a
TODO, means the fix is partial and will likely lead to similar image pull
failures later.

Examples:

.rhdh/scripts/install-rhdh-catalog-source.sh [214-215]
  # TODO: this will work by replacing the catalog index image ref in the operator bundle, but we may need to do the same
  # type of replacement for the plugin references that are mentioned in the catalog index image itself (and rebuild it).

Solution Walkthrough:

Before:

# .rhdh/scripts/install-rhdh-catalog-source.sh

if [[ "${IS_HOSTED_CONTROL_PLANE}" = "true" ]]; then
  # Unpacks bundle and replaces image refs
  newIIBImage=$(ocp_install_hosted_control_plane_cluster)
else
  # Uses ICSP/IDMS, which fails for in-container image pulls
  newIIBImage=$(ocp_install_regular_cluster)
fi

After:

# .rhdh/scripts/install-rhdh-catalog-source.sh

function ocp_install() {
  # ...
  # This function now replaces image refs in the operator bundle
  # to fix pulling the catalog index image.
  #
  # TODO: this will work by replacing the catalog index image ref in the operator bundle,
  # but we may need to do the same type of replacement for the plugin references
  # that are mentioned in the catalog index image itself (and rebuild it).
  # ...
}

# Always use the bundle modification method for all OCP clusters.
newIIBImage=$(ocp_install)
Suggestion importance[1-10]: 8

__

Why: The suggestion correctly identifies a significant limitation, acknowledged in a TODO comment, where the fix is partial and may lead to subsequent failures, impacting the overall functionality.

Medium
Organization
best practice
Validate inputs and quote expansions

Validate required inputs (e.g., IIB_IMAGE) before running ocp_install, and quote
the command substitution to prevent word-splitting/globbing issues.

.rhdh/scripts/install-rhdh-catalog-source.sh [716-727]

 if [[ "${IS_HOSTED_CONTROL_PLANE}" = "true" ]]; then
   infof "Detected an OpenShift cluster with a hosted control plane"
 fi
+: "${IIB_IMAGE:?IIB_IMAGE environment variable not set}"
 if ! command -v umoci &> /dev/null; then
   errorf "Please install umoci 0.4+. See https://github.com/opencontainers/umoci?tab=readme-ov-file#install"
   exit 1
 fi
 if ! command -v opm &> /dev/null; then
   errorf "Please install opm v1.47+. See https://github.com/operator-framework/operator-registry/releases"
   exit 1
 fi
-newIIBImage=$(ocp_install)
+newIIBImage="$(ocp_install)"
  • Apply / Chat
Suggestion importance[1-10]: 6

__

Why:
Relevant best practice - Harden shell scripts by validating required environment variables and quoting command substitutions to avoid subtle runtime errors.

Low
  • More

@rm3l rm3l changed the title fix(install-catalog-source): use the same way of installing unreleased builds of the Operator on OCP [RHDHBUGS-2489] fix(install-catalog-source): use the same logic for installing unreleased builds of the Operator on OCP [RHDHBUGS-2489] Jan 14, 2026
@rm3l rm3l changed the title fix(install-catalog-source): use the same logic for installing unreleased builds of the Operator on OCP [RHDHBUGS-2489] fix(install-catalog-source): use the same logic for installing unreleased CI builds on OCP [RHDHBUGS-2489] Jan 14, 2026
@Fortune-Ndlovu

Fortune-Ndlovu commented Jan 14, 2026

Copy link
Copy Markdown
Member

works as expected /approve

@Fortune-Ndlovu Fortune-Ndlovu left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@openshift-ci

openshift-ci Bot commented Jan 14, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Fortune-Ndlovu
Once this PR has been reviewed and has the lgtm label, please assign rm3l for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@rm3l rm3l merged commit 7a8c547 into redhat-developer:main Jan 14, 2026
10 of 11 checks passed
@rm3l rm3l deleted the rhdhbugs-2489-the-ci-job-failed-during-the-dynamic-plugin-installation-step-due-to-an-inability-to-download-the-plugin-catalog-index-image-from-the-red-hat-registry branch January 14, 2026 15:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants