Skip to content

NO-JIRA: Add E2E coverage reporting for secrets-store-csi-driver-operator #79231

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
PillaiManish wants to merge 1 commit into
base: main
Choose a base branch
from
Open

NO-JIRA: Add E2E coverage reporting for secrets-store-csi-driver-operator #79231

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
142 changes: 142 additions & 0 deletions ...t/secrets-store-csi-driver-operator/openshift-secrets-store-csi-driver-operator-main.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,21 @@ images:
- dockerfile_path: Dockerfile.mustgather
from: ocp_4.14_cli
to: secrets-store-csi-mustgather
- dockerfile_literal: |
FROM registry.ci.openshift.org/ocp/builder:rhel-9-golang-1.25-openshift-4.22 AS builder
WORKDIR /go/src/github.com/openshift/secrets-store-csi-driver-operator
COPY . .
RUN CGO_ENABLED=1 GOEXPERIMENT=strictfipsruntime go build \
-mod=vendor -trimpath \
-tags strictfipsruntime,openssl \
-cover -covermode=atomic -coverpkg=./... \
-o secrets-store-csi-driver-operator \
./cmd/secrets-store-csi-driver-operator
FROM registry.ci.openshift.org/ocp/4.22:base-rhel9
COPY --from=builder /go/src/github.com/openshift/secrets-store-csi-driver-operator/secrets-store-csi-driver-operator /usr/bin/
ENV GOCOVERDIR=/tmp/e2e-cover
ENTRYPOINT ["/bin/sh", "-c", "mkdir -p /tmp/e2e-cover && exec /usr/bin/secrets-store-csi-driver-operator \"$@\"", "--"]
to: secrets-store-csi-driver-operator-coverage
operator:
bundles:
- as: secrets-store-csi-driver-operator-bundle
Expand Down Expand Up @@ -96,9 +111,136 @@ tests:
OO_INSTALL_NAMESPACE: openshift-cluster-csi-drivers
OO_PACKAGE: secrets-store-csi-driver-operator
OO_TARGET_NAMESPACES: '!all'
post:
- as: collect-coverage
best_effort: true
cli: latest
commands: |
set -euo pipefail
NAMESPACE="openshift-cluster-csi-drivers"
DEPLOYMENT="secrets-store-csi-driver-operator"
POD_LABEL="app=secrets-store-csi-driver-operator"
GOCOVERDIR_PATH="/tmp/e2e-cover"
CODECOV_SECRET="/var/run/secrets/codecov/CODECOV_TOKEN"

artifact_dir="${ARTIFACT_DIR:-.}"
coverage_dir="${artifact_dir}/e2e-cover-data"
coverage_profile="${artifact_dir}/coverage-e2e.out"

if [ -z "${CODECOV_TOKEN:-}" ] && [ -f "${CODECOV_SECRET}" ]; then
CODECOV_TOKEN=$(cat "${CODECOV_SECRET}")
export CODECOV_TOKEN
fi

pod=$(oc get pods -n "${NAMESPACE}" -l "${POD_LABEL}" \
-o jsonpath='{.items[0].metadata.name}' 2>/dev/null)
if [ -z "${pod}" ]; then
echo "Error: no operator pod found"; exit 1
fi
echo "Found operator pod: ${pod}"

echo "Sending SIGTERM to flush coverage data (container will restart)..."
oc exec -n "${NAMESPACE}" "${pod}" -- /bin/sh -c 'kill -TERM 1' 2>/dev/null || true

echo "Waiting for container to restart and become ready..."
sleep 5
oc wait pod "${pod}" -n "${NAMESPACE}" --for=condition=Ready --timeout=120s

echo "Copying coverage data from restarted container..."
mkdir -p "${coverage_dir}"
oc cp "${NAMESPACE}/${pod}:${GOCOVERDIR_PATH}/." "${coverage_dir}"
Comment on lines +142 to +151
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot May 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical | 🏗️ Heavy lift

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Check if there's a volume mount defined for the coverage directory in the operator deployment
# or in related CSV/deployment manifests

# Search for volume definitions related to e2e-cover in the repo
rg -n "e2e-cover|GOCOVERDIR" --type yaml -C3

# Check if emptyDir or any volume is configured for /tmp
ast-grep --pattern 'volumes:
  $$$
  emptyDir
  $$$'

Repository: openshift/release

Length of output: 8693

🏁 Script executed:

# Look at the full deployment definition in the CI config file
grep -n "volumes:" ci-operator/config/openshift/secrets-store-csi-driver-operator/openshift-secrets-store-csi-driver-operator-main.yaml

# Also check if there's any volumeMount defined
grep -n "volumeMounts:" ci-operator/config/openshift/secrets-store-csi-driver-operator/openshift-secrets-store-csi-driver-operator-main.yaml

# Check the exact lines around the deployment spec
sed -n '200,250p' ci-operator/config/openshift/secrets-store-csi-driver-operator/openshift-secrets-store-csi-driver-operator-main.yaml

Repository: openshift/release

Length of output: 2048

🏁 Script executed:

# Look at lines 140-160 to see the post-step context
sed -n '140,160p' ci-operator/config/openshift/secrets-store-csi-driver-operator/openshift-secrets-store-csi-driver-operator-main.yaml

# Also check for any volumes definition in the whole file
grep -n "volumes:" ci-operator/config/openshift/secrets-store-csi-driver-operator/openshift-secrets-store-csi-driver-operator-main.yaml

# Check if there are volumeMounts anywhere
grep -n "volumeMounts:" ci-operator/config/openshift/secrets-store-csi-driver-operator/openshift-secrets-store-csi-driver-operator-main.yaml

Repository: openshift/release

Length of output: 1000

🏁 Script executed:

# Get the full test-e2e-post step definition to see complete context
grep -n "test-e2e-post" ci-operator/config/openshift/secrets-store-csi-driver-operator/openshift-secrets-store-csi-driver-operator-main.yaml -A 100

# Also look for any emptyDir or volume definitions in the entire file
cat ci-operator/config/openshift/secrets-store-csi-driver-operator/openshift-secrets-store-csi-driver-operator-main.yaml | grep -i "emptydir\|volumes\|volumemount"

Repository: openshift/release

Length of output: 43

🏁 Script executed:

# Let's read the entire CI config file to understand the full structure
wc -l ci-operator/config/openshift/secrets-store-csi-driver-operator/openshift-secrets-store-csi-driver-operator-main.yaml

# Read the file
cat ci-operator/config/openshift/secrets-store-csi-driver-operator/openshift-secrets-store-csi-driver-operator-main.yaml

Repository: openshift/release

Length of output: 14132

Coverage data will be lost after sending SIGTERM to the container.

When SIGTERM is sent to PID 1, the Go runtime flushes coverage to /tmp/e2e-cover in the current container instance. However, /tmp is part of the container's ephemeral filesystem with no persistent volume backing it. When the container restarts, Kubernetes creates a new container instance with a fresh /tmp, discarding the old container's filesystem entirely. The subsequent oc cp will copy from the new container's empty /tmp/e2e-cover directory.

To fix this, use a persistent volume (emptyDir or hostPath) mounted at the coverage directory path, or refactor to collect coverage data before triggering the container restart.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/config/openshift/secrets-store-csi-driver-operator/openshift-secrets-store-csi-driver-operator-main.yaml`
around lines 142 - 151, The current flow sends SIGTERM to PID 1 then waits for
the pod to restart and runs oc cp from ${GOCOVERDIR_PATH}, but that path is on
the container ephemeral filesystem so the coverage files are lost when the
container restarts; update the job to either (A) ensure ${GOCOVERDIR_PATH} is
backed by a persistent volume (e.g., mount an emptyDir or hostPath to the same
path in the pod spec) so files survive the restart, or (B) copy the coverage
files out of the running container before sending kill -TERM 1 (use oc cp from
${NAMESPACE}/${pod}:${GOCOVERDIR_PATH} to ${coverage_dir} prior to the oc exec
'kill -TERM 1'), referencing the variables GOCOVERDIR_PATH, coverage_dir, oc
exec, and oc cp in your changes.


echo "Coverage files:"
ls -la "${coverage_dir}/" 2>/dev/null || true

if ls "${coverage_dir}"/covmeta.* >/dev/null 2>&1; then
echo "Converting coverage data..."
go tool covdata textfmt -i="${coverage_dir}" -o="${coverage_profile}"
echo "=== E2E Coverage Summary ==="
go tool covdata percent -i="${coverage_dir}"
echo "============================="

if [ -n "${CODECOV_TOKEN:-}" ]; then
echo "Uploading to Codecov..."
codecov_bin="${artifact_dir}/codecov"
curl -sS -o "${codecov_bin}" https://uploader.codecov.io/latest/linux/codecov
curl -sS -o "${codecov_bin}.SHA256SUM" https://uploader.codecov.io/latest/linux/codecov.SHA256SUM
cd "$(dirname "${codecov_bin}")" && sha256sum -c "$(basename "${codecov_bin}").SHA256SUM" && cd - >/dev/null
chmod +x "${codecov_bin}"

codecov_flags="--file=${coverage_profile} --flags=e2e --name=E2E-Coverage --verbose"
job_type="${JOB_TYPE:-local}"
if [ "${job_type}" = "presubmit" ]; then
[ -n "${PULL_NUMBER:-}" ] && codecov_flags="${codecov_flags} --pr ${PULL_NUMBER}"
[ -n "${PULL_PULL_SHA:-}" ] && codecov_flags="${codecov_flags} --sha ${PULL_PULL_SHA}"
[ -n "${PULL_BASE_REF:-}" ] && codecov_flags="${codecov_flags} --branch ${PULL_BASE_REF}"
[ -n "${REPO_OWNER:-}" ] && [ -n "${REPO_NAME:-}" ] && codecov_flags="${codecov_flags} --slug ${REPO_OWNER}/${REPO_NAME}"
elif [ "${job_type}" = "postsubmit" ]; then
[ -n "${PULL_BASE_SHA:-}" ] && codecov_flags="${codecov_flags} --sha ${PULL_BASE_SHA}"
[ -n "${PULL_BASE_REF:-}" ] && codecov_flags="${codecov_flags} --branch ${PULL_BASE_REF}"
[ -n "${REPO_OWNER:-}" ] && [ -n "${REPO_NAME:-}" ] && codecov_flags="${codecov_flags} --slug ${REPO_OWNER}/${REPO_NAME}"
fi

${codecov_bin} ${codecov_flags} || echo "Warning: Codecov upload failed (non-fatal)"
rm -f "${codecov_bin}" "${codecov_bin}.SHA256SUM"
else
echo "CODECOV_TOKEN not set -- skipping upload. Profile saved: ${coverage_profile}"
fi
else
echo "Warning: No coverage data found"
fi
credentials:
- mount_path: /var/run/secrets/codecov
name: sscsi-driver-operator-codecov-token
namespace: test-credentials
from: src
resources:
requests:
cpu: 100m
timeout: 15m0s
- chain: gather
test:
- ref: fips-check-fips-or-die
- ref: fips-check-node-scan
- as: setup-coverage
cli: latest
commands: |
set -euo pipefail
NAMESPACE="openshift-cluster-csi-drivers"
DEPLOYMENT="secrets-store-csi-driver-operator"
GOCOVERDIR_PATH="/tmp/e2e-cover"

echo "--- E2E Coverage Setup ---"
echo "Coverage image: ${COVERAGE_IMAGE}"

echo "Discovering CSV from deployment ownerReference..."
csv=$(oc get deployment "${DEPLOYMENT}" -n "${NAMESPACE}" \
-o jsonpath='{.metadata.ownerReferences[?(@.kind=="ClusterServiceVersion")].name}')
if [ -z "${csv}" ]; then
echo "Error: no CSV found for ${DEPLOYMENT}"; exit 1
fi
echo "Found CSV: ${csv}"

echo "Patching CSV with coverage image and GOCOVERDIR env..."
oc patch csv "${csv}" -n "${NAMESPACE}" --type=json -p "[
{\"op\": \"replace\", \"path\": \"/spec/install/spec/deployments/0/spec/template/spec/containers/0/image\", \"value\": \"${COVERAGE_IMAGE}\"},
{\"op\": \"add\", \"path\": \"/spec/install/spec/deployments/0/spec/template/spec/containers/0/env/-\", \"value\": {\"name\": \"GOCOVERDIR\", \"value\": \"${GOCOVERDIR_PATH}\"}}
]"
Comment on lines +225 to +228
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot May 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

JSON patch may fail if env array doesn't exist.

The "add" operation with path ...env/- assumes the env array already exists in the container spec. If the container has no existing environment variables, this patch will fail with "path not found" error.

Consider using a test-then-add approach or ensure the operator container always has at least one env var defined:

Suggested defensive approach 
-        oc patch csv "${csv}" -n "${NAMESPACE}" --type=json -p "[
-          {\"op\": \"replace\", \"path\": \"/spec/install/spec/deployments/0/spec/template/spec/containers/0/image\", \"value\": \"${COVERAGE_IMAGE}\"},
-          {\"op\": \"add\", \"path\": \"/spec/install/spec/deployments/0/spec/template/spec/containers/0/env/-\", \"value\": {\"name\": \"GOCOVERDIR\", \"value\": \"${GOCOVERDIR_PATH}\"}}
-        ]"
+        # Check if env array exists, create if needed
+        has_env=$(oc get csv "${csv}" -n "${NAMESPACE}" -o jsonpath='{.spec.install.spec.deployments[0].spec.template.spec.containers[0].env}' 2>/dev/null)
+        if [ -z "${has_env}" ] || [ "${has_env}" = "null" ]; then
+          env_patch='{"op": "add", "path": "/spec/install/spec/deployments/0/spec/template/spec/containers/0/env", "value": [{"name": "GOCOVERDIR", "value": "'"${GOCOVERDIR_PATH}"'"}]}'
+        else
+          env_patch='{"op": "add", "path": "/spec/install/spec/deployments/0/spec/template/spec/containers/0/env/-", "value": {"name": "GOCOVERDIR", "value": "'"${GOCOVERDIR_PATH}"'"}}'
+        fi
+        oc patch csv "${csv}" -n "${NAMESPACE}" --type=json -p "[
+          {\"op\": \"replace\", \"path\": \"/spec/install/spec/deployments/0/spec/template/spec/containers/0/image\", \"value\": \"${COVERAGE_IMAGE}\"},
+          ${env_patch}
+        ]"
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/config/openshift/secrets-store-csi-driver-operator/openshift-secrets-store-csi-driver-operator-main.yaml`
around lines 225 - 228, The JSON patch that appends GOCOVERDIR to the container
env can fail if the container has no env array; modify the patching logic around
the oc patch csv "${csv}" -n "${NAMESPACE}" call so you first ensure the target
container has an env array before using the add with path ".../env/-". Implement
this by either (a) issuing a preparatory patch (e.g., an oc patch --type=merge
or a JSON patch that creates
/spec/install/spec/deployments/0/spec/template/spec/containers/0/env as an empty
array if missing) and then run the existing add for the GOCOVERDIR entry, or (b)
stop using a single JSON-patch append and instead perform a
merge/strategic-merge patch that injects the env entry (including COVERAGE_IMAGE
and GOCOVERDIR_PATH) in one safe operation so it works whether env exists or
not.


echo "Waiting for operator rollout with coverage image..."
sleep 5
oc rollout status "deployment/${DEPLOYMENT}" -n "${NAMESPACE}" --timeout=180s

oc exec -n "${NAMESPACE}" "deploy/${DEPLOYMENT}" -- env | grep GOCOVERDIR || \
echo "Warning: GOCOVERDIR not found in pod env (non-fatal)"
echo "--- Coverage setup complete ---"
dependencies:
- env: COVERAGE_IMAGE
name: secrets-store-csi-driver-operator-coverage
from: src
resources:
requests:
cpu: 100m
- as: test-e2e
cli: latest
commands: make test-e2e
Expand Down