Skip to content

OTA-1965: WIP: pkg/cvo: Add Lightspeed proposal integration #1379

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
harche wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

OTA-1965: WIP: pkg/cvo: Add Lightspeed proposal integration #1379

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 81 additions & 1 deletion AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,4 +188,84 @@ Subsystems include: `pkg/cvo`, `pkg/payload`, `lib/resourceapply`, `hack`, etc.

### Development and Testing
- Never test against production clusters - always use disposable test environments
- CVO has significant control over cluster state and can disrupt operations during development
- CVO has significant control over cluster state and can disrupt operations during development

## Lightspeed Proposal Integration

The CVO creates `Proposal` CRs (API group `agentic.openshift.io/v1alpha1`) when available updates are discovered, gated behind the `LightspeedProposals` feature gate.

### Key files
- `pkg/proposal/proposal.go` — Creates Proposal CRs with pre-collected readiness data
- `pkg/proposal/proposal_test.go` — Unit tests for proposal creation and dedup
- `pkg/cvo/cvo.go:maybeCreateLightspeedProposal` — Runs readiness checks, calls proposal creator
- `pkg/cvo/availableupdates.go:188` — Entry point after Cincinnati sync
- `pkg/featuregates/featuregates.go` — `LightspeedProposals()` feature gate
- `install/*lightspeed*` — Agent, Workflow, and system prompt manifests (applied by CVO from payload)
- Skills are consumed from the shared `agentic-skills` image (openshift/agentic-skills repo)

### Deploying dev CVO with Lightspeed proposals

Requires: lightspeed-operator already deployed on the cluster (CRDs, agent, sandbox).

```bash
# 1. Build binary and payload override (excludes CVO deployment manifest to prevent self-revert)
cd /path/to/cluster-version-operator
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -mod=vendor \
-o _output/linux/amd64/cluster-version-operator ./cmd/cluster-version-operator/
Comment on lines +213 to +214
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Apr 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Use the repository build target in the new workflow.

This section bypasses the supported CVO build path. Prefer make build, or explicitly document why this dev flow needs a raw cross-build.

Suggested doc adjustment 
-CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -mod=vendor \
-  -o _output/linux/amd64/cluster-version-operator ./cmd/cluster-version-operator/
+make build

Based on learnings, “Use make build to build the CVO binary, outputs to _output/ directory”.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -mod=vendor \
-o _output/linux/amd64/cluster-version-operator ./cmd/cluster-version-operator/
make build
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@AGENTS.md` around lines 213 - 214, Replace the raw cross-build command for
the cluster-version-operator with the supported repository build target: remove
the CGO_ENABLED/GOOS/GOARCH go build line and instead instruct users to run the
repository's make build target (e.g., "make build") which produces the CVO
binary into the _output/ directory; if a raw cross-build must be kept, add an
explicit justification explaining why the dev flow cannot use "make build" and
document expected outputs and environment differences.


# 2. Create payload override directory with release-metadata (so CVO detects correct version)
mkdir -p _output/payload-override/{manifests,release-manifests}
cp install/* _output/payload-override/manifests/
rm _output/payload-override/manifests/*30_deployment* # prevent self-revert
CLUSTER_VERSION=$(oc get clusterversion version -o jsonpath='{.status.desired.version}')
cat > _output/payload-override/release-manifests/release-metadata <<EOF
{"kind":"cincinnati-metadata-v0","version":"${CLUSTER_VERSION}"}
EOF
cat > _output/payload-override/release-manifests/image-references <<EOF
{"kind":"ImageStream","apiVersion":"image.openshift.io/v1","metadata":{"name":"${CLUSTER_VERSION}"},"spec":{"tags":[]}}
EOF

# 3. Build and push container image
cat > Dockerfile.dev <<'EOF'
FROM registry.access.redhat.com/ubi9-minimal:latest
COPY _output/linux/amd64/cluster-version-operator /usr/bin/cluster-version-operator
COPY install /manifests
COPY _output/payload-override /payload
ENTRYPOINT ["/usr/bin/cluster-version-operator"]
EOF
TAG="v$(date +%s)"
docker build --platform linux/amd64 -f Dockerfile.dev -t quay.io/harpatil/cvo-lightspeed:${TAG} .
docker push quay.io/harpatil/cvo-lightspeed:${TAG}

# 4. Deploy: single atomic patch (image + args + env) to avoid partial reverts
RELEASE_IMAGE=$(oc get clusterversion version -o jsonpath='{.status.desired.image}')
API_HOST=$(oc get infrastructure cluster -o jsonpath='{.status.apiServerInternalURI}' | sed 's|https://||' | cut -d: -f1)
oc patch deployment cluster-version-operator -n openshift-cluster-version --type json -p "[
{\"op\":\"replace\",\"path\":\"/spec/template/spec/containers/0/image\",\"value\":\"quay.io/harpatil/cvo-lightspeed:${TAG}\"},
Comment on lines +237 to +244
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Apr 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Avoid hard-coding a personal image repository in shared docs.

Use an IMAGE variable or placeholder so contributors do not copy a registry they cannot push to.

Suggested doc adjustment 
 TAG="v$(date +%s)"
-docker build --platform linux/amd64 -f Dockerfile.dev -t quay.io/harpatil/cvo-lightspeed:${TAG} .
-docker push quay.io/harpatil/cvo-lightspeed:${TAG}
+IMAGE="quay.io/yourname/cvo-lightspeed:${TAG}"
+docker build --platform linux/amd64 -f Dockerfile.dev -t "${IMAGE}" .
+docker push "${IMAGE}"
@@
-  {\"op\":\"replace\",\"path\":\"/spec/template/spec/containers/0/image\",\"value\":\"quay.io/harpatil/cvo-lightspeed:${TAG}\"},
+  {\"op\":\"replace\",\"path\":\"/spec/template/spec/containers/0/image\",\"value\":\"${IMAGE}\"},
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
docker build --platform linux/amd64 -f Dockerfile.dev -t quay.io/harpatil/cvo-lightspeed:${TAG} .
docker push quay.io/harpatil/cvo-lightspeed:${TAG}
# 4. Deploy: single atomic patch (image + args + env) to avoid partial reverts
RELEASE_IMAGE=$(oc get clusterversion version -o jsonpath='{.status.desired.image}')
API_HOST=$(oc get infrastructure cluster -o jsonpath='{.status.apiServerInternalURI}' | sed 's|https://||' | cut -d: -f1)
oc patch deployment cluster-version-operator -n openshift-cluster-version --type json -p "[
{\"op\":\"replace\",\"path\":\"/spec/template/spec/containers/0/image\",\"value\":\"quay.io/harpatil/cvo-lightspeed:${TAG}\"},
IMAGE="quay.io/yourname/cvo-lightspeed:${TAG}"
docker build --platform linux/amd64 -f Dockerfile.dev -t "${IMAGE}" .
docker push "${IMAGE}"
# 4. Deploy: single atomic patch (image + args + env) to avoid partial reverts
RELEASE_IMAGE=$(oc get clusterversion version -o jsonpath='{.status.desired.image}')
API_HOST=$(oc get infrastructure cluster -o jsonpath='{.status.apiServerInternalURI}' | sed 's|https://||' | cut -d: -f1)
oc patch deployment cluster-version-operator -n openshift-cluster-version --type json -p "[
{\"op\":\"replace\",\"path\":\"/spec/template/spec/containers/0/image\",\"value\":\"${IMAGE}\"},
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@AGENTS.md` around lines 237 - 244, Replace the hard-coded personal image
reference quay.io/harpatil/cvo-lightspeed:${TAG} with a configurable IMAGE
variable/placeholder used consistently across the build, push and oc patch
commands (the docker build, docker push lines and the value field in the oc
patch for deployment cluster-version-operator). Update the docs to reference
IMAGE or ${IMAGE} (and document that contributors should set IMAGE to their own
registry/tag) so the build/push and the patch operation use IMAGE instead of the
personal quay.io path.

{\"op\":\"replace\",\"path\":\"/spec/template/spec/containers/0/imagePullPolicy\",\"value\":\"Always\"},
{\"op\":\"replace\",\"path\":\"/spec/template/spec/containers/0/args\",\"value\":[
\"start\",\"--release-image=${RELEASE_IMAGE}\",\"--enable-auto-update=false\",
\"--listen=0.0.0.0:9099\",\"--serving-cert-file=/etc/tls/serving-cert/tls.crt\",
\"--serving-key-file=/etc/tls/serving-cert/tls.key\",\"--v=4\",\"--always-enable-capabilities=Ingress\"
]},
{\"op\":\"replace\",\"path\":\"/spec/template/spec/containers/0/env\",\"value\":[
{\"name\":\"OPERATOR_IMAGE_VERSION\",\"value\":\"${CLUSTER_VERSION}\"},
{\"name\":\"KUBERNETES_SERVICE_PORT\",\"value\":\"6443\"},
{\"name\":\"KUBERNETES_SERVICE_HOST\",\"value\":\"${API_HOST}\"},
{\"name\":\"NODE_NAME\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"spec.nodeName\"}}},
{\"name\":\"CLUSTER_PROFILE\",\"value\":\"self-managed-high-availability\"},
{\"name\":\"PAYLOAD_OVERRIDE\",\"value\":\"/payload\"}
]}
]"

# 5. Watch for proposal creation (happens after first Cincinnati update check, ~2-5 min)
oc get proposals -n openshift-lightspeed -w
```

### Gotchas
- **PAYLOAD_OVERRIDE is critical** — without it, the CVO reads `/release-manifests/release-metadata` (which doesn't exist in the dev image) and falls back to version `0.0.1-snapshot`, disabling all feature gates including `LightspeedProposals`.
- **Remove the CVO deployment manifest** from `_output/payload-override/manifests/` — otherwise the CVO reconciles its own deployment and reverts the image/env changes.
- **KUBERNETES_SERVICE_HOST must be the API hostname** (e.g., `api-int.cluster.example.com`), not a node IP. Using `fieldRef: status.hostIP` causes TLS errors.
- **All deployment changes must be in a single `oc patch`** — sequential patches trigger rollouts that can lose env vars.
- **Leader lease timeout** — after patching, the new pod must wait up to ~2 min for the old pod's lease to expire before it can start reconciling.
- **Skills image** comes from the shared `agentic-skills` repo (`registry.ci.openshift.org/ocp/4.22:agentic-skills`). The Agent CR selects specific skills via `paths`.
2 changes: 2 additions & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,3 +108,5 @@ require (
)

replace github.com/onsi/ginkgo/v2 => github.com/openshift/onsi-ginkgo/v2 v2.6.1-0.20241205171354-8006f302fd12

replace github.com/openshift/api => github.com/harche/api v0.0.0-20260414192630-b7a8e3d157cb
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Apr 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Description: Show all OpenShift API imports and direct Lightspeed feature-gate usages affected by the replace.

rg -n -C2 'github\.com/openshift/api|FeatureGateLightspeedProposals|agentic\.openshift\.io'

Repository: openshift/cluster-version-operator

Length of output: 50391

🏁 Script executed:

# Find where FeatureGateLightspeedProposals is actually used in non-vendor code
rg -n 'FeatureGateLightspeedProposals' --type=go | grep -v '^vendor/' | head -20

Repository: openshift/cluster-version-operator

Length of output: 249

🏁 Script executed:

# Search for proposal-related unstructured CR creation patterns
rg -n 'Proposal|unstructured|agentic' --type=go pkg/ | head -30

Repository: openshift/cluster-version-operator

Length of output: 3414

🏁 Script executed:

# Check the featuregates.go file to understand dependencies
fd -n 'featuregates.go' -x cat -n {}

Repository: openshift/cluster-version-operator

Length of output: 308

🏁 Script executed:

# View featuregates.go around the usage lines
sed -n '120,145p' pkg/featuregates/featuregates.go

Repository: openshift/cluster-version-operator

Length of output: 934

🏁 Script executed:

# Search for proposal CR creation patterns or resources
rg -n 'ProposalRequest|proposals\.|agentic\.openshift' pkg/ --type=go

Repository: openshift/cluster-version-operator

Length of output: 1448

🏁 Script executed:

# Check the imports in featuregates.go
head -30 pkg/featuregates/featuregates.go

Repository: openshift/cluster-version-operator

Length of output: 1734

Avoid merging with a module-wide replace to a personal fork.

The replace directive redirects all github.com/openshift/api imports—including stable types like config/v1—to github.com/harche/api. However, analysis shows that only the FeatureGateLightspeedProposals feature gate constant from the fork is actually used (in pkg/featuregates/featuregates.go lines 126 and 138), while proposal CRs are created as unstructured objects (agentic.openshift.io/v1alpha1/Proposal), not typed structs from the API.

Remove this replace directive before merging, or upstream the feature gate symbol to the main openshift/api repository instead.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@go.mod` at line 112, The go.mod replace that points github.com/openshift/api
to your personal fork must be removed; edit go.mod to delete the line "replace
github.com/openshift/api => github.com/harche/api ..." and restore use of the
upstream module, then ensure the code referencing FeatureGateLightspeedProposals
in pkg/featuregates/featuregates.go either imports the upstream
github.com/openshift/api package (if the constant exists upstream) or you
instead vendor/define the specific constant locally or upstream the symbol to
the official repo before merging.

4 changes: 2 additions & 2 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,8 @@ github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/gorilla/websocket v1.5.4-0.20250319132907-e064f32e3674 h1:JeSE6pjso5THxAzdVpqr6/geYxZytqFMBCOtn/ujyeo=
github.com/gorilla/websocket v1.5.4-0.20250319132907-e064f32e3674/go.mod h1:r4w70xmWCQKmi1ONH4KIaBptdivuRPyosB9RmPlGEwA=
github.com/harche/api v0.0.0-20260414192630-b7a8e3d157cb h1:8rHr8NpecxNNNjEqLJBLprSit1WFWJTEHsKePFnlURc=
github.com/harche/api v0.0.0-20260414192630-b7a8e3d157cb/go.mod h1:pyVjK0nZ4sRs4fuQVQ4rubsJdahI1PB94LnQ8sGdvxo=
github.com/imdario/mergo v0.3.12 h1:b6R2BslTbIEToALKP7LxUvijTsNI9TAe80pLWN2g/HU=
github.com/imdario/mergo v0.3.12/go.mod h1:jmQim1M+e3UYxmgPu/WyfjB3N3VflVyUjjjwH0dnCYA=
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
Expand Down Expand Up @@ -104,8 +106,6 @@ github.com/onsi/gomega v1.38.2 h1:eZCjf2xjZAqe+LeWvKb5weQ+NcPwX84kqJ0cZNxok2A=
github.com/onsi/gomega v1.38.2/go.mod h1:W2MJcYxRGV63b418Ai34Ud0hEdTVXq9NW9+Sx6uXf3k=
github.com/openshift-eng/openshift-tests-extension v0.0.0-20250220212757-b9c4d98a0c45 h1:hXpbYtP3iTh8oy/RKwKkcMziwchY3fIk95ciczf7cOA=
github.com/openshift-eng/openshift-tests-extension v0.0.0-20250220212757-b9c4d98a0c45/go.mod h1:6gkP5f2HL0meusT0Aim8icAspcD1cG055xxBZ9yC68M=
github.com/openshift/api v0.0.0-20260302174620-dcac36b908db h1:MOQ5JSIlbP4apwTrEdNpApT6PsnB0/1S6y9aKODp5Ks=
github.com/openshift/api v0.0.0-20260302174620-dcac36b908db/go.mod h1:pyVjK0nZ4sRs4fuQVQ4rubsJdahI1PB94LnQ8sGdvxo=
github.com/openshift/client-go v0.0.0-20260302182750-20813ce71ca6 h1:wJv4Ia+R4OxoaJcTUyvMtBc5rWFvfTiEA8d5f1MBPqI=
github.com/openshift/client-go v0.0.0-20260302182750-20813ce71ca6/go.mod h1:3lkVff575BlbDUUhMsrD1IyvfkZ+oKUB7iZuVy1m0W0=
github.com/openshift/library-go v0.0.0-20260303171201-5d9eb6295ff6 h1:xjqy0OolrFdJ+ofI/aD0+2k9+MSk5anP5dXifFt539Q=
Expand Down
Loading