Skip to content

Improve troubleshooting support #217

Description

@porridge

Roxie does a great job automating stackrox deployment.
However when things fail, users are on their own -- typically need to resort to manual inspection (when running roxie directly) or custom shell scripts (when it's being run by other process) to find out what exactly was wrong.

Example output
2026-06-22T12:33:27.033319781Z 00:00 Using kubeconfig /data/kubeconfig.yaml
2026-06-22T12:33:28.667009062Z 00:01 Running without a controlling terminal.
2026-06-22T12:33:28.667081152Z 00:01 Looking up main image tag
2026-06-22T12:33:28.667093272Z 00:01 No MAIN_IMAGE_TAG found in the environment, using default main image tag 4.9.2 for deployment
2026-06-22T12:33:28.667100922Z 00:01 To use a different tag, set the MAIN_IMAGE_TAG environment variable
2026-06-22T12:33:28.667108832Z 00:01 Alternatively, execute roxie from within the stackrox repository, in which case the currently checked out stackrox tag will be used
2026-06-22T12:33:28.667114152Z 00:01 Detected cluster type: Unknown
2026-06-22T12:33:28.667120692Z 00:01 Enabling port-forwarding due to no exposure
2026-06-22T12:33:28.667127232Z 00:01 � ACS Deployer initialized
2026-06-22T12:33:28.667134332Z 00:01 
2026-06-22T12:33:28.667141602Z 00:01 ��������������������������������������������������������������
2026-06-22T12:33:28.667147422Z 00:01 � Deployment Configuration                                   �
2026-06-22T12:33:28.667154762Z 00:01 ��������������������������������������������������������������
2026-06-22T12:33:28.667161482Z 00:01 �              Component: Secured Cluster                    �
2026-06-22T12:33:28.667167232Z 00:01 �           Cluster Type: Unknown                            �
2026-06-22T12:33:28.667192692Z 00:01 �               Main Tag: 4.9.2                              �
2026-06-22T12:33:28.667196982Z 00:01 �     Kubernetes Context: tngkwtoh-prod                      �
2026-06-22T12:33:28.667200942Z 00:01 �       Central Endpoint: 34.61.239.98:443                   �
2026-06-22T12:33:28.667204352Z 00:01 ��������������������������������������������������������������
2026-06-22T12:33:28.667208352Z 00:01 
2026-06-22T12:33:28.667211962Z 00:01 Preparing and verifying Docker credentials...
2026-06-22T12:33:28.914640751Z 00:01 Successfully verified credentials for quay.io (repository: rhacs-eng/main)
2026-06-22T12:33:28.915392241Z 00:01 Docker credentials verified successfully
2026-06-22T12:33:28.915448131Z 00:01 Initiating deployment of Secured Cluster
2026-06-22T12:33:29.328337104Z 00:02 Missing CRDs detected (centrals.platform.stackrox.io, securedclusters.platform.stackrox.io, securitypolicies.config.stackrox.io)
2026-06-22T12:33:29.328390024Z 00:02 Fetching bundle quay.io/rhacs-eng/stackrox-operator-bundle:v4.9.2
2026-06-22T12:33:29.328398754Z 00:02 Created temporary directory: /tmp/roxie-deployer-2411910525/stackrox-operator-bundle-578185967
2026-06-22T12:33:29.328405504Z 00:02 Pulling and extracting operator bundle image...
2026-06-22T12:33:29.328484474Z 00:02 Using temporary directory: /tmp/oci-image-3217397312
2026-06-22T12:33:29.328537274Z 00:02 Fetching image quay.io/rhacs-eng/stackrox-operator-bundle:v4.9.2
2026-06-22T12:33:29.758326476Z 00:02 � Image fetched successfully
2026-06-22T12:33:29.758388956Z 00:02 Extracting /manifests/ directory from image layers...
2026-06-22T12:33:29.758398856Z 00:02 Found 3 layer(s) in image
2026-06-22T12:33:29.758406006Z 00:02 Extracting layer 1/3...
2026-06-22T12:33:30.346955051Z 00:03 Extracting layer 2/3...
2026-06-22T12:33:30.455281447Z 00:03 Extracting layer 3/3...
2026-06-22T12:33:30.550211152Z 00:03 Copying manifests to destination...
2026-06-22T12:33:30.551342652Z 00:03 � Manifests extracted to: /tmp/roxie-deployer-2411910525/stackrox-operator-bundle-578185967
2026-06-22T12:33:30.552144172Z 00:03 � Bundle extracted to: /tmp/roxie-deployer-2411910525/stackrox-operator-bundle-578185967
2026-06-22T12:33:30.568118382Z 00:03 Applying 3 CRD(s) to cluster
2026-06-22T12:33:30.737046145Z 00:03 � Successfully applied CRD config.stackrox.io_securitypolicies.yaml
2026-06-22T12:33:30.930913246Z 00:03 � Successfully applied CRD platform.stackrox.io_centrals.yaml
2026-06-22T12:33:31.126749518Z 00:04 � Successfully applied CRD platform.stackrox.io_securedclusters.yaml
2026-06-22T12:33:31.127427388Z 00:04 Cleaned up CRD bundle directory: /tmp/roxie-deployer-2411910525/stackrox-operator-bundle-578185967
2026-06-22T12:33:31.436202155Z 00:04 Operator tag: 4.9.2
2026-06-22T12:33:31.436318735Z 00:04 Created temporary directory: /tmp/roxie-deployer-2411910525/stackrox-operator-bundle-1192811949
2026-06-22T12:33:31.436336295Z 00:04 Pulling and extracting operator bundle image...
2026-06-22T12:33:31.436342665Z 00:04 Using temporary directory: /tmp/oci-image-3535094181
2026-06-22T12:33:31.436350035Z 00:04 Fetching image quay.io/rhacs-eng/stackrox-operator-bundle:v4.9.2
2026-06-22T12:33:31.686637904Z 00:04 � Image fetched successfully
2026-06-22T12:33:31.686689814Z 00:04 Extracting /manifests/ directory from image layers...
2026-06-22T12:33:31.686700824Z 00:04 Found 3 layer(s) in image
2026-06-22T12:33:31.686707684Z 00:04 Extracting layer 1/3...
2026-06-22T12:33:31.750913531Z 00:04 Extracting layer 2/3...
2026-06-22T12:33:31.812981558Z 00:04 Extracting layer 3/3...
2026-06-22T12:33:31.874371136Z 00:04 Copying manifests to destination...
2026-06-22T12:33:31.875774836Z 00:04 � Manifests extracted to: /tmp/roxie-deployer-2411910525/stackrox-operator-bundle-1192811949
2026-06-22T12:33:31.876551916Z 00:04 � Bundle extracted to: /tmp/roxie-deployer-2411910525/stackrox-operator-bundle-1192811949
2026-06-22T12:33:31.876594386Z 00:04 Bundle image: quay.io/rhacs-eng/stackrox-operator-bundle:v4.9.2
2026-06-22T12:33:31.892765685Z 00:04 Applying 3 CRD(s) to cluster
2026-06-22T12:33:32.032471339Z 00:04 � Successfully applied CRD config.stackrox.io_securitypolicies.yaml
2026-06-22T12:33:32.618881745Z 00:05 � Successfully applied CRD platform.stackrox.io_centrals.yaml
2026-06-22T12:33:32.864099834Z 00:05 � Successfully applied CRD platform.stackrox.io_securedclusters.yaml
2026-06-22T12:33:32.864161214Z 00:05 � Parsing ClusterServiceVersion deployment specification
2026-06-22T12:33:32.872033574Z 00:05 � Operator deployment plan:
2026-06-22T12:33:32.872079844Z 00:05   � Namespace: rhacs-operator-system
2026-06-22T12:33:32.872088524Z 00:05   � ServiceAccount: rhacs-operator-controller-manager
2026-06-22T12:33:32.872094874Z 00:05   � Setting up pull secrets: false
2026-06-22T12:33:32.872156994Z 00:05 Preparing namespace rhacs-operator-system
2026-06-22T12:33:32.981659599Z 00:05 Creating namespace rhacs-operator-system
2026-06-22T12:33:34.673473288Z 00:07 � Waiting for operator deployment to become ready...
2026-06-22T12:33:53.787442233Z 00:26 � Operator deployment is ready (1 replicas)
2026-06-22T12:33:53.787531573Z 00:26 � Operator deployment completed successfully!
2026-06-22T12:33:53.788286903Z 00:26 Cleaned up operator bundle directory: /tmp/roxie-deployer-2411910525/stackrox-operator-bundle-1192811949
2026-06-22T12:33:53.788338203Z 00:26 Deploying SecuredCluster to namespace stackrox
2026-06-22T12:33:53.911096578Z 00:26 � Deploying SecuredCluster via Operator...
2026-06-22T12:33:53.911153878Z 00:26 Preparing namespace stackrox
2026-06-22T12:33:54.015246243Z 00:26 Creating namespace stackrox
2026-06-22T12:33:54.548997011Z 00:27 Using cluster name: production
2026-06-22T12:33:54.549082491Z 00:27 Generating CRS named "production-crs" with roxctl...
2026-06-22T12:33:54.681019065Z 00:27 � CRS generated
2026-06-22T12:33:54.681102145Z 00:27 Applying CRS to sensor namespace
2026-06-22T12:33:54.952737974Z 00:27 � CRS applied
2026-06-22T12:33:54.952876504Z 00:27 Applying SecuredCluster custom resource
2026-06-22T12:33:55.235666632Z 00:28 � SecuredCluster CR applied
2026-06-22T12:33:55.235731472Z 00:28 � Waiting for Secured Cluster to become ready (timeout: 20m0s)...
2026-06-22T12:33:55.235738941Z 00:28 Waiting for resource deployment/sensor to exist in namespace stackrox...
2026-06-22T12:34:11.362040534Z 00:44   � Deployment 'scanner' created (/ replicas ready)
2026-06-22T12:34:11.362094924Z 00:44   � Deployment 'scanner-db' created (/ replicas ready)
2026-06-22T12:34:11.362104344Z 00:44   � Deployment 'scanner-v4-db' created (/ replicas ready)
2026-06-22T12:34:11.362110644Z 00:44   � Deployment 'scanner-v4-indexer' created (/ replicas ready)
2026-06-22T12:34:11.687709311Z 00:44     � Pod 'collector-4zrns' starting...
2026-06-22T12:34:12.540371444Z 00:45   � Deployment 'admission-control' created (/ replicas ready)
2026-06-22T12:34:12.540482294Z 00:45   � Deployment 'sensor' created (/ replicas ready)
2026-06-22T12:34:12.761577565Z 00:45     � Pod 'admission-control-8795b4596-2z7mt' starting...
2026-06-22T12:34:12.761649175Z 00:45     � Pod 'collector-99rsz' starting...
2026-06-22T12:34:12.761658215Z 00:45     � Pod 'scanner-654dcf5986-2wk69' starting...
2026-06-22T12:34:12.761690905Z 00:45     � Pod 'scanner-654dcf5986-h4rmx' starting...
2026-06-22T12:34:12.761694695Z 00:45     � Pod 'scanner-654dcf5986-sxgwn' starting...
2026-06-22T12:34:12.761698675Z 00:45     � Pod 'scanner-db-5c45f54d54-2tk97' starting...
2026-06-22T12:34:12.761702505Z 00:45     � Pod 'scanner-v4-db-847d754787-dsqcv' starting...
2026-06-22T12:34:12.761710965Z 00:45     � Pod 'scanner-v4-indexer-7dfc9f45-mbwjk' starting...
2026-06-22T12:34:13.377989289Z 00:46   � Deployment 'admission-control' progressing (/3 replicas ready)
2026-06-22T12:34:13.378059409Z 00:46   � Deployment 'scanner' progressing (/3 replicas ready)
2026-06-22T12:34:13.378069229Z 00:46   � Deployment 'scanner-db' progressing (/1 replicas ready)
2026-06-22T12:34:13.378075389Z 00:46   � Deployment 'scanner-v4-db' progressing (/1 replicas ready)
2026-06-22T12:34:13.378081119Z 00:46   � Deployment 'scanner-v4-indexer' progressing (/2 replicas ready)
2026-06-22T12:34:13.378086509Z 00:46   � Deployment 'sensor' progressing (/1 replicas ready)
2026-06-22T12:34:13.589102210Z 00:46     � Pod 'admission-control-8795b4596-cd94n' starting...
2026-06-22T12:34:13.589183450Z 00:46     � Pod 'admission-control-8795b4596-jvghn' starting...
2026-06-22T12:34:13.589193260Z 00:46     � Pod 'scanner-v4-indexer-7dfc9f45-8bcjw' starting...
2026-06-22T12:34:13.589200040Z 00:46     � Pod 'sensor-54d7b4d46f-f9hfp' starting...
2026-06-22T12:35:16.516144766Z 01:49 Still waiting for component Secured Cluster in namespace stackrox
2026-06-22T12:36:16.533735396Z 02:49 Still waiting for component Secured Cluster in namespace stackrox
2026-06-22T12:37:17.520001820Z 03:50 Still waiting for component Secured Cluster in namespace stackrox
2026-06-22T12:38:17.527889582Z 04:50 Still waiting for component Secured Cluster in namespace stackrox
2026-06-22T12:39:17.540303378Z 05:50 Still waiting for component Secured Cluster in namespace stackrox
2026-06-22T12:40:17.564659464Z 06:50 Still waiting for component Secured Cluster in namespace stackrox
2026-06-22T12:41:18.522324097Z 07:51 Still waiting for component Secured Cluster in namespace stackrox
2026-06-22T12:42:18.528357917Z 08:51 Still waiting for component Secured Cluster in namespace stackrox
2026-06-22T12:43:18.536278720Z 09:51 Still waiting for component Secured Cluster in namespace stackrox
2026-06-22T12:44:19.516268265Z 10:52 Still waiting for component Secured Cluster in namespace stackrox
2026-06-22T12:45:19.529500059Z 11:52 Still waiting for component Secured Cluster in namespace stackrox
2026-06-22T12:46:19.586497084Z 12:52 Still waiting for component Secured Cluster in namespace stackrox
2026-06-22T12:47:20.571782070Z 13:53 Still waiting for component Secured Cluster in namespace stackrox
2026-06-22T12:48:21.519315586Z 14:54 Still waiting for component Secured Cluster in namespace stackrox
2026-06-22T12:49:21.549073568Z 15:54 Still waiting for component Secured Cluster in namespace stackrox
2026-06-22T12:50:22.508834869Z 16:55 Still waiting for component Secured Cluster in namespace stackrox
2026-06-22T12:51:22.539633219Z 17:55 Still waiting for component Secured Cluster in namespace stackrox
2026-06-22T12:52:23.516971795Z 18:56 Still waiting for component Secured Cluster in namespace stackrox
2026-06-22T12:53:24.526592100Z 19:57 Still waiting for component Secured Cluster in namespace stackrox
2026-06-22T12:53:55.309953834Z 20:28 Transient error in kubectl command: error: timed out waiting for the condition on deployments/sensor
2026-06-22T12:53:55.310041204Z 
2026-06-22T12:53:55.310054554Z 20:28 Retrying kubectl command (attempt 2/3) after 4s...
2026-06-22T12:54:00.235965330Z Error: deployment failed: failed to deploy secured cluster: failed waiting for SecuredCluster: timeout reached

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions