inference-only-stack.md

Inference Only Stack

Table of Contents

• Inference Only Stack
  • Table of Contents
  • Overview
  • Prerequisites
  • What Gets Installed
  • Values Override
  • Scripted Installation (Helm)
    • 1. Clone the repository
    • 2. Install operators (first Helm run)
    • 3. Wait for CRDs
    • 4. Create CRs (second Helm run)
    • 5. Enable Authorino TLS (post-install)
    • 6. Verify the installation
  • Enabling Authorino TLS
    • CLI method
  • Verification
    • Check operator CSVs
    • Check Authorino TLS
    • Check DataScienceCluster status
    • Comprehensive verification
  • Troubleshooting
    • CRs not being created
    • Authorino TLS issues
    • Dependencies not being installed

Overview

The inference-only stack deploys KServe with distributed inference (llm-d) without the full RHOAI/ODH platform. This is useful when you only need model serving capabilities and want to avoid installing components like pipelines, workbenches, dashboards, training operators, and monitoring.

The stack installs a minimal set of dependency operators required by KServe:

Operator	Purpose	Namespace
cert-manager	Certificate management and TLS provisioning	cert-manager-operator
Leader Worker Set	Distributed inference workflows	openshift-lws-operator
Red Hat Connectivity Link (RHCL)	API management (Kuadrant/Authorino)	kuadrant-system

Prerequisites

• OpenShift cluster (version 4.19.9 or later)
• kubectl or oc CLI installed
• Cluster admin permissions
• Helm v4

What Gets Installed

Since all components and monitoring default to Removed, the values override only needs to enable KServe:

1. ODH/RHOAI operator (via OLM)
2. Dependency operators (via OLM): cert-manager, Leader Worker Set, RHCL (Kuadrant) are auto-enabled by KServe
3. DSCInitialization (DSCI) with monitoring disabled (default)
4. DataScienceCluster (DSC) with only KServe set to Managed

Values Override

The simplest way to deploy the inference-only stack is using the built-in rhaii profile:

helm upgrade --install rhaii ./charts/rhai-on-openshift-chart \
  --set profile=rhaii \
  --set operator.type=rhoai \
  -n rhai-gitops --create-namespace

The rhaii profile automatically enables KServe with its required dependencies and disables unnecessary ones (like jobSet). See the Deploy Profiles section in the chart README for more details.

Alternatively, you can use the values override file at docs/examples/values-inference-only.yaml for full control over the configuration.

Note

The chart's tri-state dependency resolution (auto/true/false) handles transitive dependencies automatically. For example, RHCL auto-pulls cert-manager and Leader Worker Set as its own dependencies. Components set to Removed (the default) do not trigger dependency installation.

Scripted Installation (Helm)

1. Clone the repository

git clone https://github.com/opendatahub-io/odh-gitops.git
cd odh-gitops

2. Install operators (first Helm run)

The first run installs the OLM subscriptions (Namespace, OperatorGroup, Subscription). CRs are skipped because their CRDs do not exist yet.

# Using profile (recommended)
helm upgrade --install rhaii ./charts/rhai-on-openshift-chart \
  --set profile=rhaii \
  --set operator.type=rhoai \
  -n rhai-gitops --create-namespace

# Or using values file for full control
helm upgrade --install rhaii ./charts/rhai-on-openshift-chart \
  -f docs/examples/values-inference-only.yaml \
  -n rhai-gitops --create-namespace

3. Wait for CRDs

Wait for the operators to install and register their CRDs before creating CRs:

kubectl wait --for=condition=Established \
  crd/leaderworkersetoperators.operator.openshift.io --timeout=300s

kubectl wait --for=condition=Established \
  crd/kuadrants.kuadrant.io --timeout=300s

kubectl wait --for=condition=Established \
  crd/datascienceclusters.datasciencecluster.opendatahub.io --timeout=300s

kubectl wait --for=condition=Established \
  crd/dscinitializations.dscinitialization.opendatahub.io --timeout=300s

Note

You can also use the provided script ./scripts/wait-for-crds.sh which waits for all known CRDs.

4. Create CRs (second Helm run)

Now that CRDs exist, the second run creates the CR resources (DSCInitialization, DataScienceCluster, Kuadrant, LeaderWorkerSetOperator, etc.):

# Using profile
helm upgrade --install rhaii ./charts/rhai-on-openshift-chart \
  --set profile=rhaii \
  --set operator.type=rhoai \
  -n rhai-gitops

# Or using values file
helm upgrade --install rhaii ./charts/rhai-on-openshift-chart \
  -f docs/examples/values-inference-only.yaml \
  -n rhai-gitops

5. Enable Authorino TLS (post-install)

Warning

This step is required for KServe to function correctly. See Enabling Authorino TLS for why this is a post-install step.

KUSTOMIZE_MODE=false ./scripts/prepare-authorino-tls.sh

6. Verify the installation

See Verification for commands to confirm everything is running.

Enabling Authorino TLS

The Kuadrant operator automatically creates the Authorino resource when the Kuadrant CR is applied. Because Authorino is created by the operator (not by the Helm chart), TLS must be enabled as a post-install step after Authorino exists.

CLI method

Use the provided script which handles waiting for resources, annotating the service, and patching the Authorino CR:

KUSTOMIZE_MODE=false ./scripts/prepare-authorino-tls.sh

The script:

1. Waits for the Authorino service to be created
2. Annotates the service to trigger TLS certificate generation
3. Waits for the TLS certificate secret
4. Patches the Authorino CR to enable TLS

Verification

Check operator CSVs

Verify that all operators are installed and in Succeeded phase:

kubectl get csv -A | grep -E "(cert-manager|leader-worker|rhcl|opendatahub|rhods)"

Check Authorino TLS

Verify that Authorino has TLS enabled:

kubectl get authorino authorino -n kuadrant-system \
  -o jsonpath='{.spec.listener.tls}'

Check DataScienceCluster status

kubectl get datasciencecluster -o jsonpath='{.items[0].status.phase}'

Comprehensive verification

Use the provided verification script for a full check of all operator subscriptions and pod readiness:

./scripts/verify-dependencies.sh

Troubleshooting

CRs not being created

If CR resources (DataScienceCluster, Kuadrant, LeaderWorkerSetOperator, etc.) are not being created after the Helm install:

1. Verify the operator is installed and the CRD exists:

   kubectl get crd datascienceclusters.datasciencecluster.opendatahub.io
   kubectl get crd kuadrants.kuadrant.io

2. Run helm upgrade again. CRs are skipped until their CRDs exist:

   helm upgrade --install rhaii ./charts/rhai-on-openshift-chart \
     --set profile=rhaii --set operator.type=rhoai \
     -n rhai-gitops

Authorino TLS issues

If Authorino TLS is not working:

1. Check that the service annotation exists:

   kubectl get svc authorino-authorino-authorization -n kuadrant-system \
     -o jsonpath='{.metadata.annotations}'

2. Check that the TLS secret was created:

   kubectl get secret authorino-server-cert -n kuadrant-system

3. Verify the Authorino CR has TLS enabled:

   kubectl get authorino authorino -n kuadrant-system \
     -o jsonpath='{.spec.listener.tls}'

4. If the secret does not exist, re-run the TLS preparation script:

   KUSTOMIZE_MODE=false ./scripts/prepare-authorino-tls.sh

Dependencies not being installed

If a dependency operator is not being installed:

1. Verify the component requiring it has managementState: Managed (not Removed)
2. Check that the dependency is not explicitly set to false in the component's dependencies
3. Verify the top-level dependencies.<name>.enabled is not set to false