Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions _topic_maps/_topic_map.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1637,6 +1637,24 @@ Topics:
Topics:
- Name: DPU Operator
File: dpu-operator
- Name: NVIDIA DPF Operator
Dir: dpf_operator
Distros: openshift-enterprise,openshift-origin
Topics:
- Name: About the NVIDIA DPF Operator
File: about-dpf-operator
- Name: Setting up the environment for DPF
File: dpf-environment-setup
- Name: Installing and configuring the DPF Operator
File: dpf-operator-installation
- Name: Provisioning the DPU hosted cluster
File: dpf-hosted-cluster-provisioning
- Name: Adding worker nodes and provisioning DPUs
File: dpf-worker-nodes-provisioning
- Name: Validating traffic and configuring telemetry
File: dpf-validation-telemetry
- Name: Troubleshooting DPF
File: dpf-troubleshooting
- Name: Network Observability Operator
Dir: network_observability_operator
Distros: openshift-enterprise,openshift-origin
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/nw-dpf-lab-topology.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
10 changes: 10 additions & 0 deletions modules/nw-dpf-adding-worker-nodes-overview.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
// Module included in the following assemblies:
//
// * networking/networking_operators/dpf_operator/dpf-worker-nodes-provisioning.adoc

:_mod-docs-content-type: CONCEPT
[id="adding-worker-nodes_{context}"]
= Worker nodes

[role="_abstract"]
You can add DPU-equipped worker nodes to the management cluster by using the Assisted Installer or the Bare Metal Operator.
38 changes: 38 additions & 0 deletions modules/nw-dpf-adding-workers-assisted-installer.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
// Module included in the following assemblies:
//
// * networking/networking_operators/dpf_operator/dpf-worker-nodes-provisioning.adoc

:_mod-docs-content-type: PROCEDURE
[id="nw-dpf-adding-workers-assisted-installer_{context}"]
= Add worker nodes by using the Assisted Installer

[role="_abstract"]
You can add DPU-equipped worker nodes to the management cluster by using the Assisted Installer on the {hybrid-console}.

.Prerequisites

* You have access to the management cluster as a user with the `cluster-admin` role.
* The management cluster is registered on the {hybrid-console}.
* You have iDRAC or BMC access to the worker node.
* The worker node meets the hardware and network prerequisites. For details, see "DPF hardware requirements" and "DPF network infrastructure requirements".

.Procedure

. Log in to the link:https://console.redhat.com/openshift[{hybrid-console}].

. Select your cluster from the cluster list.

. Click *Add hosts*.

. Click *Download Discovery ISO* and save the discovery ISO to your local system.

. Upload the ISO to the worker node by using the iDRAC or BMC interface and boot the server from the ISO.

. Wait for the node to boot from the discovery ISO.

. After the node is discovered, note the DPU interface name from the Assisted Installer interface.
This value is used for the `DPU_P0` environment variable.
+
image::nw-dpf-assisted-installer-dpu-interface.png[Assisted Installer showing DPU interface name]

. Click *Install ready host* to start the installation and monitor the installation progress.
226 changes: 226 additions & 0 deletions modules/nw-dpf-adding-workers-baremetal-operator.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,226 @@
// Module included in the following assemblies:
//
// * networking/networking_operators/dpf_operator/dpf-worker-nodes-provisioning.adoc

:_mod-docs-content-type: PROCEDURE
[id="nw-dpf-adding-workers-baremetal-operator_{context}"]
= Add worker nodes by using the Bare Metal Operator

[role="_abstract"]
You can add DPU-equipped worker nodes to the management cluster by using the Bare Metal Operator to automate provisioning through the cluster API.

.Prerequisites

* You have access to the management cluster as a user with the `cluster-admin` role.
* The OpenShift CLI (`oc`) is installed.
* The Bare Metal Operator is installed on the management cluster.
* Physical worker servers with Redfish-compatible BMC, iDRAC, or iLO access are available.
* Network connectivity exists from the management cluster to the worker BMC interfaces.
* You have the BMC IP address and access credentials for each server.
* You have the MAC address of the management network interface for each server.
* You have the name of the root disk device for each server.

.Procedure

. Set the following environment variables for the worker node:
+
[source,terminal]
----
$ export BMC_IP=<bmc_ip_address>
$ export BMC_USER=<bmc_username>
$ export BMC_PASSWORD=<bmc_password>
$ export WORKER_NAME=<worker_name>
$ export BOOT_MAC=<management_interface_mac>
$ export ROOT_DEVICE=<root_device_path>
----
+
where:
+
--
`<bmc_ip_address>`:: Specifies the IP address of the worker node BMC interface.
`<bmc_username>`:: Specifies the username for BMC access.
`<bmc_password>`:: Specifies the password for BMC access.
`<worker_name>`:: Specifies a name for the worker node, such as `worker-01`.
`<management_interface_mac>`:: Specifies the MAC address of the out-of-band management interface, such as `00:00:5E:00:53:01`.
`<root_device_path>`:: Specifies the path to the root disk device, such as `/dev/nvme0n1`.
--

. Verify BMC connectivity from one of the control plane nodes:
+
[source,terminal]
----
$ ping $BMC_IP
----
+
[source,terminal]
----
$ curl -k https://$BMC_IP/redfish/v1/
----
+
[source,terminal]
----
$ curl -k -u $BMC_USER:$BMC_PASSWORD https://$BMC_IP/redfish/v1/Systems
----

. Verify that the Bare Metal Operator is available:
+
[source,terminal]
----
$ oc get clusteroperator baremetal
----
+
.Example output
[source,terminal]
----
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
baremetal 4.22.x True False False ...
----

. Create a file named `provisioning.yaml` with the following content to disable the provisioning network:
+
[source,yaml]
----
apiVersion: metal3.io/v1alpha1
kind: Provisioning
metadata:
name: provisioning-configuration
spec:
provisioningNetwork: "Disabled"
watchAllNamespaces: false
----
+
[IMPORTANT]
====
When `provisioningNetwork` is set to `Disabled`, servers boot by using Redfish virtual media instead of PXE.
====

. Apply the `Provisioning` resource:
+
[source,terminal]
----
$ oc apply -f provisioning.yaml
----

. Create a file named `bmc-secret.yaml` with the following content to store the BMC credentials:
+
[source,yaml]
----
apiVersion: v1
kind: Secret
metadata:
name: ${WORKER_NAME}-bmc-secret
namespace: openshift-machine-api
type: Opaque
stringData:
username: ${BMC_USER}
password: ${BMC_PASSWORD}
----

. Apply the BMC credentials secret:
+
[source,terminal]
----
$ envsubst < bmc-secret.yaml | oc apply -f -
----

. Create a file named `machineset-dpu.yaml` with the following content:
+
[source,yaml]
----
apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
name: worker-dpu
namespace: openshift-machine-api
spec:
replicas: 1
selector:
matchLabels:
machine.openshift.io/cluster-api-machineset: worker-dpu
template:
metadata:
labels:
machine.openshift.io/cluster-api-machineset: worker-dpu
node-role.kubernetes.io/worker-dpu: ""
spec:
metadata:
labels:
node-role.kubernetes.io/worker-dpu: ""
providerSpec:
value:
hostSelector:
matchLabels:
dpu-capable: "true"
customDeploy:
method: install_coreos
userData:
name: worker-dpu-user-data-managed
namespace: openshift-machine-api
----
+
The `MachineSet` must be created before the `BareMetalHost` resources.
It automatically selects `BareMetalHost` resources that are labeled `dpu-capable: "true"`, provisions them, and applies the `node-role.kubernetes.io/worker-dpu=""` label.
Set `replicas` to match the number of DPU worker nodes.

. Apply the `MachineSet` resource:
+
[source,terminal]
----
$ oc apply -f machineset-dpu.yaml
----
+
.Example output
[source,terminal]
----
machineset.machine.openshift.io/worker-dpu created
----

. Create a file named `baremetalhost.yaml` with the following content:
+
[source,yaml]
----
apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
name: $WORKER_NAME
namespace: openshift-machine-api
labels:
dpu-capable: "true"
annotations:
inspect.metal3.io: disabled
spec:
online: false
bootMACAddress: $BOOT_MAC
rootDeviceHints:
deviceName: $ROOT_DEVICE
bmc:
address: redfish-virtualmedia+https://$BMC_IP
credentialsName: $WORKER_NAME-bmc-secret
disableCertificateVerification: true
----

. Apply the `BareMetalHost` resource:
+
[source,terminal]
----
$ envsubst < baremetalhost.yaml | oc apply -f -
----

.Verification

* Monitor the provisioning progress:
+
[source,terminal]
----
$ oc get bmh -n openshift-machine-api -w
----
+
.Example output
[source,terminal]
----
NAME STATE CONSUMER ONLINE ERROR AGE
worker-01 registering true 10s
worker-01 available true 30s
worker-01 provisioning true 1m
worker-01 provisioned true 10m
----
10 changes: 10 additions & 0 deletions modules/nw-dpf-adjusting-ovnk-cni-overview.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
// Module included in the following assemblies:
//
// * networking/networking_operators/dpf_operator/dpf-worker-nodes-provisioning.adoc

:_mod-docs-content-type: CONCEPT
[id="ovnk-cni-adjustments_{context}"]
= OVN-Kubernetes CNI settings for DPF

[role="_abstract"]
You must adjust the cluster CNI so that it supports DPU acceleration.
72 changes: 72 additions & 0 deletions modules/nw-dpf-approving-dpu-csrs.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
// Module included in the following assemblies:
//
// * networking/networking_operators/dpf_operator/dpf-worker-nodes-provisioning.adoc

:_mod-docs-content-type: PROCEDURE
[id="nw-dpf-approving-dpu-csrs_{context}"]
= Approve DPU node CSRs

[role="_abstract"]
You must approve the pending certificate signing requests (CSRs) for DPU nodes on the hosted cluster so that the DPU nodes can join the hosted cluster and complete provisioning.

[NOTE]
====
When you use the `dpf-hcp-provisioner-operator`, DPU CSR approval is handled automatically.
Manual approval is provided as a fallback if automatic approval is not functioning.
====

.Prerequisites

* You have access to the hosted cluster as a user with the `cluster-admin` role.
* The OpenShift CLI (`oc`) is installed.
* The `KUBECONFIG` environment variable is set to the hosted cluster kubeconfig file.
* DPU provisioning has reached the `DPU Cluster Config` stage.

.Procedure

. Watch for pending CSRs from the DPU nodes:
+
[source,terminal]
----
$ oc get csr -w
----
+
The DPU node name typically follows the pattern `<host_worker_node_name>-<dpu_serial_number>`.

. Approve all pending CSRs:
+
[source,terminal]
----
$ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs oc adm certificate approve
----
+
.Example output
[source,terminal]
----
certificatesigningrequest.certificates.k8s.io/csr-6jx22 approved
certificatesigningrequest.certificates.k8s.io/csr-tb6nd approved
----
+
Repeat this step until no pending CSRs remain.

.Verification

* Verify that the DPU nodes joined the hosted cluster and are in a `Ready` state:
+
[source,terminal]
----
$ oc get nodes
----
+
.Example output
[source,terminal]
----
NAME STATUS ROLES AGE VERSION
host-worker1-mt2413xz0b65 Ready worker 2m48s v1.33.5
host-worker2-mt2413xz0aw6 Ready worker 2m45s v1.33.5
----
+
[NOTE]
====
After the DPU nodes join the hosted cluster, the DPU provisioning process continues to the remaining stages.
====
Loading