Skip to content

Commit 79f695b

Browse files
ajfloederajfloeder
authored
Merge pull request #285 from NearNodeFlash/drive-firmware-upgrade
Update drive firmware procedure for new drive firmware release
2 parents fb4267f + 4f85c8d commit 79f695b

1 file changed

Lines changed: 161 additions & 87 deletions

File tree

‎docs/guides/firmware-upgrade/readme.md‎

Lines changed: 161 additions & 87 deletions
Original file line numberDiff line numberDiff line change
@@ -3,111 +3,185 @@ authors: Tony Floeder <anthony.floeder@hpe.com>, Nate Thornton <nate.thornton@hp
33
categories: setup
44
---
55

6-
# Firmware Upgrade Procedures
6+
# NVMe Drive Firmware
77

8-
This guide presents the firmware upgrade procedures to upgrade firmware from the Rabbit using tools present in the operating system.
8+
| Documentation Section | Link |
9+
|------------------------|---------------------|
10+
| Required Tools | [Required Tools](#required-tools) |
11+
| Installed Firmware | [Querying Firmware](#querying-firmware) |
12+
| Upgrade Procedure | [Firmware Upgrade Procedure](#firmware-upgrade-procedure) |
13+
| Notes | [Upgrade Notes](#upgrade-notes) |
14+
| PAX Firmware | [PCIe Switch Firmware Upgrade](#pcie-switch-firmware-upgrade) |
915

10-
## PCIe Switch Firmware Upgrade
11-
12-
In order to upgrade the firmware on the PCIe switch, the `switchtec` kernel driver and utility of the same name must be installed. Rabbit hardware consists of two PCIe switches, which can be managed by devices typically located at `/dev/switchtec0` and `/dev/switchtec1`.
13-
14-
!!! danger
15-
Upgrading the switch firmware will cause the switch to reset. Prototype Rabbit units not supporting hotplug should undergo a power-cycle to ensure switch initialization following firmware uprade. Similarily, compute nodes not supporting hotplug may lose connectivity after firmware upgrade and should also be power-cycled.
16-
17-
```bash
18-
IMAGE=$1 # Provide the path to the firmware image file
19-
SWITCHES=("/dev/switchtec0" "/dev/switchtec1")
20-
for SWITCH in "${SWITCHES[@]}"; do switchtec fw-update "$SWITCH" "$IMAGE" --yes; done
21-
```
22-
23-
## NVMe Drive Firmware Upgrade
24-
25-
In order to upgrade the firmware on NVMe drives attached to Rabbit, the `switchtec` and `switchtec-nvme` executables must be installed. All firmware downloads to drives are sent to the physical function of the drive which is accessible only using the `switchtec-nvme` executable.
26-
27-
### Batch Method
16+
This guide presents the firmware upgrade procedures for NVMe drives attached to the Rabbit using tools present in the operating system.
2817

29-
#### Download and Commit New Firmware
18+
## Required Tools
3019

31-
The [nvme.sh](https://github.com/NearNodeFlash/nnf-ec/blob/master/tools/nvme.sh) helper script applies the same command to each physical device fabric ID in the system. It provides a convenient way to upgrade the firmware on all drives in the system. Please see [fw-download](https://www.mankier.com/1/nvme-fw-download) and [fw-commit](https://www.mankier.com/1/nvme-fw-commit) for details about the individual commands.
20+
In order to upgrade the firmware on NVMe drives attached to Rabbit, the `switchtec` and `switchtec-nvme` executables must be installed. All firmware downloads to drives are sent to the physical function of the drive (the main interface of the NVMe device, as opposed to any virtual or secondary functions). The physical function is the primary PCIe endpoint that manages the device and is required for firmware operations. Access to the physical function is possible only using the `switchtec-nvme` executable. The helper script: [nnf-nvme.sh](https://github.com/NearNodeFlash/nnf-ec/blob/master/tools/nnf-nvme.sh) provides the mechanism to update drive firmware and query existing versions. [nnf-switch.sh](https://github.com/NearNodeFlash/nnf-ec/blob/master/tools/nnf-switch.sh) provides additional information about the PAX switch's view of the drive firmware configuration.
3221

33-
```bash
34-
# Download firmware to all drives
35-
./nvme.sh cmd fw-download --fw=</path/to/nvme.fw>
22+
## Querying Firmware
3623

37-
# Commit the new firmware
38-
# action=3: The image is requested to be activated immediately
39-
./nvme.sh cmd fw-commit --action=3
40-
```
41-
42-
#### Rebind the PCIe Connections
24+
Running the `nnf-switch.sh slot-info` command is the simplest method to see the active firmware for each drive.
4325

44-
In order to use the drives at this point, they must be unbound and bound to the PCIe fabric to reset device connections. The [bind.sh](https://github.com/NearNodeFlash/nnf-ec/blob/master/tools/bind.sh) helper script performs these two actions. Its use is illustrated below.
26+
```sh
27+
./nnf-switch.sh slot-info
28+
DEVICE: /dev/switchtec0 PAX_ID: 1
4529

46-
```bash
47-
# Unbind all drives from the Rabbit to disconnect the PCIe connection to the drives
48-
./bind.sh unbind
30+
PDFID: 0x1900 SLOT: 04 KIOXIA YCU0A03H03A1 1TCRS104 /dev/nvme9 0000:83:00.0 Physical Port ID 8 (EP attached):
31+
PDFID: 0x1500 SLOT: 05 KIOXIA YCU0A05Z03A1 1TCRS104 /dev/nvme10 0000:84:00.0 Physical Port ID 10 (EP attached):
32+
PDFID: 0x1800 SLOT: 06 KIOXIA YCU0A02V03A1 1TCRS104 /dev/nvme11 0000:85:00.0 Physical Port ID 12 (EP attached):
33+
PDFID: 0x1400 SLOT: 02 KIOXIA YCU0A04303A1 1TCRS104 /dev/nvme12 0000:86:00.0 Physical Port ID 14 (EP attached):
34+
PDFID: 0x1600 SLOT: 09 KIOXIA YCU0A03B03A1 1TCRS104 /dev/nvme13 0000:88:00.0 Physical Port ID 18 (EP attached):
35+
PDFID: 0x1700 SLOT: 10 KIOXIA YCU0A06103A1 1TCRS104 /dev/nvme14 0000:89:00.0 Physical Port ID 20 (EP attached):
36+
PDFID: 0x1300 SLOT: 11 KIOXIA YCU0A00V03A1 1TCRS104 /dev/nvme15 0000:8a:00.0 Physical Port ID 22 (EP attached):
37+
PDFID: 0x1a00 SLOT: 03 KIOXIA YCT0A06P03A1 1TCRS104 /dev/nvme16 0000:8b:00.0 Physical Port ID 48 (EP attached):
4938

50-
# Bind all drives to the Rabbit to reconnect the PCIe bus
51-
./bind.sh bind
39+
DEVICE: /dev/switchtec1 PAX_ID: 0
5240

53-
# At this point, your drives should be running the new firmware.
54-
# Verify the firmware...
55-
./nvme.sh cmd id-ctrl | grep -E "^fr "
41+
PDFID: 0x4200 SLOT: 08 KIOXIA YCT0A00H03A1 1TCRS104 /dev/nvme1 0000:05:00.0 Physical Port ID 8 (EP attached):
42+
PDFID: 0x3d00 SLOT: 07 KIOXIA YCT0A06W03A1 1TCRS104 /dev/nvme2 0000:06:00.0 Physical Port ID 10 (EP attached):
43+
PDFID: 0x3b00 SLOT: 15 KIOXIA YCU0A00F03A1 1TCRS104 /dev/nvme3 0000:07:00.0 Physical Port ID 12 (EP attached):
44+
PDFID: 0x3e00 SLOT: 16 KIOXIA YCU0A05D03A1 1TCRS104 /dev/nvme4 0000:08:00.0 Physical Port ID 14 (EP attached):
45+
PDFID: 0x4000 SLOT: 17 KIOXIA YCT0A00X03A1 1TCRS104 /dev/nvme5 0000:09:00.0 Physical Port ID 16 (EP attached):
46+
PDFID: 0x3f00 SLOT: 18 KIOXIA YCU0A05E03A1 1TCRS104 /dev/nvme6 0000:0a:00.0 Physical Port ID 18 (EP attached):
47+
PDFID: 0x3c00 SLOT: 14 KIOXIA YCT0A07A03A1 1TCRS104 /dev/nvme7 0000:0b:00.0 Physical Port ID 20 (EP attached):
48+
PDFID: 0x4100 SLOT: 12 KIOXIA YCU0A00Z03A1 1TCRS104 /dev/nvme8 0000:0d:00.0 Physical Port ID 48 (EP attached):
5649
```
5750

58-
### Individual Drive Method
51+
Each NVMe drive supports three firmware slots, allowing up to three different firmware versions to be installed at once. To check the installed firmware versions on Rabbit drives, use the `nnf-nvme.sh cmd fw-log` command. The example below shows a factory configuration with `1TCRS104` firmware in all three slots; the `afi` value ("Active Firmware Image") indicates which slot is currently active (e.g., `afi : 0x1` means slot 1 is active, `afi : 0x2` means slot 2 is active, etc.).
5952

60-
#### Determine Physical Device Fabric ID
53+
**Note:** In this document, a "slot" refers to a firmware storage location on the NVMe drive. Each drive can hold up to three firmware images, one per slot.
6154

62-
The first step is to determine a drive's unique Physical Device Fabric Identifier (PDFID). The following code fragment demonstrates one way to list the physcial device fabric ids of all the NVMe drives in the system.
55+
```sh
56+
./nnf-nvme.sh cmd fw-log
57+
Executing fw-log for each drive on /dev/switchtec0
58+
Executing fw-log for each drive on /dev/switchtec1
59+
Execute on 0x1900@/dev/switchtec0 fw-log
60+
Firmware Log for device:switchtec0
61+
afi : 0x1
62+
frs1 : 0x3430315352435431 (1TCRS104)
63+
frs2 : 0x3430315352435431 (1TCRS104)
64+
frs3 : 0x3430315352435431 (1TCRS104)
6365

64-
```bash
65-
#!/bin/bash
66+
... # The other 7 drives attached to /dev/switchtec0...
6667

67-
SWITCHES=("/dev/switchtec0" "/dev/switchtec1")
68-
for SWITCH in "${SWITCHES[@]}";
69-
do
70-
mapfile -t PDFIDS < <(sudo switchtec fabric gfms-dump "${SWITCH}" | grep "Function 0 " -A1 | grep PDFID | awk '{print $2}')
71-
for INDEX in "${!PDFIDS[@]}";
72-
do
73-
echo "${PDFIDS[$INDEX]}@$SWITCH"
74-
done
75-
done
76-
```
68+
Execute on 0x4200@/dev/switchtec1 fw-log
69+
Firmware Log for device:switchtec1
70+
afi : 0x1
71+
frs1 : 0x3430315352435431 (1TCRS104)
72+
frs2 : 0x3430315352435431 (1TCRS104)
73+
frs3 : 0x3430315352435431 (1TCRS104)
7774

78-
```bash
79-
# Produces a list like this:
80-
0x1300@/dev/switchtec0
81-
0x1600@/dev/switchtec0
82-
0x1700@/dev/switchtec0
83-
0x1400@/dev/switchtec0
84-
0x1800@/dev/switchtec0
85-
0x1900@/dev/switchtec0
86-
0x1500@/dev/switchtec0
87-
0x1a00@/dev/switchtec0
88-
0x4100@/dev/switchtec1
89-
0x3c00@/dev/switchtec1
90-
0x4000@/dev/switchtec1
91-
0x3e00@/dev/switchtec1
92-
0x4200@/dev/switchtec1
93-
0x3b00@/dev/switchtec1
94-
0x3d00@/dev/switchtec1
95-
0x3f00@/dev/switchtec1
75+
... # The other 7 drives attached to /dev/switchtec1
9676
```
9777

98-
#### Download Firmware
99-
100-
Using the physical device fabric identifier, the following commands update the firmware for specified drive.
78+
## Firmware Upgrade Procedure
79+
80+
This section documents the drive firmware update procedure.
81+
NOTE: The PAX switch allows a single drive firmware download at a time, but since there are 2 PAX switches in the Rabbit, `nnf-nvme.sh` downloads drive firmware to 2 drives concurrently. The output for the `update-firmware` operation thus waits until all operations complete.
82+
83+
1. **Initiate Firmware Update**
84+
85+
```sh
86+
time ./nnf-nvme.sh update-firmware ./1TCRS105.std --verbose
87+
88+
KIOXIA NVMe Drive Firmware Update
89+
=================================
90+
91+
Firmware file: ./1TCRS105.std
92+
New version: 105
93+
Dry run: false
94+
Verbose: true
95+
Force update: false
96+
97+
Starting firmware update for 0x1900@/dev/switchtec0
98+
Processing device: 0x1900@/dev/switchtec0
99+
Found 3 firmware slots for 0x1900@/dev/switchtec0:
100+
Slot 1: 1TCRS104
101+
Slot 2: 1TCRS104
102+
Slot 3: 1TCRS104
103+
Oldest firmware found in slot 1: 1TCRS104
104+
New firmware version: 105
105+
Updating firmware on 0x1900@/dev/switchtec0 (slot 1) from 1TCRS104 to 105
106+
Downloading firmware to 0x1900@/dev/switchtec0...
107+
Firmware download success
108+
Firmware downloaded successfully to 0x1900@/dev/switchtec0
109+
Committing firmware to slot 1 on 0x1900@/dev/switchtec0...
110+
Success committing firmware action:0 slot:1
111+
Firmware committed successfully to slot 1 on 0x1900@/dev/switchtec0
112+
Activating firmware on 0x1900@/dev/switchtec0...
113+
Success committing firmware action:3 slot:0
114+
Firmware activated successfully on 0x1900@/dev/switchtec0
115+
Successfully updated firmware on 0x1900@/dev/switchtec0
116+
SUCCESS: Firmware update completed for 0x1900@/dev/switchtec0
117+
118+
... # The other 7 drives attached to /dev/switchtec0...
119+
120+
Starting firmware update for 0x4200@/dev/switchtec1
121+
Processing device: 0x4200@/dev/switchtec1
122+
Found 3 firmware slots for 0x4200@/dev/switchtec1:
123+
Slot 1: 1TCRS104
124+
Slot 2: 1TCRS104
125+
Slot 3: 1TCRS104
126+
Oldest firmware found in slot 1: 1TCRS104
127+
New firmware version: 105
128+
Updating firmware on 0x4200@/dev/switchtec1 (slot 1) from 1TCRS104 to 105
129+
Downloading firmware to 0x4200@/dev/switchtec1...
130+
Firmware download success
131+
Firmware downloaded successfully to 0x4200@/dev/switchtec1
132+
Committing firmware to slot 1 on 0x4200@/dev/switchtec1...
133+
Success committing firmware action:0 slot:1
134+
Firmware committed successfully to slot 1 on 0x4200@/dev/switchtec1
135+
Activating firmware on 0x4200@/dev/switchtec1...
136+
Success committing firmware action:3 slot:0
137+
Firmware activated successfully on 0x4200@/dev/switchtec1
138+
Successfully updated firmware on 0x4200@/dev/switchtec1
139+
SUCCESS: Firmware update completed for 0x4200@/dev/switchtec1
140+
141+
... # The other 7 drives attached to /dev/switchtec1...
142+
143+
real 4m24.395s
144+
user 0m0.161s
145+
sys 3m12.450s
146+
```
147+
148+
- This command runs the `update-firmware` function of `nnf-nvme.sh` targeting all drives and using the specified firmware file (`1TCRS105.std`).
149+
- The `--verbose` flag provides detailed output for each drive.
150+
- The duration for this operation was `4m 25s`.
151+
152+
2. **Verify Firmware Activation**
153+
154+
```sh
155+
./nnf-nvme.sh cmd fw-log
156+
Executing fw-log for each drive on /dev/switchtec0
157+
Executing fw-log for each drive on /dev/switchtec1
158+
Execute on 0x1900@/dev/switchtec0 fw-log
159+
Firmware Log for device:switchtec0
160+
afi : 0x1
161+
frs1 : 0x3530315352435431 (1TCRS105)
162+
frs2 : 0x3430315352435431 (1TCRS104)
163+
frs3 : 0x3430315352435431 (1TCRS104)
164+
... # The other 7 drives attached to /dev/switchtec0...
165+
Execute on 0x4200@/dev/switchtec1 fw-log
166+
Firmware Log for device:switchtec1
167+
afi : 0x1
168+
frs1 : 0x3530315352435431 (1TCRS105)
169+
frs2 : 0x3430315352435431 (1TCRS104)
170+
frs3 : 0x3430315352435431 (1TCRS104)
171+
... # The other 7 drives attached to /dev/switchtec1
172+
173+
```
174+
175+
- The firmware upgrade installed `1TCRS105` into slot 1.
176+
177+
3. **Power Cycle Rabbit System**
178+
179+
- Power cycle the Rabbit system, including all drives and PAX switches, to ensure reliable initialization.
180+
181+
## Upgrade Notes
182+
183+
`nnf-nvme.sh update-firmware` selects the slot containing the oldest firmware and installs the new drive firmware into that slot. If you run `nnf-nvme.sh update-firmware` 3 times, you would end up overwriting all 3 slots with the newer firmware. This is not recommended, but possible.
101184

102-
```bash
103-
# Download firmware to the drive
104-
sudo switchtec-nvme fw-download <PhysicalDeviceFabricID> --fw=</path/to/nvme.fw>
105-
106-
# Activate the new firmware
107-
# action=3: The image is requested to be activated immediately without reset.
108-
sudo switchtec-nvme fw-commit --action=3
109-
```
110-
111-
#### Rebind PCIe Connection
185+
## PCIe Switch Firmware Upgrade
112186

113-
Once the firmware has been downloaded and committed, the PCIe connection from the Rabbit to the drive must be unbound and rebound. Please see [bind.sh](https://github.com/NearNodeFlash/nnf-ec/blob/master/tools/bind.sh) for details.
187+
The supported approach to upgrade PCIe switch (PAX) firmware is via the redfish endpoint.

0 commit comments

Comments
 (0)