|
| 1 | +<div id="readme" class="Box-body readme blob js-code-block-container"> |
| 2 | +<article class="markdown-body entry-content p-3 p-md-6" itemprop="text"> |
| 3 | +<p align="right"> |
| 4 | +<a href="/?tab=readme-ov-file#--hacc-development">Back to top</a> |
| 5 | +</p> |
| 6 | + |
| 7 | +# CMDB Structure |
| 8 | + |
| 9 | +The Configuration Management DataBase (CMDB) stores all information about the nodes in the cluster and the devices inside those nodes in a structure of text files. The CMDB is located in the `hdev/cli/cmdb/` directory and is structured as follows: |
| 10 | + |
| 11 | +``` |
| 12 | +cmdb/ |
| 13 | +├── node-01.example.com/ |
| 14 | +│ ├── devices_acap_fpga |
| 15 | +│ ├── devices_gpu |
| 16 | +│ └── devices_network |
| 17 | +├── node-02.example.com/ |
| 18 | +│ ├── devices_acap_fpga |
| 19 | +│ ├── devices_gpu |
| 20 | +│ └── devices_network |
| 21 | +├── node-03.example.com/ |
| 22 | +│ ├── devices_acap_fpga |
| 23 | +│ ├── devices_gpu |
| 24 | +│ └── devices_network |
| 25 | +└── < etc ... > |
| 26 | +``` |
| 27 | + |
| 28 | +Each node is represented by a directory with the same name as the hostname of the node. Inside this node directory there are up to three text files, each representing a specific category of device. |
| 29 | + |
| 30 | +### Device files |
| 31 | + |
| 32 | +These files contain an entry per row for a particular device. A row has multiple attributes related to the entry, of which only the values are notated. The attribute is inferred from the location on the row. The values of attributes on each row are separated by a space. If an attribute can take a list, then the list items are separated by a forward slash '/'. |
| 33 | + |
| 34 | +Below are the schemas of the three different device files: |
| 35 | + |
| 36 | +#### devices\_acap\_fpga |
| 37 | + |
| 38 | + |
| 39 | + |
| 40 | +The reconfigurable devices are PCIe add-on cards that provide a reconfigurable fabric. Hdev only supports AMD/Xilinx FPGAs at this moment. |
| 41 | + |
| 42 | +There is one row per reconfigurable device, and the columns represent the following information: |
| 43 | + |
| 44 | +``` |
| 45 | +<device index> <upstream bdf> <root bdf> <lnkctl> <device type> <device name> <serial number> <ip addresses> <mac addresses> <platform> <part> |
| 46 | +``` |
| 47 | + |
| 48 | +| Attribute | Type | Description | |
| 49 | +|---------------------|------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------| |
| 50 | +| Device Index | int | Unique index of the device row. First row has index 1, the second has index 2, etc. | |
| 51 | +| Upstream BDF | string | The PCI BDF of the Xilinx reconfigurable device. See [instructions below](#upstream-bdf). | |
| 52 | +| Root BDF | string | The PCI BDF of the port that the reconfigurable device is connected to. See [instructions below](#root-bdf). | |
| 53 | +| LnkCtl | hex | The capabilities register address of the LinkCtl for the root bdf. See [instructions below](#lnkctl). | |
| 54 | +| Device Type | string | Type of reconfigurable device. Choose between "acap", "asoc", or "fpga". See [instructions below](#device-type-for-reconfigurable-devices). | |
| 55 | +| Device Name | string | Name of the device as reported by Vivado Hardware Manager. | |
| 56 | +| Serial Number | string | The Serial number of the device. See [instructions below](#serial-number). | |
| 57 | +| IP Addresses | list | A list of IP addresses for the interfaces of the reconfigurable device. Use reserved IPs, dont' rely on DHCP for this type of device. | |
| 58 | +| MAC Addresses | list | A list of MAC addresses of the interfaces of the reconfigurable device. List items correspond to the items in the IP list. See [instructions below](#mac-addresses). | |
| 59 | +| Platform | string | The default platform (also called shell) that is loaded on the reconfigurable device. See [instructions below](#platform). | |
| 60 | +| Part Number | string | The part name of the reconfigurable device. See [instructions below](#part-number). | |
| 61 | + |
| 62 | + |
| 63 | +#### devices\_gpu |
| 64 | + |
| 65 | + |
| 66 | + |
| 67 | +The Graphics Processing Units (GPU) here are targeting AMD GPUs that use the rocm library. |
| 68 | + |
| 69 | +There is one row per GPU, and the columns represent the following information: |
| 70 | + |
| 71 | +``` |
| 72 | +<device index> <bus> <device type> <gpu id> <serial number> <unique id> |
| 73 | +``` |
| 74 | + |
| 75 | +| Attribute | Type | Description | |
| 76 | +|---------------------|------------|-----------------------------------------------------------------------------------------------------------------------------------------------| |
| 77 | +| Device Index | int | Unique index of the device row. First row has index 1, the second has index 2, etc. | |
| 78 | +| Upstream BDF | string | The PCI BDF of the GPU. See [instructions below](#upstream-bdf). | |
| 79 | +| Device Type | string | Always set to "gpu". | |
| 80 | +| GPU ID | string | The Device ID as denoted by `rocm-smi -i`. | |
| 81 | +| Serial Number | string | The Serial number of the device. See [instructions below](#serial-number). | |
| 82 | +| Unique ID | string | The Unique ID as provided by `rocm-smi --showuniqueid`. | |
| 83 | + |
| 84 | + |
| 85 | +#### devices\_network |
| 86 | + |
| 87 | + |
| 88 | + |
| 89 | +The network devices are the Network Interface Cards (NIC) that interface with a high-speed data network. Low-speed access and management connections should not be added here. |
| 90 | + |
| 91 | +There is one row per networking device, and the columns represent the following information: |
| 92 | + |
| 93 | +``` |
| 94 | +<device index> <bdf> <device type> <device name> <ip addresses> <mac addresses> |
| 95 | +``` |
| 96 | + |
| 97 | +| Attribute | Type | Description | |
| 98 | +|---------------------|------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------| |
| 99 | +| Device Index | int | Unique index of the device row. First row has index 1, the second has index 2, etc. | |
| 100 | +| Upstream BDF | string | The PCI BDF of the network device. See [instructions below](#upstream-bdf). | |
| 101 | +| Device Type | string | Always set to "nic". | |
| 102 | +| Device Name | string | A representative string of your choosing that identifies the vendor or model of your NIC. | |
| 103 | +| IP Addresses | list | A list of IP addresses for the interfaces of the network device. It is expected in a data center setting, IP addresses are reserved or static. | |
| 104 | +| MAC Addresses | list | A list of MAC addresses of the interfaces of the network device. List items correspond to the items in the IP list. See [instructions below](#mac-addresses). | |
| 105 | + |
| 106 | + |
| 107 | +How to find the ... |
| 108 | +------------------- |
| 109 | + |
| 110 | +#### Upstream BDF |
| 111 | +BDF is an abbreviation of [Bus/Device/Function](https://en.wikipedia.org/wiki/PCI_configuration_space#Technical_information) and is the identifier for a certain PCIe device. The BDF follows a format of BB:DD.F, where BB, DD and F are hexadecimal numbers that represent the Bus, Device and Function respectively. We can find the BDF for our device using the `lspci` command. We can search in the lspci command for a Vendor or Model name that would identify our device. For example `lspci | grep Xilinx`, will most likely give us all BDFs for Xilinx devices. |
| 112 | + |
| 113 | + |
| 114 | + |
| 115 | +In this example, we only have one V80 FPGA, but see two BDFs. As the Bus and Device are the same, we know this is the same FPGA device, that provides two functions. If this is the case for you always pick the BDF with Function 0 for the entry in the CMDB. |
| 116 | + |
| 117 | +We can also search in lspci using a vendor ID, which may result in more reliable results. You will need to know the Vendor ID of your device, which you often can find using a search engine. You can also find it when you already know a device to be of that vendor, for example the Xilinx devices from before. We can get the Vendor ID using: `setpci -s c4:00.0 VENDOR_ID` (yes, confusingly the program is called `setpci`, but this specific command will only read). This will give us `10ee` for a Xilinx Device. So when we look for a Xilinx Device in the future we can also run (note the `:` at the end, this is important) |
| 118 | +```bash |
| 119 | +lspci -d 10ee: |
| 120 | +``` |
| 121 | + |
| 122 | +Here is a table of Vendor IDs for a selection of vendors: |
| 123 | + |
| 124 | +| Vendor | Vendor ID | |
| 125 | +|------------------|-----------| |
| 126 | +| Xilinx | 10ee: | |
| 127 | +| Broadcom | 14e4: | |
| 128 | +| Mellanox | 15b3: | |
| 129 | +| AMD motherboards | 1022: | |
| 130 | +| AMD GPUs | 1002: | |
| 131 | + |
| 132 | +The tools that come with reconfigurable devices often also show the BDF of the devices, for example `xbutil examine` and `ami_tool overview`. Additionally for AMD GPUs you can also find the upstream BDF using `rocm-smi --showbus`. |
| 133 | + |
| 134 | +#### Root BDF |
| 135 | +The root BDF is the BDF of the device that the Upstream BDF is connected to. We can easily find this root BDF with |
| 136 | +```bash |
| 137 | +lspci -s <upstream bdf> -PP |
| 138 | +``` |
| 139 | +The `-s <upstream bdf>` flag selects the device with the Upstream BDF (fill in the BDF that is relevant to you). The `-PP` flag will enhance the lspci output to also show the root BDF. The lspci will show this as `<root bdf>/<upstream bdf>`. |
| 140 | + |
| 141 | + |
| 142 | + |
| 143 | +You can also see the whole structure of the PCIe connection tree using `lspci -t`. Which will show you something like: |
| 144 | + |
| 145 | + |
| 146 | + |
| 147 | +We here see that under `0000:c0` there are multiple devices (the first `0000` represent the PCIe domain or PCIe segment, which most systems only have one of). So we see `[c4]` (the Bus of our Xilinx device) next to `01.1`, which is the Device and Function part of the root BDF. So when we construct this, the `c4:00.0` device is under the `c0.01.1` root BDF, which is also what we saw with the `lspci -s c4:00.0 -PP` command. |
| 148 | + |
| 149 | +#### LnkCtl |
| 150 | +The LnkCtl is the register address that has the controls over the link on the root bdf. We find this address by first showing the verbose lspci output of the root bdf. |
| 151 | + |
| 152 | +```bash |
| 153 | +lspci -s c0.01.1 -vv |
| 154 | +``` |
| 155 | + |
| 156 | +We look for the section of `LnkCtl`. This is inside a `Capabilities` section and we look at the value that is in square brackets after this `Capabilities` header. In our example we see `[58]`, so we fill in `58` for the LnkCtl. |
| 157 | + |
| 158 | +#### Device Type for reconfigurable devices |
| 159 | +The Device type for the reconfigurable devices have three options: `acap`, `asoc` and `fpga`. |
| 160 | + |
| 161 | +- Select `acap` for the [AMD Adaptive Compute Acceleration Platforms](https://docs.amd.com/r/en-US/am016-versal-cpm-ccix/Revision-History), such as the `AMD Versal VCK5000`. |
| 162 | +- Select `asoc` for the [AMD Adaptive System on Chip](https://www.amd.com/en/products/accelerators/alveo/v80.html), such as the `AMD Versal V80`. |
| 163 | +- Select `fpga` for all other AMD Alveo devices, such as the `AMD U55c`. |
| 164 | + |
| 165 | + |
| 166 | +#### Serial number |
| 167 | +You can often find the Serial number in the very verbose output of lspci using `lspci -vvv`. You will then find it after `[SN]`. |
| 168 | + |
| 169 | +For reconfigurable devices, this method often does not work. Then you will use the provided tools to look this up. For devices using XRT use `xbmgmt examine` and for devices using AVED use `ami_tool overview`. |
| 170 | + |
| 171 | +For AMD GPUs you can also use `rocm-smi --showserial` to find the serial number. |
| 172 | + |
| 173 | + |
| 174 | +#### MAC Addresses |
| 175 | + |
| 176 | +###### fpga and acap |
| 177 | +For the FPGAs and ACAPs that use XRT, we can retrieve the MAC addresses using the following command |
| 178 | +```bash |
| 179 | +xbmgmt examine --device <bdf> -r platform |
| 180 | +``` |
| 181 | + |
| 182 | +Note that using the `xbmgmt` command we need to use the `.0` function, whereas the user-space `xbutil` uses the `.1` function. |
| 183 | + |
| 184 | +###### asoc |
| 185 | +For the ASOCs that use AVED, we can retrieve the MAC addresses using the following command |
| 186 | +```bash |
| 187 | +ami_tool mfg_info -d <bdf> |
| 188 | +``` |
| 189 | + |
| 190 | +###### network |
| 191 | +To get the mac address of a network device, we can simply use a standard Linux command |
| 192 | +```bash |
| 193 | +ip a |
| 194 | +``` |
| 195 | +Look for the network interface of your NIC and note down the mac address. If your NIC has multiple physical ports, then it will also most likely have multiple interfaces in the output of the `ip a` command. These interface names are often closely related and only increase a certain value in the name. |
| 196 | + |
| 197 | +#### Platform |
| 198 | +The platform is the shell that is installed on the FPGA/ASOC/ACAP. |
| 199 | + |
| 200 | +For devices using XRT we can list all our devices and their platform using this command |
| 201 | +```bash |
| 202 | +xbutil examine |
| 203 | +``` |
| 204 | + |
| 205 | +For devices using AVED, we can list all our devices and their platform using this command (TODO: probably, but still need to verify) |
| 206 | +```bash |
| 207 | +ami_tool overview |
| 208 | +``` |
| 209 | + |
| 210 | +#### Part Number |
| 211 | +TODO: unknown how to get this at the moment |
0 commit comments