Move BDT to more models. SR to follow next.

Author: bo3z

part5_bdt.ipynb 6_more_models/6a_bdt.ipynbpart5_bdt.ipynb renamed to 6_more_models/6a_bdt.ipynb

@@ -6,17 +6,20 @@
    "source": [
     "<img src=\"https://github.com/thesps/conifer/blob/master/conifer_v1.png?raw=true\" width=\"250\" alt=\"conifer\" />\n",
     "\n",
-    "In this notebook we will take the first steps with training a BDT with `xgboost`, then translating it to HLS code for FPGA with `conifer`\n",
+    "In this notebook we will take the first steps with training a boosted decision tree (BDT) with `xgboost`, then translating it to HLS code for FPGA inference with `conifer`.\n",
     "\n",
-    "Key concepts:\n",
-    "- model training\n",
-    "- model evaluation\n",
-    "- `conifer` configuration and conversion\n",
-    "- model emulation\n",
-    "- model synthesis\n",
-    "- accelerator creation\n",
+    "## What is a Boosted Decision Tree?\n",
     "\n",
-    "For some use cases, the Forest Processing Unit might be an easier entry point as no FPGA synthesis is required for supported boards. Read more about the FPU here: https://ssummers.web.cern.ch/conifer/fpu.html"
+    "A Boosted Decision Tree (BDT) is an ensemble learning method that builds a strong classifier by combining many shallow decision trees. Each tree is trained to correct the residual errors of the previous ones. `XGBoost` is a particularly efficient and widely used gradient boosting framework that adds regularisation and second-order gradient information to improve generalisation and training speed. BDTs are popular in high-energy physics because they train quickly, are interpretable, and are often competitive with deep neural networks on tabular data. Their tree-structured computation also maps naturally to FPGA hardware: each tree can be evaluated in parallel, making BDTs well-suited for low-latency trigger and online inference applications.\n",
+    "\n",
+    "## Key notebook parts\n",
+    "\n",
+    "- **Model training**: train a multi-class `XGBClassifier` on the jet tagging dataset and compare its accuracy to the Keras/PyTorch baseline from Part 1\n",
+    "- **Model evaluation**: measure classification performance using ROC and accuracy\n",
+    "- **`conifer` configuration and conversion**: configure the `xilinxhls` backend and convert the trained XGBoost model into `conifer`'s intermediate representation, which generates synthesisable HLS C++ code\n",
+    "- **Model emulation**: compile the generated HLS C++ on the CPU and run bit-accurate predictions to verify conversion correctness and numerical precision before FPGA synthesis\n",
+    "- **Model synthesis**: run Vitis HLS C Synthesis followed by Vivado RTL synthesis\n",
+    "- **Accelerator creation**: configure a board-specific deployment target and build a complete bitfile for a `pynq-z2` board, ready for on-device inference\n"
    ]
   },
   {
@@ -27,30 +30,38 @@
    "source": [
     "import xgboost as xgb\n",
     "import matplotlib.pyplot as plt\n",
+    "import sys\n",
+    "sys.path.append('..')\n",
     "import plotting\n",
     "import numpy as np\n",
     "from scipy.special import softmax\n",
     "from sklearn.preprocessing import LabelEncoder, OneHotEncoder\n",
     "import conifer\n",
     "import json\n",
     "import os\n",
-    "import sys\n",
-    "\n",
-    "os.environ['PATH'] = os.environ['XILINX_VITIS'] + '/bin:' + os.environ['PATH']\n",
     "\n",
-    "# enable more output from conifer\n",
+    "# Enable more outputs from conifer\n",
     "import logging\n",
     "\n",
     "logging.basicConfig(stream=sys.stdout, level=logging.WARNING)\n",
     "logger = logging.getLogger('conifer')\n",
     "logger.setLevel('DEBUG')\n",
     "\n",
-    "# create a random seed at we use to make the results repeatable\n",
+    "# Create a random seed at we use to make the results repeatable\n",
     "seed = int('hls4ml-tutorial'.encode('utf-8').hex(), 16) % 2**31\n",
     "\n",
     "print(f'Using conifer version {conifer.__version__}')"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "MODEL_TYPE = 'keras'  # set to 'pytorch' if you used the PyTorch notebook in Part 1"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -59,7 +70,7 @@
     "\n",
     "Load the jet tagging dataset.\n",
     "\n",
-    "**Note**: you need to run part1 first."
+    "**Note**: you need to run part 1 first to generate the dataset files."
    ]
   },
   {
@@ -68,11 +79,11 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "X_train_val = np.load('X_train_val.npy')\n",
-    "X_test = np.load('X_test.npy')\n",
-    "y_train_val_one_hot = np.load('y_train_val.npy')\n",
-    "y_test_one_hot = np.load('y_test.npy')\n",
-    "classes = np.load('classes.npy', allow_pickle=True)"
+    "X_train_val = np.load('../data/X_train_val.npy')\n",
+    "X_test = np.load('../data/X_test.npy')\n",
+    "y_train_val_one_hot = np.load('../data/y_train_val.npy')\n",
+    "y_test_one_hot = np.load('../data/y_test.npy')\n",
+    "classes = np.load('../data/classes.npy', allow_pickle=True)"
    ]
   },
   {
@@ -131,32 +142,50 @@
    "outputs": [],
    "source": [
     "from sklearn.metrics import accuracy_score\n",
-    "from tensorflow.keras.models import load_model\n",
-    "\n",
-    "# load the KERAS model from part 1\n",
-    "model_ref = load_model('model_1/KERAS_check_best_model.h5')\n",
-    "y_ref = model_ref.predict(X_test)\n",
     "\n",
-    "# compute predictions of the xgboost model\n",
+    "if MODEL_TYPE == 'keras':\n",
+    "    from tensorflow.keras.models import load_model\n",
+    "    model_ref = load_model('../models/keras_model_part1.h5')\n",
+    "    y_ref = model_ref.predict(X_test)\n",
+    "\n",
+    "elif MODEL_TYPE == 'pytorch':\n",
+    "    import torch\n",
+    "    import torch.nn as nn\n",
+    "\n",
+    "    class JetTagger(nn.Module):\n",
+    "        def __init__(self):\n",
+    "            super().__init__()\n",
+    "            self.fc1    = nn.Linear(16, 64)\n",
+    "            self.fc2    = nn.Linear(64, 32)\n",
+    "            self.fc3    = nn.Linear(32, 32)\n",
+    "            self.output = nn.Linear(32, 5)\n",
+    "\n",
+    "        def forward(self, x):\n",
+    "            x = torch.relu(self.fc1(x))\n",
+    "            x = torch.relu(self.fc2(x))\n",
+    "            x = torch.relu(self.fc3(x))\n",
+    "            return torch.softmax(self.output(x), dim=1)\n",
+    "\n",
+    "    model_ref = JetTagger()\n",
+    "    model_ref.load_state_dict(torch.load('../models/pytorch_weights_part1.pt'))\n",
+    "    model_ref.eval()\n",
+    "    with torch.no_grad():\n",
+    "        y_ref = model_ref(torch.FloatTensor(X_test)).numpy()\n",
+    "\n",
+    "# Compute predictions of the xgboost model\n",
     "y_xgb = clf.predict_proba(X_test)\n",
-    "print(f'Accuracy baseline:  {accuracy_score(np.argmax(y_test_one_hot, axis=1), np.argmax(y_ref, axis=1)):.5f}')\n",
+    "print(f'Accuracy {MODEL_TYPE}:    {accuracy_score(np.argmax(y_test_one_hot, axis=1), np.argmax(y_ref, axis=1)):.5f}')\n",
     "print(f'Accuracy xgboost:   {accuracy_score(np.argmax(y_test_one_hot, axis=1), np.argmax(y_xgb, axis=1)):.5f}')\n",
     "\n",
     "fig, ax = plt.subplots(figsize=(9, 9))\n",
     "_ = plotting.makeRoc(y_test_one_hot, y_ref, classes, linestyle='--')\n",
-    "plt.gca().set_prop_cycle(None)  # reset the colors\n",
+    "plt.gca().set_prop_cycle(None)\n",
     "_ = plotting.makeRoc(y_test_one_hot, y_xgb, classes, linestyle='-')\n",
     "\n",
-    "# add a legend\n",
     "from matplotlib.lines import Line2D\n",
-    "\n",
-    "lines = [\n",
-    "    Line2D([0], [0], ls='--'),\n",
-    "    Line2D([0], [0], ls='-'),\n",
-    "]\n",
     "from matplotlib.legend import Legend\n",
-    "\n",
-    "leg = Legend(ax, lines, labels=['part1 Keras', 'xgboost'], loc='lower right', frameon=False)\n",
+    "leg = Legend(ax, [Line2D([0], [0], ls='--'), Line2D([0], [0], ls='-')],\n",
+    "             labels=[f'part1 {MODEL_TYPE}', 'xgboost'], loc='lower right', frameon=False)\n",
     "ax.add_artist(leg)"
    ]
   },
@@ -170,7 +199,7 @@
     "\n",
     "We will print the configuration, modify it, and print it again. The modifications are:\n",
     "- set the `OutputDirectory` to something descriptive\n",
-    "- set the `XilinxPart` to the part number of the FPGA on the Alveo U50"
+    "- set the `XilinxPart` to the part number of the FPGA on the Alveo U250"
    ]
   },
   {
@@ -181,16 +210,16 @@
    "source": [
     "cfg = conifer.backends.xilinxhls.auto_config()\n",
     "\n",
-    "# print the config\n",
+    "# Print the config\n",
     "print('Default Configuration\\n' + '-' * 50)\n",
     "plotting.print_dict(cfg)\n",
     "print('-' * 50)\n",
     "\n",
-    "# modify the config\n",
-    "cfg['OutputDir'] = 'model_5/'\n",
+    "# Set output directory and target device\n",
+    "cfg['OutputDir'] = '../hls4ml_prjs/conifer_prj_bdt_part6a'\n",
     "cfg['XilinxPart'] = 'xcu250-figd2104-2L-e'\n",
     "\n",
-    "# print the config again\n",
+    "# Print the config again (to verify change)\n",
     "print('Modified Configuration\\n' + '-' * 50)\n",
     "plotting.print_dict(cfg)\n",
     "print('-' * 50)"
@@ -220,14 +249,17 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# convert the model to the conifer representation\n",
+    "# Convert the model to the conifer representation\n",
     "conifer_model = conifer.converters.convert_from_xgboost(clf, cfg)\n",
-    "# print the help to see the API on the conifer_model\n",
+    "\n",
+    "# Print the help to see the API of the conifer_model\n",
     "help(conifer_model)\n",
-    "# write the project (writing HLS project to disk)\n",
+    "\n",
+    "# Write the project (writing HLS project to disk)\n",
     "conifer_model.write()\n",
-    "# save the conifer model - we can load this again later\n",
-    "clf.save_model('model_5/xgboost_model.json')"
+    "\n",
+    "# Save the xgboost model alongside the conifer project\n",
+    "clf.save_model('../hls4ml_prjs/conifer_prj_bdt_part6a/xgboost_model.json')"
    ]
   },
   {
@@ -237,10 +269,10 @@
     "## Explore\n",
     "Browse the files in the newly created project directory to take a look at the HLS code.\n",
     "\n",
-    "The output of `!tree model_5` is:\n",
+    "The output of `!tree ../hls4ml_prjs/conifer_prj_bdt_part6a` is:\n",
     "\n",
     "```\n",
-    "model_5/\n",
+    "conifer_prj_bdt_part6a/\n",
     "├── bridge.cpp\n",
     "├── build_hls.tcl\n",
     "├── firmware\n",
@@ -306,29 +338,22 @@
    "source": [
     "y_hls_proba = softmax(y_hls)  # compute class probabilities from the raw predictions\n",
     "\n",
-    "print(f'Accuracy baseline:  {accuracy_score(np.argmax(y_test_one_hot, axis=1), np.argmax(y_ref, axis=1)):.5f}')\n",
+    "print(f'Accuracy {MODEL_TYPE}:    {accuracy_score(np.argmax(y_test_one_hot, axis=1), np.argmax(y_ref, axis=1)):.5f}')\n",
     "print(f'Accuracy xgboost:   {accuracy_score(np.argmax(y_test_one_hot, axis=1), np.argmax(y_xgb, axis=1)):.5f}')\n",
     "print(f'Accuracy conifer:   {accuracy_score(np.argmax(y_test_one_hot, axis=1), np.argmax(y_hls_proba, axis=1)):.5f}')\n",
     "\n",
-    "\n",
     "fig, ax = plt.subplots(figsize=(9, 9))\n",
     "_ = plotting.makeRoc(y_test_one_hot, y_ref, classes, linestyle='--')\n",
-    "plt.gca().set_prop_cycle(None)  # reset the colors\n",
+    "plt.gca().set_prop_cycle(None)\n",
     "_ = plotting.makeRoc(y_test_one_hot, y_xgb, classes, linestyle=':')\n",
-    "plt.gca().set_prop_cycle(None)  # reset the colors\n",
+    "plt.gca().set_prop_cycle(None)\n",
     "_ = plotting.makeRoc(y_test_one_hot, y_hls_proba, classes, linestyle='-')\n",
     "\n",
-    "# add a legend\n",
     "from matplotlib.lines import Line2D\n",
-    "\n",
-    "lines = [\n",
-    "    Line2D([0], [0], ls='--'),\n",
-    "    Line2D([0], [0], ls=':'),\n",
-    "    Line2D([0], [0], ls='-'),\n",
-    "]\n",
     "from matplotlib.legend import Legend\n",
-    "\n",
-    "leg = Legend(ax, lines, labels=['part1 Keras', 'xgboost', 'conifer'], loc='lower right', frameon=False)\n",
+    "leg = Legend(ax,\n",
+    "    [Line2D([0], [0], ls='--'), Line2D([0], [0], ls=':'), Line2D([0], [0], ls='-')],\n",
+    "    labels=[f'part1 {MODEL_TYPE}', 'xgboost', 'conifer'], loc='lower right', frameon=False)\n",
     "ax.add_artist(leg)"
    ]
   },
@@ -337,11 +362,11 @@
    "metadata": {},
    "source": [
     "## Build\n",
-    "Now we'll run the Vitis HLS and Vivado synthesis. HLS C Synthesis compiles our C++ to RTL, performing scheduling and resource mapping. Vivado synthesis synthesizes the RTL from the previous step into a netlist, and produces a more realistic resource estimation. The latency can't change during Vivado synthesis, it's fixed in the RTL description.\n",
+    "Now we'll run the Vitis HLS and Vivado synthesis. HLS C Synthesis compiles our C++ to RTL, performing scheduling and resource mapping. Vivado synthesis synthesizes the RTL from the previous step into a netlist, and produces a more realistic resource estimation. \n",
     "\n",
     "After the build completes we can also browse the new log files and reports that are generated.\n",
     "\n",
-    "**Warning**: this step might take around 10 minutes"
+    "**This step takes around 10 minutes.**"
    ]
   },
   {
@@ -397,7 +422,7 @@
    "outputs": [],
    "source": [
     "pynq_model_cfg = conifer.backends.xilinxhls.auto_config()\n",
-    "pynq_model_cfg['OutputDir'] = 'model_5_pynq'  # choose a new project directory\n",
+    "pynq_model_cfg['OutputDir'] = '../hls4ml_prjs/conifer_prj_bdt_part6a_pynq'\n",
     "pynq_model_cfg['ProjectName'] = 'conifer_jettag'\n",
     "pynq_model_cfg['AcceleratorConfig'] = {\n",
     "    'Board': 'pynq-z2',  # choose a pynq-z2 board\n",
@@ -444,7 +469,7 @@
    "source": [
     "### Load the model\n",
     "\n",
-    "We load the JSON for the conifer model we previously used, applying the new configuration just defined. We'll see that the FPGA part specified by the board overrides the `XilinxPart` specified in the default."
+    "We load the JSON for the conifer model we previously saved, applying the new configuration just defined. We'll see that the FPGA part specified by the board overrides the `XilinxPart` specified in the default."
    ]
   },
   {
@@ -453,7 +478,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "pynq_model = conifer.model.load_model('model_5/my_prj.json', new_config=pynq_model_cfg)\n",
+    "pynq_model = conifer.model.load_model('../hls4ml_prjs/conifer_prj_bdt_part6a/my_prj.json', new_config=pynq_model_cfg)\n",
     "pynq_model.write()"
    ]
   },
@@ -465,11 +490,11 @@
     "\n",
     "Now we run `build` again, running HLS Synthesis, Logic Synthesis and Place & Route, finally producing a bitfile and an archive of files that we'll need to run inference on the pynq-z2 board. \n",
     "\n",
-    "**Warning**: this step might take around 20 minutes to complete.\n",
+    "**This step takes around 20 minutes.**\n",
     "\n",
     "The floorplan of the bitfile should like something like this, where the individual tree modules are highlighted in different colours:\n",
     "\n",
-    "<img src=\"./images/part5_floorplan.png\" width=\"300\" />"
+    "<img src=\"../images/part5_floorplan.png\" width=\"300\" />"
    ]
   },
   {
@@ -488,9 +513,9 @@
     "## Inference on pynq-z2\n",
     "\n",
     "Running inference on the `pynq-z2` would look like this:\n",
-    "- download the `model_5/conifer_jettag.zip` archive from this notebook\n",
-    "- upload `conifer_jettag.zip` to the pynq-z2 device and unzip it\n",
-    "- start a jupyter notebook on the `pynq-z2` and run the following code:\n",
+    "- Download the `conifer_bdt_pynq/conifer_jettag.zip` archive from this notebook\n",
+    "- Upload `conifer_jettag.zip` to the pynq-z2 device and unzip it\n",
+    "- Start a jupyter notebook on the `pynq-z2` and run the following code:\n",
     "\n",
     "```\n",
     "import conifer\n",
@@ -503,7 +528,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
+   "display_name": "hls4ml-tutorial",
    "language": "python",
    "name": "python3"
   },
@@ -517,7 +542,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.10"
+   "version": "3.10.16"
   }
  },
  "nbformat": 4,