Merge branch 'main' into qonnx_binary_quant

Author: jmitrevs

.pre-commit-config.yaml

@@ -10,7 +10,7 @@ repos:
            '--skip-string-normalization']
 
 - repo: https://github.com/tox-dev/pyproject-fmt
-  rev: v2.5.1
+  rev: v2.6.0
   hooks:
     - id: pyproject-fmt
 
@@ -35,7 +35,7 @@ repos:
   - id: isort
 
 - repo: https://github.com/asottile/pyupgrade
-  rev: v3.19.1
+  rev: v3.20.0
   hooks:
   - id: pyupgrade
     args: ["--py310-plus"]

docs/frontend/keras.rst

@@ -1,11 +1,19 @@
-================
-Keras and QKeras
-================
+================================
+Keras and its quantized variants
+================================
 
-Keras and the quantization library QKeras are well supported in ``hls4ml``. Currently, the Keras v2 (``tf.keras``) is the preferred version, and the future versions of ``hls4ml`` will expand support for Keras v3. The frontend is based on the parsing the serialized json representation of the model.
+Keras and the quantization library QKeras are well supported in ``hls4ml``. Both Keras v2 (``tf.keras``) and the new Keras v3 are supported. While the Keras v2 support is based on parsing the serialized json representation of the model, the Keras v3 support uses direct model inspection.
 
-Currently, ``hls4ml`` can parse most Keras layers, including core layers, convolutional layers, pooling layers, recurrent layers, merging/reshaping layers and activation layers, implemented either via sequential or functional API. Notably missing are the attention and normalization layers. The equivalent QKeras API and quantizers are also supported. The ``Lambda`` layers don't save their state in the serialized format and are thus impossible to parse. In this case, the ``Lambda`` layers can be implemented as custom layers and parsed via the :ref:`Extension API`.
+Currently, ``hls4ml`` can parse most Keras layers, including core layers, convolutional layers, pooling layers, recurrent layers, merging/reshaping layers and activation layers, implemented either via sequential or functional API. Notably missing are the attention and normalization layers. The ``Lambda`` layers don't save their state in the serialized format and are thus impossible to parse. In this case, the ``Lambda`` layers can be implemented as custom layers and parsed via the :ref:`Extension API`.
 
 The ``data_format='channels_first'`` parameter of Keras layers is supported, but not extensively tested. All HLS implementations in ``hls4ml`` are based on ``channels_last`` data format and need to be converted to that format before the HLS code can be emitted. We encourage users of ``channels_first`` to report their experiences to developers on GitHub.
 
+
+* `QKeras <https://github.com/fastmachinelearning/qkeras>`_
+    The equivalent QKeras API and its quantizers are also supported by ``hls4ml``. QKeras is not compatible with Keras v3. Currently, only HGQ2 is compatible with Keras v3 (see below).
+* `HGQ <https://github.com/calad0i/HGQ>`_
+    The equivalent HGQ API is also supported. HGQ is not compatible with Keras v3. See `advanced/HGQ <../advanced/hgq.html>`__ for more information.
+* `HGQ2 <https://github.com/calad0i/HGQ2>`_
+    HGQ2 is based on Keras v3. Its support in hls4ml is currently under development.
+
 The development team of ``hls4ml`` is currently exploring options for QKeras alternative and will provide a drop-in replacement API compatible with Keras v3.

docs/intro/setup.rst

@@ -37,14 +37,26 @@ version can be installed directly from ``git``:
 Dependencies
 ============
 
-The ``hls4ml`` library requires python 3.10 or later, and depends on a number of Python packages and external tools for synthesis and simulation. Python dependencies are automatically managed
-by ``pip`` or ``conda``.
+.. note::
+   As of version 1.1.0+, all conversion frontend specific packages are optional. Only install the packages you need.
 
-* `TensorFlow <https://pypi.org/project/tensorflow/>`_ (version 2.8 to 2.14) and `QKeras <https://pypi.org/project/qkeras/>`_ are required by the Keras converter. One may want to install newer versions of QKeras from GitHub. Newer versions of TensorFlow can be used, but QKeras and hl4ml do not currently support Keras v3.
+The ``hls4ml`` library requires python 3.10 or later, and depends on a number of Python packages and external tools for synthesis and simulation. Python dependencies are automatically managed by ``pip`` or ``conda``.
+
+The following Python packages are all optional and are only required if you intend to use the corresponding converter.
+
+* `Keras <https://pypi.org/project/keras/>`_ is required by the Keras converter.
+   * `TensorFlow <https://pypi.org/project/tensorflow/>`_ (version 2.8 to 2.14) is required by the Keras v2 converter (keras v2 is included in TensorFlow).
+   * `Keras <https://pypi.org/project/keras/>` 3.0 or above is required by the Keras v3 converter. Keras v3 supports multiple backends for training and inference, and the conversion is not tied any specific backend. Notice that Keras v3 may **not** coexist with Keras v2 in the same Python environment.
 
 * `ONNX <https://pypi.org/project/onnx/>`_ (version 1.4.0 and newer) is required by the ONNX converter.
 
-* `PyTorch <https://pytorch.org/get-started>`_ package is optional. If not installed, the PyTorch converter will not be available.
+* `PyTorch <https://pytorch.org/get-started>`_ is required by the PyTorch converter.
+
+* Quantization support
+   * `QKeras <https://github.com/fastmachinelearning/qkeras>`_: based on Keras v2. See `frontend/keras <../frontend/keras.html>`_ for more details
+   * `HGQ <https://github.com/calad0i/HGQ>`_: Based on Keras v2. See `advanced/HGQ <../advanced/hgq.html>`_ for more details.
+   * `Brevitas <https://xilinx.github.io/brevitas/>`_: Based on PyTorch. See `frontend/pytorch <../frontend/pytorch.html>`_ for more details.
+   * `QONNX <https://github.com/fastmachinelearning/qonnx>`_: Based on ONNX. See `frontend/onnx <../frontend/onnx.html>`_ for more details.
 
 Running C simulation from Python requires a C++11-compatible compiler. On Linux, a GCC C++ compiler ``g++`` is required. Any version from a recent
 Linux should work. On MacOS, the *clang*-based ``g++`` is enough. For the oneAPI backend, one must have oneAPI installed, along with the FPGA compiler,

hls4ml/backends/fpga/fpga_backend.py

@@ -917,33 +917,6 @@ def generate_conv2d_line_buffer_fn(
 
         return generated_code
 
-    @staticmethod
-    def permute_config_gen(name: str, shape: tuple[int, ...], perm: tuple[int, ...]):
-        """
-        Generate new shape and perm_strides for a permute operation. Operates by mapping the output index
-        to input input index by:
-        - unravel the output index
-        - map each dimension to the corresponding stride in the input tensor, sum
-        The operation can be expressed as:
-
-        new_shape = tuple(shape[i] for i in perm)
-        strides = np.cumprod((shapes[1:] + (1,))[::-1])[::-1]
-        perm_strides = [strides[i] for i in perm]
-        out[index] = inp[np.dot(np.unravel_index(index, new_shape), perm_strides)]
-
-        Args:
-            name (str): The name of the configuration.
-            shape (tuple[int, ...]): The shape of the input tensor.
-            perm (tuple[int, ...]): The permutation of the dimensions.
-
-        Returns:
-            (new_shape, perm_strides) (tuple, tuple):  the output shape and permutation strides.
-        """
-        new_shape = tuple(shape[i] for i in perm)
-        strides = np.cumprod((shape[1:] + (1,))[::-1])[::-1]
-        perm_strides = tuple(int(strides[i]) for i in perm)
-        return (new_shape, perm_strides)
-
     @model_optimizer()
     def write_hls(self, model):
         self.writer.write_hls(model)

hls4ml/backends/fpga/passes/clone.py

@@ -79,9 +79,9 @@ def transform(self, model, node):
             n_outputs = len(output_map[output]) + in_output
             if n_outputs == 1:
                 continue
-            if n_outputs > 3:
+            if n_outputs > 7:
                 msg = f'ERROR: Cloning output {output} of {node.class_name}\
-                      ({node.name}) more than 3 times not currently supported'
+                      ({node.name}) more than 7 times not currently supported'
                 raise ValueError(msg)
 
             out_var = node.get_output_variable(output)

hls4ml/backends/oneapi/oneapi_backend.py

@@ -130,12 +130,15 @@ def get_default_flow(self):
     def get_writer_flow(self):
         return self._writer_flow
 
-    def create_initial_config(self, part='Arria10', clock_period=5, io_type='io_parallel', write_tar=False, **_):
+    def create_initial_config(
+        self, part='Agilex7', clock_period=5, hyperopt_handshake=False, io_type='io_parallel', write_tar=False, **_
+    ):
         """Create initial configuration of the oneAPI backend.
 
         Args:
-            part (str, optional): The FPGA part to be used. Defaults to 'Arria10'.
-            clock_period (int, optional): The clock period. Defaults to 5.
+            part (str, optional): The FPGA part to be used. Defaults to 'Agilex7'.
+            clock_period (int, optional): The clock period in ns. Defaults to 5.
+            hyperopt_handshake (bool, optional): Should hyper-optimized handshaking be used? Defaults to False
             io_type (str, optional): Type of implementation used. One of
                 'io_parallel' or 'io_stream'. Defaults to 'io_parallel'.
             write_tar (bool, optional): If True, compresses the output directory into a .tar.gz file. Defaults to False.
@@ -146,8 +149,9 @@ def create_initial_config(self, part='Arria10', clock_period=5, io_type='io_para
 
         config = {}
 
-        config['Part'] = part if part is not None else 'Arria10'
+        config['Part'] = part if part is not None else 'Agilex7'
         config['ClockPeriod'] = clock_period
+        config['HyperoptHandshake'] = hyperopt_handshake
         config['IOType'] = io_type
         config['HLSConfig'] = {}
         config['WriterConfig'] = {
@@ -167,7 +171,7 @@ def compile(self, model):
             Exception: If the project failed to compile
 
         Returns:
-            string: Returns the name of the compiled library.
+            Path: Returns the name of the compiled library.
         """
         outdir = Path(Path.cwd(), model.config.get_output_dir())
         builddir = outdir / 'build'

hls4ml/backends/oneapi/passes/reshaping_templates.py

@@ -3,6 +3,7 @@
 from hls4ml.backends.oneapi.oneapi_template import StreamFunctionCallTemplate, TaskSequenceTemplate
 from hls4ml.backends.template import FunctionCallTemplate, LayerConfigTemplate
 from hls4ml.model.layers import Reshape, Resize, Transpose, ZeroPadding1D, ZeroPadding2D
+from hls4ml.utils.transpose_utils import transpose_config_gen
 
 # ZeroPadding templates
 
@@ -185,16 +186,8 @@ def format(self, node):
         perm = tuple(node.get_attr('perm'))
         name = f'config{node.index}'
 
-        new_shape, perm_strides = node.model.config.backend.permute_config_gen(name, shape, perm)
-        return transpose_config_template.format(
-            dims=len(shape),
-            N=int(np.prod(shape)),
-            from_shape=', '.join(str(x) for x in shape),
-            perm=', '.join(str(x) for x in perm),
-            perm_strides=', '.join(str(x) for x in perm_strides),
-            to_shape=', '.join(str(x) for x in new_shape),
-            config_name=name,
-        )
+        conf = transpose_config_gen(name, shape, perm)
+        return transpose_config_template.format(**conf)
 
 
 class TransposeFunctionTemplate(FunctionCallTemplate):

hls4ml/backends/vivado/passes/einsum.py

@@ -0,0 +1,109 @@
+from math import ceil
+
+from hls4ml.backends.backend import get_backend
+from hls4ml.backends.template import FunctionCallTemplate, LayerConfigTemplate
+from hls4ml.model.layers import Einsum
+from hls4ml.utils.transpose_utils import transpose_config_gen
+
+from .reshaping_templates import transpose_config_template
+
+# Shared Dense template
+# Einsum template
+
+einsum_config_template = '''
+struct config{index} {{
+    typedef config{index}_tpose_inp0 tpose_inp0_config;
+    typedef config{index}_tpose_inp1 tpose_inp1_config;
+    typedef config{index}_tpose_out tpose_out_conf;
+
+    typedef {accum_t.name} accum_t;
+
+    // Layer Sizes
+    static const unsigned n_free0 = {n_free0};
+    static const unsigned n_free1 = {n_free1};
+    static const unsigned n_contract = {n_contract};
+    static const unsigned n_inplace = {n_inplace};
+
+    // Resource reuse info
+    static const unsigned io_type = nnet::{iotype};
+    static const unsigned strategy = nnet::{strategy};
+    static const unsigned reuse_factor = {reuse_factor};
+    static const unsigned multiplier_limit = {multiplier_limit};
+    static const bool store_weights_in_bram = false; // NOT USED
+
+    template <class x_T, class y_T>
+    using product = nnet::product::{product_type}<x_T, y_T>;
+}};
+'''
+
+einsum_function_template = 'nnet::einsum<{input0_t}, {input1_t}, {output_t}, {config}>({input0}, {input1}, {output});'
+
+einsum_include_list = ['nnet_utils/nnet_einsum.h']
+
+
+class EinsumConfigTemplate(LayerConfigTemplate):
+    def __init__(self):
+        super().__init__(Einsum)
+        self.template = einsum_config_template
+
+    def format(self, node: Einsum):
+        default_params = self._default_config_params(node)
+
+        strategy = node.attributes['strategy']
+        io_type = node.model.config.get_config_value('IOType')
+
+        assert io_type == 'io_parallel', 'EinsumDense layer only supports io_parallel for now'
+        assert strategy.lower() == 'latency', 'EinsumDense layer only supports Latency strategy for now'
+
+        # EinsumDense config
+        params = default_params.copy()
+        params['strategy'] = strategy
+        params['n_free0'] = node.attributes['n_free0']
+        params['n_free1'] = node.attributes['n_free1']
+        params['n_contract'] = node.attributes['n_contract']
+        params['n_inplace'] = node.attributes['n_inplace']
+        inp0_t = node.get_input_variable(node.inputs[0]).type.precision
+        inp1_t = node.get_input_variable(node.inputs[1]).type.precision
+        params['product_type'] = get_backend('vivado').product_type(inp0_t, inp1_t)
+
+        total_mults = params['n_free0'] * params['n_free1'] * params['n_contract'] * params['n_inplace']
+        params['multiplier_limit'] = ceil(total_mults / params['reuse_factor'])
+
+        einsum_conf = self.template.format(**params)
+
+        # inp/out transpose config
+        inp0_shape = node.attributes['inp0_shape']
+        inp1_shape = node.attributes['inp1_shape']
+        out_interpert_shape = node.attributes['out_interpert_shape']
+        inp0_tpose_idxs = node.attributes['inp0_tpose_idxs']
+        inp1_tpose_idxs = node.attributes['inp1_tpose_idxs']
+        out_tpose_idxs = node.attributes['out_tpose_idxs']
+        tpose_inp0_config_name = f'config{node.index}_tpose_inp0'
+        tpose_inp1_config_name = f'config{node.index}_tpose_inp1'
+        tpose_out_conf_name = f'config{node.index}_tpose_out'
+
+        conf = transpose_config_gen(tpose_inp0_config_name, inp0_shape, inp0_tpose_idxs)
+        inp0_tpose_conf = transpose_config_template.format(**conf)
+        conf = transpose_config_gen(tpose_inp1_config_name, inp1_shape, inp1_tpose_idxs)
+        inp1_tpose_conf = transpose_config_template.format(**conf)
+        conf = transpose_config_gen(tpose_out_conf_name, out_interpert_shape, out_tpose_idxs)
+        out_tpose_conf = transpose_config_template.format(**conf)
+
+        return '\n\n'.join((inp0_tpose_conf, inp1_tpose_conf, out_tpose_conf, einsum_conf))
+
+
+class EinsumFunctionTemplate(FunctionCallTemplate):
+    def __init__(self):
+        super().__init__(Einsum, include_header=einsum_include_list)
+        self.template = einsum_function_template
+
+    def format(self, node: Einsum):
+        params = {}
+        params['config'] = f'config{node.index}'
+        params['input0_t'] = node.get_input_variable(node.inputs[0]).type.name
+        params['input1_t'] = node.get_input_variable(node.inputs[1]).type.name
+        params['output_t'] = node.get_output_variable().type.name
+        params['input0'] = node.get_input_variable(node.inputs[0]).name
+        params['input1'] = node.get_input_variable(node.inputs[1]).name
+        params['output'] = node.get_output_variable().name
+        return self.template.format(**params)