feat: add sparse CNN support (sparsepixels)

[hftsoi]

Dear experts, this PR is to add feature support for sparse CNN models built from the sparsepixels library, which is the code implementation of the paper arXiv:2512.06208. Issue is opened at #<1467>. Please see below the descriptions. Thanks.

Description

Add end-to-end conversion of sparse CNN models, built with the sparsepixels Keras package (HGQ2-based), to HLS firmware via the Vivado and Vitis backends. Sparse CNNs propagate only active pixels (those above a threshold) plus their spatial hash coordinates through the network instead of operating on the full image grid, drastically reducing FPGA latency and resource usage compared to dense convolution on naturally sparse inputs (e.g., particle detector data, neutrino events, sparse imaging).

What's added:

• Five new layer classes in hls4ml/model/layers.py: SparseInputReduce, SparseConv2D, SparseActivation, SparsePooling2D, SparseFlatten
• Keras v3 handlers for the sparse-pixels layers (InputReduce, QConv2DSparse, AveragePooling2DSparse)
• Vivado backend: graph optimizer, input-precision fix pass, and config/function templates for each sparse layer
• New HLS kernel header nnet_sparsepixels.h
• Bit-exact precision dispatches for all 5 sparse layer types

Patch is fully self-contained. Every new pass and dispatch is gated on the new layer types via isinstance, so no existing model conversion path is touched.

Backends: Vivado, Vitis, io_parallel. Other backends and io_stream are out of scope at the moment.

Dependency: sparsepixels (optional for pytest only).

Type of change

• New feature (non-breaking change which adds functionality)
• A new research paper code implementation

Tests

test/pytest/test_sparsepixels.py builds a small sparse CNN, converts it via hls4ml, compiles, and compares HLS vs Keras outputs over random inputs on both Vivado and Vitis backends.

Test Configuration:

Checklist

• I have read the guidelines for contributing.
• I have commented my code, particularly in hard-to-understand areas.
• I have made corresponding changes to the documentation.
• My changes generate no new warnings.
• I have installed and run pre-commit on the files I edited or added.
• I have added tests that prove my fix is effective or that my feature works.

Happy to add to docs as a separate PR.

[hftsoi]

Adding how sparse CNN flows in HLS

Every sparse layer operates on two parallel arrays that travel together through the network:

• sparse_arr_feat[N_sparse * C]: the feature values, one per active pixel per channel
• sparse_arr_hash[N_sparse * 2]: the (h, w) spatial coordinates (1-based) of each active pixel

Keeping the hash alongside the features is what makes sparse convolution possible without materializing the full H * W grid. Layer walkthrough:

1. SparseInputReduce: takes the dense input x_in[H * W * C], scans the first channel for pixels above threshold, and emits the top N_sparse active pixels (leftmost-first, via a binary-tree reduction). Outputs the two arrays above for the rest of the network. All channels of each selected pixel are preserved.

2. SparseConv2D: for every output pixel, iterates over all N_sparse input pixels and uses the difference of their hash coordinates to index into the kernel. Pairs whose offset lies outside the kernel radius contribute 0 and are skipped. Each output pixel's hash is identical to its input hash, so sparsity pattern is preserved across the conv layer.

3. SparseActivation: element-wise activation on the sparse feature array. The hash array is passed through untouched.

4. SparsePooling2D: pooling implemented on the sparse representation. Each input hash gets its new pooled cell. Pixels mapping to the same pooled cell have the pooling operation applied, and the duplicates are zeroed. A new hash array is emitted for the pooled layout.

5. SparseFlatten: the sparse->dense transition. The sparse representation is scattered back to a dense H_out * W_out * C_out grid using the final hash coordinates, then flattened. From this point on, the network runs on a normal dense tensor, so any standard Dense/Activation/Softmax layer can follow.

[JanFSchulte]

Just to double check: Is it correct to drop the bias if the MAC happens to sum to zero incidentally?

[hftsoi]

Thanks for flagging this, it is an intended design choice in this first version. The sparse arrays are fixed length (n_max_pixels slots) and currently carry no explicit per-slot validity flag, so an invalid (inactive) pixel is encoded as exact zero in the feature values. With this representation, the kernel uses the non-zero acc as a guard to avoid adding bias to inactive pixels, so an inactive pixel always stays inactive in sparse convolution. The corner case you mentioned is indeed possible but is rare practically, and we have made the training library to exactly mirror the HLS side so there is no mismatch. A possible improvement on this is to add a separate validity flag per slot to guard the acc, which requires some tests and could be done in future.

[JanFSchulte]

Thanks for the clarification. That's all good then.

[JanFSchulte]

So I guess currently this test will never be run? I think it should be added to the keras 3 test list here https://github.com/fastmachinelearning/hls4ml/blob/main/test/pytest/generate_ci_yaml.py#L38 and add the sparsepixels package to the ones installed for these tests in https://github.com/fastmachinelearning/hls4ml/blob/main/pyproject.toml#L72

[hftsoi]

Added now

[hftsoi]

I think the ci pytests failed with the unrelated pquant_keras, ours has passed?

[JanFSchulte]

Indeed, just a time-out. Test succeeded on the second try. Looks good to me, let others have a chance to chime in, otherwise I'll merge tomorrow morning.

[vloncar]

I was hoping that this can be refactored not to be here, rather to be a separate pass later on, but didn't investigate the feasibility. What's the blocker for moving it to a "convert" flow on the parsed model?

[hftsoi]

Now refactored that to the convert flow. Tested locally, the conversion stays the same.