Support for Transformer / Attention (QKV) layers and torch.matmul in hls4ml, and resource optimization strategies?

[hub-sszhang]

Hi hls4ml team,

I am working on deploying a small transformer-like model (self-attention + MLP) to FPGA using hls4ml.

My model includes:

• Linear layers for Q, K, V projections
• Attention computation using torch.matmul (QK^T and attention * V)
• Softmax over attention scores

I understand that hls4ml mainly supports Dense and convolutional layers, so I would like to ask:

1. Does hls4ml support general matrix multiplications (e.g., torch.matmul) outside of Dense layers?
2. Is there any current or planned support for attention/transformer-style layers?

Additionally, I am trying to manually implement the attention part in Vitis HLS.

Currently, I have already applied:

• #pragma HLS PIPELINE
• #pragma HLS ARRAY_PARTITION

However, resource usage (DSP/LUT) is still quite high.

So I would also like to ask:

4. Are there recommended strategies to further optimize resource usage for matrix multiplications in HLS (e.g., QK^T and attention × V)?

5. Are there best practices for implementing or approximating softmax efficiently in FPGA (for low latency and low resource usage)?

Both latency and resource usage are critical for my study.

Any suggestions or references would be greatly appreciated.

Thanks a lot for your help!