Time multiplexing multipliers and adders to reduce resource usage

[gaborszita]

@fdxmw As I said back in December, floating point multipliers and adders are so resource-heavy that when I attempt to simulate neural network training, it is extremely slow and consumes a massive amount of RAM. Hence, time-multiplexing multipliers and adders is necessary to lower resource usage.

I cooked up time-multiplexing multipliers with Claude, and overall the design seems good to me, so I was wondering if you could check if the general idea looks good. Of course, this is not final, and there are multiple changes I would like to add, like time-multiplexing adders too. I also don't like the way Claude handled the MemBlock inputs, so just disregard the _make_systolic_array_memblock_inputs and num_systolic_array_cycles functions for now, I'll come up with a solution for the MemBlock async problem later. I was just wondering if you could skim it and let me know whether you think the general idea is good.

[fdxmw]

Sorry for the slow reply! There's a lot to think about here. Some ideas:

• If you want to save area, the elementwise adders are the lowest hanging fruit. These elementwise additions currently run fully parallel, so we could save quite a bit of space by running them serially. Serializing these additions should be pretty straightforward.

• For the systolic array, I think there are two ways we could approach this problem, with various tradeoffs:

  1. Multiplexing PEs over time to save space, which seems to be the approach in this PR. This design is pretty straightforward, but the benefits are limited as tensors scale up, since we still need all the registers and wiring to feed all the the input data to the systolic array, and store the full product matrix in accumulator registers.
  2. We could implement traditional block-matrix multiplication, which internally uses a lower-level ordinary systolic array for matrix multiplication, which has a nice hierarchical structure. But this requires a lot more control logic, since the higher-level block-matrix multiplier needs to orchestrate the inputs for the lower-level systolic array (keep track of which blocks we're currently multiplying). This is a more complicated design, but it should use a lot less registers and wires, especially as tensors scale up, since we don't need registers for all the inputs or the full product matrix (only need accumulator registers for the smaller block products; we can store the full product matrix in a MemBlock).

I'm totally fine with (i) for the short-term, but (ii) does seem like the right long-term solution to me, especially if people want to run larger neural networks with pyrtlnet. Let me know if you see other options, and how you plan to proceed.

Regardless of which approach we take, there are a couple details to keep in mind:

• It'd be great to support non-square block shapes, since pyrtlnet's tensors are often very not-square. The first layer's output has shape (batch_size, 18), and batch_size is 1 by default, which wouldn't fit nicely into a R×R block of PEs.

• Padding inputs should be sufficient; we shouldn't need to pad the systolic array itself. I think this comment in the PR points out the issue:

  The padded positions get zero inputs and their accumulator outputs are dropped before returning.

  The hardware for these padded positions sounds pretty useless, so we should try to remove it :)