From 7ff2ef221eea233a4ccacd0f4e5cd33a24c3ef06 Mon Sep 17 00:00:00 2001 From: Arnaud de Grandmaison Date: Mon, 30 Jun 2025 17:44:22 +0200 Subject: [PATCH] Add support for devices with SME2 hardware. --- .../1-get-started.md | 321 +++++++++----- .../10-going-further.md | 73 ++++ .../2-check-your-environment.md | 292 ++++++++----- .../3-streaming-mode.md | 78 ++++ .../4-outer-product.md | 108 ----- ...-vanilla-matmul.md => 4-vanilla-matmul.md} | 54 ++- .../5-outer-product.md | 122 ++++++ .../5-sme2-matmul-asm.md | 202 --------- .../6-sme2-matmul-asm.md | 412 ++++++++++++++++++ .../6-sme2-matmul-intr.md | 344 --------------- .../7-sme2-matmul-intr.md | 307 +++++++++++++ .../8-benchmarking.md | 94 ++++ .../8-going-further.md | 51 --- .../{7-debugging.md => 9-debugging.md} | 49 ++- .../multiplying-matrices-with-sme2/_index.md | 28 +- .../overview.md | 16 +- 16 files changed, 1556 insertions(+), 995 deletions(-) create mode 100644 content/learning-paths/cross-platform/multiplying-matrices-with-sme2/10-going-further.md create mode 100644 content/learning-paths/cross-platform/multiplying-matrices-with-sme2/3-streaming-mode.md delete mode 100644 content/learning-paths/cross-platform/multiplying-matrices-with-sme2/4-outer-product.md rename content/learning-paths/cross-platform/multiplying-matrices-with-sme2/{3-vanilla-matmul.md => 4-vanilla-matmul.md} (59%) create mode 100644 content/learning-paths/cross-platform/multiplying-matrices-with-sme2/5-outer-product.md delete mode 100644 content/learning-paths/cross-platform/multiplying-matrices-with-sme2/5-sme2-matmul-asm.md create mode 100644 content/learning-paths/cross-platform/multiplying-matrices-with-sme2/6-sme2-matmul-asm.md delete mode 100644 content/learning-paths/cross-platform/multiplying-matrices-with-sme2/6-sme2-matmul-intr.md create mode 100644 content/learning-paths/cross-platform/multiplying-matrices-with-sme2/7-sme2-matmul-intr.md create mode 100644 content/learning-paths/cross-platform/multiplying-matrices-with-sme2/8-benchmarking.md delete mode 100644 content/learning-paths/cross-platform/multiplying-matrices-with-sme2/8-going-further.md rename content/learning-paths/cross-platform/multiplying-matrices-with-sme2/{7-debugging.md => 9-debugging.md} (78%) diff --git a/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/1-get-started.md b/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/1-get-started.md index 932afa453c..562ba90582 100644 --- a/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/1-get-started.md +++ b/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/1-get-started.md @@ -6,66 +6,159 @@ weight: 3 layout: learningpathall --- -## Installing software for this Learning Path +To follow this Learning Path, you will need to set up an environment to develop +with SME2 and download the code examples. This learning path assumes two +different ways of working, and you will need to select the one appropriate for +your machine: +- Case #1: Your machine has native SME2 support --- check the [list of devices + with SME2 support](#devices-with-sme2-support). +- Case #2: Your machine does not have native SME2 support. This learning path + supports this use case by enabling you to run code with SME2 instructions in + an emulator in bare metal mode, i.e., the emulator runs the SME2 code + *without* an operating system. + +## Code examples + +[Download the code examples](https://gitlab.arm.com/learning-code-examples/code-examples/-/archive/main/code-examples-main.tar.gz?path=learning-paths/cross-platform/multiplying-matrices-with-sme2) for this learning path, expand the archive, and change your current directory to: +``code-examples/learning-paths/cross-platform/multiplying-matrices-with-sme2`` : -To follow this Learning Path, you will need to set up an environment to develop with SME2. +```BASH +tar xfz code-examples-main-learning-paths-cross-platform-multiplying-matrices-with-sme2.tar.gz -s /code-examples-main-learning-paths-cross-platform-multiplying-matrices-with-sme2/code-examples/ +cd code-examples/learning-paths/cross-platform/multiplying-matrices-with-sme2 +``` -You will require: +The listing of the content of this directory should look like this: - - A compiler with support for SME2 instructions. You can use [Clang](https://www.llvm.org/) - version 18 or later, or [GCC](https://gcc.gnu.org/) version 14, or later. This Learning - Path uses ``Clang``. +```TXT +code-examples/learning-paths/cross-platform/multiplying-matrices-with-sme2/ +├── .clang-format +├── .devcontainer/ +│ └── devcontainer.json +├── .git/ +├── .gitignore +├── Makefile +├── README.rst +├── docker/ +│ ├── assets.source_me +│ ├── build-all-containers.sh +│ ├── build-my-container.sh +│   └── sme2-environment.docker +├── hello.c +├── main.c +├── matmul.h +├── matmul_asm.c +├── matmul_asm_impl.S +├── matmul_intr.c +├── matmul_vanilla.c +├── misc.c +├── misc.h +├── preprocess_l_asm.S +├── preprocess_vanilla.c +├── run-fvp.sh +└── sme2_check.c +``` - - An emulator to execute code with the SME2 instructions. This Learning - Path uses [Arm's Fixed Virtual Platform (FVP) model](https://developer.arm.com/Tools%20and%20Software/Fixed%20Virtual%20Platforms). +It contains: +- The code examples that will be used throughout this learning path. +- A ``Makefile`` that builds the code examples. +- A shell script called ``run-fvp.sh`` that runs the FVP (used in the emulated + SME2 case). +- A directory called ``docker`` that contains materials related to Docker, which + are: + - A script called ``assets.source_me`` that provides the FVP and compiler + toolchain references. + - A Docker recipe called ``sme2-environment.docker`` to build the container + that you will use. + - A shell script called ``build-my-container.sh`` that you can use if you want + to build the Docker container. This is not essential; ready-made images are + available for you. + - A script called ``build-all-containers.sh`` that was used to create the + image for you to download to provide multi-architecture support for both + x86_64 and AArch64. +- A configuration script for VS Code to be able to use the container from the + IDE called ``.devcontainer/devcontainer.json``. -You will also require Git and Docker installed on your machine. +{{% notice Note %}} +From this point in the Learning Path, all instructions assume that your current +directory is +``code-examples/learning-paths/cross-platform/multiplying-matrices-with-sme2``. +{{% /notice %}} -### Set up Git +## Platforms with native SME2 support -To check if Git is already installed on your machine, use the following command line in a terminal: +If your machine has native support for SME2, then you only need to ensure that +you have a compiler with support for SME2 instructions. -```BASH { output_lines=2 } -git --version -git version 2.47.1 -``` +A recent enough version of the compiler is required because SME2 is a recent +addition to the Arm instruction set. Compiler versions that are too old will +have incomplete or no SME2 support, leading to compilation errors or +non-functional code. You can use [Clang](https://www.llvm.org/) version 18 or +later, or [GCC](https://gcc.gnu.org/) version 14 or later. This Learning Path +uses ``clang``. -If the above command line fails with a message similar to "``git: command not found``", then install Git following the steps for your machine's OS. +At the time of writing, the ``clang`` version shipped with macOS is ``17.0.0``, +which forces us to use the version from ``homebrew`` (which has version +``20.1.7``). Ensure the ``clang`` compiler you are using is recent enough with +``clang --version``: {{< tabpane code=true >}} + {{< tab header="Linux/Ubuntu" language="bash">}} -sudo apt install git + sudo apt install clang {{< /tab >}} + {{< tab header="macOS" language="bash">}} -brew install git + brew install clang {{< /tab >}} + {{< /tabpane >}} -### Docker +You are now all set to start hacking with SME2! -To enable you to get started easily and with the tools that you need, you can fetch a Docker container with the required compiler and FVP. Alternatively, if you do wish to build the container yourself, the ``Dockerfile`` is also available. +## Platforms with emulated SME2 support +If your machine does not have SME2 support or if you want to run SME2 with an +emulator, you will need to install Docker. Docker containers provide +functionality to execute commands in an isolated environment, where you have all +the necessary tools you require without cluttering your machine. The containers +run independently, meaning they do not interfere with other containers on the +same machine or server. + +This learning path provides a Docker image that has a compiler and [Arm's Fixed +Virtual Platform (FVP) +model](https://developer.arm.com/Tools%20and%20Software/Fixed%20Virtual%20Platforms) +for emulating code with SME2 instructions. The Docker image recipe is provided +(with the code examples) so you can study it and build it yourself. You could +also decide not to use the Docker image and follow the +``sme2-environment.docker`` Docker file instructions to install the tools on +your machine. + +### Docker {{% notice Note %}} -This Learning Path works without ``docker``, but the compiler and the FVP must be available in your search path. +This Learning Path works without ``docker``, but the compiler and the FVP must +be available in your search path. {{% /notice %}} -Start by checking that ``docker`` is installed on your machine by typing the following -command line in a terminal: +Start by checking that ``docker`` is installed on your machine by typing the +following command line in a terminal: ```BASH { output_lines="2" } docker --version Docker version 27.3.1, build ce12230 ``` -If the above command fails with a message similar to "``docker: command not found``" -then follow the steps from the [Docker Install Guide](https://learn.arm.com/install-guides/docker/). +If the above command fails with a message similar to "``docker: command not +found``" then follow the steps from the [Docker Install +Guide](https://learn.arm.com/install-guides/docker/). {{% notice Note %}} -You might need to login again or restart your machine for the changes to take effect. +You might need to log in again or restart your machine for the changes to take +effect. {{% /notice %}} -Once you have confirmed that Docker is installed on your machine, you can check that it is operating normally with the following: +Once you have confirmed that Docker is installed on your machine, you can check +that it is operating normally with the following: ```BASH { output_lines="2-27" } docker run hello-world @@ -79,131 +172,125 @@ Hello from Docker! This message shows that your installation appears to be working correctly. To generate this message, Docker followed these steps: - - 1. The Docker client contacted the Docker daemon. - - 2. The Docker daemon pulled the "hello-world" image from the Docker Hub. + +1. The Docker client contacted the Docker daemon. + +2. The Docker daemon pulled the "hello-world" image from the Docker Hub. + (arm64v8) - - 3. The Docker daemon created a new container from that image which runs the + +3. The Docker daemon created a new container from that image which runs the + executable that produces the output you are currently reading. - - 4. The Docker daemon streamed that output to the Docker client, which sent it + +4. The Docker daemon streamed that output to the Docker client, which sent it + to your terminal. To try something more ambitious, you can run an Ubuntu container with: - $ docker run -it ubuntu bash + +$ docker run -it ubuntu bash Share images, automate workflows, and more with a free Docker ID: - https://hub.docker.com/ + +https://hub.docker.com/ For more examples and ideas, visit: - https://docs.docker.com/get-started/ + +https://docs.docker.com/get-started/ ``` -## Environment +You can use Docker in the following ways: +- Directly from the command line. For example, when you are working from a + terminal on your local machine. +- Within a containerized environment. Configure VS Code to execute all the + commands inside a Docker container, allowing you to work seamlessly within the + Docker environment. -Now, [download the code examples](https://gitlab.arm.com/learning-code-examples/code-examples/-/archive/main/code-examples-main.tar.gz?path=learning-paths/cross-platform/multiplying-matrices-with-sme2) -for this learning path, expand the archive and change your current directory to -``code-examples/learning-paths/cross-platform/sme2`` : +### Working with Docker from a terminal -```BASH -tar xfz code-examples-main-learning-paths-cross-platform-multiplying-matrices-with-sme2.tar.gz -s /code-examples-main-learning-paths-cross-platform-multiplying-matrices-with-sme2/code-examples/ -cd code-examples/learning-paths/cross-platform/multiplying-matrices-with-sme2 -``` +When a command is executed in the Docker container environment, you must prepend +it with instructions on the command line so that your shell executes it within +the container. -This list of content in this directory should look like this : +For example, to execute ``COMMAND ARGUMENTS`` in the SME2 Docker container, the +command line looks like this: -```TXT -code-examples/learning-paths/cross-platform/multiplying-matrices-with-sme2/ -├── .clang-format -├── .devcontainer/ -│ └── devcontainer.json -├── .git/ -├── .gitignore -├── Makefile -├── README.rst -├── docker/ -│ ├── assets.source_me -│ ├── build-all-containers.sh -│ ├── build-my-container.sh -│   └── sme2-environment.docker -├── hello.c -├── main.c -├── matmul.h -├── matmul_asm.c -├── matmul_asm_impl.S -├── matmul_intr.c -├── matmul_vanilla.c -├── misc.c -├── misc.h -├── preprocess_l_asm.S -├── preprocess_vanilla.c -├── run-fvp.sh -└── sme2_check.c +```BASH +docker run --rm -v "$PWD:/work" -w /work armswdev/sme2-learning-path:sme2-environment-v2 COMMAND ARGUMENTS ``` -It contains: -- Code examples. -- A ``Makefile`` that builds the code examples. -- A shell script called ``run-fvp.sh`` that runs the FVP. -- A directory called ``docker`` that contains materials related to Docker, which are: - - A script called ``assets.source_me`` that provides the FVP and compiler toolchain references. - - A Docker recipe called ``sme2-environment.docker`` to build the container that - you will use. - - A shell script called ``build-my-container.sh`` that you can use if you want to build the Docker container. This is not essential however, as ready-made images are made available for you. - - A script called ``build-all-containers.sh`` that was used to create the image for you to download to provide multi-architecture support for both x86_64 and AArch64. -- A configuration script for VS Code to be able to use the container from the IDE called ``.devcontainer/devcontainer.json``. - -{{% notice Note %}} -From this point in the Learning Path, all instructions assume that your current -directory is ``code-examples/learning-paths/cross-platform/multiplying-matrices-with-sme2``.{{% /notice %}} +This invokes Docker, using the +``armswdev/sme2-learning-path:sme2-environment-v2`` container image, and mounts +the current working directory (the +``code-examples/learning-paths/cross-platform/multiplying-matrices-with-sme2``) +inside the container to ``/work``, then sets ``/work`` as the working directory +and runs ``COMMAND ARGUMENTS`` in this environment. +For example, to run ``make``, you need to enter: -## Using the environment +```BASH +docker run --rm -v "$PWD:/work" -w /work armswdev/sme2-learning-path:sme2-environment-v2 make +``` -Docker containers provide you with the functionality to execute commands in an isolated environment, where you have all the necessary tools that you require without having to clutter your machine. The containers runs independently, which means that they do not interfere with other containers on the same machine or server. +### Working within the Docker container from the terminal -You can use Docker in the following ways: -- Directly from the command line. For example, when you are working from a terminal on your local machine. -- Within a containerized environment. Configure VS Code to execute run all the commands inside a Docker container, allowing you to work seamlessly within the Docker environment. +The above commands are long and error-prone, so you can instead choose to work +interactively within the terminal, which would save you from prepending the +``docker run ...`` magic before each command you want to execute. To work in +this mode, run Docker without any command (note the ``-it`` command line +argument to the Docker invocation): -### Working from a terminal - -When a command is executed in the Docker container environment, you must prepend it with instructions on the command line so that your shell executes it within the container. +```BASH +docker run --rm -it -v "$PWD:/work" -w /work armswdev/sme2-learning-path:sme2-environment-v2 +``` -For example, to execute ``COMMAND ARGUMENTS`` in the SME2 Docker container, the command line looks like this: +You are now in the Docker container; you can execute all commands directly. For +example, the ``make`` command can now be simply invoked with: -```SH -docker run --rm -v "$PWD:/work" -w /work armswdev/sme2-learning-path:sme2-environment-v1 COMMAND ARGUMENTS +```BASH +make ``` -This invokes Docker, using the -``armswdev/sme2-learning-path:sme2-environment-v1``container -image, and mounts the current working directory (the ``code-examples/learning-paths/cross-platform/multiplying-matrices-with-sme2``) -inside the container to ``/work``, then sets ``/work`` as the -working directory and runs ``COMMAND ARGUMENTS`` in this environment. +To exit the container, simply hit CTRL+D. Note that the container is not +persistent (it was invoked with ``--rm``), so each invocation will use a +container freshly built from the image. All the files reside outside the +container, so changes you make to them will be persistent. -For example, to run ``make``, you need to enter: +### Working within the Docker container with VSCode -```SH -docker run --rm -v "$PWD:/work" -w /work armswdev/sme2-learning-path:sme2-environment-v1 make -``` +If you are using Visual Studio Code as your IDE, it can use the container as is. -### Working from within the Docker container +Make sure you have the [Microsoft Dev Containers](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) extension installed. -Make sure you have the [Microsoft Dev -Containers](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) -extension installed. - -Then select the **Reopen in Container** menu entry as Figure 1 shows. +Then select the **Reopen in Container** menu entry as Figure 1 shows. It automatically finds and uses ``.devcontainer/devcontainer.json``: ![example image alt-text#center](VSCode.png "Figure 1: Setting up the Docker Container.") -All your commands now run within the container, so there is no need to prepend them with a Docker invocation, as VS Code handles all this seamlessly for you. +All your commands now run within the container, so there is no need to prepend +them with a Docker invocation, as VS Code handles all this seamlessly for you. {{% notice Note %}} -For the rest of this Learning Path, shell commands include the full Docker invocation so that users not using VS Code can copy the complete command line. However, if you are using VS Code, you only need to use the `COMMAND ARGUMENTS` part. +For the rest of this Learning Path, shell commands include the full Docker +invocation so that users not using VS Code can copy the complete command line. +However, if you are using VS Code, you only need to use the `COMMAND ARGUMENTS` +part. {{% /notice %}} + +### Devices with SME2 support + +By chip: + +| Manufacturer | Chip | Devices | +|--------------|--------|---------| +| Apple | M4 | iPad Pro 11" & 13", iMac, Mac mini, MacBook Air 13" & 15"| +| Apple | M4 Pro | Mac mini, MacBook Pro 14" & 16" | +| Apple | M4 Max | MacBook Pro 14" & 16", Mac Studio | + +By product: + +| Manufacturer | Product family | Models | +|--------------|----------------|--------| +| Apple | iPhone 16 | iPhone 16, iPhone 16 Plus, iPhone 16e, iPhone 16 Pro, iPhone 16 Pro Max | \ No newline at end of file diff --git a/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/10-going-further.md b/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/10-going-further.md new file mode 100644 index 0000000000..441c3be98f --- /dev/null +++ b/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/10-going-further.md @@ -0,0 +1,73 @@ +--- +title: Going further +weight: 12 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +In this section, you will learn about the many different optimizations that are +available to you. + +## Generalize the algorithms + +In this Learning Path, you focused on using SME2 for matrix multiplication with +floating point numbers. However in practice, any library or framework supporting +matrix multiplication should also handle various integer types. + +You can see that the algorithm structure for matrix preprocessing as well as +multiplication with the outer product does not change at all for other data +types - they only need to be adapted. + +This is suitable for languages with [generic +programming](https://en.wikipedia.org/wiki/Generic_programming) like C++ with +templates. You can even make the template manage a case where the value +accumulated during the product uses a larger type than the input matrices. SME2 +has the instructions to deal efficiently with this common case scenario. + +This enables the library developer to focus on the algorithm, testing, and +optimizations, while allowing the compiler to generate multiple variants. + +## Unroll further + +You might have noticed that ``matmul_intr_impl`` computes only one tile at a +time, for the sake of simplicity. + +SME2 does support multi-vector instructions, and some were used in +``preprocess_l_intr``, for example, ``svld1_x2``. + +Loading two vectors at a time enables the simultaneous computing of more tiles, +and as the input matrices have been laid out in memory in a neat way, the +consecutive loading of the data is efficient. Implementing this approach can +make improvements to the ``macc`` to load ``ratio``. + +In order to check your understanding of SME2, you can try to implement this +unrolling yourself in the intrinsic version (the asm version already has this +optimization). You can check your work by comparing your results to the expected +reference values. + +## Apply strategies + +One method for optimization is to use strategies that are flexible depending on +the matrices' dimensions. This is especially easy to set up when working in C or +C++, rather than directly in assembly language. + +By playing with the mathematical properties of matrix multiplication and the +outer product, it is possible to minimize data movement as well as reduce the +overall number of operations to perform. + +For example, it is common that one of the matrices is actually a vector, meaning +that it has a single row or column, and then it becomes advantageous to +transpose it. Can you see why? + +The answer is that as the elements are stored contiguously in memory, an ``Nx1`` +and ``1xN`` matrices have the exact same memory layout. The transposition +becomes a no-op, and the matrix elements stay in the same place in memory. + +An even more *degenerated* case that is easy to manage is when one of the +matrices is essentially a scalar, which means that it is a matrix with one row +and one column. + +Although our current code handles it correctly from a results point of view, a +different algorithm and use of instructions might be more efficient. Can you +think of another way? diff --git a/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/2-check-your-environment.md b/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/2-check-your-environment.md index d188f3aca5..0e824002e2 100644 --- a/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/2-check-your-environment.md +++ b/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/2-check-your-environment.md @@ -6,55 +6,89 @@ weight: 4 layout: learningpathall --- -In this section, you will check that your environment is all set up and ready to develop with SME2. This will be your first hands-on experience with the environment. +In this section, you will verify that your environment is set up and ready to +develop with SME2. This will be your first hands-on experience with the +environment. ## Compile the examples -First, compile the example code with Clang: - -```BASH { output_lines="2-19" } -docker run --rm -v "$PWD:/work" -w /work armswdev/sme2-learning-path:sme2-environment-v1 make -clang --target=aarch64-none-elf -march=armv9.4-a+sme2 -fno-exceptions -fno-rtti -mno-unaligned-access -O2 -Wall -std=c99 -nostartfiles -lcrt0-semihost -lsemihost -Wl,--defsym=__boot_flash=0x80000000 -Wl,--defsym=__flash=0x80001000 -Wl,--defsym=__ram=0x81000000 -T picolibc.ld -o hello hello.c -clang --target=aarch64-none-elf -march=armv9.4-a+sme2 -fno-exceptions -fno-rtti -mno-unaligned-access -O2 -Wall -std=c99 -c -o sme2_check.o sme2_check.c -clang --target=aarch64-none-elf -march=armv9.4-a+sme2 -fno-exceptions -fno-rtti -mno-unaligned-access -O2 -Wall -std=c99 -c -o misc.o misc.c -clang --target=aarch64-none-elf -march=armv9.4-a+sme2 -fno-exceptions -fno-rtti -mno-unaligned-access -O2 -Wall -std=c99 -nostartfiles -lcrt0-semihost -lsemihost -Wl,--defsym=__boot_flash=0x80000000 -Wl,--defsym=__flash=0x80001000 -Wl,--defsym=__ram=0x81000000 -T picolibc.ld -o sme2_check sme2_check.o misc.o -clang --target=aarch64-none-elf -march=armv9.4-a+sme2 -fno-exceptions -fno-rtti -mno-unaligned-access -O2 -Wall -std=c99 -DIMPL=asm -c -o main_asm.o main.c -clang --target=aarch64-none-elf -march=armv9.4-a+sme2 -fno-exceptions -fno-rtti -mno-unaligned-access -O2 -Wall -std=c99 -c -o matmul_asm.o matmul_asm.c -clang --target=aarch64-none-elf -march=armv9.4-a+sme2 -fno-exceptions -fno-rtti -mno-unaligned-access -O2 -Wall -std=c99 -c -o matmul_asm_impl.o matmul_asm_impl.S -clang --target=aarch64-none-elf -march=armv9.4-a+sme2 -fno-exceptions -fno-rtti -mno-unaligned-access -O2 -Wall -std=c99 -c -o preprocess_l_asm.o preprocess_l_asm.S -clang --target=aarch64-none-elf -fno-exceptions -fno-rtti -mno-unaligned-access -O2 -Wall -std=c99 -c -o matmul_vanilla.o matmul_vanilla.c -clang --target=aarch64-none-elf -fno-exceptions -fno-rtti -mno-unaligned-access -O2 -Wall -std=c99 -c -o preprocess_vanilla.o preprocess_vanilla.c -clang --target=aarch64-none-elf -march=armv9.4-a+sme2 -fno-exceptions -fno-rtti -mno-unaligned-access -O2 -Wall -std=c99 -nostartfiles -lcrt0-semihost -lsemihost -Wl,--defsym=__boot_flash=0x80000000 -Wl,--defsym=__flash=0x80001000 -Wl,--defsym=__ram=0x81000000 -T picolibc.ld -o sme2_matmul_asm main_asm.o matmul_asm.o matmul_asm_impl.o preprocess_l_asm.o matmul_vanilla.o preprocess_vanilla.o misc.o -clang --target=aarch64-none-elf -march=armv9.4-a+sme2 -fno-exceptions -fno-rtti -mno-unaligned-access -O2 -Wall -std=c99 -DIMPL=intr -c -o main_intr.o main.c -clang --target=aarch64-none-elf -march=armv9.4-a+sme2 -fno-exceptions -fno-rtti -mno-unaligned-access -O2 -Wall -std=c99 -c -o matmul_intr.o matmul_intr.c -clang --target=aarch64-none-elf -march=armv9.4-a+sme2 -fno-exceptions -fno-rtti -mno-unaligned-access -O2 -Wall -std=c99 -nostartfiles -lcrt0-semihost -lsemihost -Wl,--defsym=__boot_flash=0x80000000 -Wl,--defsym=__flash=0x80001000 -Wl,--defsym=__ram=0x81000000 -T picolibc.ld -o sme2_matmul_intr main_intr.o matmul_intr.o matmul_vanilla.o preprocess_vanilla.o misc.o +First, build the code examples by running `make`: + +{{< tabpane code=true >}} + {{< tab header="Native SME2 support" language="bash" output_lines="2-19">}} +make +/opt/homebrew/opt/llvm/bin/clang -O2 -Wall -std=c99 -march=native+sme2 -DBAREMETAL=0 -o hello hello.c +/opt/homebrew/opt/llvm/bin/clang -O2 -Wall -std=c99 -march=native+sme2 -DBAREMETAL=0 -c -o sme2_check.o sme2_check.c +/opt/homebrew/opt/llvm/bin/clang -O2 -Wall -std=c99 -march=native+sme2 -DBAREMETAL=0 -c -o misc.o misc.c +/opt/homebrew/opt/llvm/bin/clang -O2 -Wall -std=c99 -march=native+sme2 -o sme2_check sme2_check.o misc.o +/opt/homebrew/opt/llvm/bin/clang -O2 -Wall -std=c99 -march=native+sme2 -DBAREMETAL=0 -DIMPL=asm -c -o main_asm.o main.c +/opt/homebrew/opt/llvm/bin/clang -O2 -Wall -std=c99 -march=native+sme2 -DBAREMETAL=0 -c -o matmul_asm.o matmul_asm.c +/opt/homebrew/opt/llvm/bin/clang -Wall -march=native+sve+sme2 -DBAREMETAL=0 -c -o matmul_asm_impl.o matmul_asm_impl.S +/opt/homebrew/opt/llvm/bin/clang -Wall -march=native+sve+sme2 -DBAREMETAL=0 -c -o preprocess_l_asm.o preprocess_l_asm.S +/opt/homebrew/opt/llvm/bin/clang -O2 -Wall -std=c99 -c -o matmul_vanilla.o matmul_vanilla.c +/opt/homebrew/opt/llvm/bin/clang -O2 -Wall -std=c99 -c -o preprocess_vanilla.o preprocess_vanilla.c +/opt/homebrew/opt/llvm/bin/clang -O2 -Wall -std=c99 -march=native+sme2 -o sme2_matmul_asm main_asm.o matmul_asm.o matmul_asm_impl.o preprocess_l_asm.o matmul_vanilla.o preprocess_vanilla.o misc.o +/opt/homebrew/opt/llvm/bin/clang -O2 -Wall -std=c99 -march=native+sme2 -DBAREMETAL=0 -DIMPL=intr -c -o main_intr.o main.c +/opt/homebrew/opt/llvm/bin/clang -O2 -Wall -std=c99 -march=native+sme2 -DBAREMETAL=0 -c -o matmul_intr.o matmul_intr.c +/opt/homebrew/opt/llvm/bin/clang -O2 -Wall -std=c99 -march=native+sme2 -o sme2_matmul_intr main_intr.o matmul_intr.o matmul_vanilla.o preprocess_vanilla.o misc.o +/opt/homebrew/opt/llvm/bin/llvm-objdump --demangle -d hello > hello.lst +/opt/homebrew/opt/llvm/bin/llvm-objdump --demangle -d sme2_check > sme2_check.lst +/opt/homebrew/opt/llvm/bin/llvm-objdump --demangle -d sme2_matmul_asm > sme2_matmul_asm.lst +/opt/homebrew/opt/llvm/bin/llvm-objdump --demangle -d sme2_matmul_intr > sme2_matmul_intr.lst + {{< /tab >}} + + {{< tab header="Emulated SME2 support" language="bash" output_lines="2-19">}} +docker run --rm -v "$PWD:/work" -w /work armswdev/sme2-learning-path:sme2-environment-v2 make +clang -O2 -Wall -std=c99 --target=aarch64-none-elf -march=armv9.4-a+sme2 -fno-exceptions -fno-rtti -mno-unaligned-access -DBAREMETAL=1 -nostartfiles -lcrt0-semihost -lsemihost -nostartfiles -lcrt0-semihost -lsemihost -Wl,--defsym=__boot_flash=0x80000000 -Wl,--defsym=__flash=0x80001000 -Wl,--defsym=__ram=0x81000000 -T picolibc.ld -o hello hello.c +clang -O2 -Wall -std=c99 --target=aarch64-none-elf -march=armv9.4-a+sme2 -fno-exceptions -fno-rtti -mno-unaligned-access -DBAREMETAL=1 -c -o sme2_check.o sme2_check.c +clang -O2 -Wall -std=c99 --target=aarch64-none-elf -march=armv9.4-a+sme2 -fno-exceptions -fno-rtti -mno-unaligned-access -DBAREMETAL=1 -c -o misc.o misc.c +clang -O2 -Wall -std=c99 --target=aarch64-none-elf -march=armv9.4-a+sme2 -fno-exceptions -fno-rtti -mno-unaligned-access -nostartfiles -lcrt0-semihost -lsemihost -nostartfiles -lcrt0-semihost -lsemihost -Wl,--defsym=__boot_flash=0x80000000 -Wl,--defsym=__flash=0x80001000 -Wl,--defsym=__ram=0x81000000 -T picolibc.ld -o sme2_check sme2_check.o misc.o +clang -O2 -Wall -std=c99 --target=aarch64-none-elf -march=armv9.4-a+sme2 -fno-exceptions -fno-rtti -mno-unaligned-access -DBAREMETAL=1 -DIMPL=asm -c -o main_asm.o main.c +clang -O2 -Wall -std=c99 --target=aarch64-none-elf -march=armv9.4-a+sme2 -fno-exceptions -fno-rtti -mno-unaligned-access -DBAREMETAL=1 -c -o matmul_asm.o matmul_asm.c +clang -Wall --target=aarch64-none-elf -march=armv9.4-a+sme2 -DBAREMETAL=1 -c -o matmul_asm_impl.o matmul_asm_impl.S +clang -Wall --target=aarch64-none-elf -march=armv9.4-a+sme2 -DBAREMETAL=1 -c -o preprocess_l_asm.o preprocess_l_asm.S +clang -O2 -Wall -std=c99 --target=aarch64-none-elf -fno-exceptions -fno-rtti -mno-unaligned-access -c -o matmul_vanilla.o matmul_vanilla.c +clang -O2 -Wall -std=c99 --target=aarch64-none-elf -fno-exceptions -fno-rtti -mno-unaligned-access -c -o preprocess_vanilla.o preprocess_vanilla.c +clang -O2 -Wall -std=c99 --target=aarch64-none-elf -march=armv9.4-a+sme2 -fno-exceptions -fno-rtti -mno-unaligned-access -nostartfiles -lcrt0-semihost -lsemihost -nostartfiles -lcrt0-semihost -lsemihost -Wl,--defsym=__boot_flash=0x80000000 -Wl,--defsym=__flash=0x80001000 -Wl,--defsym=__ram=0x81000000 -T picolibc.ld -o sme2_matmul_asm main_asm.o matmul_asm.o matmul_asm_impl.o preprocess_l_asm.o matmul_vanilla.o preprocess_vanilla.o misc.o +clang -O2 -Wall -std=c99 --target=aarch64-none-elf -march=armv9.4-a+sme2 -fno-exceptions -fno-rtti -mno-unaligned-access -DBAREMETAL=1 -DIMPL=intr -c -o main_intr.o main.c +clang -O2 -Wall -std=c99 --target=aarch64-none-elf -march=armv9.4-a+sme2 -fno-exceptions -fno-rtti -mno-unaligned-access -DBAREMETAL=1 -c -o matmul_intr.o matmul_intr.c +clang -O2 -Wall -std=c99 --target=aarch64-none-elf -march=armv9.4-a+sme2 -fno-exceptions -fno-rtti -mno-unaligned-access -nostartfiles -lcrt0-semihost -lsemihost -nostartfiles -lcrt0-semihost -lsemihost -Wl,--defsym=__boot_flash=0x80000000 -Wl,--defsym=__flash=0x80001000 -Wl,--defsym=__ram=0x81000000 -T picolibc.ld -o sme2_matmul_intr main_intr.o matmul_intr.o matmul_vanilla.o preprocess_vanilla.o misc.o llvm-objdump --demangle -d hello > hello.lst llvm-objdump --demangle -d sme2_check > sme2_check.lst llvm-objdump --demangle -d sme2_matmul_asm > sme2_matmul_asm.lst llvm-objdump --demangle -d sme2_matmul_intr > sme2_matmul_intr.lst -``` + {{< /tab >}} +{{< /tabpane >}} - Executed within the docker ``armswdev/sme2-learning-path:sme2-environment-v1`` environment, the ``make`` command performs the following tasks: +The `make` command performs the following tasks: +- It builds four executables: `hello`, `sme2_check`, `sme2_matmul_asm`, and + `sme2_matmul_intr`. +- It creates the assembly listings for the four executables: `hello.lst`, + `sme2_check.lst`, `sme2_matmul_asm.lst`, and `sme2_matmul_intr.lst`. -- It builds four executables: ``hello``, ``sme2_check``, ``sme2_matmul_asm``, and ``sme2_matmul_intr``. -- It creates the assembly listings for the four executables: ``hello.lst``, ``sme2_check.lst``, ``sme2_matmul_asm.lst``, and ``sme2_matmul_intr.lst``. +At any point, you can clean the directory of all the files that have been built +by invoking `make clean`: -{{% notice Note %}} -At any point, you can clean the directory of all the files that have been built by invoking ``make clean``: +{{< tabpane code=true >}} + {{< tab header="Native SME2 support" language="bash" output_lines="2">}} + make clean + rm hello sme2_check sme2_matmul_asm sme2_matmul_intr hello.lst sme2_check.lst sme2_matmul_asm.lst sme2_matmul_intr.lst *.o + {{< /tab >}} -```BASH -$ docker run --rm -v "$PWD:/work" -w /work armswdev/sme2-learning-path:sme2-environment-v1 make clean -``` -{{% /notice %}} + {{< tab header="Emulated SME2 support" language="bash" output_lines="2">}} + docker run --rm -v "$PWD:/work" -w /work armswdev/sme2-learning-path:sme2-environment-v2 make clean + rm hello sme2_check sme2_matmul_asm sme2_matmul_intr hello.lst sme2_check.lst sme2_matmul_asm.lst sme2_matmul_intr.lst *.o + {{< /tab >}} +{{< /tabpane >}} -## Basic checks +## Basic Checks -The very first program that you should run is the famous "Hello, world !" example that -will tell you if your environment is set up correctly. +The very first program that you should run is the famous "Hello, world!" example +that will tell you if your environment is set up correctly. -The source code is contained in ``hello.c`` and looks like this: +The source code is contained in `hello.c` and looks like this: -```C +```C { line_numbers="true" } #include #include @@ -64,73 +98,71 @@ int main(int argc, char *argv[]) { } ``` -Run the FVP simulation of the ``hello`` program with: +Run the `hello` program with: -```BASH { output_lines="2-4" } -docker run --rm -v "$PWD:/work" -w /work armswdev/sme2-learning-path:sme2-environment-v1 ./run-fvp.sh hello -Hello, world ! +{{< tabpane code=true >}} + {{< tab header="Native SME2 support" language="bash" output_lines="2">}} + ./hello + Hello, world ! + {{< /tab >}} -Info: /OSCI/SystemC: Simulation stopped by user. -``` + {{< tab header="Emulated SME2 support" language="bash" output_lines="2-4">}} + docker run --rm -v "$PWD:/work" -w /work armswdev/sme2-learning-path:sme2-environment-v2 ./run-fvp.sh hello + Hello, world ! -The important line here is "``Hello, world !``" as it demonstrates that the generic code -can be compiled and run on the FVP. + Info: /OSCI/SystemC: Simulation stopped by user. + {{< /tab >}} +{{< /tabpane >}} + +In the emulated case, there are extra lines that are printed out by the FVP, but +the important line here is "Hello, world!": it demonstrates that the generic +code can be compiled and executed. ## SME2 checks -You will now run the ``sme2_check`` program, which checks that SME2 works as -expected, in both the compiler and in the FVP. +You will now run the `sme2_check` program, which verifies that SME2 works as +expected. This checks both the compiler and the CPU (or the emulated CPU) are +properly supporting SME2. + +The source code is found in `sme2_check.c`: -The source code is found in -``sme2_check.c``: +```C { line_numbers="true" } +#include "misc.h" -```C +#include #include #include -#include "misc.h" - #ifdef __ARM_FEATURE_SME2 #include #else #error __ARM_FEATURE_SME2 is not defined #endif -#define get_cpu_ftr(regId, feat, msb, lsb) \ - ({ \ - unsigned long __val; \ - __asm__("mrs %0, " #regId : "=r"(__val)); \ - printf("%-20s: 0x%016lx\n", #regId, __val); \ - printf(" - %-10s: 0x%08lx\n", #feat, \ - (__val >> lsb) & ((1 << (msb - lsb)) - 1)); \ - }) +__arm_locally_streaming void function_in_streaming_mode() { + printf("In streaming_mode: %d, SVL: %" PRIu64 " bits\n", + __arm_in_streaming_mode(), svcntb() * 8); +} int main(int argc, char *argv[]) { - get_cpu_ftr(ID_AA64PFR0_EL1, SVE, 35, 32); - get_cpu_ftr(ID_AA64PFR1_EL1, SME, 27, 24); - int n = 0; -#ifdef __ARM_FEATURE_SME2 - setup_sme(); - n = svcntb() * 8; +#if BAREMETAL == 1 + setup_sme_baremetal(); #endif - if (n) { - printf("SVE is available with length %d\n", n); - } else { - printf("SVE is unavailable.\n"); + + if (!display_cpu_features()) { + printf("SME2 is not supported on this CPU.\n"); exit(EXIT_FAILURE); } - printf("Checking has_sme: %d\n", __arm_has_sme()); - printf("Checking in_streaming_mode: %d\n", __arm_in_streaming_mode()); + printf("Checking initial in_streaming_mode: %d\n", + __arm_in_streaming_mode()); - printf("Starting streaming mode...\n"); - __asm__("smstart"); + printf("Switching to streaming mode...\n"); - printf("Checking in_streaming_mode: %d\n", __arm_in_streaming_mode()); + function_in_streaming_mode(); - printf("Stopping streaming mode...\n"); - __asm__("smstop"); + printf("Switching back from streaming mode...\n"); printf("Checking in_streaming_mode: %d\n", __arm_in_streaming_mode()); @@ -138,36 +170,82 @@ int main(int argc, char *argv[]) { } ``` -The ``sme2_check`` program displays the SVE field of the ``ID_AA64PFR0_EL1`` system register and the SME field of the ``ID_AA64PFR1_EL1`` system register. It will then check if SVE and SME are available, then finally will switch into streaming mode and back from streaming mode. - -The ``__ARM_FEATURE_SME2`` macro is provided by the compiler when it targets an SME-capable target, which is specified with the ``-march=armv9.4-a+sme2`` command line option to ``clang`` in -file ``Makefile``. - -The ``arm_sme.h`` include file is part of the Arm C Library -Extension ([ACLE](https://arm-software.github.io/acle/main/)). - -The ACLE provides types and function declarations to enable C/C++ programmers to make the best possible use of the Arm architecture. You can use the SME-related part of the library, but it does also provide support for Neon or other Arm architectural extensions. - -```BASH -docker run --rm -v "$PWD:/work" -w /work armswdev/sme2-learning-path:sme2-environment-v1 ./run-fvp.sh sme2_check -``` - -The output should be similar to: - -```TXT -ID_AA64PFR0_EL1 : 0x1101101131111112 - - SVE : 0x00000001 -ID_AA64PFR1_EL1 : 0x0000101002000001 - - SME : 0x00000002 -SVE is available with length 512 -Checking has_sme: 1 -Checking in_streaming_mode: 0 -Starting streaming mode... -Checking in_streaming_mode: 1 -Stopping streaming mode... -Checking in_streaming_mode: 0 - -Info: /OSCI/SystemC: Simulation stopped by user. -``` - -You have now checked that the code can be compiled and run with full SME2 support, and are all set to move to the next section. +The ``__ARM_FEATURE_SME2`` macro (line 7) is provided by the compiler when it +targets an SME-capable target, which is specified with the ``+sme2`` +architectural feature in ``-march=armv9.4-a+sme2`` (emulated environment) or +``-march=native+sme2`` command line option to ``clang`` in file ``Makefile``. + +The ``arm_sme.h`` file included at line 8 is part of the Arm C Library Extension +([ACLE](https://arm-software.github.io/acle/main/)). The ACLE provides types and +function declarations to enable C/C++ programmers to make the best possible use +of the Arm architecture. You can use the SME-related part of the library, but it +does also provide support for Neon or other Arm architectural extensions. + +In order to run in a baremetal environment (like the one being used in the +emulated SME2 support), where no operating system has done the setup of the +processor for the user land programs, an additional step is required to turn +SME2 on. This is the purpose of the ``setup_sme_baremetal()`` call at line 21. +In environments where SME2 is natively supported, nothing needs to be done, +which is why the execution of this function is condionned by the ``BAREMETAl`` +macro. ``BAREMETAL`` is set to 1 in the ``Makefile`` when the FVP is targeted, +and set to 0 otherwise. The body of the ``setup_sme_baremetal`` function is +defined in ``misc.c``. + +The ``sme2_check`` program then displays whether SVE, SME and SME2 are supported +at line 24. The checking of SVE, SME and SME2 is done differently depending on +``BAREMETAL``. This platform specific behaviour is abstract by the +``display_cpu_features()`` : +- In baremetal mode, our program has access to system registers and can thus do + some low level peek at what the silicon actually supports. The program will + print the SVE field of the ``ID_AA64PFR0_EL1`` system register and the SME + field of the ``ID_AA64PFR1_EL1`` system register. +- In non baremetal mode, on an Apple platform the program needs to use a higher + level API call. + +The body of the ``display_cpu_features`` function is defined in ``misc.c``. + +If SME2 is not available, ``sme2_check`` will emit a diagnostic message (line +25) and exit (line 26). + +``sme2_check`` will then print the initial streaming mode state at line 29 +(which is expected to be 0), then will switch to streaming mode (line 34) when +invoking function ``function_in_streaming_mode`` to show the Streaming Vector +Length (a.k.a ``SVL``), and then switch back to non streaming mode (when +returning from ``function_in_streaming_mode``). Function +``function_in_streaming_mode`` is defined at line 13. Note that it has been +annotated with the ``__arm_locally_streaming`` attribute, which instructs the +compiler to automatically switch to streaming mode when invoking this function. +Streaming mode will be discussed in more depth in the next section. + +{{< tabpane code=true >}} + {{< tab header="Native SME2 support" language="bash" output_lines="2-9">}} + ./sme2_check + HAS_SVE: 0 + HAS_SME: 1 + HAS_SME2: 1 + Checking initial in_streaming_mode: 0 + Switching to streaming mode... + In streaming_mode: 1, SVL: 512 bits + Switching back from streaming mode... + Checking in_streaming_mode: 0 + {{< /tab >}} + + {{< tab header="Emulated SME2 support" language="bash" output_lines="2-12">}} + docker run --rm -v "$PWD:/work" -w /work armswdev/sme2-learning-path:sme2-environment-v2 ./run-fvp.sh sme2_check + ID_AA64PFR0_EL1 : 0x1101101131111112 + - SVE : 0x00000001 + ID_AA64PFR1_EL1 : 0x0000101002000001 + - SME : 0x00000002 + Checking has_sme: 1 + Checking initial in_streaming_mode: 0 + Switching to streaming mode... + In streaming_mode: 1, SVL: 512 bits + Switching back from streaming mode... + Checking in_streaming_mode: 0 + + Info: /OSCI/SystemC: Simulation stopped by user. + {{< /tab >}} +{{< /tabpane >}} + +You have now checked that the code can be compiled and run with full SME2 +support. You are all set to move to the next section. diff --git a/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/3-streaming-mode.md b/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/3-streaming-mode.md new file mode 100644 index 0000000000..0240b6efd8 --- /dev/null +++ b/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/3-streaming-mode.md @@ -0,0 +1,78 @@ +--- +title: Streaming mode +weight: 5 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +In real-world large-scale software, a program moves back and forth from +streaming mode, and some streaming mode routines call other streaming mode +routines, which means that some state needs to be saved and restored. This +includes the ZA storage. This is defined in the ACLE and supported by the +compiler: the programmer *just* has to annotate the functions with some keywords +and let the compiler automatically perform the low-level tasks of managing the +streaming mode. This frees the developer from a tedious and error-prone task. +See [Introduction to streaming and non-streaming +mode](https://arm-software.github.io/acle/main/acle.html#controlling-the-use-of-streaming-mode) +for further information. The rest of this section references information from +the ACLE. + +## About streaming mode + +The AArch64 architecture defines a concept called *streaming mode*, controlled +by a processor state bit called `PSTATE.SM`. At any given point in time, the +processor is either in streaming mode (`PSTATE.SM==1`) or in non-streaming mode +(`PSTATE.SM==0`). There is an instruction called `SMSTART` to enter streaming mode +and an instruction called `SMSTOP` to return to non-streaming mode. + +Streaming mode has three main effects on C and C++ code: + +- It can change the length of SVE vectors and predicates: the length of an SVE + vector in streaming mode is called the “streaming vector length” (SVL), which + might be different from the normal non-streaming vector length. See + [Effect of streaming mode on VL](https://arm-software.github.io/acle/main/acle.html#effect-of-streaming-mode-on-vl) + for more details. +- Some instructions can only be executed in streaming mode, which means that + their associated ACLE intrinsics can only be used in streaming mode. These + intrinsics are called “streaming intrinsics”. +- Some other instructions can only be executed in non-streaming mode, which + means that their associated ACLE intrinsics can only be used in non-streaming + mode. These intrinsics are called “non-streaming intrinsics”. + +The C and C++ standards define the behavior of programs in terms of an *abstract +machine*. As an extension, the ACLE specification applies the distinction +between streaming mode and non-streaming mode to this abstract machine: at any +given point in time, the abstract machine is either in streaming mode or in +non-streaming mode. + +This distinction between processor mode and abstract machine mode is mostly just +a specification detail. However, the usual “as if” rule applies: the +processor's actual mode at runtime can be different from the abstract machine's +mode, provided that this does not alter the behavior of the program. One +practical consequence of this is that C and C++ code does not specify the exact +placement of `SMSTART` and `SMSTOP` instructions; the source code simply places +limits on where such instructions go. For example, when stepping through a +program in a debugger, the processor mode might sometimes be different from the +one implied by the source code. + +ACLE provides attributes that specify whether the abstract machine executes statements: + +- In non-streaming mode, in which case they are called *non-streaming statements*. +- In streaming mode, in which case they are called *streaming statements*. +- In either mode, in which case they are called *streaming-compatible statements*. + +SME provides an area of storage called ZA, of size `SVL.B` x `SVL.B` bytes. It +also provides a processor state bit called `PSTATE.ZA` to control whether ZA +is enabled. + +In C and C++ code, access to ZA is controlled at function granularity: a +function either uses ZA or it does not. Another way to say this is that a +function either “has ZA state” or it does not. + +If a function does have ZA state, the function can either share that ZA state +with the function's caller or create new ZA state “from scratch”. In the latter +case, it is the compiler's responsibility to free up ZA so that the function can +use it; see the description of the lazy saving scheme in +[AAPCS64](https://arm-software.github.io/acle/main/acle.html#AAPCS64) for details +about how the compiler does this. diff --git a/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/4-outer-product.md b/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/4-outer-product.md deleted file mode 100644 index ea55642846..0000000000 --- a/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/4-outer-product.md +++ /dev/null @@ -1,108 +0,0 @@ ---- -title: Outer product -weight: 6 - -### FIXED, DO NOT MODIFY -layout: learningpathall ---- -## Matrix multiplication with the outer product - -In this section, you will learn how you can use the outer product with the SME engine to improve matrix multiplication. - -In this standard matrix multiplication example, the core of the computation can be represented as: - -```C - acc += matLeft[m * K + k] * matRight[k * N + n]; -``` - -Which translates to 1 multiply-accumulate, which is also known as ``macc``, for two loads (``matLeft[m * K + k]`` -and ``matRight[k *N + n]``). It therefore has a 1:2 ``macc`` to ``load`` ratio. - -From a memory system perspective, this is not effective, especially since this -computation is done within a triple-nested loop, repeatedly loading data from -memory. - -To exacerbate matters, large matrices might not fit in cache. In order to improve the matrix multiplication efficiency, the goal is to increase the ``macc`` to ``load`` ratio, which means to increase the number of multiply-accumulate operations per load. - -Figure 3 below shows how the matrix multiplication of ``matLeft`` (3 rows, 2 -columns) by ``matRight`` (2 rows, 3 columns) can be decomposed as the sum of the -outer products: - -![example image alt-text#center](outer_product.png "Figure 3: Outer Product-based Matrix Multiplication.") - -The SME engine builds on the -[Outer Product](https://en.wikipedia.org/wiki/Outer_product) as matrix -multiplication can be expressed as the -[sum of column-by-row outer products](https://en.wikipedia.org/wiki/Outer_product#Connection_with_the_matrix_product). - - -## About transposition - -From the previous page, you will recall that matrices are laid out in row-major -order. This means that loading row-data from memory is efficient as the memory -system operates efficiently with contiguous data. An example of this is where caches are loaded row by row, and data prefetching is simple - just load the data from ``current address + sizeof(data)``. This is not the case for loading column-data from memory though, as it requires more work from the memory system. - -In order to further improve the effectiveness of the matrix multiplication, it -is therefore desirable to change the layout in memory of the left-hand side -matrix, which is called ``matLeft`` in the code examples in this Learning Path. -The improved layout would ensure that elements from the same column are located -next to each other in memory. This is essentially a matrix transposition, -which changes ``matLeft`` from row-major order to column-major order. - -{{% notice Important %}} -It is important to note here that this reorganizes the layout of the matrix in -memory in order for the algorithm implementation to be more efficient. The -transposition affects only the memory layout. ``matLeft`` is transformed to -column-major order, but from a mathematical perspective, ``matleft`` is -*not* transposed. -{{% /notice %}} - -### Transposition in the real world - -In the same way that trees don't reach the sky, the SME engine has physical implementation limits. It operates with tiles in the ZA storage. Tiles are 2D portions of the matrices being processed. SME has dedicated instructions to load data to, and store data from tiles efficiently, as well as instructions to operate with and on tiles, for example the [fmopa](https://developer.arm.com/documentation/ddi0602/latest/SME-Instructions/FMOPA--non-widening---Floating-point-outer-product-and-accumulate-?lang=en) -instruction which takes two vectors as inputs and accumulate all the outer -products to a 2D tile. The tile in ZA storage is what allows SME to increase the -``macc`` to ``load`` ratio, as all the tile elements are loaded to the tile, to -be used with the SME outer product instructions. - -Taking into account that the ZA storage is finite, the desirable transposition -of the ``matLeft`` matrix that was discussed in the previous section needs to -adapted to the tile dimensions, so that a tile is easy to access. The -``matLeft`` preprocessing has thus some aspects of transpositions, but takes -into account the tiling as well and is referred to in the code as -``preprocess``. - -Here is at the algorithmic level what ``preprocess_l`` does in practice: - -```C -void preprocess_l(uint64_t nbr, uint64_t nbc, uint64_t SVL, - const float *restrict a, float *restrict a_mod) { - - // For all tiles of SVL x SVL data - for (uint64_t By = 0; By < nbr; By += SVL) { - for (uint64_t Bx = 0; Bx < nbc; Bx += SVL) { - // For this tile - const uint64_t dest = By * nbc + Bx * SVL; - for (uint64_t j = 0; j < SVL; j++) { - for (uint64_t i = 0; i < SVL && (Bx + i) < nbc; i++) { - if (By + j < nbr) { - a_mod[dest + i * SVL + j] = a[(By + j) * nbc + Bx + i]; - } else { - // These elements are outside of matrix a, so zero them. - a_mod[dest + i * SVL + j] = 0.0; - } - } - } - } - } -} -``` - -``preprocess_l`` will be used to check the assembly and intrinsic versions of -the matrix multiplication perform the preprocessing step correctly. This code is -located in file ``preprocess_vanilla.c``. - -{{% notice Note %}} -In real-world applications, it might be possible to arrange for ``matLeft`` to -be stored in column-major order, eliminating the need for transposition, and making the preprocessing step unnecessary. Matrix processing frameworks and libraries often have attributes within the matrix object to track if it is row- or column-major order, and whether it has been transposed to avoid unnecessary computations. -{{% /notice %}} diff --git a/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/3-vanilla-matmul.md b/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/4-vanilla-matmul.md similarity index 59% rename from content/learning-paths/cross-platform/multiplying-matrices-with-sme2/3-vanilla-matmul.md rename to content/learning-paths/cross-platform/multiplying-matrices-with-sme2/4-vanilla-matmul.md index e78e399bbc..c91630a18a 100644 --- a/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/3-vanilla-matmul.md +++ b/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/4-vanilla-matmul.md @@ -1,16 +1,14 @@ --- title: Vanilla matrix multiplication -weight: 5 +weight: 6 ### FIXED, DO NOT MODIFY layout: learningpathall --- -## Vanilla matrix multiplication - In this section, you will learn about an example of standard matrix multiplication in C. -### Algorithm description +## Vanilla matrix multiplication algorithm The vanilla matrix multiplication operation takes two input matrices, A [Ar rows x Ac columns] and B [Br rows x Bc columns], to produce an output matrix C @@ -22,7 +20,6 @@ element in the B column then summing all these products, as Figure 2 shows. This implies that the A, B, and C matrices have some constraints on their dimensions: - - A's number of columns must match B's number of rows: Ac == Br. - C has the dimensions Cr == Ar and Cc == Bc. @@ -31,22 +28,21 @@ properties and use, by reading this [Wikipedia article on Matrix Multiplication](https://en.wikipedia.org/wiki/Matrix_multiplication). In this Learning Path, you will see the following variable names: - -- ``matLeft`` corresponds to the left-hand side argument of the matrix +- `matLeft` corresponds to the left-hand side argument of the matrix multiplication. -- ``matRight``corresponds to the right-hand side of the matrix multiplication. -- ``M`` is ``matLeft`` number of rows. -- ``K`` is ``matLeft`` number of columns (and ``matRight`` number of rows). -- ``N`` is ``matRight`` number of columns. -- ``matResult``corresponds to the result of the matrix multiplication, with - ``M`` rows and ``N`` columns. +- `matRight`corresponds to the right-hand side of the matrix multiplication. +- `M` is `matLeft` number of rows. +- `K` is `matLeft` number of columns (and `matRight` number of rows). +- `N` is `matRight` number of columns. +- `matResult`corresponds to the result of the matrix multiplication, with + `M` rows and `N` columns. -### C implementation +## C implementation A literal implementation of the textbook matrix multiplication algorithm, as -described above, can be found in file ``matmul_vanilla.c``: +described above, can be found in file `matmul_vanilla.c`: -```C +```C { line_numbers="true" } void matmul(uint64_t M, uint64_t K, uint64_t N, const float *restrict matLeft, const float *restrict matRight, float *restrict matResult) { @@ -65,16 +61,16 @@ void matmul(uint64_t M, uint64_t K, uint64_t N, ``` In this Learning Path, the matrices are laid out in memory as contiguous -sequences of elements, in [Row-Major -Order](https://en.wikipedia.org/wiki/Row-_and_column-major_order). The -``matmul`` function performs the algorithm described above. - -The pointers to ``matLeft``, ``matRight`` and ``matResult`` have been annotated as -``restrict``, which informs the compiler that the memory areas designated by -those pointers do not alias. This means that they do not overlap in any way, so that the -compiler does not need to insert extra instructions to deal with these cases. -The pointers to ``matLeft`` and ``matRight`` are marked as ``const`` as neither of these two matrices are modified by ``matmul``. - -You now have a reference standard matrix multiplication function. You will use it later -on in this Learning Path to ensure that the assembly version and the intrinsics -version of the multiplication algorithm do not contain errors. \ No newline at end of file +sequences of elements, in [Row-Major Order](https://en.wikipedia.org/wiki/Row-_and_column-major_order). +The `matmul` function performs the algorithm described above. + +The pointers to `matLeft`, `matRight` and `matResult` have been annotated +as `restrict`, which informs the compiler that the memory areas designated by +those pointers do not alias. This means that they do not overlap in any way, so +that the compiler does not need to insert extra instructions to deal with these +cases. The pointers to `matLeft` and `matRight` are marked as `const` as +neither of these two matrices are modified by `matmul`. + +You now have a reference standard matrix multiplication function. You will use +it later on in this Learning Path to ensure that the assembly version and the +intrinsics version of the multiplication algorithm do not contain errors. \ No newline at end of file diff --git a/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/5-outer-product.md b/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/5-outer-product.md new file mode 100644 index 0000000000..b49a5d2ba8 --- /dev/null +++ b/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/5-outer-product.md @@ -0,0 +1,122 @@ +--- +title: Outer product +weight: 7 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +In this section, you will learn how to use the outer product with the SME engine +to improve matrix multiplication execution performances. + +## Matrix multiplication with the outer product + +In the vanilla matrix multiplication example, the core of the computation is: + +```C + acc += matLeft[m * K + k] * matRight[k * N + n]; +``` + +This translates to one multiply-accumulate operation, known as `macc`, for two +loads (`matLeft[m * K + k]` and `matRight[k * N + n]`). It therefore has a 1:2 +`macc` to `load` ratio. + +From a memory system perspective, this is not efficient, especially since this +computation is done within a triple-nested loop, repeatedly loading data from +memory. + +To make matters worse, large matrices might not fit in cache. To improve matrix +multiplication efficiency, the goal is to increase the `macc` to `load` ratio, +which means increasing the number of multiply-accumulate operations per load. + +Figure 3 below illustrates how the matrix multiplication of `matLeft` (3 rows, 2 +columns) by `matRight` (2 rows, 3 columns) can be decomposed as the sum of outer +products: + +![example image alt-text#center](outer_product.png "Figure 3: Outer Product-based Matrix Multiplication.") + +The SME engine builds on the [Outer +Product](https://en.wikipedia.org/wiki/Outer_product) because matrix +multiplication can be expressed as the [sum of column-by-row outer +products](https://en.wikipedia.org/wiki/Outer_product#Connection_with_the_matrix_product). + +## About transposition + +From the previous page, you will recall that matrices are laid out in row-major +order. This means that loading row-data from memory is efficient as the memory +system operates efficiently with contiguous data. An example of this is where +caches are loaded row by row, and data prefetching is simple - just load the +data from `current address + sizeof(data)`. This is not the case for loading +column-data from memory though, as it requires more work from the memory system. + +To further improve matrix multiplication effectiveness, it is therefore +desirable to change the layout in memory of the left-hand side matrix, called +`matLeft` in the code examples in this Learning Path. The improved layout would +ensure that elements from the same column are located next to each other in +memory. This is essentially a matrix transposition, which changes `matLeft` from +row-major order to column-major order. + +{{% notice Important %}} +It is important to note here that this reorganizes the layout of the matrix in +memory to make the algorithm implementation more efficient. The transposition +affects only the memory layout. `matLeft` is transformed to column-major order, +but from a mathematical perspective, `matLeft` is *not* transposed. +{{% /notice %}} + +### Transposition in the real world + +Just as trees don't reach the sky, the SME engine has physical implementation +limits. It operates with tiles in the ZA storage. Tiles are 2D portions of the +matrices being processed. SME has dedicated instructions to load and store data +from tiles efficiently, as well as instructions to operate with and on tiles. +For example, the +[fmopa](https://developer.arm.com/documentation/ddi0602/latest/SME-Instructions/FMOPA--non-widening---Floating-point-outer-product-and-accumulate-?lang=en) +instruction takes two vectors as inputs and accumulates all the outer products +into a 2D tile. The tile in ZA storage allows SME to increase the `macc` to +`load` ratio by loading all the tile elements to be used with the SME outer +product instructions. + +Considering that ZA storage is finite, the desired transposition of the +`matLeft` matrix discussed in the previous section needs to be adapted to the +tile dimensions, so that a tile is easy to access. The `matLeft` preprocessing +thus involves some aspects of transposition but also takes into account tiling, +referred to in the code as `preprocess`. + +Here is what `preprocess_l` does in practice, at the algorithmic level: + +```C { line_numbers = "true" } +void preprocess_l(uint64_t nbr, uint64_t nbc, uint64_t SVL, + const float *restrict a, float *restrict a_mod) { + + // For all tiles of SVL x SVL data + for (uint64_t By = 0; By < nbr; By += SVL) { + for (uint64_t Bx = 0; Bx < nbc; Bx += SVL) { + // For this tile + const uint64_t dest = By * nbc + Bx * SVL; + for (uint64_t j = 0; j < SVL; j++) { + for (uint64_t i = 0; i < SVL && (Bx + i) < nbc; i++) { + if (By + j < nbr) { + a_mod[dest + i * SVL + j] = a[(By + j) * nbc + Bx + i]; + } else { + // These elements are outside of matrix a, so zero them. + a_mod[dest + i * SVL + j] = 0.0; + } + } + } + } + } +} +``` + +`preprocess_l` will be used to check that the assembly and intrinsic versions of +the matrix multiplication perform the preprocessing step correctly. This code is +located in the file `preprocess_vanilla.c`. + +{{% notice Note %}} +In real-world applications, it might be possible to arrange for `matLeft` to be +stored in column-major order, eliminating the need for transposition and making +the preprocessing step unnecessary. Matrix processing frameworks and libraries +often have attributes within the matrix object to track if it is in row- or +column-major order, and whether it has been transposed to avoid unnecessary +computations. +{{% /notice %}} \ No newline at end of file diff --git a/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/5-sme2-matmul-asm.md b/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/5-sme2-matmul-asm.md deleted file mode 100644 index 9504eff06d..0000000000 --- a/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/5-sme2-matmul-asm.md +++ /dev/null @@ -1,202 +0,0 @@ ---- -title: SME2 assembly matrix multiplication -weight: 7 - -### FIXED, DO NOT MODIFY -layout: learningpathall ---- -## Matrix multiplication with SME2 in assembly - -In this chapter, you will use an SME2-optimized matrix multiplication written -directly in assembly. - -### Description - -This Learning Path reuses the assembly version provided in the [SME Programmer's -Guide](https://developer.arm.com/documentation/109246/0100/matmul-fp32--Single-precision-matrix-by-matrix-multiplication) -where you will find a high-level and an in-depth description of the two steps -performed. - -The assembly versions have been modified so they coexist nicely with -the intrinsic versions. In this Learning Path, the ``preprocess`` function is -defined in ``preprocess_l_asm.S`` and the outer product-based matrix -multiplication is found in ``matmul_asm_impl.S``. - -These two functions have been stitched together in ``matmul_asm.c`` with the same prototype as the reference implementation of matrix multiplication, so that a top-level ``matmul_asm`` can -be called from the ``main`` function: - -```C -void matmul_asm(uint64_t M, uint64_t K, uint64_t N, - const float *restrict matLeft, const float *restrict matRight, - float *restrict matLeft_mod, float *restrict matResult) { - __asm volatile("" - : - : - : "p0", "p1", "p2", "p3", "p4", "p5", "p6", "p7", "p8", "p9", - "p10", "p11", "p12", "p13", "p14", "p15", "z0", "z1", "z2", - "z3", "z4", "z5", "z6", "z7", "z8", "z9", "z10", "z11", - "z12", "z13", "z14", "z15", "z16", "z17", "z18", "z19", - "z20", "z21", "z22", "z23", "z24", "z25", "z26", "z27", - "z28", "z29", "z30", "z31"); - - preprocess_l_asm(M, K, matLeft, matLeft_mod); - matmul_asm_impl(M, K, N, matLeft_mod, matRight, matResult); - - __asm volatile("" - : - : - : "p0", "p1", "p2", "p3", "p4", "p5", "p6", "p7", "p8", "p9", - "p10", "p11", "p12", "p13", "p14", "p15", "z0", "z1", "z2", - "z3", "z4", "z5", "z6", "z7", "z8", "z9", "z10", "z11", - "z12", "z13", "z14", "z15", "z16", "z17", "z18", "z19", - "z20", "z21", "z22", "z23", "z24", "z25", "z26", "z27", - "z28", "z29", "z30", "z31"); -} -``` - -Note here the use of the ``__asm`` statement forcing the compiler to save the SVE/SME registers. - -The high-level ``matmul_asm`` function is called from ``main.c``: - -```C -#include "matmul.h" -#include "misc.h" - -#include -#include -#include -#include - -#ifndef __ARM_FEATURE_SME2 -#error __ARM_FEATURE_SME2 is not defined -#endif - -#ifndef IMPL -#error matmul implementation selection macro IMPL is not defined -#endif - -#define STRINGIFY_(I) #I -#define STRINGIFY(I) STRINGIFY_(I) -#define FN(M, I) M##I -#define MATMUL(I, M, K, N, mL, mR, mM, m) FN(matmul_, I)(M, K, N, mL, mR, mM, m) - -// Assumptions: -// nbr in matLeft (M): any -// nbc in matLeft, nbr in matRight (K): any K > 2 (for the asm version) -// nbc in matRight (N): any - -int main(int argc, char **argv) { - - /* Size parameters */ - uint64_t M, N, K; - if (argc >= 4) { - M = strtoul(argv[1], NULL, 0); - K = strtoul(argv[2], NULL, 0); - N = strtoul(argv[3], NULL, 0); - } else { - /* Default: 125x35x70 */ - M = 125; - K = 35; - N = 70; - } - - printf("\nSME2 Matrix Multiply fp32 *%s* example with args %lu %lu %lu\n", - STRINGIFY(IMPL), M, K, N); - - setup_sme(); - - const uint64_t SVL = svcntsw(); - - /* Calculate M of transformed matLeft. */ - const uint64_t M_mod = SVL * (M / SVL + (M % SVL != 0 ? 1 : 0)); - - float *matRight = (float *)malloc(K * N * sizeof(float)); - - float *matLeft = (float *)malloc(M * K * sizeof(float)); - float *matLeft_mod = (float *)malloc(M_mod * K * sizeof(float)); - float *matLeft_mod_ref = (float *)malloc(M_mod * K * sizeof(float)); - - float *matResult = (float *)malloc(M * N * sizeof(float)); - float *matResult_ref = (float *)malloc(M * N * sizeof(float)); - -#ifdef DEBUG - initialize_matrix(matLeft, M * K, LINEAR_INIT); - initialize_matrix(matRight, K * N, LINEAR_INIT); - initialize_matrix(matLeft_mod, M_mod * K, DEAD_INIT); - initialize_matrix(matResult, M * N, DEAD_INIT); - - print_matrix(M, K, matLeft, "matLeft"); - print_matrix(K, N, matRight, "matRight"); -#else - initialize_matrix(matLeft, M * K, RANDOM_INIT); - initialize_matrix(matRight, K * N, RANDOM_INIT); -#endif - - MATMUL(IMPL, M, K, N, matLeft, matRight, matLeft_mod, matResult); - - // Compute the reference values with the vanilla implementations. - matmul(M, K, N, matLeft, matRight, matResult_ref); - preprocess_l(M, K, SVL, matLeft, matLeft_mod_ref); - - unsigned error = compare_matrices(K, M_mod, matLeft_mod_ref, matLeft_mod, - "Matrix preprocessing"); - if (!error) - error = compare_matrices(M, N, matResult_ref, matResult, - "Matrix multiplication"); - - free(matRight); - - free(matLeft); - free(matLeft_mod); - free(matLeft_mod_ref); - - free(matResult); - free(matResult_ref); - - return error ? EXIT_FAILURE : EXIT_SUCCESS; -} -``` - -The same ``main.c`` file is used for the assembly and intrinsic-based versions -of the matrix multiplication. It first sets the ``M``, ``K`` and ``N`` -parameters, to either the arguments supplied on the command line or uses the default -value. - -Depending on the ``M``, ``K``, ``N`` dimension parameters, ``main`` allocates memory for all the matrices and initializes ``matLeft`` and ``matRight`` with random data. The actual matrix multiplication implementation is provided through the ``IMPL`` macro. - -It then runs the matrix multiplication from ``IMPL`` and computes the reference values for the preprocessed matrix as well as the result matrix. It then compares the actual values to the reference values and reports errors, if there are any. Finally, all the memory is deallocated before exiting the program with a success or failure return code. - -### Compile and run it - -First, make sure that the ``sme2_matmul_asm`` executable is up-to-date: - -```BASH -docker run --rm -v "$PWD:/work" -w /work armswdev/sme2-learning-path:sme2-environment-v1 make sme2_matmul_asm -``` - -Then execute ``sme2_matmul_asm`` on the FVP: - -```BASH -docker run --rm -v "$PWD:/work" -w /work armswdev/sme2-learning-path:sme2-environment-v1 ./run-fvp.sh sme2_matmul_asm -``` - -The output should be something similar to: - -```TXT -SME2 Matrix Multiply fp32 *asm* example with args 125 35 70 -Matrix preprocessing: PASS ! -Matrix multiplication: PASS ! - -Info: /OSCI/SystemC: Simulation stopped by user. -``` - -{{% notice Tip %}} -The example above uses the default values for the ``M`` (125), ``K``(25) and ``N``(70) -parameters. You can override this and provide your own values on the command line: - -```BASH -docker run --rm -v "$PWD:/work" -w /work armswdev/sme2-learning-path:sme2-environment-v1 ./run-fvp.sh sme2_matmul_asm 7 8 9 -``` - -Here the values ``M=7``, ``K=8`` and ``N=9`` are used instead. -{{% /notice %}} \ No newline at end of file diff --git a/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/6-sme2-matmul-asm.md b/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/6-sme2-matmul-asm.md new file mode 100644 index 0000000000..61c5821116 --- /dev/null +++ b/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/6-sme2-matmul-asm.md @@ -0,0 +1,412 @@ +--- +title: SME2 assembly matrix multiplication +weight: 8 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +In this chapter, you will use an SME2-optimized matrix multiplication written +directly in assembly. + +## Matrix multiplication with SME2 in assembly + +### Description + +This Learning Path reuses the assembly version provided in the [SME Programmer's +Guide](https://developer.arm.com/documentation/109246/0100/matmul-fp32--Single-precision-matrix-by-matrix-multiplication) +where you will find a high-level and an in-depth description of the two steps +performed. + +The assembly versions have been modified so they coexist nicely with +the intrinsic versions. The modifications include: +- let the compiler manage the switching back and forth from streaming mode, +- don't use register `x18` which is used as a platform register. + +In this Learning Path: +- the `preprocess` function is named `preprocess_l_asm` and is defined in + `preprocess_l_asm.S` +- the outer product-based matrix multiplication is named `matmul_asm_impl`and + is defined in `matmul_asm_impl.S`. + +Those 2 functions are declared in `matmul.h`: + +```C +// Matrix preprocessing, in assembly. +void preprocess_l_asm(uint64_t M, uint64_t K, const float *restrict a, + float *restrict a_mod) __arm_streaming __arm_inout("za"); + +// Matrix multiplication (with the *transposed* RHS), in assembly. +void matmul_asm_impl( + uint64_t M, uint64_t K, uint64_t N, const float *restrict matLeft_mod, + const float *restrict matRight, + float *restrict matResult) __arm_streaming __arm_inout("za"); +``` + +You will note that they have been marked with 2 attributes: `__arm_streaming` +and `__arm_inout("za")`. This instructs the compiler that these functions +expect the streaming mode to be active, and that they don't new to save / +restore the ZA storage. + +These two functions are stitched together in `matmul_asm.c` with the +same prototype as the reference implementation of matrix multiplication, so that +a top-level `matmul_asm` can be called from the `main` function: + +```C +__arm_new("za") __arm_locally_streaming void matmul_asm( + uint64_t M, uint64_t K, uint64_t N, const float *restrict matLeft, + const float *restrict matRight, float *restrict matLeft_mod, + float *restrict matResult) { + + preprocess_l_asm(M, K, matLeft, matLeft_mod); + matmul_asm_impl(M, K, N, matLeft_mod, matRight, matResult); +} +``` + +Note that `matmul_asm` has been annotated with 2 attributes: +`__arm_new("za")` and `__arm_locally_streaming`. This instructs the compiler +to swith to streaming mode and save the ZA storage (and restore it when the +function returns). + +The high-level `matmul_asm` function is called from `main.c`. This file might look a bit complex at first sight, but fear not, here are some explanations: +- the same `main.c` is used for the assembly- and intrinsic-based versions of the matrix multiplication --- this is parametrized at compilation time with the `IMPL` macro. This avoids code duplication and improves maintenance. +- on a baremetal platform, the program only works in *verification mode*, where it compares the results of the assembly-based (resp. intrinsic-based) matrix multiplication with the vanilla reference implementation. When targeting a non-baremetal platform, a *benchmarking mode* is also available. + +```C { line_numbers="true" } +#ifndef __ARM_FEATURE_SME2 +#error __ARM_FEATURE_SME2 is not defined +#endif + +#ifndef IMPL +#error matmul implementation selection macro IMPL is not defined +#endif + +#include "matmul.h" +#include "misc.h" + +#include +#include +#include +#include + +#define STRINGIFY_(I) #I +#define STRINGIFY(I) STRINGIFY_(I) +#define FN(M, I) M##I +#define MATMUL(I, M, K, N, mL, mR, mM, m) FN(matmul_, I)(M, K, N, mL, mR, mM, m) + +void usage(const char *prog_name) { +#if BAREMETAL == 1 + printf("Usage: %s \n", prog_name); + printf(" M: number of rows in matLeft (default: 125)\n"); + printf(" K: number of columns in matLeft and matRight (default: 35)\n"); + printf(" N: number of columns in matRight (default: 70)\n"); + printf("Example: matmul 125 35 70\n"); +#else + printf("Depending on the number of arguments, the program can be invoked " + "in two modes:\n"); + printf(" - verification mode. The program will run the assembly or " + "intrinsics implementatation of the matrix multiplication and " + "compare the results with a reference implementation.\n"); + printf(" - benchmarking mode. The program will run the assembly or " + "intrinsics implementation of the matrix multiplication a number of " + "times and print the time taken to perform the operation.\n"); + + printf("\n"); + printf("Verification mode:\n"); + printf(" %s\n", prog_name); + printf(" %s \n", prog_name); + printf("with:\n"); + printf(" - M: number of rows in matLeft (default: 125)\n"); + printf(" - K: number of columns in matLeft and number of rows in matRight " + "(default: 35). Must be > 2 for assembly version of matmul.\n"); + printf(" - N: number of columns in matRight (default: 70)\n"); + printf("Example: %s 67 18 23\n", prog_name); + + printf("\n"); + printf("Benchmarking mode:\n"); + printf(" %s \n", prog_name); + printf(" %s \n", prog_name); + printf("with:\n"); + printf(" - I: number of iterations to perform. Must be > 0.\n"); + printf(" - M: number of rows in matLeft (default: 125)\n"); + printf(" - K: number of columns in matLeft and number of rows in matRight " + "(default: 35). Must be > 2 for assembly version of matmul.\n"); + printf(" - N: number of columns in matRight (default: 70)\n"); + printf("Example: %s 1000 67 18 23\n", prog_name); +#endif +} + +int main(int argc, char **argv) { + + /* Matrices size parameters, defaults to 125x35x70. + Assumptions (for assembly handwritten matmul) are: + - number of rows in matLeft (M): any + - number of columns in matLeft and number of rows in matRight (K): any K > 2 + - number of columns in matRight (N): any + */ + uint64_t I = 0; // Number of iterations to perform for benchmarking. + uint64_t M = 125; // Number of rows in matLeft. + uint64_t N = 35; // Number of columns in matRight. + uint64_t K = 70; // Number of columns (resp. rows) in matLeft (resp. matRight). + + switch (argc) { + case 1: + // Verification mode, with default matrix sizes. + break; +#if BAREMETAL == 0 + case 2: + // Benchmarking mode, with default matrix sizes. + I = strtoull(argv[1], NULL, 0); + if (I == 0) { + printf("Error, in benchmarking mode, I must be > 0.\n"); + return EXIT_FAILURE; + } + break; +#endif + case 4: + // Verification mode, with user-defined matrix sizes. + M = strtoul(argv[1], NULL, 0); + K = strtoul(argv[2], NULL, 0); + N = strtoul(argv[3], NULL, 0); + break; +#if BAREMETAL == 0 + case 5: + // Benchmarking mode, with user-defined matrix sizes. + I = strtoull(argv[1], NULL, 0); + if (I == 0) { + printf("Error, in benchmarking mode, I must be > 0.\n"); + return EXIT_FAILURE; + } + M = strtoul(argv[2], NULL, 0); + K = strtoul(argv[3], NULL, 0); + N = strtoul(argv[4], NULL, 0); + break; +#endif + default: + usage(argv[0]); + return EXIT_FAILURE; + } + + // Check assumptions hold. + if (strcmp(STRINGIFY(IMPL), "asm")==0 && K <= 2) { + printf("Error, for assembly implementation of matmul, K must be > 2.\n"); + return EXIT_FAILURE; + } + + // Describe the operation that will be performed. + printf("SME2 Matrix Multiply fp32 *%s* ", STRINGIFY(IMPL)); + if (I != 0) + printf("[benchmarking mode, %" PRIu64 " iterations] ", I); + else + printf("[verification mode] "); + printf("with M=%" PRIu64 ", K=%" PRIu64 ", N=%" PRIu64 "\n", M, K, N); + +#if BAREMETAL == 1 + setup_sme_baremetal(); +#endif + + const uint64_t SVL = svcntsw(); + + // Calculate M of transformed matLeft. + const uint64_t M_mod = SVL * (M / SVL + (M % SVL != 0 ? 1 : 0)); + + // Allocate memory for all matrices. + float *matRight = (float *)malloc(K * N * sizeof(float)); + + float *matLeft = (float *)malloc(M * K * sizeof(float)); + float *matLeft_mod = (float *)malloc(M_mod * K * sizeof(float)); + float *matLeft_mod_ref = (float *)malloc(M_mod * K * sizeof(float)); + + float *matResult = (float *)malloc(M * N * sizeof(float)); + float *matResult_ref = (float *)malloc(M * N * sizeof(float)); + + // Initialize matrices. Input matrices are initialized with random values in + // non debug mode. In debug mode, all matrices are initialized with linear + // or known values values for easier debugging. +#ifdef DEBUG + initialize_matrix(matLeft, M * K, LINEAR_INIT); + initialize_matrix(matRight, K * N, LINEAR_INIT); + initialize_matrix(matLeft_mod, M_mod * K, DEAD_INIT); + initialize_matrix(matResult, M * N, DEAD_INIT); + + print_matrix(M, K, matLeft, "matLeft"); + print_matrix(K, N, matRight, "matRight"); +#else + initialize_matrix(matLeft, M * K, RANDOM_INIT); + initialize_matrix(matRight, K * N, RANDOM_INIT); +#endif + + unsigned error = 0; + if (I == 0) { + // Verification mode. + MATMUL(IMPL, M, K, N, matLeft, matRight, matLeft_mod, matResult); + + // Compute the reference values with the vanilla implementations. + preprocess_l(M, K, SVL, matLeft, matLeft_mod_ref); + matmul(M, K, N, matLeft, matRight, matResult_ref); + + error = compare_matrices(K, M_mod, matLeft_mod_ref, matLeft_mod, + "Matrix preprocessing"); + if (!error) + error = compare_matrices(M, N, matResult_ref, matResult, + "Matrix multiplication"); + } else { +#if BAREMETAL == 0 + // Benchmarking mode. + uint64_t min_time = UINT64_MAX; + uint64_t max_time = 0; + double sum = 0.0; + + // Warm-up runs to ensure the CPU is ready for benchmarking. + for (uint64_t i = 0; i < 10; i++) + matmul(M, K, N, matLeft, matRight, matResult_ref); + + // Measure the time taken by the matrix multiplication. + for (uint64_t i = 0; i < I; i++) { + const uint64_t start_time = get_time_microseconds(); + matmul(M, K, N, matLeft, matRight, matResult_ref); + const uint64_t elapsed_time = get_time_microseconds() - start_time; + + if (elapsed_time < min_time) + min_time = elapsed_time; + if (elapsed_time > max_time) + max_time = elapsed_time; + sum += elapsed_time; + } + printf("Reference implementation: min time = %" PRIu64 " us, " + "max time = %" PRIu64 " us, avg time = %.2f us\n", + min_time, max_time, sum / I); + + // Benchmarking mode (SME2 implementation). + min_time = UINT64_MAX; + max_time = 0; + sum = 0.0; + + // Warm-up runs to ensure the CPU is ready for benchmarking. + for (uint64_t i = 0; i < 10; i++) + MATMUL(IMPL, M, K, N, matLeft, matRight, matLeft_mod, matResult); + + // Measure the time taken by the SME2 matrix multiplication. + for (uint64_t i = 0; i < I; i++) { + const uint64_t start_time = get_time_microseconds(); + MATMUL(IMPL, M, K, N, matLeft, matRight, matLeft_mod, matResult); + const uint64_t elapsed_time = get_time_microseconds() - start_time; + + if (elapsed_time < min_time) + min_time = elapsed_time; + if (elapsed_time > max_time) + max_time = elapsed_time; + sum += elapsed_time; + } + printf("SME2 implementation *%s*: min time = %" PRIu64 " us, " + "max time = %" PRIu64 " us, avg time = %.2f us\n", + STRINGIFY(IMPL), min_time, max_time, sum / I); +#else + printf("Error, can not run in benchmarking mode in baremetal.\n"); + return EXIT_FAILURE; +#endif + } + + // Free allocated memory. + free(matRight); + + free(matLeft); + free(matLeft_mod); + free(matLeft_mod_ref); + + free(matResult); + free(matResult_ref); + + return error ? EXIT_FAILURE : EXIT_SUCCESS; +} +``` + +The same `main.c` file is used for the assembly and intrinsic-based versions +of the matrix multiplication. It first sets the `M`, `K` and `N` +parameters, to either the arguments supplied on the command line (lines 93-95) +or uses the default value (lines 73-75). In non-baremetal mode, it also accepts +(lines 82-89 and lines 98-108), as first parameter, an iteration count `I` +used for benchmarking. + +Depending on the `M`, `K`, `N` dimension parameters, `main` allocates +memory for all the matrices and initializes `matLeft` and `matRight` with +random data. The actual matrix multiplication implementation is provided through +the `IMPL` macro. + +In *verification mode*, it then runs the matrix multiplication from `IMPL` +(line 167) and computes the reference values for the preprocessed matrix as well +as the result matrix (lines 170 and 171). It then compares the actual values to +the reference values and reports errors, if there are any (lines 173-177). +Finally, all the memory is deallocated (lines 236-243) before exiting the +program with a success or failure return code at line 245. + +In *benchmarking mode*, it will first run the vanilla reference matrix +multiplication (resp. assembly- or intrinsic-based matrix multiplication) 10 +times without measuring elapsed time to warm-up the CPU. It will then measure +the elapsed execution time of the vanilla reference matrix multiplication (resp. +assembly- or intrinsic-based matrix multiplication) `I` times and then compute +and report the minimum, maximum and average execution times. + +{{% notice Note %}} +Benchmarking and profiling are not simple tasks. The purpose of this learning path +is to provide some basic guidelines on the performance improvement that can be +obtained with SME2. +{{% /notice %}} + +### Compile and run it + +First, make sure that the `sme2_matmul_asm` executable is up-to-date: + +{{< tabpane code=true >}} + {{< tab header="Native SME2 support" language="bash" output_lines="2">}} + make sme2_matmul_asm + make: `sme2_matmul_asm' is up to date. + {{< /tab >}} + + {{< tab header="Emulated SME2 support" language="bash" output_lines="2">}} + docker run --rm -v "$PWD:/work" -w /work armswdev/sme2-learning-path:sme2-environment-v2 make sme2_matmul_asm + make: 'sme2_matmul_asm' is up to date. + {{< /tab >}} +{{< /tabpane >}} + +Then execute `sme2_matmul_asm` either natively or on the FVP: + +{{< tabpane code=true >}} + {{< tab header="Native SME2 support" language="bash" output_lines="2-4">}} + ./sme2_matmul_asm + SME2 Matrix Multiply fp32 *asm* [verification mode] with M=125, K=70, N=35 + Matrix preprocessing: PASS ! + Matrix multiplication: PASS ! + {{< /tab >}} + + {{< tab header="Emulated SME2 support" language="bash" output_lines="2-6">}} + docker run --rm -v "$PWD:/work" -w /work armswdev/sme2-learning-path:sme2-environment-v2 ./run-fvp.sh sme2_matmul_asm + SME2 Matrix Multiply fp32 *asm* [verification mode] with M=125, K=70, N=35 + Matrix preprocessing: PASS ! + Matrix multiplication: PASS ! + + Info: /OSCI/SystemC: Simulation stopped by user. + {{< /tab >}} +{{< /tabpane >}} + +`sme2_matmul_asm` prints the version of the matrix multiplication performed +(asm or intr) as well as the `M`, `K` and `N` parameters. It also prints +whether the preprocessing and matrix multiplication passed (`PASS`) or failed +(`FAILED`) the comparison the vanilla reference implementation. + +{{% notice Tip %}} +The example above uses the default values for the `M` (125), `K`(25) and `N`(70) +parameters. You can override this and provide your own values on the command line: + +{{< tabpane code=true >}} + {{< tab header="Native SME2 support" language="bash">}} + ./sme2_matmul_asm 7 8 9 + {{< /tab >}} + + {{< tab header="Emulated SME2 support" language="bash">}} + docker run --rm -v "$PWD:/work" -w /work armswdev/sme2-learning-path:sme2-environment-v2 ./run-fvp.sh sme2_matmul_asm 7 8 9 + {{< /tab >}} +{{< /tabpane >}} + +Here the values `M=7`, `K=8` and `N=9` are used instead. +{{% /notice %}} \ No newline at end of file diff --git a/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/6-sme2-matmul-intr.md b/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/6-sme2-matmul-intr.md deleted file mode 100644 index 413f54bf48..0000000000 --- a/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/6-sme2-matmul-intr.md +++ /dev/null @@ -1,344 +0,0 @@ ---- -title: SME2 intrinsics matrix multiplication -weight: 8 - -### FIXED, DO NOT MODIFY -layout: learningpathall ---- -## Matrix multiplication with SME2 intrinsics - -In this section, you will write an SME2 optimized matrix multiplication in C using the intrinsics that the compiler provides. - -*Intrinsics*, also know known as *compiler intrinsics* or *intrinsic functions*, are the functions available to application developers that the compiler has an -intimate knowledge of. This enables the compiler to either translate the function to a specific instruction or to perform specific optimizations, or both. - -You can learn more about intrinsics in this [Wikipedia -Article on Intrinsic Function](https://en.wikipedia.org/wiki/Intrinsic_function). - -Using intrinsics allows the programmer to use the specific instructions -required to achieve the required performance while writing in C all the typically-required standard code, such as loops. This produces performance close to what can be reached with hand-written assembly whilst being significantly more maintainable and portable. - -All Arm-specific intrinsics are specified in the -[ACLE](https://github.com/ARM-software/acle), which is the Arm C Language Extension. ACLE -is supported by the main compilers, most notably [GCC](https://gcc.gnu.org/) and -[Clang](https://clang.llvm.org). - -## Streaming mode - -On the previous page, assembly language provided the programmer with full access to processor features. However, this comes at the cost of increased complexity and maintenance, particularly when managing large codebases with deeply nested function calls. Additionally, the assembly version operates at a very low level and does not fully handle the SME state. - -In real-world large-scale software, the program moves back and forth from streaming mode, and some streaming mode routines call other streaming mode routines, which means that some state needs to be saved and restored. This includes the ZA storage. This is defined in the ACLE and -supported by the compiler: the programmer *just* has to annotate the function -with some keywords and set up some registers (see function ``setup_sme`` in -``misc.c`` for an example). See [Introduction to streaming and non-streaming mode](https://arm-software.github.io/acle/main/acle.html#controlling-the-use-of-streaming-mode) -for further information. The rest of this section references information from the ACLE. - -The AArch64 architecture defines a concept called *streaming mode*, controlled -by a processor state bit called ``PSTATE.SM``. At any given point in time, the -processor is either in streaming mode (``PSTATE.SM==1``) or in non-streaming mode -(``PSTATE.SM==0``). There is an instruction called ``SMSTART`` to enter streaming mode -and an instruction called ``SMSTOP`` to return to non-streaming mode. - -Streaming mode has three main effects on C and C++ code: - -- It can change the length of SVE vectors and predicates: the length of an SVE - vector in streaming mode is called the “streaming vector length” (SVL), which - might be different from the normal non-streaming vector length. See - [Effect of streaming mode on VL](https://arm-software.github.io/acle/main/acle.html#effect-of-streaming-mode-on-vl) - for more details. -- Some instructions can only be executed in streaming mode, which means that - their associated ACLE intrinsics can only be used in streaming mode. These - intrinsics are called “streaming intrinsics”. -- Some other instructions can only be executed in non-streaming mode, which - means that their associated ACLE intrinsics can only be used in non-streaming - mode. These intrinsics are called “non-streaming intrinsics”. - -The C and C++ standards define the behavior of programs in terms of an *abstract -machine*. As an extension, the ACLE specification applies the distinction -between streaming mode and non-streaming mode to this abstract machine: at any -given point in time, the abstract machine is either in streaming mode or in -non-streaming mode. - -This distinction between processor mode and abstract machine mode is mostly just -a specification detail. However, the usual “as if” rule applies: the -processor's actual mode at runtime can be different from the abstract machine's -mode, provided that this does not alter the behavior of the program. One -practical consequence of this is that C and C++ code does not specify the exact -placement of ``SMSTART`` and ``SMSTOP`` instructions; the source code simply places -limits on where such instructions go. For example, when stepping through a -program in a debugger, the processor mode might sometimes be different from the -one implied by the source code. - -ACLE provides attributes that specify whether the abstract machine executes statements: - -- In non-streaming mode, in which case they are called *non-streaming statements*. -- In streaming mode, in which case they are called *streaming statements*. -- In either mode, in which case they are called *streaming-compatible statements*. - -SME provides an area of storage called ZA, of size ``SVL.B`` x ``SVL.B`` bytes. It -also provides a processor state bit called ``PSTATE.ZA`` to control whether ZA -is enabled. - -In C and C++ code, access to ZA is controlled at function granularity: a -function either uses ZA or it does not. Another way to say this is that a -function either “has ZA state” or it does not. - -If a function does have ZA state, the function can either share that ZA state -with the function's caller or create new ZA state “from scratch”. In the latter -case, it is the compiler's responsibility to free up ZA so that the function can -use it; see the description of the lazy saving scheme in -[AAPCS64](https://arm-software.github.io/acle/main/acle.html#AAPCS64) for details -about how the compiler does this. - -## Implementation - -Here again, a top level function named ``matmul_intr`` in ``matmul_intr.c`` -will be used to stitch together the preprocessing and the multiplication: - -```C "{ line_numbers = true }" -__arm_new("za") __arm_locally_streaming void matmul_intr( - uint64_t M, uint64_t K, uint64_t N, const float *restrict matLeft, - const float *restrict matRight, float *restrict matLeft_mod, - float *restrict matResult) { - uint64_t SVL = svcntsw(); - preprocess_l_intr(M, K, SVL, matLeft, matLeft_mod); - matmul_intr_impl(M, K, N, SVL, matLeft_mod, matRight, matResult); -} -``` - -Note the ``__arm_new("za")`` and ``__arm_locally_streaming`` at line 1 that will -make the compiler save the ZA storage so we can use it without destroying its -content if it was still in use by one of the callers. - -``SVL``, the dimension of the ZA storage, is requested from the underlying -hardware with the ``svcntsw()`` function call at line 5, and passed down to the -``preprocess_l_intr`` and ``matmul_intr_impl`` functions. ``svcntsw()`` is a -function provided be the ACLE library. - -### Matrix preprocessing - -```C "{ line_numbers = true }" -void preprocess_l_intr( - uint64_t M, uint64_t K, uint64_t SVL, const float *restrict a, - float *restrict a_mod) __arm_streaming __arm_inout("za") { - const uint64_t M_mod = SVL * (M / SVL + (M % SVL != 0 ? 1 : 0)); - - // The outer loop, iterating over rows (M dimension) - for (uint64_t row = 0; row < M; row += SVL) { - - svbool_t pMDim = svwhilelt_b32(row, M); - - // The inner loop, iterating on columns (K dimension). - for (uint64_t col = 0; col < K; col += 2 * SVL) { - - svcount_t pKDim = svwhilelt_c32(col, K, 2); - - // Load-as-rows - for (uint64_t trow = 0; trow < SVL; trow += 4) { - svcount_t p0 = svpsel_lane_c32(pKDim, pMDim, trow + 0); - svcount_t p1 = svpsel_lane_c32(pKDim, pMDim, trow + 1); - svcount_t p2 = svpsel_lane_c32(pKDim, pMDim, trow + 2); - svcount_t p3 = svpsel_lane_c32(pKDim, pMDim, trow + 3); - - const uint64_t tile_UL_corner = (row + trow) * K + col; - svfloat32x2_t zp0 = svld1_x2(p0, &a[tile_UL_corner + 0 * K]); - svfloat32x2_t zp1 = svld1_x2(p1, &a[tile_UL_corner + 1 * K]); - svfloat32x2_t zp2 = svld1_x2(p2, &a[tile_UL_corner + 2 * K]); - svfloat32x2_t zp3 = svld1_x2(p3, &a[tile_UL_corner + 3 * K]); - - svfloat32x4_t zq0 = svcreate4(svget2(zp0, 0), svget2(zp1, 0), - svget2(zp2, 0), svget2(zp3, 0)); - svfloat32x4_t zq1 = svcreate4(svget2(zp0, 1), svget2(zp1, 1), - svget2(zp2, 1), svget2(zp3, 1)); - svwrite_hor_za32_f32_vg4( - /* tile: */ 0, /* slice: */ trow, zq0); - svwrite_hor_za32_f32_vg4( - /* tile: */ 1, /* slice: */ trow, zq1); - } - - // Read-as-columns and store - const uint64_t dest_0 = row * K + col * SVL; - const uint64_t dest_1 = dest_0 + SVL * SVL; - for (uint64_t tcol = 0; tcol < SVL; tcol += 4) { - svcount_t p0 = svwhilelt_c32(dest_0 + tcol * SVL, K * M_mod, 4); - svcount_t p1 = svwhilelt_c32(dest_1 + tcol * SVL, K * M_mod, 4); - svfloat32x4_t zq0 = - svread_ver_za32_f32_vg4(/* tile: */ 0, /* slice: */ tcol); - svfloat32x4_t zq1 = - svread_ver_za32_f32_vg4(/* tile: */ 1, /* slice: */ tcol); - svst1(p0, &a_mod[dest_0 + tcol * SVL], zq0); - svst1(p1, &a_mod[dest_1 + tcol * SVL], zq1); - } - } - } -} -``` - -Note that ``preprocess_l_intr`` has been annotated at line 3 with: - -- ``__arm_streaming``, because this function is using streaming instructions, - -- ``__arm_inout("za")``, because ``preprocess_l_intr`` reuses the ZA storage - from its caller. - -The matrix preprocessing is performed in a double nested loop, over the ``M`` -(line 7) and ``K`` (line 12) dimensions of the input matrix ``a``. Both loops -have an ``SVL`` step increment, which corresponds to the horizontal and vertical -dimensions of the ZA storage that will be used. The dimensions of ``a`` may not -be perfect multiples of ``SVL`` though... which is why the predicates ``pMDim`` -(line 9) and ``pKDim`` (line 14) are computed in order to know which rows (respectively -columns) are valid. - -The core of ``preprocess_l_intr`` is made of two parts: - -- Lines 17 - 37: load matrix tile as rows. In this part, loop unrolling has been - used at 2 different levels. At the lowest level, 4 rows are loaded at a time - (lines 24-27). But this goes much further because as SME2 has multi-vectors - operations (hence the ``svld1_x2`` intrinsic to load 2 rows in 2 vector - registers), this allows the function to load the consecutive row, which - happens to be the row from the neighboring tile on the right: this means two - tiles are processed at once. At line 29-32, the pairs of vector registers are - rearranged on quads of vector registers so they can be stored horizontally in - the two tiles' ZA storage at lines 33-36 with the ``svwrite_hor_za32_f32_vg4`` - intrinsic. Of course, as the input matrix may not have dimensions that are - perfect multiples of ``SVL``, the ``p0``, ``p1``, ``p2`` and ``p3`` predicates - are computed with the ``svpsel_lane_c32`` intrinsic (lines 18-21) so that - elements outside of the input matrix are set to 0 when they are loaded at - lines 24-27. - -- Lines 39 - 51: read the matrix tile as columns and store them. Now that the 2 - tiles have been loaded *horizontally*, they will be read *vertically* with the - ``svread_ver_za32_f32_vg4`` intrinsic to quad-registers of vectors (``zq0`` - and ``zq1``) at lines 45-48 and then stored with the ``svst1`` intrinsic to - the relevant location in the destination matrix ``a_mod`` (lines 49-50). Note - again the usage of predicates ``p0`` and ``p1`` (computed at lines 43-44) to - ``svst1`` to prevent writing out of the matrix bounds. - -Using intrinsics simplifies function development significantly, provided one has a good understanding of the SME2 instruction set. -Predicates, which are fundamental to SVE and SME, enable a natural expression of algorithms while handling corner cases efficiently. -Notably, there is no explicit condition checking within the loops to account for rows or columns extending beyond matrix bounds. - -### Outer-product multiplication - -```C "{ line_numbers = true }" -void matmul_intr_impl( - uint64_t M, uint64_t K, uint64_t N, uint64_t SVL, - const float *restrict matLeft_mod, const float *restrict matRight, - float *restrict matResult) __arm_streaming __arm_inout("za") { - - // Build the result matrix tile by tile. - for (uint64_t row = 0; row < M; row += SVL) { - - svbool_t pMDim = svwhilelt_b32(row, M); - - for (uint64_t col = 0; col < N; col += SVL) { - - svbool_t pNDim = svwhilelt_b32(col, N); - - // Outer product + accumulation - svzero_za(); - const uint64_t matLeft_pos = row * K; - const uint64_t matRight_UL_corner = col; - for (uint64_t k = 0; k < K; k++) { - svfloat32_t zL = - svld1(pMDim, &matLeft_mod[matLeft_pos + k * SVL]); - svfloat32_t zR = - svld1(pNDim, &matRight[matRight_UL_corner + k * N]); - svmopa_za32_m(0, pMDim, pNDim, zL, zR); - } - - // Store ZA to matResult. - const uint64_t result_tile_UL_corner = row * N + col; - for (uint64_t trow = 0; trow < SVL && row + trow < M; trow += 4) { - svbool_t p0 = svpsel_lane_b32(pNDim, pMDim, row + trow + 0); - svbool_t p1 = svpsel_lane_b32(pNDim, pMDim, row + trow + 1); - svbool_t p2 = svpsel_lane_b32(pNDim, pMDim, row + trow + 2); - svbool_t p3 = svpsel_lane_b32(pNDim, pMDim, row + trow + 3); - - svst1_hor_za32( - /* tile: */ 0, /* slice: */ trow + 0, p0, - &matResult[result_tile_UL_corner + (trow + 0) * N]); - svst1_hor_za32( - /* tile: */ 0, /* slice: */ trow + 1, p1, - &matResult[result_tile_UL_corner + (trow + 1) * N]); - svst1_hor_za32( - /* tile: */ 0, /* slice: */ trow + 2, p2, - &matResult[result_tile_UL_corner + (trow + 2) * N]); - svst1_hor_za32( - /* tile: */ 0, /* slice: */ trow + 3, p3, - &matResult[result_tile_UL_corner + (trow + 3) * N]); - } - } - } -} -``` - -Note again that ``matmul_intr_impl`` function has been annotated at line 4 with: - -- ``__arm_streaming``, because the function is using streaming instructions, - -- ``__arm_inout("za")``, because the function reuses the ZA storage from its caller. - -The multiplication with the outer product is performed in a double-nested loop, -over the ``M`` (line 7) and ``N`` (line 11) dimensions of the input matrices -``matLeft_mod`` and ``matRight``. Both loops have an ``SVL`` step increment, -which corresponds to the horizontal and vertical dimensions of the ZA storage -that will be used as one tile at a time will be processed. The ``M`` and ``N`` -dimensions of the inputs may not be perfect multiples of ``SVL`` so the -predicates ``pMDim`` (line 9) (respectively ``pNDim`` at line 13) are computed in order -to know which rows (respectively columns) are valid. - -The core of the multiplication is done in 2 parts: - -- Outer-product and accumulation at lines 15-25. As ``matLeft`` has been - laid-out perfectly in memory with ``preprocess_l_intr``, this part becomes - straightforward. First, the tile is zeroed with the ``svzero_za`` intrinsics - at line 16 so the outer products can be accumulated in the tile. The outer - products are computed and accumulation over the ``K`` common dimension with - the loop at line 19: the column of ``matleft_mod`` and the row of ``matRight`` - are loaded with the ``svld1`` intrinsics at line 20-23 to vector registers - ``zL`` and ``zR``, which are then used at line 24 with the ``svmopa_za32_m`` - intrinsic to perform the outer product and accumulation (to tile 0). This - is exactly what was shown in Figure 2 earlier in the Learning Path. - Note again the usage of the ``pMDim`` and ``pNDim`` predicates to deal - correctly with the rows and columns respectively which are out of bounds. - -- Storing of the result matrix at lines 27-46. The previous section computed the matrix multiplication result for the current tile, which now needs - to be written back to memory. This is done with the loop at line 29 which will - iterate over all rows of the tile: the ``svst1_hor_za32`` intrinsic at lines - 35-46 stores directly from the tile to memory. Note that the loop has been - unrolled by a factor of 4 (thus the ``trow += 4`` increment, line 29) and the - 4 ``svst1_hor_za32``. Again, the ``pMDim`` and ``pNDim`` predicates deal - gracefully with the parts of the tile which are out-of-bound for the - destination matrix ``matResult``. - -Once again, intrinsics makes it easy to fully leverage SME2, provided you have a solid understanding of its available instructions. -Predicates handle corner cases elegantly, ensuring robust execution. Most importantly, the code adapts to different SVL values across various hardware implementations without requiring recompilation. -This follows the key principle of compile-once, run-everywhere, allowing systems with larger SVL to execute computations more efficiently while using the same binary. - -### Compile and run - -The main function is exactly the same that was used for the assembly version, -with the ``IMPL`` macro defined to be ``intr`` in the ``Makefile``. - -First, make sure that the ``sme2_matmul_intr`` executable is up-to-date: - -```BASH -docker run --rm -v "$PWD:/work" -w /work armswdev/sme2-learning-path:sme2-environment-v1 make sme2_matmul_intr -``` - -Then execute ``sme2_matmul_intr`` on the FVP: - -```BASH -docker run --rm -v "$PWD:/work" -w /work armswdev/sme2-learning-path:sme2-environment-v1 ./run-fvp.sh sme2_matmul_intr -``` - -This should output something similar to: - -```TXT -SME2 Matrix Multiply fp32 *intr* example with args 125 35 70 -Matrix preprocessing: PASS ! -Matrix multiplication: PASS ! - -Info: /OSCI/SystemC: Simulation stopped by user. -``` diff --git a/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/7-sme2-matmul-intr.md b/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/7-sme2-matmul-intr.md new file mode 100644 index 0000000000..a170de702d --- /dev/null +++ b/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/7-sme2-matmul-intr.md @@ -0,0 +1,307 @@ +--- +title: SME2 intrinsics matrix multiplication +weight: 9 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +In this section, you will write an SME2 optimized matrix multiplication in C +using the intrinsics that the compiler provides. + +## Matrix multiplication with SME2 intrinsics + +*Intrinsics*, also know known as *compiler intrinsics* or *intrinsic functions*, +are the functions available to application developers that the compiler has an +intimate knowledge of. This enables the compiler to either translate the +function to a specific instruction or to perform specific optimizations, or +both. + +You can learn more about intrinsics in this [Wikipedia +Article on Intrinsic Function](https://en.wikipedia.org/wiki/Intrinsic_function). + +Using intrinsics allows the programmer to use the specific instructions required +to achieve the required performance while writing in C all the +typically-required standard code, such as loops. This produces performance close +to what can be reached with hand-written assembly whilst being significantly +more maintainable and portable. + +All Arm-specific intrinsics are specified in the +[ACLE](https://github.com/ARM-software/acle), which is the Arm C Language Extension. ACLE +is supported by the main compilers, most notably [GCC](https://gcc.gnu.org/) and +[Clang](https://clang.llvm.org). + +## Implementation + +Here again, a top level function named `matmul_intr` in `matmul_intr.c` +will be used to stitch together the preprocessing and the multiplication: + +```C "{ line_numbers = true }" +__arm_new("za") __arm_locally_streaming void matmul_intr( + uint64_t M, uint64_t K, uint64_t N, const float *restrict matLeft, + const float *restrict matRight, float *restrict matLeft_mod, + float *restrict matResult) { + uint64_t SVL = svcntsw(); + preprocess_l_intr(M, K, SVL, matLeft, matLeft_mod); + matmul_intr_impl(M, K, N, SVL, matLeft_mod, matRight, matResult); +} +``` + +Note the `__arm_new("za")` and `__arm_locally_streaming` at line 1 that will +make the compiler save the ZA storage so we can use it without destroying its +content if it was still in use by one of the callers. + +`SVL`, the dimension of the ZA storage, is requested from the underlying +hardware with the `svcntsw()` function call at line 5, and passed down to the +`preprocess_l_intr` and `matmul_intr_impl` functions. `svcntsw()` is a +function provided be the ACLE library. + +### Matrix preprocessing + +```C "{ line_numbers = true }" +void preprocess_l_intr( + uint64_t M, uint64_t K, uint64_t SVL, const float *restrict a, + float *restrict a_mod) __arm_streaming __arm_inout("za") { + const uint64_t M_mod = SVL * (M / SVL + (M % SVL != 0 ? 1 : 0)); + + // The outer loop, iterating over rows (M dimension) + for (uint64_t row = 0; row < M; row += SVL) { + + svbool_t pMDim = svwhilelt_b32(row, M); + + // The inner loop, iterating on columns (K dimension). + for (uint64_t col = 0; col < K; col += 2 * SVL) { + + svcount_t pKDim = svwhilelt_c32(col, K, 2); + + // Load-as-rows + for (uint64_t trow = 0; trow < SVL; trow += 4) { + svcount_t p0 = svpsel_lane_c32(pKDim, pMDim, trow + 0); + svcount_t p1 = svpsel_lane_c32(pKDim, pMDim, trow + 1); + svcount_t p2 = svpsel_lane_c32(pKDim, pMDim, trow + 2); + svcount_t p3 = svpsel_lane_c32(pKDim, pMDim, trow + 3); + + const uint64_t tile_UL_corner = (row + trow) * K + col; + svfloat32x2_t zp0 = svld1_x2(p0, &a[tile_UL_corner + 0 * K]); + svfloat32x2_t zp1 = svld1_x2(p1, &a[tile_UL_corner + 1 * K]); + svfloat32x2_t zp2 = svld1_x2(p2, &a[tile_UL_corner + 2 * K]); + svfloat32x2_t zp3 = svld1_x2(p3, &a[tile_UL_corner + 3 * K]); + + svfloat32x4_t zq0 = svcreate4(svget2(zp0, 0), svget2(zp1, 0), + svget2(zp2, 0), svget2(zp3, 0)); + svfloat32x4_t zq1 = svcreate4(svget2(zp0, 1), svget2(zp1, 1), + svget2(zp2, 1), svget2(zp3, 1)); + svwrite_hor_za32_f32_vg4( + /* tile: */ 0, /* slice: */ trow, zq0); + svwrite_hor_za32_f32_vg4( + /* tile: */ 1, /* slice: */ trow, zq1); + } + + // Read-as-columns and store + const uint64_t dest_0 = row * K + col * SVL; + const uint64_t dest_1 = dest_0 + SVL * SVL; + for (uint64_t tcol = 0; tcol < SVL; tcol += 4) { + svcount_t p0 = svwhilelt_c32(dest_0 + tcol * SVL, K * M_mod, 4); + svcount_t p1 = svwhilelt_c32(dest_1 + tcol * SVL, K * M_mod, 4); + svfloat32x4_t zq0 = + svread_ver_za32_f32_vg4(/* tile: */ 0, /* slice: */ tcol); + svfloat32x4_t zq1 = + svread_ver_za32_f32_vg4(/* tile: */ 1, /* slice: */ tcol); + svst1(p0, &a_mod[dest_0 + tcol * SVL], zq0); + svst1(p1, &a_mod[dest_1 + tcol * SVL], zq1); + } + } + } +} +``` + +Note that `preprocess_l_intr` has been annotated at line 3 with: + +- `__arm_streaming`, because this function is using streaming instructions, + +- `__arm_inout("za")`, because `preprocess_l_intr` reuses the ZA storage + from its caller. + +The matrix preprocessing is performed in a double nested loop, over the `M` +(line 7) and `K` (line 12) dimensions of the input matrix `a`. Both loops +have an `SVL` step increment, which corresponds to the horizontal and vertical +dimensions of the ZA storage that will be used. The dimensions of `a` may not +be perfect multiples of `SVL` though... which is why the predicates `pMDim` +(line 9) and `pKDim` (line 14) are computed in order to know which rows +(respectively columns) are valid. + +The core of `preprocess_l_intr` is made of two parts: + +- Lines 17 - 37: load matrix tile as rows. In this part, loop unrolling has been + used at 2 different levels. At the lowest level, 4 rows are loaded at a time + (lines 24-27). But this goes much further because as SME2 has multi-vectors + operations (hence the `svld1_x2` intrinsic to load 2 rows in 2 vector + registers), this allows the function to load the consecutive row, which + happens to be the row from the neighboring tile on the right: this means two + tiles are processed at once. At line 29-32, the pairs of vector registers are + rearranged on quads of vector registers so they can be stored horizontally in + the two tiles' ZA storage at lines 33-36 with the `svwrite_hor_za32_f32_vg4` + intrinsic. Of course, as the input matrix may not have dimensions that are + perfect multiples of `SVL`, the `p0`, `p1`, `p2` and `p3` predicates + are computed with the `svpsel_lane_c32` intrinsic (lines 18-21) so that + elements outside of the input matrix are set to 0 when they are loaded at + lines 24-27. + +- Lines 39 - 51: read the matrix tile as columns and store them. Now that the 2 + tiles have been loaded *horizontally*, they will be read *vertically* with the + `svread_ver_za32_f32_vg4` intrinsic to quad-registers of vectors (`zq0` + and `zq1`) at lines 45-48 and then stored with the `svst1` intrinsic to + the relevant location in the destination matrix `a_mod` (lines 49-50). Note + again the usage of predicates `p0` and `p1` (computed at lines 43-44) to + `svst1` to prevent writing out of the matrix bounds. + +Using intrinsics simplifies function development significantly, provided one has +a good understanding of the SME2 instruction set. Predicates, which are +fundamental to SVE and SME, enable a natural expression of algorithms while +handling corner cases efficiently. Notably, there is no explicit condition +checking within the loops to account for rows or columns extending beyond matrix +bounds. + +### Outer-product multiplication + +```C "{ line_numbers = true }" +void matmul_intr_impl( + uint64_t M, uint64_t K, uint64_t N, uint64_t SVL, + const float *restrict matLeft_mod, const float *restrict matRight, + float *restrict matResult) __arm_streaming __arm_inout("za") { + + // Build the result matrix tile by tile. + for (uint64_t row = 0; row < M; row += SVL) { + + svbool_t pMDim = svwhilelt_b32(row, M); + + for (uint64_t col = 0; col < N; col += SVL) { + + svbool_t pNDim = svwhilelt_b32(col, N); + + // Outer product + accumulation + svzero_za(); + const uint64_t matLeft_pos = row * K; + const uint64_t matRight_UL_corner = col; + for (uint64_t k = 0; k < K; k++) { + svfloat32_t zL = + svld1(pMDim, &matLeft_mod[matLeft_pos + k * SVL]); + svfloat32_t zR = + svld1(pNDim, &matRight[matRight_UL_corner + k * N]); + svmopa_za32_m(0, pMDim, pNDim, zL, zR); + } + + // Store ZA to matResult. + const uint64_t result_tile_UL_corner = row * N + col; + for (uint64_t trow = 0; trow < SVL && row + trow < M; trow += 4) { + svbool_t p0 = svpsel_lane_b32(pNDim, pMDim, row + trow + 0); + svbool_t p1 = svpsel_lane_b32(pNDim, pMDim, row + trow + 1); + svbool_t p2 = svpsel_lane_b32(pNDim, pMDim, row + trow + 2); + svbool_t p3 = svpsel_lane_b32(pNDim, pMDim, row + trow + 3); + + svst1_hor_za32( + /* tile: */ 0, /* slice: */ trow + 0, p0, + &matResult[result_tile_UL_corner + (trow + 0) * N]); + svst1_hor_za32( + /* tile: */ 0, /* slice: */ trow + 1, p1, + &matResult[result_tile_UL_corner + (trow + 1) * N]); + svst1_hor_za32( + /* tile: */ 0, /* slice: */ trow + 2, p2, + &matResult[result_tile_UL_corner + (trow + 2) * N]); + svst1_hor_za32( + /* tile: */ 0, /* slice: */ trow + 3, p3, + &matResult[result_tile_UL_corner + (trow + 3) * N]); + } + } + } +} +``` + +Note again that `matmul_intr_impl` function has been annotated at line 4 with: + +- `__arm_streaming`, because the function is using streaming instructions, + +- `__arm_inout("za")`, because the function reuses the ZA storage from its caller. + +The multiplication with the outer product is performed in a double-nested loop, +over the `M` (line 7) and `N` (line 11) dimensions of the input matrices +`matLeft_mod` and `matRight`. Both loops have an `SVL` step increment, +which corresponds to the horizontal and vertical dimensions of the ZA storage +that will be used as one tile at a time will be processed. The `M` and `N` +dimensions of the inputs may not be perfect multiples of `SVL` so the +predicates `pMDim` (line 9) (respectively `pNDim` at line 13) are computed in order +to know which rows (respectively columns) are valid. + +The core of the multiplication is done in 2 parts: + +- Outer-product and accumulation at lines 15-25. As `matLeft` has been + laid-out perfectly in memory with `preprocess_l_intr`, this part becomes + straightforward. First, the tile is zeroed with the `svzero_za` intrinsics + at line 16 so the outer products can be accumulated in the tile. The outer + products are computed and accumulation over the `K` common dimension with + the loop at line 19: the column of `matleft_mod` and the row of `matRight` + are loaded with the `svld1` intrinsics at line 20-23 to vector registers + `zL` and `zR`, which are then used at line 24 with the `svmopa_za32_m` + intrinsic to perform the outer product and accumulation (to tile 0). This + is exactly what was shown in Figure 2 earlier in the Learning Path. + Note again the usage of the `pMDim` and `pNDim` predicates to deal + correctly with the rows and columns respectively which are out of bounds. + +- Storing of the result matrix at lines 27-46. The previous section computed the + matrix multiplication result for the current tile, which now needs to be + written back to memory. This is done with the loop at line 29 which will + iterate over all rows of the tile: the `svst1_hor_za32` intrinsic at lines + 35-46 stores directly from the tile to memory. Note that the loop has been + unrolled by a factor of 4 (thus the `trow += 4` increment, line 29) and the + 4 `svst1_hor_za32`. Again, the `pMDim` and `pNDim` predicates deal + gracefully with the parts of the tile which are out-of-bound for the + destination matrix `matResult`. + +Once again, intrinsics makes it easy to fully leverage SME2, provided you have a +solid understanding of its available instructions. The compiler is automatically +handling many low-level aspects (saving / restoring of the difeerent contexts), +as well as not using registers that are reserved on specific plaforms (like +`x18`). Predicates handle corner cases elegantly, ensuring robust execution. +Most importantly, the code adapts to different SVL values across various +hardware implementations without requiring recompilation. This follows the key +principle of compile-once, run-everywhere, allowing systems with larger SVL to +execute computations more efficiently while using the same binary. + +### Compile and run + +The main function is exactly the same that was used for the assembly version, +with the `IMPL` macro defined to be `intr` in the `Makefile`. + +First, make sure that the `sme2_matmul_intr` executable is up-to-date: + +{{< tabpane code=true >}} + {{< tab header="Native SME2 support" language="bash" output_lines="2">}} + make sme2_matmul_intr + make: `sme2_matmul_intr' is up to date. + {{< /tab >}} + + {{< tab header="Emulated SME2 support" language="bash" output_lines="2">}} + docker run --rm -v "$PWD:/work" -w /work armswdev/sme2-learning-path:sme2-environment-v2 make sme2_matmul_intr + make: 'sme2_matmul_intr' is up to date. + {{< /tab >}} +{{< /tabpane >}} + +Then execute `sme2_matmul_intr` either natively or on the FVP: + +{{< tabpane code=true >}} + {{< tab header="Native SME2 support" language="bash" output_lines="2-4">}} + ./sme2_matmul_intr + SME2 Matrix Multiply fp32 *intr* [verification mode] with M=125, K=70, N=35 + Matrix preprocessing: PASS ! + Matrix multiplication: PASS ! + {{< /tab >}} + + {{< tab header="Emulated SME2 support" language="bash" output_lines="2-6">}} + docker run --rm -v "$PWD:/work" -w /work armswdev/sme2-learning-path:sme2-environment-v2 ./run-fvp.sh sme2_matmul_intr + SME2 Matrix Multiply fp32 *intr* [verification mode] with M=125, K=70, N=35 + Matrix preprocessing: PASS ! + Matrix multiplication: PASS ! + + Info: /OSCI/SystemC: Simulation stopped by user. + {{< /tab >}} +{{< /tabpane >}} diff --git a/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/8-benchmarking.md b/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/8-benchmarking.md new file mode 100644 index 0000000000..c38a7d3a94 --- /dev/null +++ b/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/8-benchmarking.md @@ -0,0 +1,94 @@ +--- +title: Benchmarking +weight: 10 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +In this section, if your machine supports native execution of SME2 instructions, +you will perform benchmarking of the matrix multiplication improvement thanks to +SME2. + +## About benchmarking and emulation + +Emulation is generally not the best way to assess the performance of a piece of +code. Emulation focuses on correctly simulating instructions and leaves out many +details necessary for precise execution time measurement. For example, as +explained in the section on the outer product, the goal was to increase the +`macc` to `load` ratio. Emulators, including the FVP, do not model in detail the +cache effects or the timing effects of the memory accesses. At best, an emulator +can provide an instruction count for the vanilla reference implementation versus +the assembly-/intrinsic-based versions of the matrix multiplication, but this is +known to be a poor proxy for execution time comparisons. + +## Benchmarking on platform with native SME2 support + +{{% notice Note %}} +Benchmarking and profiling are not simple tasks. The purpose of this learning path +is to provide some basic guidelines on the performance improvement that can be +obtained with SME2. +{{% /notice %}} + +If your machine natively supports SME2, then benchmarking becomes possible. When +`sme2_matmul_asm` and `sme2_matmul_intr` were compiled with `BAREMETAL=0`, the +*benchmarking mode* becomes available. + +*Benchmarking mode* is enabled by prepending the `M`, `K`, `N` optional +parameters with an iteration count (`I`). + +Now measure the execution time of `sme2_matmul_intr` for 1000 multiplications of +matrices with the default sizes: + +```BASH { output_lines="2-4"} +./sme2_matmul_intr 1000 +SME2 Matrix Multiply fp32 *intr* [benchmarking mode, 1000 iterations] with M=125, K=70, N=35 +Reference implementation: min time = 101 us, max time = 438 us, avg time = 139.42 us +SME2 implementation *intr*: min time = 1 us, max time = 8 us, avg time = 1.82 us +``` + +The execution time is reported in microseconds. A wide spread between the +minimum and maximum figures can be noted and is expected as the way of doing the +benchmarking is simplified for the purpose of simplicity. You will, however, +note that the intrinsic version of the matrix multiplication brings on average a +76x execution time reduction. + +{{% notice Tip %}} +You can override the default values for `M` (125), `K` (25), and `N` (70) and +provide your own values on the command line. For example, you can benchmark the +`M=7`, `K=8`, and `N=9` case with: + +```BASH { output_lines="2-4"} +./sme2_matmul_intr 1000 7 8 9 +SME2 Matrix Multiply fp32 *intr* [benchmarking mode, 1000 iterations] with M=7, K=8, N=9 +Reference implementation: min time = 0 us, max time = 14 us, avg time = 0.93 us +SME2 implementation *intr*: min time = 0 us, max time = 1 us, avg time = 0.61 us +``` +{{% /notice %}} + +Now measure the execution time of `sme2_matmul_asm` for 1000 multiplications of +matrices with the default sizes: + +```BASH { output_lines="2-4"} +./sme2_matmul_asm 1000 +SME2 Matrix Multiply fp32 *asm* [benchmarking mode, 1000 iterations] with M=125, K=70, N=35 +Reference implementation: min time = 101 us, max time = 373 us, avg time = 136.49 us +SME2 implementation *asm*: min time = 1 us, max time = 8 us, avg time = 1.44 us +``` + +You can note that, although the vanilla reference matrix multiplication is the +same, there is some variability in the execution time. + +You'll also note that the assembly version of the SME2 matrix multiplication is +slightly faster (1.44 us compared to 1.82 us for the intrinsic-based version). +This *must not* convince you that assembly is better though! The comparison done +here is far from being an apples-to-apples comparison: +- Firstly, the assembly version has some requirements on the `K` parameter that + the intrinsic version does not have. +- Second, the assembly version has an optimization that the intrinsic version, + for the sake of readability in this learning path, does not have (see the + [Going further + section](/learning-paths/cross-platform/multiplying-matrices-with-sme2/10-going-further/) + to know more). +- Last, but not least, the intrinsic version is *easily* readable and + maintainable. \ No newline at end of file diff --git a/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/8-going-further.md b/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/8-going-further.md deleted file mode 100644 index be2ac04bad..0000000000 --- a/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/8-going-further.md +++ /dev/null @@ -1,51 +0,0 @@ ---- -title: Going further -weight: 10 - -### FIXED, DO NOT MODIFY -layout: learningpathall ---- - -## Generalize the algorithms - -In this Learning Path, you focused on using SME2 for matrix -multiplication with floating point numbers. However in practice, any library or framework supporting matrix multiplication should -also handle various integer types. - -You can see that the algorithm structure for matrix preprocessing as well -as multiplication with the outer product does not change at all for other data -types - they only need to be adapted. - -This is suitable for languages with [generic -programming](https://en.wikipedia.org/wiki/Generic_programming) like C++ with -templates. You can even make the template manage a case where the value -accumulated during the product uses a larger type than the input matrices. SME2 has the instructions to deal efficiently with this common case scenario. - -This enables the library developer to focus on the algorithm, testing, and optimizations, while allowing the compiler to generate multiple variants. - -## Unroll further - -You might have noticed that ``matmul_intr_impl`` computes only one tile at a time, for the sake of simplicity. - -SME2 does support multi-vector instructions, and some were used in ``preprocess_l_intr``, for example, ``svld1_x2``. - -Loading two vectors at a time enables the simultaneous computing of more tiles, and as the input matrices have been laid out in memory in a neat way, the consecutive -loading of the data is efficient. Implementing this approach can make improvements to the ``macc`` to load ``ratio``. - -In order to check your understanding of SME2, you can try to implement this unrolling yourself. You can check your work by comparing your results to the expected -reference values. - -## Apply strategies - -One method for optimization is to use strategies that are flexible depending on the matrices' dimensions. This is especially easy to set up when working in C or C++, -rather than directly in assembly language. - -By playing with the mathematical properties of matrix multiplication and the outer product, it is possible to minimize data movement as well as reduce the overall number of operations to perform. - -For example, it is common that one of the matrices is actually a vector, meaning that it has a single row or column, and then it becomes advantageous to transpose it. Can you see why? - -The answer is that as the elements are stored contiguously in memory, an ``Nx1`` and ``1xN`` matrices have the exact same memory layout. The transposition becomes a no-op, and the matrix elements stay in the same place in memory. - -An even more *degenerated* case that is easy to manage is when one of the matrices is essentially a scalar, which means that it is a matrix with one row and one column. - -Although our current code handles it correctly from a results point of view, a different algorithm and use of instructions might be more efficient. Can you think of another way? diff --git a/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/7-debugging.md b/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/9-debugging.md similarity index 78% rename from content/learning-paths/cross-platform/multiplying-matrices-with-sme2/7-debugging.md rename to content/learning-paths/cross-platform/multiplying-matrices-with-sme2/9-debugging.md index 3d490bd8af..a6debde6f4 100644 --- a/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/7-debugging.md +++ b/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/9-debugging.md @@ -1,19 +1,24 @@ --- title: Debugging -weight: 9 +weight: 11 ### FIXED, DO NOT MODIFY layout: learningpathall --- -## Debugging +In practice, writing code can be complex and debugging code is required. -### Looking at the generated code +In this section, you will learn about the different ways to debug SME2 code. -In some cases, it is useful to look at the code generated by the compiler. In this Learning Path, the assembly listings have been produced and you can -inspect them. +## Looking at the generated code -For example, the inner loop with the outer product and the accumulation of the matrix multiplication with intrinsics from the listing file ``sme2_matmul_intr.lst`` looks like this: +In some cases, it is useful to look at the code generated by the compiler. In +this Learning Path, the assembly listings have been produced and you can inspect +them. + +For example, the inner loop with the outer product and the accumulation of the +matrix multiplication with intrinsics from the listing file +`sme2_matmul_intr.lst` looks like this: ```TXT ... @@ -29,18 +34,25 @@ For example, the inner loop with the outer product and the accumulation of the m ### With debuggers -Both of the main debuggers, ``gdb`` and ``lldb``, have some support for debugging SME2 code. Their usage is not shown in this Learning Path though, the main -reason for this being that this Learning Path focuses on the CPU in *baremetal* mode. +Both of the main debuggers, `gdb` and `lldb`, have some support for +debugging SME2 code. Their usage is not shown in this Learning Path though. -This is a simplistic, and minimalistic environment, without an operating system, for example. Debug mode requires a debug monitor to interface between the debugger, the program, and the CPU. +Note that debugging on the emulator might require some more steps as this is a +simplistic, and minimalistic environment, without an operating system, for +example. Debug mode requires a debug monitor to interface between the debugger, +the program, and the CPU. ### With trace -The FVP can emit an instruction trace file in text format, known as the Tarmac trace. This provides a convenient way for you to understand what the program is doing. +The FVP can emit an instruction trace file in text format, known as the Tarmac +trace. This provides a convenient way for you to understand what the program is +doing. -In the excerpt shown below, you can see that the SVE register ``z0`` has been loaded with 16 values, as predicate ``p0`` was true, with an ``LD1W`` -instruction, whereas ``z1`` was loaded with only two values, as ``p1``. ``z0``, and ``z1`` are later used by the ``fmopa`` instruction to compute the -outer product, and the trace displays the content of the ZA storage. +In the excerpt shown below, you can see that the SVE register `z0` has been +loaded with 16 values, as predicate `p0` was true, with an `LD1W` +instruction, whereas `z1` was loaded with only two values, as `p1`. `z0`, +and `z1` are later used by the `fmopa` instruction to compute the outer +product, and the trace displays the content of the ZA storage. ```TXT 923530000 ps IT (92353) 80001b08 a540a1a0 O EL3h_s : LD1W {z0.S},p0/Z,[x13] @@ -91,19 +103,20 @@ outer product, and the trace displays the content of the ZA storage. 923580000 ps R ZA0H_S_15 00000000_00000000_00000000_00000000_00000000_00000000_00000000_00000000_00000000_00000000_00000000_00000000_00000000_00000000_4479e70a_44f4223e ``` -You can get a Tarmac trace when invoking ``run-fvp.sh`` by adding the ``--trace`` option as the *first* argument, for example: +You can get a Tarmac trace when invoking `run-fvp.sh` by adding the +`--trace` option as the *first* argument, for example: ```BASH -docker run --rm -v "$PWD:/work" -w /work armswdev/sme2-learning-path:sme2-environment-v1 ./run-fvp.sh --trace sme2_matmul_asm +docker run --rm -v "$PWD:/work" -w /work armswdev/sme2-learning-path:sme2-environment-v2 ./run-fvp.sh --trace sme2_matmul_asm ``` Tracing is not enabled by default. It slows down the simulation significantly and the trace file can become very large for programs with large matrices. {{% notice Debugging tip %}} It can be helpful when debugging to understand where an element in the -Tile is coming from. The current code base allows you to do that in ``debug`` -mode, when ``-DDEBUG`` is passed to the compiler in the ``Makefile``. If you -look into ``main.c``, you will notice that the matrix initialization is no +Tile is coming from. The current code base allows you to do that in `debug` +mode, when `-DDEBUG` is passed to the compiler in the `Makefile`. If you +look into `main.c`, you will notice that the matrix initialization is no longer random, but instead initializes each element with its linear index. This makes it *easier* to find where the matrix elements are loaded in the tile in tarmac trace, for example. diff --git a/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/_index.md b/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/_index.md index c869b5af7f..11fa78ba65 100644 --- a/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/_index.md +++ b/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/_index.md @@ -9,17 +9,17 @@ learning_objectives: - Implement a reference matrix multiplication without using SME2. - Use SME2 assembly instructions to improve the matrix multiplication performance. - Use SME2 intrinsics to improve the matrix multiplication performance using the C programming language. - - Compile and run code with SME2 instructions. + - Compile code with SME2 instructions. + - Run code with SME2 instructions, on a platform with SME2 support or with an emulator. prerequisites: - Basic knowledge of Arm's Scalable Matrix Extension (SME). - - Basic knowledge of Arm's Scalable Vector Extension (SVE). + - Basic knowledge of Arm's Scalable Vector Extension (SVE). - An intermediate understanding of C programming language and assembly language. - - A computer running Linux, MacOS, or Windows. + - A computer running Linux, macOS, or Windows. - Installations of Git and Docker. - - An emulator to run code with SME2 instructions. - - A compiler with support for SME2 instructions. - + - A platform that support SME2 (see the [list of devices with SME2 support](/learning-paths/cross-platform/multiplying-matrices-with-sme2/1-get-started/#devices-with-sme2-support)) or an emulator to run code with SME2 instructions. + - A compiler with support for SME2 instructions. author: Arnaud de Grandmaison @@ -52,18 +52,26 @@ further_reading: title: Port Code to Arm Scalable Vector Extension (SVE) link: https://learn.arm.com/learning-paths/servers-and-cloud-computing/sve type: website - - resource: - title: Arm Scalable Matrix Extension (SME) Introduction (Part 1) - link: https://community.arm.com/arm-community-blogs/b/architectures-and-processors-blog/posts/arm-scalable-matrix-extension-introduction - type: blog - resource: title: Introducing the Scalable Matrix Extension for the Armv9-A Architecture link: https://community.arm.com/arm-community-blogs/b/architectures-and-processors-blog/posts/scalable-matrix-extension-armv9-a-architecture type: website + - resource: + title: Arm Scalable Matrix Extension (SME) Introduction (Part 1) + link: https://community.arm.com/arm-community-blogs/b/architectures-and-processors-blog/posts/arm-scalable-matrix-extension-introduction + type: blog - resource: title: Arm Scalable Matrix Extension (SME) Introduction (Part 2) link: https://community.arm.com/arm-community-blogs/b/architectures-and-processors-blog/posts/arm-scalable-matrix-extension-introduction-p2 type: blog + - resource: + title: (Part 3) Matrix-matrix multiplication. Neon, SVE, and SME compared + link: https://community.arm.com/arm-community-blogs/b/architectures-and-processors-blog/posts/matrix-matrix-multiplication-neon-sve-and-sme-compared + type: blog + - resource: + title: Build adaptive libraries with multiversioning + link: https://learn.arm.com/learning-paths/cross-platform/function-multiversioning/ + type: website - resource: title: SME Programmer's Guide link: https://developer.arm.com/documentation/109246/latest diff --git a/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/overview.md b/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/overview.md index db395e1ff9..008b274479 100644 --- a/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/overview.md +++ b/content/learning-paths/cross-platform/multiplying-matrices-with-sme2/overview.md @@ -6,7 +6,7 @@ weight: 2 layout: learningpathall --- -# Overview of Arm's Scalable Matrix Extension Version 2 +# Overview of Arm's Scalable Matrix Extension Version 2 ### What is SME2? @@ -30,11 +30,9 @@ Additional architectural features of SME2 include: If you are not familiar with matrix multiplication, or would benefit from refreshing your knowledge, this [Wikipedia article on Matrix multiplication](https://en.wikipedia.org/wiki/Matrix_multiplication) is a good start. -This Learning Path assumes some basic understanding of SVE and SME. If you are not familiar with SVE or SME, these are some useful resources that you can read first: - - - [Introducing the Scalable Matrix Extension for the Armv9-A - Architecture](https://community.arm.com/arm-community-blogs/b/architectures-and-processors-blog/posts/scalable-matrix-extension-armv9-a-architecture). - - [Arm Scalable Matrix Extension (SME) Introduction (Part - 1)](https://community.arm.com/arm-community-blogs/b/architectures-and-processors-blog/posts/arm-scalable-matrix-extension-introduction). - - [Arm Scalable Matrix Extension (SME) Introduction (Part - 2)](https://community.arm.com/arm-community-blogs/b/architectures-and-processors-blog/posts/arm-scalable-matrix-extension-introduction-p2). \ No newline at end of file +This Learning Path assumes some basic understanding of SVE and SME. If you are not familiar with SVE or SME, these are some useful resources that you can read first: +- [Introducing the Scalable Matrix Extension for the Armv9-A Architecture](https://community.arm.com/arm-community-blogs/b/architectures-and-processors-blog/posts/scalable-matrix-extension-armv9-a-architecture). +- [Arm Scalable Matrix Extension (SME) Introduction (Part 1)](https://community.arm.com/arm-community-blogs/b/architectures-and-processors-blog/posts/arm-scalable-matrix-extension-introduction). +- [Arm Scalable Matrix Extension (SME) Introduction (Part 2)](https://community.arm.com/arm-community-blogs/b/architectures-and-processors-blog/posts/arm-scalable-matrix-extension-introduction-p2). +- [Part 3: Matrix-matrix multiplication. Neon, SVE, and SME compared](https://community.arm.com/arm-community-blogs/b/architectures-and-processors-blog/posts/matrix-matrix-multiplication-neon-sve-and-sme-compared). +- [Build adaptive libraries with multiversioning](https://learn.arm.com/learning-paths/cross-platform/function-multiversioning/) \ No newline at end of file