Arm backend: Make upstream testing more target specific (#15920)

This patch splits up our upstream testing such that only relevant items
are installed and only required things are tested upstream. It removes
any duplicate testing and makes the upstream testing structure more
transparent and readable.

Change-Id: I14cf329eed03d9859aef97204a9f12185b9533a2


cc @freddan80 @per @zingo @oscarandersson8218 @digantdesai

---------

Signed-off-by: Michiel Olieslagers <michiel.olieslagers@arm.com>
Co-authored-by: Zingo Andersen <zingo.andersen@arm.com>

Author: Michiel-Olieslagers

.github/workflows/pull.yml

@@ -490,17 +490,19 @@ jobs:
       build-tool: buck2
       docker-image: ci-image:executorch-ubuntu-22.04-clang12
 
-  unittest-arm-backend-with-no-fvp:
-    name: unittest-arm-backend-with-no-fvp
+  unittest-arm-backend-with-no-deps:
+    name: unittest-arm-backend-with-no-deps
     uses: pytorch/test-infra/.github/workflows/linux_job_v2.yml@main
     permissions:
       id-token: write
       contents: read
     strategy:
       matrix:
         include:
-          - test_arm_baremetal: test_pytest_ops
-          - test_arm_baremetal: test_pytest_models
+          - test_arm_baremetal: test_pytest_ops_no_target
+          - test_arm_baremetal: test_pytest_ops_tosa
+          - test_arm_baremetal: test_pytest_models_tosa
+          - test_arm_baremetal: test_run_tosa
       fail-fast: false
     with:
       runner: linux.2xlarge
@@ -516,7 +518,7 @@ jobs:
         source .ci/scripts/utils.sh
         install_executorch "--use-pt-pinned-commit"
 
-        .ci/scripts/setup-arm-baremetal-tools.sh
+        .ci/scripts/setup-arm-baremetal-tools.sh --disable-ethos-u-deps
 
         ARM_TEST=${{ matrix.test_arm_baremetal }}
 

.github/workflows/trunk.yml

@@ -273,21 +273,21 @@ jobs:
         # Test selective build
         PYTHON_EXECUTABLE=python bash examples/portable/scripts/test_demo_backend_delegation.sh "${BUILD_TOOL}"
 
-  test-arm-backend:
-    name: test-arm-backend
+  test-arm-backend-ethos-u:
+    name: test-arm-backend-ethos-u
     uses: pytorch/test-infra/.github/workflows/linux_job_v2.yml@main
     permissions:
       id-token: write
       contents: read
     strategy:
       matrix:
         include:
-          - test_arm_baremetal: test_pytest_ops_ethosu_fvp
-          - test_arm_baremetal: test_pytest_models_ethosu_fvp
-          - test_arm_baremetal: test_run_ethosu_fvp
-          - test_arm_baremetal: test_models_tosa
-          - test_arm_baremetal: test_models_ethos-u55
-          - test_arm_baremetal: test_models_ethos-u85
+          - test_arm_baremetal: test_pytest_ops_ethos_u55
+          - test_arm_baremetal: test_pytest_models_ethos_u55
+          - test_arm_baremetal: test_run_ethos_u55
+          - test_arm_baremetal: test_pytest_ops_ethos_u85
+          - test_arm_baremetal: test_pytest_models_ethos_u85
+          - test_arm_baremetal: test_run_ethos_u85
           - test_arm_baremetal: test_smaller_stories_llama
           - test_arm_baremetal: test_memory_allocation
           - test_arm_baremetal: test_model_smollm2-135M

backends/arm/README.md

@@ -110,13 +110,24 @@ These scripts also install the necessary dependencies to run the tests.
 Below is an overview of some of the testing options this script provides:
 
 | Command                                        | Description                                  |
-| ---------------------------------------------- | -------------------------------------------- |
-| `test_arm_baremetal.sh test_pytest`            | Runs all unit tests.                         |
-| `test_arm_baremetal.sh test_pytest_ethosu_fvp` | Same as `test_pytest` but uses Corstone FVP. |
-| `test_arm_baremetal.sh test_run_ethosu_fvp`    | Runs some models with Corstone FVP.          |
-| `test_arm_baremetal.sh test_full_ethosu_fvp`   | Runs E2E model tests on Corstone FVP.        |
-| `test_arm_baremetal.sh test_pytest_vkml`       | Runs all unit tests with Vulkan ML.          |
-| `test_arm_baremetal.sh test_full_vkml`         | Run E2E models test with Vulkan ML.          |
+| ---------------------------------------------------- | ------------------------------------------------------------ |
+| `test_arm_baremetal.sh test_pytest_ops_no_target`    | Runs operator unit tests for non-target specific use-cases.  |
+| `test_arm_baremetal.sh test_pytest_models_no_target` | Runs model unit tests for non-target specific use-cases.     |
+| `test_arm_baremetal.sh test_pytest_ops_tosa`         | Runs operator unit tests for TOSA specific use-cases.        |
+| `test_arm_baremetal.sh test_pytest_models_tosa`      | Runs model unit tests for TOSA specific use-cases.           |
+| `test_arm_baremetal.sh test_run_tosa`                | Runs end-to-end unit tests for TOSA specific use-cases.      |
+| `test_arm_baremetal.sh test_pytest_ops_ethos_u55`    | Runs operator unit tests for Ethos-U55 specific use-cases.   |
+| `test_arm_baremetal.sh test_pytest_models_ethos_u55` | Runs model unit tests for Ethos-U55 specific use-cases.      |
+| `test_arm_baremetal.sh test_run_ethos_u55`           | Runs end-to-end unit tests for Ethos-U55 specific use-cases. |
+| `test_arm_baremetal.sh test_pytest_ops_ethos_u85`    | Runs operator unit tests for Ethos-U85 specific use-cases.   |
+| `test_arm_baremetal.sh test_pytest_models_ethos_u85` | Runs model unit tests for Ethos-U85 specific use-cases.      |
+| `test_arm_baremetal.sh test_run_ethos_u85`           | Runs end-to-end unit tests for Ethos-U85 specific use-cases. |
+| `test_arm_baremetal.sh test_pytest_ops_vkml`         | Runs operator unit tests for VGF specific use-cases.         |
+| `test_arm_baremetal.sh test_pytest_models_vkml`      | Runs model unit tests for VGF specific use-cases.            |
+| `test_arm_baremetal.sh test_run_vkml`                | Runs end-to-end unit tests for VGF specific use-cases. |
+| `test_arm_baremetal.sh test_model_smollm2-135M`      | Runs some models with Corstone FVP.          |
+| `test_arm_baremetal.sh test_smaller_stories_llama`   | Runs E2E model tests on Corstone FVP.        |
+| `test_arm_baremetal.sh test_memory_allocation`       | Runs memory allocation tests for Ethos-U specific targets    |
 
 For more information, please refer to the `backends/arm/test/test_arm_baremetal.sh` script.