Add Android benchmark script (#18802) #10995
GregoryComermetal.yml
on: push
Matrix: export-model-metal-artifact
test-executorch-metal-build
/
macos-job
5m 25s
test-metal-backend-modules
/
macos-job
9m 21s
Matrix: test-model-metal-e2e
Waiting for pending jobs
Annotations
2 errors and 11 warnings
|
export-model-metal-artifact (mistralai, Voxtral-Mini-4B-Realtime-2602, quantized-int4-metal) / macos-job
The job has exceeded the maximum execution time of 1h30m0s
|
|
export-model-metal-artifact (mistralai, Voxtral-Mini-4B-Realtime-2602, quantized-int4-metal) / macos-job
The operation was canceled.
|
|
test-executorch-metal-build / macos-job
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/cache@v3, actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683, pmeier/pytest-results-action@a2c1430e2bddadbad9f49a6f9b879f062c6b19b1. Actions will be forced to run with Node.js 24 by default starting June 2nd, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
export-model-metal-artifact (openai, whisper-small, non-quantized) / macos-job
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/cache@v3, actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683, actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02, pmeier/pytest-results-action@a2c1430e2bddadbad9f49a6f9b879f062c6b19b1. Actions will be forced to run with Node.js 24 by default starting June 2nd, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
export-model-metal-artifact (openai, whisper-small, quantized-int4-metal) / macos-job
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/cache@v3, actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683, actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02, pmeier/pytest-results-action@a2c1430e2bddadbad9f49a6f9b879f062c6b19b1. Actions will be forced to run with Node.js 24 by default starting June 2nd, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
export-model-metal-artifact (openai, whisper-large-v3-turbo, quantized-int4-metal) / macos-job
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/cache@v3, actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683, actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02, pmeier/pytest-results-action@a2c1430e2bddadbad9f49a6f9b879f062c6b19b1. Actions will be forced to run with Node.js 24 by default starting June 2nd, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
export-model-metal-artifact (openai, whisper-large-v3-turbo, non-quantized) / macos-job
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/cache@v3, actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683, actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02, pmeier/pytest-results-action@a2c1430e2bddadbad9f49a6f9b879f062c6b19b1. Actions will be forced to run with Node.js 24 by default starting June 2nd, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
test-metal-backend-modules / macos-job
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/cache@v3, actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683, pmeier/pytest-results-action@a2c1430e2bddadbad9f49a6f9b879f062c6b19b1. Actions will be forced to run with Node.js 24 by default starting June 2nd, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
export-model-metal-artifact (nvidia, parakeet-tdt, non-quantized) / macos-job
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/cache@v3, actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683, actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02, pmeier/pytest-results-action@a2c1430e2bddadbad9f49a6f9b879f062c6b19b1. Actions will be forced to run with Node.js 24 by default starting June 2nd, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
export-model-metal-artifact (nvidia, parakeet-tdt, quantized-int4-metal) / macos-job
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/cache@v3, actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683, actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02, pmeier/pytest-results-action@a2c1430e2bddadbad9f49a6f9b879f062c6b19b1. Actions will be forced to run with Node.js 24 by default starting June 2nd, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
export-model-metal-artifact (mistralai, Voxtral-Mini-3B-2507, quantized-int4-metal) / macos-job
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/cache@v3, actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683, actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02, pmeier/pytest-results-action@a2c1430e2bddadbad9f49a6f9b879f062c6b19b1. Actions will be forced to run with Node.js 24 by default starting June 2nd, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
export-model-metal-artifact (mistralai, Voxtral-Mini-3B-2507, non-quantized) / macos-job
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/cache@v3, actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683, actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02, pmeier/pytest-results-action@a2c1430e2bddadbad9f49a6f9b879f062c6b19b1. Actions will be forced to run with Node.js 24 by default starting June 2nd, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
|
export-model-metal-artifact (mistralai, Voxtral-Mini-4B-Realtime-2602, quantized-int4-metal) / macos-job
Node.js 20 actions are deprecated. The following actions are running on Node.js 20 and may not work as expected: actions/cache@v3, actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683, actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02, pmeier/pytest-results-action@a2c1430e2bddadbad9f49a6f9b879f062c6b19b1. Actions will be forced to run with Node.js 24 by default starting June 2nd, 2026. Node.js 20 will be removed from the runner on September 16th, 2026. Please check if updated versions of these actions are available that support Node.js 24. To opt into Node.js 24 now, set the FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true environment variable on the runner or in your workflow file. Once Node.js 24 becomes the default, you can temporarily opt out by setting ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
|
Artifacts
Produced during runtime
| Name | Size | Digest | |
|---|---|---|---|
|
mistralai-Voxtral-Mini-3B-2507-metal-non-quantized
|
6.82 GB |
sha256:08726fc55925632fe7f245574415cfa6af9dd5c79e01b3c7af0726ea5ae5b8bf
|
|
|
mistralai-Voxtral-Mini-3B-2507-metal-quantized-int4-metal
|
3.2 GB |
sha256:57ca070f25a06a1cc2ebd6c201bd28a8aa903900fb4dd6372108b519f54d0d12
|
|
|
mistralai-Voxtral-Mini-4B-Realtime-2602-metal-quantized-int4-metal
|
7.82 KB |
sha256:34277dc6d408b9500653b76e5786c7853b3454289c6ac0d93f5a0bed6fd36d8c
|
|
|
nvidia-parakeet-tdt-metal-non-quantized
|
951 MB |
sha256:bbf93fe83944dcff15e19d7f50e0eded5797b4bb2157f51f133850f92fee501c
|
|
|
nvidia-parakeet-tdt-metal-quantized-int4-metal
|
436 MB |
sha256:82f72dde457d4d612d62c7c5d0e139b02f0aec5a658a8a7faab5891c37e7554a
|
|
|
openai-whisper-large-v3-turbo-metal-non-quantized
|
1.18 GB |
sha256:a2094c25fa805a4f36817a947e7c15506dd551e1dbdbe12670bc30d2d9f1ed80
|
|
|
openai-whisper-large-v3-turbo-metal-quantized-int4-metal
|
476 MB |
sha256:67e04d7227c7f4d1df3798f2eed9fc069f062000c1c9fb20aebb032eaaa2c91c
|
|
|
openai-whisper-small-metal-non-quantized
|
362 MB |
sha256:e775ebe3f3f08335e879a32a3dda28ba49226999d882f179b47e21bfb2c8c67c
|
|
|
openai-whisper-small-metal-quantized-int4-metal
|
169 MB |
sha256:e9a5048d3e01ef72e937f80bb14a4205e13cd95fd31d0ccbcf35da346c23ce5b
|
|