You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/learning-paths/servers-and-cloud-computing/tune-pytorch-cpu-perf-with-threads/tune.md
+4-4Lines changed: 4 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,7 +9,7 @@ layout: learningpathall
9
9
10
10
## Tuning thread size
11
11
12
-
We will run inference on Google's [Gemma-3-1B-it](https://huggingface.co/google/gemma-3-1b-it) and measure how inference performance vaires with thread count for both the 270-million-parameter and 1-billion-parameter models. We will be running the `transformers_llm_text_gen.py` script which by default applies groupwise, layout-aware INT4 quantization of our model.
12
+
We will run inference on Google's [Gemma-3](https://huggingface.co/google/gemma-3-1b-it) model and measure how inference performance vaires with thread count for both the 270-million-parameter and 1-billion-parameter models. We will be running the `transformers_llm_text_gen.py` script which by default applies groupwise, layout-aware INT4 quantization of our model.
13
13
14
14
Create a file names `comparison-1b.sh` and paste in the following script.
15
15
@@ -34,7 +34,7 @@ for t in 2 4 8 16 32 64 96; do
34
34
done
35
35
```
36
36
37
-
Likewise create a separate script, `comparison-1b.sh` for comparing the 270m model
37
+
Likewise create a separate script, `comparison-270m.sh` for comparing the 270m model
38
38
39
39
```bash
40
40
#!/usr/bin/env bash
@@ -114,7 +114,7 @@ So far, we have been running in PyTorch's eager execution mode, we can observe t
@@ -146,5 +146,5 @@ Decode Tokens per second: 107.37
146
146
147
147
### Summary
148
148
149
-
In this learning path, we explored how the number of OpenMP threads is a tunable parameter that can significantly impact the performance of a large language model. This is especially important when running such models on Arm systems with high core counts. You should also take the model’s parameter size into account. In practice, using a heuristic or trial-and-error approach is often the fastest way to determine the optimal thread count for a given model and system.
149
+
In this learning path, we explored how the number of OpenMP threads is a tunable parameter that can impact the performance of a large language model. This is especially important when running such models on Arm systems with high core counts. You should also take the model’s parameter size into account. In practice, using a heuristic or trial-and-error approach is often the fastest way to determine the optimal thread count for a given model and system.
0 commit comments