Skip to content

Commit 133bc3d

Browse files
committed
Minor update based on review feedback.
Signed-off-by: Anna Mayne <anna.mayne@arm.com>
1 parent 6f5a3fc commit 133bc3d

1 file changed

Lines changed: 1 addition & 1 deletion

File tree

  • content/learning-paths/servers-and-cloud-computing/vllm-benchmark-quantisation

content/learning-paths/servers-and-cloud-computing/vllm-benchmark-quantisation/4-benchmarking.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -90,7 +90,7 @@ Repeat with the quantised model.
9090
lm_eval --model vllm --model_args pretrained=RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8,dtype=bfloat16,max_model_len=4096 --tasks mmlu,gsm8k --batch_size auto
9191
```
9292

93-
We would expect to see the precision is slightly lower with INT8.
93+
We expect INT8 inference to show a slight accuracy drop compared to BF16. For reference results and expected accuracy differences, see the Red Hat model card: https://huggingface.co/RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8#accuracy
9494

9595
## Summary of results
9696

0 commit comments

Comments
 (0)