feat: FP8 ViT Attention w/ FlashInfer#1660
Conversation
Signed-off-by: AlpinDale <alpindale@gmail.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7cec407bab
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| mm_cfg = get_multimodal_config() | ||
| scale_path = mm_cfg.mm_encoder_fp8_scale_path if mm_cfg is not None else None | ||
| if scale_path is None: |
There was a problem hiding this comment.
process_weights_after_loading fetches mm_encoder_fp8_scale_path via get_multimodal_config(), but this method is invoked after model init when set_current_aphrodite_config is no longer active (see BaseModelLoader.load_model calling process_weights_after_loading after the init context). In that normal path mm_cfg is None, so scale_path becomes None even when --mm-encoder-fp8-scale-path is set; static scales are never loaded, scale buffers stay at 1.0, and because _fp8_dynamic_scale is already False, scales are never updated dynamically either.
Useful? React with 👍 / 👎.
| self.fp8_enabled = True | ||
| self._fp8_dynamic_scale = mm_cfg.mm_encoder_fp8_scale_path is None | ||
| self.fp8_quant = QuantFP8(static=True, group_shape=GroupShape.PER_TENSOR) |
There was a problem hiding this comment.
This branch enables FP8 whenever mm_encoder_attn_dtype == "fp8", but it does not ensure self.attn_backend is FLASHINFER. With default CUDA backend selection, ViT prefers FLASH_ATTN before FLASHINFER, so users can set --mm-encoder-attn-dtype fp8 and still execute _forward_fa/_forward_sdpa paths with no FP8 quantization at all. That creates a silent config mismatch where the FP8 option appears enabled but is effectively ignored unless backend is separately forced.
Useful? React with 👍 / 👎.
1 participant
vllm-project/vllm#38065