Skip to content

bf16 asm mha: enable doubleq and kv reverse to improve perf#4068

Open
tingchen988 wants to merge 1 commit into
mainfrom
tingchen_mha_doubleq
Open

bf16 asm mha: enable doubleq and kv reverse to improve perf#4068
tingchen988 wants to merge 1 commit into
mainfrom
tingchen_mha_doubleq

Conversation

@tingchen988

Copy link
Copy Markdown
Contributor

Motivation

[gfx1250] fmha fwd bf16 hdim128/64 support per tensor bf16 for gfx1250: enable doubleq and kv reverse to improve perf

Technical Details

change a runtime arg in csrc/py_itfs_cu/asm_fmha_fwd_with_sink_varlen.cu

Test Plan

python3 op_tests/test_fmha_fwd_with_sink_varlen_asm.py

Test Result

Function test:
pytest op_tests/test_fmha_fwd_with_sink_varlen_asm.py::test_fmha_fwd_with_sink_varlen_asm_correctness -v -s
test pass

Perf test:
pytest op_tests/test_fmha_fwd_with_sink_varlen_asm.py::test_fmha_fwd_with_sink_varlen_asm_perf -v -s
perf would be improved by ~2% with doubleq and kv reverse

Submission Checklist

@tingchen988 tingchen988 requested a review from a team July 3, 2026 03:09
@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every PR:

  • ✅ Pre-checks (submodule verification, code formatting)
  • ✅ Aiter op tests (gfx942 + gfx950)
  • ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label Tests
ci:triton-300x Run an additional Triton test job on MI300X in PRs; main branch always runs both MI35X and MI300X
ci:sglang SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm vLLM benchmark: GPT-OSS-120B, DeepSeek-R1-0528, Kimi-K2.5
ci:all All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 4068 --add-label <label>

// 6 = 0b110 -> reverse_kv=0, double_q=1, remap_xy=1. Must match how the
// shipped VARLEN .co was built.
args.opt = 4;
args.opt = 7;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no tg num change?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tg num would be automatically changed according to args.opt in line 252.

@zufayu zufayu requested a review from amd-ruitang3 July 6, 2026 02:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants