Skip to content

feat: Change paged FP8 prefill back to regular attention#171

Merged
ani300 merged 3 commits intomainfrom
fp8_paged_prefill_opt
Aug 5, 2025
Merged

feat: Change paged FP8 prefill back to regular attention#171
ani300 merged 3 commits intomainfrom
fp8_paged_prefill_opt

Conversation

@ani300
Copy link
Copy Markdown
Contributor

@ani300 ani300 commented Aug 1, 2025

Description of the change

Deeptools developers prefer the prefill for paged fp8 attention to be regular attention instead of the math fp8 attention, given that the FP8 projections return FP16 anyways and running attention in FP8 for prefill essentially requires casting activations to FP8 twice.

Related issues or PRs

Internal discussion.

How to verify the PR

Was the PR tested

  • I have added >=1 unit test(s) for every new method I have added (if that coverage is difficult, please briefly explain the reason)
  • I have ensured all unit tests pass

Checklist for passing CI/CD:

  • All commits are signed showing "Signed-off-by: Name <email@domain.com>" with git commit -signoff or equivalent
  • PR title and commit messages adhere to Conventional Commits
  • Contribution is formatted with tox -e fix
  • Contribution passes linting with tox -e lint
  • Contribution passes spellcheck with tox -e spellcheck
  • Contribution passes all unit tests with tox -e unit

Note: CI/CD performs unit tests on multiple versions of Python from a fresh install. There may be differences with your local environment and the test environment.

Signed-off-by: Antoni Viros i Martin <aviros@ibm.com>
@ani300 ani300 changed the title Change paged FP8 prefill back to regular attention feat: Change paged FP8 prefill back to regular attention Aug 1, 2025
@github-actions github-actions Bot added the feat label Aug 1, 2025
ani300 added 2 commits August 1, 2025 20:25
Signed-off-by: Antoni Viros i Martin <aviros@ibm.com>
Signed-off-by: Antoni Viros i Martin <aviros@ibm.com>
Copy link
Copy Markdown
Collaborator

@andrea-fasoli andrea-fasoli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

followed the discussion. Change looks good.

@ani300 ani300 merged commit 39db419 into main Aug 5, 2025
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants