Skip to content

feat: Per-sequence scaling in FP8 attention, FP8 fixes#162

Merged
andrea-fasoli merged 1 commit intofoundation-model-stack:mainfrom
ani300:fp8_sequence_fixes
Jul 11, 2025
Merged

feat: Per-sequence scaling in FP8 attention, FP8 fixes#162
andrea-fasoli merged 1 commit intofoundation-model-stack:mainfrom
ani300:fp8_sequence_fixes

Conversation

@ani300
Copy link
Copy Markdown
Contributor

@ani300 ani300 commented Jul 11, 2025

Description of the change

  • Per-sequence scaling in FP8 attention for both static and continuous batching
  • FP8 fixes for the CPU fallback in both eager and torch compile modes

Related issues or PRs

Internal to IBM issue

How to verify the PR

Was the PR tested

  • I have added >=1 unit test(s) for every new method I have added (if that coverage is difficult, please briefly explain the reason)
  • I have ensured all unit tests pass

Checklist for passing CI/CD:

  • All commits are signed showing "Signed-off-by: Name <email@domain.com>" with git commit -signoff or equivalent
  • PR title and commit messages adhere to Conventional Commits
  • Contribution is formatted with tox -e fix
  • Contribution passes linting with tox -e lint
  • Contribution passes spellcheck with tox -e spellcheck
  • Contribution passes all unit tests with tox -e unit

Note: CI/CD performs unit tests on multiple versions of Python from a fresh install. There may be differences with your local environment and the test environment.

…PU fallback errors

Signed-off-by: Antoni Viros i Martin <aviros@ibm.com>
@ani300 ani300 added bug Something isn't working enhancement New feature or request labels Jul 11, 2025
@ani300 ani300 changed the title Per-sequence scaling in FP8 attention, FP8 fixes feat: Per-sequence scaling in FP8 attention, FP8 fixes Jul 11, 2025
@github-actions github-actions Bot added the feat label Jul 11, 2025
@andrea-fasoli
Copy link
Copy Markdown
Collaborator

tested the fp8_spyre_op fix locally and confirm it addresses the _scaled_mm issue when compiling the model.
Waiting for further tests before merging.

Copy link
Copy Markdown
Collaborator

@andrea-fasoli andrea-fasoli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tested as functional

@andrea-fasoli andrea-fasoli merged commit c920911 into foundation-model-stack:main Jul 11, 2025
11 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working enhancement New feature or request feat

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants