Skip to content

fix: feat: fix for new transformers (>4.48) and new QLinear for INT8 training with HW emulation#141

Merged
chichun-charlie-liu merged 7 commits into
foundation-model-stack:mainfrom
chichun-charlie-liu:triton_aiu_sim
Jun 24, 2025
Merged

fix: feat: fix for new transformers (>4.48) and new QLinear for INT8 training with HW emulation#141
chichun-charlie-liu merged 7 commits into
foundation-model-stack:mainfrom
chichun-charlie-liu:triton_aiu_sim

Conversation

@chichun-charlie-liu

Copy link
Copy Markdown
Collaborator

Description of the change

  1. For new transformers (>4.48), model output could be simply a python tuple instead of a special object. Adjust Evaluator.evaluate() accordingly to accommodate both old and new versions.
  2. Added a new QLinear wrapper QLinearINT8Train for INT8 training with HW emulation taken into account.
  3. fix a setting in triton unit test to match with previous results. A default for kernel has been changed previously. It will increase the relative error of the triton matmul slightly and marginally fail some of the tests. We only apply <this flag>=False in unit tests and keep it True for real use.

Related issues or PRs

Will close #122 and #127 after this PR is merged.

How to verify the PR

dq example was tested with new version of transformers (4.52), which will show correct perplexity instead of throwing errors.

Was the PR tested

  • I have added >=1 unit test(s) for every new method I have added (if that coverage is difficult, please briefly explain the reason)
  • I have ensured all unit tests pass

Checklist for passing CI/CD:

  • All commits are signed showing "Signed-off-by: Name <email@domain.com>" with git commit -signoff or equivalent
  • PR title and commit messages adhere to Conventional Commits
  • Contribution is formatted with tox -e fix
  • Contribution passes linting with tox -e lint
  • Contribution passes spellcheck with tox -e spellcheck
  • Contribution passes all unit tests with tox -e unit

Note: CI/CD performs unit tests on multiple versions of Python from a fresh install. There may be differences with your local environment and the test environment.

Signed-off-by: cliu-us <cliu@us.ibm.com>
…king HW simulation into account

Signed-off-by: cliu-us <cliu@us.ibm.com>
…y increase in rel errs, change settings in tests to match what we had before

Signed-off-by: cliu-us <cliu@us.ibm.com>
Comment thread fms_mo/modules/linear.py Outdated
@andrea-fasoli

Copy link
Copy Markdown
Collaborator

fix a setting in triton unit test to match with previous results. A default for kernel has been changed previously. It will increase the relative error of the triton matmul slightly and marginally fail some of the tests. We only apply =False in unit tests and keep it True for real use

Can we update the unit tests to reflect this new default value for the triton kernels, instead of adding this flag?

Comment thread fms_mo/modules/bmm.py
Comment thread fms_mo/modules/conv.py
Comment thread fms_mo/modules/linear.py
Comment thread fms_mo/modules/linear.py Outdated
Signed-off-by: cliu-us <cliu@us.ibm.com>
Signed-off-by: cliu-us <cliu@us.ibm.com>
…t being fp32

Signed-off-by: cliu-us <cliu@us.ibm.com>
@chichun-charlie-liu chichun-charlie-liu self-assigned this Jun 23, 2025
@chichun-charlie-liu chichun-charlie-liu linked an issue Jun 23, 2025 that may be closed by this pull request
Comment thread fms_mo/utils/aiu_utils.py Outdated
Signed-off-by: cliu-us <cliu@us.ibm.com>
@chichun-charlie-liu chichun-charlie-liu merged commit 988add0 into foundation-model-stack:main Jun 24, 2025
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add accumulation triton kernel

3 participants