[models] Support for IBM Granite Models 3.2, 3.3 & 4.0 with FP16 and Q8 by mikepapadim · Pull Request #92 · beehive-lab/GPULlama3.java

Add Granite model support in inference pipeline

mikepapadim

committed

Add Granite model-specific implementations for inference pipeline

mikepapadim

committed

Fix Makefile indentation on 'install' target

mikepapadim

committed

Refactor `GraniteTokenizer` to simplify special token handling and remove unused methods.

mikepapadim

committed

Add Granite-specific TornadoVM components and FP16 inference pipelines

mikepapadim

committed

Introduce parallelized attention computation in Granite kernels

mikepapadim

committed

Remove unused `LlamaTornadoWeights` import and clean up formatting in `GraniteFP16FFNLayers`

mikepapadim

committed

Update README to include IBM Granite 3.1+ model support and link to Granite 3.3 collection

mikepapadim

committed

Update README to replace `TORNADO_SDK` with `TORNADOVM_HOME` for consistency in environment variable naming

mikepapadim

committed

Update `GraniteFP16LayerPlanner` to use `LogitsGraniteFP16Layer` for logits initialization

mikepapadim

committed

Refactor `LogitsGraniteFP16Layer` to extend `LogitsFP16Layer` and update constructor for enhanced configurability

mikepapadim

committed

Add TornadoVM Q8_0 support for Granite inference pipeline

mikepapadim

committed

Use `GraniteTokenizer` for EOS token retrieval in `GraniteChatFormat` to eliminate hardcoded values.

mikepapadim

committed

Add `ropeRotationWithCacheCopy` kernel to `GraniteKernels` for RoPE computation and fused cache writes

mikepapadim

committed

Refactor `GraniteLoader`: enhance metadata handling, improve formatting, and streamline tensor-loading logic.

mikepapadim

committed

Replace `TransformerComputeKernelsLayered::ropeRotationWithCacheCopy` call with `GraniteKernels::ropeRotationWithCacheCopy` in `GraniteFP16FFNLayers` and add `ropeTheta` to arguments.

mikepapadim

committed

Refactor `GraniteTokenizer`: add support for Granite 4.0, enhance metadata handling, and improve pretokenizer logic.

mikepapadim

committed

Replace `TransformerComputeKernelsLayered::ropeRotationWithCacheCopy` with `GraniteKernels::ropeRotationWithCacheCopy` in `GraniteQ8_0FFNLayers` and add `ropeTheta` to kernel arguments.

mikepapadim

committed

Update README to add IBM Granite 4.0 collection link

mikepapadim

committed

Update README to include support for Phi-3, IBM Granite 3.2+, and IBM Granite 4.0 models

mikepapadim

committed

Add build step for running Granite 3.2 model in CI pipelineAdd workflow step to run Granite-3.2-2b-instruct-f16.gguf during CI

mikepapadim

committed

Add workflow step to run Granite-3.2-2b-instruct-Q8.gguf during CI

mikepapadim

committed

Add workflow steps to run Granite-4.0-1b-F16.gguf and Granite-4.0-1b-Q8_0.gguf during CI

mikepapadim

committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[models] Support for IBM Granite Models 3.2, 3.3 & 4.0 with FP16 and Q8#92

[models] Support for IBM Granite Models 3.2, 3.3 & 4.0 with FP16 and Q8#92
mikepapadim merged 26 commits intomainfrom
feat/models/ibm-granite

Commits on Dec 17, 2025

Add Granite model support in inference pipeline

Add Granite model-specific implementations for inference pipeline

Fix Makefile indentation on 'install' target

Refactor `GraniteTokenizer` to simplify special token handling and remove unused methods.

Add Granite-specific TornadoVM components and FP16 inference pipelines

Introduce parallelized attention computation in Granite kernels

Remove unused `LlamaTornadoWeights` import and clean up formatting in `GraniteFP16FFNLayers`

Update README to include IBM Granite 3.1+ model support and link to Granite 3.3 collection

Update README to replace `TORNADO_SDK` with `TORNADOVM_HOME` for consistency in environment variable naming

Update `GraniteFP16LayerPlanner` to use `LogitsGraniteFP16Layer` for logits initialization

Refactor `LogitsGraniteFP16Layer` to extend `LogitsFP16Layer` and update constructor for enhanced configurability

Add TornadoVM Q8_0 support for Granite inference pipeline

Use `GraniteTokenizer` for EOS token retrieval in `GraniteChatFormat` to eliminate hardcoded values.

Add `ropeRotationWithCacheCopy` kernel to `GraniteKernels` for RoPE computation and fused cache writes

Refactor `GraniteLoader`: enhance metadata handling, improve formatting, and streamline tensor-loading logic.

Replace `TransformerComputeKernelsLayered::ropeRotationWithCacheCopy` call with `GraniteKernels::ropeRotationWithCacheCopy` in `GraniteFP16FFNLayers` and add `ropeTheta` to arguments.

Refactor `GraniteTokenizer`: add support for Granite 4.0, enhance metadata handling, and improve pretokenizer logic.

Replace `TransformerComputeKernelsLayered::ropeRotationWithCacheCopy` with `GraniteKernels::ropeRotationWithCacheCopy` in `GraniteQ8_0FFNLayers` and add `ropeTheta` to kernel arguments.

Update README to add IBM Granite 4.0 collection link

Update README to include support for Phi-3, IBM Granite 3.2+, and IBM Granite 4.0 models

Add build step for running Granite 3.2 model in CI pipelineAdd workflow step to run Granite-3.2-2b-instruct-f16.gguf during CI

Add workflow step to run Granite-3.2-2b-instruct-Q8.gguf during CI

Add workflow steps to run Granite-4.0-1b-F16.gguf and Granite-4.0-1b-Q8_0.gguf during CI

Commits on Dec 18, 2025

Fix logits scaling in `GraniteKernels`: correct scaling order for `hb` and `output` writes.

Remove unused imports from `GraniteFP16LayerPlanner`.

Rename `Granite8_0LayerPlanner` to `GraniteQ8_0LayerPlanner` for consistency with naming conventions across Q8_0 models.