Skip to content

[GGUF] Introduce GGUF Frontend#36579

Open
mvafin wants to merge 4 commits into
openvinotoolkit:masterfrom
mvafin:mvafin/gguf/frontend-port
Open

[GGUF] Introduce GGUF Frontend#36579
mvafin wants to merge 4 commits into
openvinotoolkit:masterfrom
mvafin:mvafin/gguf/frontend-port

Conversation

@mvafin

@mvafin mvafin commented Jun 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Details:

This PR upstreams the ggml-openvino frontend from the llama.cpp OpenVINO backend as a
new, in-tree GGUF frontend (ENABLE_OV_GGUF_FRONTEND, OFF by default). The first
commit is a verbatim copy; the important changes relative to the llama.cpp version are:

  • Renamed ggml → gguf across directories, headers, namespace (ov::frontend::gguf),
    GgmlDecoder → GgufDecoder, macros, CMake target and build option — to reflect its role
    as an OpenVINO frontend for GGUF models (the internal llama.cpp name was "ggml"). The
    runtime GGML_OP_* graph op-type keys are kept, since they are ggml graph identifiers, not
    frontend symbols.
  • Hidden but linkable frontend: added FrontEndPluginInfo::m_hidden in the common
    frontend manager. The GGUF frontend installs and links like any other frontend
    (openvino::frontend::gguf), but is excluded from FrontEndManager::get_available_front_ends()
    and load_by_model / load_by_framework, so it isn't auto-selected by the generic loading API
    while remaining usable by direct linkers (llama.cpp backend, OpenVINO GenAI).
  • Device-/mode-agnostic conversion & slimmer decoder API: made GgufDecoder node-scoped,
    surfaced GGUF weights as real GGML_OP_NONE leaves, folded the rope helpers into a single
    rope_config attribute, and dropped backend-internal/unused decoder methods and dead code
    (unused translators, utils helpers, weight splitters, the never-registered FuseToSDPA pass).
  • Stateless-by-default KV cache: the frontend always emits the stateless, llama.cpp-faithful
    graph — each ggml SET_ROWS becomes an internal SetRows op lowered to ScatterUpdate. A
    backend may opt into an OpenVINO stateful ReadValue/Assign subgraph via a
    DecoderTransformationExtension (selected structurally, no decoder flag). The NPU-only
    SqueezeMatmul pass was moved out to the backend.
  • Self-contained GGUF quant/dequant implementation with a test suite validating dequant
    against real-ggml reference data, plus the Q4_K integer (u8) zero-point so the CPU/GPU
    plugin fuses the dequant into MatMul (matching the original backend's performance).

Tickets:

AI Assistance:

  • AI assistance used: yes

@mvafin
mvafin requested review from a team as code owners June 25, 2026 23:46
@mvafin
mvafin requested review from mlukasze and removed request for a team June 25, 2026 23:46
@github-actions github-actions Bot added category: build OpenVINO cmake script / infra category: CI OpenVINO public CI github_actions Pull requests that update GitHub Actions code no-match-files labels Jun 25, 2026
@mvafin
mvafin force-pushed the mvafin/gguf/frontend-port branch 15 times, most recently from 52d34f8 to a4886aa Compare July 1, 2026 16:07
@mvafin
mvafin requested a review from a team as a code owner July 1, 2026 16:07
@github-actions github-actions Bot added category: Core OpenVINO Core (aka ngraph) category: CPP API OpenVINO CPP API bindings labels Jul 1, 2026
@mvafin
mvafin force-pushed the mvafin/gguf/frontend-port branch from a4886aa to 0899f97 Compare July 1, 2026 19:34
@github-actions github-actions Bot added the category: packaging OpenVINO packaging / distribution label Jul 1, 2026
@mvafin
mvafin force-pushed the mvafin/gguf/frontend-port branch from 0899f97 to 9c2d652 Compare July 1, 2026 20:13
@mvafin mvafin changed the title [GGUF][DO NOT REVIEW] Introduce GGUF Frontend [GGUF] Introduce GGUF Frontend Jul 1, 2026
@mvafin
mvafin force-pushed the mvafin/gguf/frontend-port branch from 9c2d652 to 0aa3c4c Compare July 8, 2026 15:34
@mvafin
mvafin requested a review from a team as a code owner July 8, 2026 15:34
@mvafin
mvafin requested review from akuporos and artanokhov and removed request for a team July 8, 2026 15:34
@github-actions github-actions Bot added the category: dependency_changes Pull requests that update a dependency file label Jul 8, 2026
@mvafin
mvafin force-pushed the mvafin/gguf/frontend-port branch 2 times, most recently from 7290ca9 to b18361b Compare July 8, 2026 16:01
@mvafin
mvafin requested review from cavusmustafa and ravi9 July 8, 2026 16:40
@ravi9

ravi9 commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

FYI: @zhaixuejun1993 @wine99

@mvafin
mvafin force-pushed the mvafin/gguf/frontend-port branch from 3dafa1b to 6fab1e8 Compare July 9, 2026 16:33
cavusmustafa
cavusmustafa reviewed Jul 10, 2026
View reviewed changes
Comment thread src/frontends/gguf/include/openvino/frontend/gguf/decoder.hpp Outdated
cavusmustafa
cavusmustafa reviewed Jul 10, 2026
View reviewed changes

@cavusmustafa cavusmustafa left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requant is a separate pre-convert step in llama.cpp OpenVINO backend (https://github.com/ravi9/llama.cpp/blob/4fc4ec5541b243957ae5099edb67372f8f3b550e/ggml/src/ggml-openvino/ggml-openvino.cpp#L243). Can we still do this in the new frontend?

@mvafin
mvafin force-pushed the mvafin/gguf/frontend-port branch 2 times, most recently from 4d4155d to 753978c Compare July 15, 2026 11:25
mvafin added 3 commits July 15, 2026 13:38
@mvafin
[GGML FE] Copy ggml-openvino frontend translation layer from llama.cpp
ae57118 
Initial MVP port of the OpenVINO frontend that lives in llama.cpp
(ggml/src/ggml-openvino). This commit copies the OpenVINO-side
translation layer as-is from llama.cpp master:

  - frontend / input_model / translate_session / op_table
  - all op translators (op/*.cpp)
  - transformation passes (pass/*) and rt_info attribute
  - decoder.h / frontend.h public headers

Only the include paths are adjusted to the OpenVINO source layout
(decoder.h / frontend.h become public headers under
include/openvino/frontend/ggml/) and the unused "ggml-impl.h" include
is dropped: the translation layer operates purely on the abstract
GgmlDecoder interface and OpenVINO ops, so it has no dependency on
llama.cpp / ggml. The ggml-coupled ingestion layer (cgraph decoder,
quantization, gguf reader) is intentionally not copied.

Build wiring:
  - new ENABLE_OV_GGML_FRONTEND option, OFF by default
  - SKIP_INSTALL so the frontend is never shipped in the release package
  - LINKABLE_FRONTEND so tests can drive FrontEnd::convert directly

Verified: builds into libopenvino_ggml_frontend.so with no undefined
ggml symbols.
@mvafin
[GGUF FE] Rename ggml frontend to gguf frontend
e620926 
Rename the frontend from "ggml" to "gguf" to match its role as an
OpenVINO frontend for GGUF models (the llama.cpp-internal name was
"ggml"):

  - directory src/frontends/ggml -> src/frontends/gguf
  - public headers include/openvino/frontend/ggml -> .../gguf
  - namespace ov::frontend::ggml -> ov::frontend::gguf
  - class GgmlDecoder -> GgufDecoder
  - visibility macros GGML_FRONTEND_(C_)API -> GGUF_FRONTEND_(C_)API
  - op-converter macro GGML_OP_CONVERTER -> GGUF_OP_CONVERTER
  - local rope helpers / variables ggml_* -> gguf_*
  - CMake target openvino_ggml_frontend -> openvino_gguf_frontend
  - build option ENABLE_OV_GGML_FRONTEND -> ENABLE_OV_GGUF_FRONTEND

Runtime op-type string keys ("GGML_OP_ADD", "GGML_OP_NONE",
"GGML_UNARY_OP_*", "GGML_GLU_OP_*") are kept as-is: they are the op
identifiers produced by the ggml graph and matched by the decoder, not
frontend symbols.
@mvafin
[GGUF FE] Add tests and typed decoder API; Q4_K integer zero-point
33a4fb5 
Add the frontend's test suite (per-op tests, weight/dequant-vs-ggml tests with
real-ggml reference data) and adopt the typed GgufDecoder API the translators
consume. Includes the Q4_K integer (u8) zero-point so the CPU/GPU plugin fuses
the dequant into the MatMul, matching the original ggml-openvino backend's
performance.
@mvafin
mvafin force-pushed the mvafin/gguf/frontend-port branch from ae0d7f8 to 87208c6 Compare July 16, 2026 20:12
@mvafin
[GGUF FE] Simplify the decoder API and refactor KV-cache handling
87208c6 
Consolidate the GgufDecoder interface into a small, typed, ggml-free contract
and move execution-mode concerns out of the frontend. The decoder shrinks from
the ported llama.cpp GgmlDecoder (node-index-threaded, raw ggml op_params, a
large get_view_input_* family) to a node-scoped interface the op translators
consume without ever touching ggml memory.

Decoder API:
- Node-scoped: visit_subgraph binds a decoder to a single node; per-node
  accessors take no node index. Model-scope queries (inputs, outputs, rope
  config, iteration) go through ov::frontend::gguf::InputModel, which becomes
  the model-scope facade over the decoder.
- Operation parameters are exposed as typed get_attribute<T>(key) values
  ("op_case", "output_type", "scale", "eps", "bias", "swapped", "max_bias",
  "is_swa", "view_seq_offset", "input_ggml_shape", "rope_config", ...) instead
  of raw int32 op_params arrays.
- Removed redundant/raw virtuals: get_input_stride, get_input_type,
  get_output_type, get_op_case, get_output_op_params/get_input_op_params, the
  get_view_input_* family, get_model_extra_inputs, and the model-scope
  duplicates. get_input_view_offset is renamed get_input_view_element_offset
  (returns elements, not bytes). get_input_shape is kept: MUL_MAT / FLASH_ATTN_EXT
  need the static ggml batch/head dims, which are dynamic on the live OV node in
  the stateful KV-cache path.
- Weights are surfaced as GGML_OP_NONE leaf nodes carrying a "data" attribute
  (translate_weight dequantizes), so there is no separate get_model_weights().

KV cache / execution mode:
- The frontend emits a typed SetRows placeholder op for every ggml SET_ROWS and
  lowers it to the stateless ScatterUpdate form by default. A caller wanting
  stateful execution registers an alternative lowering via
  DecoderTransformationExtension, so statefulness stays a backend concern and
  the frontend conversion is identical regardless of execution mode.
- Drop the "naive" flag: get_rope_config() tolerates a missing attribute
  (returns a default RopeConfig), so the shared LLM scaffolding (sliced mask +
  rope sin/cos table) is built only when the inputs are present and self-gates
  for a bare op / small cgraph. Both are loaded through FrontEnd::load().

Also: correctness fixes for dequant / ops and Q2_K/Q3_K packing; an MSVC C4244
warning fix; and header-include minimization with forward declarations.
@zhaixuejun1993

zhaixuejun1993 commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Seems like this version is too old, maybe u can update to latest.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cavusmustafa cavusmustafa cavusmustafa left review comments
@mlukasze mlukasze Awaiting requested review from mlukasze mlukasze is a code owner automatically assigned from openvinotoolkit/openvino-maintainers
@akuporos akuporos Awaiting requested review from akuporos akuporos is a code owner automatically assigned from openvinotoolkit/openvino-configuration-mgmt
@artanokhov artanokhov Awaiting requested review from artanokhov artanokhov is a code owner automatically assigned from openvinotoolkit/openvino-configuration-mgmt
@ravi9 ravi9 Awaiting requested review from ravi9

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

category: build OpenVINO cmake script / infra category: CI OpenVINO public CI category: Core OpenVINO Core (aka ngraph) category: CPP API OpenVINO CPP API bindings category: dependency_changes Pull requests that update a dependency file category: packaging OpenVINO packaging / distribution github_actions Pull requests that update GitHub Actions code no-match-files

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@mvafin @ravi9 @zhaixuejun1993 @cavusmustafa