Merge branch 'JamePeng:main' into main

Author: alcoftTAO

.github/workflows/build-wheels-metal.yaml

@@ -54,7 +54,7 @@ jobs:
             -DGGML_METAL=on
             -DGGML_METAL_USE_BF16=on
             -DGGML_METAL_EMBED_LIBRARY=off
-            -DGGML_METAL_SHADER_DEBUG=on
+            -DGGML_METAL_SHADER_DEBUG=on"
         with:
           package-dir: .
           output-dir: wheelhouse2

CHANGELOG.md

@@ -7,6 +7,79 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.3.27]
+
+- feat: add `PaddleOCR-VL-1.5` multimodal chat handler `PaddleOCRChatHandler`
+
+- fix: resolve VRAM leak in multimodal models by explicitly closing mtmd context
+    - Remove the `ExitStack` closure in `Llava15ChatHandler` to break circular references preventing garbage collection of the vision context.
+    - Implement explicit `close()` and `__del__()` methods in the chat handler to safely free `mtmd_ctx`.
+    - Integrate `chat_handler.close()` into the main `Llama.close()` lifecycle and nullify related attributes for immediate memory reclamation.
+
+- fix: resolve reload memory leaks by breaking circular references in cleanup
+    - Remove closure-based `ExitStack` callbacks in `LlamaModel`, `LlamaContext`, and `LlamaBatch` to eliminate circular references.
+    - Move explicit C-level memory freeing (`llama_*_free`) directly into `close()` methods.
+    - Nullify additional attributes (`tokenizer_`, `model_params`, etc.) in `Llama.close()` to guarantee immediate garbage collection during unload.
+    - Add todo comment to LoRA process logic, the current LoRa loading logic is outdated and needs to be refactored.
+
+- feat: add memory breakdown and sampler performance timings APIs
+
+- feat: Search system paths for shared libraries(`by @benniekiss`)
+
+- Optimize `longest_token_prefix` to use zero-copy NumPy arrays and drop .tolist() overhead
+
+- Free _candidates and large numpy arrays during explicit close()
+
+- fix: resolve memory leaks in sampling context lifecycle
+    - Safely close temporary `LlamaSamplingContext` in `sample()` using a try-finally block.
+    - Explicitly release the previous `_sampling_ctx` in `generate()` before re-assignment to prevent orphaned pointers.
+    - Ensure `_sampling_ctx` is properly freed in `Llama.close()`.
+
+- Fix custom sampler memory cleanup and improve lifecycle management
+    - Add explicit `close()` and `__del__()` to CustomSampler to safely free C resources and break Python reference cycles
+    - Ensure custom samplers are properly detached and freed in `LlamaSampler.close()`
+    - Add minor documentation comments for clarity
+
+- feat: Update llama.cpp to [ggml-org/llama.cpp/commit/cacc371f99fb3b5b431d3fa89ac0c752bbd62a3b](https://github.com/ggml-org/llama.cpp/commit/cacc371f99fb3b5b431d3fa89ac0c752bbd62a3b)
+
+- feat: Sync llama.cpp llama/mtmd API Binding 20260223
+
+More information see: https://github.com/JamePeng/llama-cpp-python/compare/3d0fd1b75ee564361a4babf21f88855225ba1fe0...1f8341ee74a2fea15a1008487cbecaccf918c755
+
+## [0.3.26]
+- perf(llama-cpp): optimize LlamaTokenDataArray memory operations
+    - Cache NumPy field views for 'id', 'logit', and 'p' to bypass expensive property lookups.
+    - Refactor copy_logits to use pre-generated ID sequences and cached views.
+    - Ensure logical consistency by resetting token IDs every sampling step to counter C++ reordering.
+    - Minimize redundant memory allocations during the inference loop.
+
+- feat: Add explicit memory cleanup for sampling contexts
+    - Implements `close()` and `__del__` for LlamaTokenDataArray and expands LlamaSamplingContext cleanup.
+    - Ensures NumPy views and internal C-references are properly released to allow Python GC to reclaim memory.
+
+- optimize(memory): reduce scores buffer size and optimize state saving
+    - Update save_state and load_state API use.
+    - Refactored self.scores to allocate only a single row (1, n_vocab) when logits_all=False, significantly reducing memory usage for large vocabulary models.
+    - Optimized save_state to eliminate redundant memory allocations and copies by using ctypes.string_at.
+    - Updated load_state, eval, and sampler adapters to correctly handle the dynamic shape of self.scores.
+
+- Fix CMake install layout to avoid top-level bin directory in site-packages
+
+- ggml: Load ggml library from candidate path list
+    - Auto-select lib/ or bin/ directories
+    - Add backend loading functions
+
+- feat(loader): extend default library search paths on Linux and macOS
+    - `load_shared_library` to include a path list feature (allowing you to add custom paths in addition to the default ones). You can later add your own paths to the `libggml_base_paths` candidate list in `_ggml.py`, such as those not commonly used Python paths.
+    - fix: Enhance the handling logic for non-existent file paths.
+    - Improves reliability of shared library discovery for system-wide installations.
+
+- feat: Update llama.cpp to [ggml-org/llama.cpp/commit/abb9f3c42b5e6acee9e8e37836ef691d1a41bdb8](https://github.com/ggml-org/llama.cpp/commit/abb9f3c42b5e6acee9e8e37836ef691d1a41bdb8)
+
+- feat: Sync llama.cpp llama/mtmd API Binding 20260219
+
+More information see: https://github.com/JamePeng/llama-cpp-python/compare/b03224b2dde3c8cbdd8bf529794e3a41ac7f5751...6b38182a17effd0cedbf8736bee2464dd6c7b4da
+
 ## [0.3.25]
 - feat: [Refactor Llama class to use new LlamaSampler chain API from _internals](https://github.com/JamePeng/llama-cpp-python/commit/1e6094a327f0fb9dc35d52f84d8ebabc1faa1e95)
 This commit refactors the high-level Llama class to fully utilize the new C++ `llama_sampler` chain architecture via `LlamaSamplingContext`.

CMakeLists.txt

@@ -5,6 +5,12 @@ project(llama_cpp)
 option(LLAMA_BUILD "Build llama.cpp shared library and install alongside python package" ON)
 option(MTMD_BUILD "Build mtmd shared library and install alongside python package" ON)
 
+
+set(CMAKE_INSTALL_BINDIR llama_cpp/bin CACHE PATH "" FORCE)
+set(CMAKE_INSTALL_LIBDIR llama_cpp/lib CACHE PATH "" FORCE)
+set(CMAKE_INSTALL_INCLUDEDIR llama_cpp/include CACHE PATH "" FORCE)
+
+
 # Helper function to install targets to Python package directories
 function(llama_cpp_python_install_target target)
     if(NOT TARGET ${target})
@@ -37,7 +43,7 @@ function(llama_cpp_python_install_target target)
         endif()
     endforeach()
 
-    # Configure RPATH
+    # Proper RPATH handling
     if(UNIX)
         set(INSTALL_RPATH_VAL "$ORIGIN")
         if(APPLE)
@@ -68,6 +74,9 @@ if (LLAMA_BUILD)
     # Enable building of the common library
     set(LLAMA_BUILD_COMMON ON CACHE BOOL "llama.cpp: build common utils library" FORCE)
 
+    # Enable build and link OpenSSL
+    set(LLAMA_OPENSSL ON CACHE BOOL "llama.cpp: build and link OpenSSL" FORCE)
+
     # Disable building of examples
     set(LLAMA_BUILD_EXAMPLES OFF CACHE BOOL "llama.cpp: build examples" FORCE)
 
@@ -77,9 +86,6 @@ if (LLAMA_BUILD)
     # Disable building curl support
     set(LLAMA_CURL OFF CACHE BOOL "llama.cpp: use libcurl to download model from an URL" FORCE)
 
-    # Enable build and link OpenSSL
-    set(LLAMA_OPENSSL ON CACHE BOOL "llama.cpp: build and link OpenSSL" FORCE)
-
     # Architecture detection and settings for Apple platforms
     if (APPLE)
         # If CMAKE_OSX_ARCHITECTURES is not set, use the host architecture
@@ -107,10 +113,8 @@ if (LLAMA_BUILD)
 
     add_subdirectory(vendor/llama.cpp)
 
-    if (WIN32)
-        if (TARGET llama)
-            set_target_properties(llama PROPERTIES WINDOWS_EXPORT_ALL_SYMBOLS ON)
-        endif()
+    if (WIN32 AND TARGET llama)
+        set_target_properties(llama PROPERTIES WINDOWS_EXPORT_ALL_SYMBOLS ON)
     endif()
 
     # Define list of GGML targets to install

README.md

@@ -519,6 +519,7 @@ Below are the supported multi-modal models and their respective chat handlers (P
 | [glm4.6v](https://huggingface.co/unsloth/GLM-4.6V-Flash-GGUF) | `GLM46VChatHandler` | `glm4.6v` |
 | [granite-docling](https://huggingface.co/ibm-granite/granite-docling-258M-GGUF) | `GraniteDoclingChatHandler` | `granite-docling` |
 | [lfm2-vl](https://huggingface.co/LiquidAI/LFM2-VL-3B-GGUF) | `LFM2VLChatHandler` | `lfm2-vl` |
+| [paddleocr-vl-1.5](https://huggingface.co/JamePeng2023/PaddleOCR-VL-1.5-GGUF) | `PaddleOCRChatHandler` | `paddleocr` |
 | [qwen2.5-vl](https://huggingface.co/unsloth/Qwen2.5-VL-3B-Instruct-GGUF) | `Qwen25VLChatHandler` | `qwen2.5-vl` |
 | [qwen3-vl](https://huggingface.co/unsloth/Qwen3-VL-8B-Thinking-GGUF) | `Qwen3VLChatHandler` | `qwen3-vl` |
 

llama_cpp/__init__.py

@@ -1,4 +1,4 @@
 from .llama_cpp import *
 from .llama import *
 
-__version__ = "0.3.25"
+__version__ = "0.3.27"

llama_cpp/_ctypes_extensions.py

@@ -5,7 +5,7 @@
 import ctypes
 import functools
 import pathlib
-
+from ctypes.util import find_library
 from typing import (
     Any,
     Callable,
@@ -20,25 +20,37 @@
 
 
 # Load the library
-def load_shared_library(lib_base_name: str, base_path: pathlib.Path):
-    """Platform independent shared library loader"""
-    # Searching for the library in the current directory under the name "libllama" (default name
-    # for llamacpp) and "llama" (default name for this repo)
-    lib_paths: List[pathlib.Path] = []
-    # Determine the file extension based on the platform
+def load_shared_library(lib_base_name: str, base_paths: Union[pathlib.Path, list[pathlib.Path]]):
+    if isinstance(base_paths, pathlib.Path):
+        base_paths = [base_paths]
+
+    lib_names = []
+
     if sys.platform.startswith("linux") or sys.platform.startswith("freebsd"):
-        lib_paths += [
-            base_path / f"lib{lib_base_name}.so",
-        ]
+        lib_names = [f"lib{lib_base_name}.so"]
+
+        base_paths.extend([
+            "/usr/local/lib",
+            "/usr/lib",
+            "/usr/lib64",
+        ])
+
     elif sys.platform == "darwin":
-        lib_paths += [
-            base_path / f"lib{lib_base_name}.so",
-            base_path / f"lib{lib_base_name}.dylib",
+        lib_names = [
+            f"lib{lib_base_name}.dylib",
+            f"lib{lib_base_name}.so",
         ]
+
+        base_paths.extend([
+            "/usr/local/lib",
+            "/opt/homebrew/lib",
+            "/usr/lib",
+        ])
+
     elif sys.platform == "win32":
-        lib_paths += [
-            base_path / f"{lib_base_name}.dll",
-            base_path / f"lib{lib_base_name}.dll",
+        lib_names = [
+            f"{lib_base_name}.dll",
+            f"lib{lib_base_name}.dll",
         ]
     else:
         raise RuntimeError("Unsupported platform")
@@ -47,11 +59,17 @@ def load_shared_library(lib_base_name: str, base_path: pathlib.Path):
 
     # Add the library directory to the DLL search path on Windows (if needed)
     if sys.platform == "win32":
-        os.add_dll_directory(str(base_path))
-        os.environ["PATH"] = str(base_path) + os.pathsep + os.environ["PATH"]
-
-    if sys.platform == "win32" and sys.version_info >= (3, 8):
-        os.add_dll_directory(str(base_path))
+        for base_path in base_paths:
+            p = pathlib.Path(base_path)
+            if p.exists() and p.is_dir():
+                os.add_dll_directory(str(p))
+                os.environ["PATH"] = str(p) + os.pathsep + os.environ["PATH"]
+
+    if sys.platform == "win32" and sys.version_info >= (3, 9):
+        for base_path in base_paths:
+            p = pathlib.Path(base_path)
+            if p.exists() and p.is_dir():
+                os.add_dll_directory(str(p))
         if "CUDA_PATH" in os.environ:
             cuda_path = os.environ["CUDA_PATH"]
             sub_dirs_to_add = [
@@ -75,16 +93,30 @@ def load_shared_library(lib_base_name: str, base_path: pathlib.Path):
 
         cdll_args["winmode"] = ctypes.RTLD_GLOBAL
 
-    # Try to load the shared library, handling potential errors
-    for lib_path in lib_paths:
-        if lib_path.exists():
-            try:
-                return ctypes.CDLL(str(lib_path), **cdll_args)  # type: ignore
-            except Exception as e:
-                raise RuntimeError(f"Failed to load shared library '{lib_path}': {e}")
-
-    raise FileNotFoundError(
-        f"Shared library with base name '{lib_base_name}' not found"
+    errors = []
+
+    # First, try to find an available library through the system
+    lib_path = find_library(lib_base_name)
+    if lib_path:
+        try:
+            return ctypes.CDLL(lib_path, **cdll_args)
+        except Exception as e:
+            errors.append(f"{lib_path}: {e}")
+
+    # Then fallback to manually checking the list of paths.
+    for base_path in base_paths:
+        for lib_name in lib_names:
+            lib_path = pathlib.Path(base_path) / lib_name
+
+            if lib_path.exists():
+                try:
+                    return ctypes.CDLL(str(lib_path), **cdll_args)
+                except Exception as e:
+                    errors.append(f"{lib_path}: {e}")
+
+    raise RuntimeError(
+        f"Failed to load '{lib_base_name}' from {base_paths}\n"
+        + "\n".join(errors)
     )
 
 

llama_cpp/_ggml.py

@@ -2,12 +2,16 @@
 
 This module provides a minimal interface for working with ggml tensors from llama-cpp-python
 """
+import ctypes
 import enum
 import os
 import pathlib
-import ctypes
 
-import llama_cpp._ctypes_extensions as ctypes_ext
+from llama_cpp._ctypes_extensions import (
+    load_shared_library,
+    byref,
+    ctypes_function_for_shared_library,
+)
 
 from typing import (
     Callable,
@@ -17,8 +21,15 @@
     TYPE_CHECKING,
 )
 
-libggml_base_path = pathlib.Path(os.path.abspath(os.path.dirname(__file__))) / "lib"
-libggml = ctypes_ext.load_shared_library("ggml", libggml_base_path)
+libggml_base_path = pathlib.Path(os.path.abspath(os.path.dirname(__file__)))
+libggml_base_paths = [
+    libggml_base_path / "lib",
+    libggml_base_path / "bin",
+]
+
+libggml = load_shared_library("ggml", libggml_base_paths)
+
+ggml_function = ctypes_function_for_shared_library(libggml)
 
 # // ====== ggml.h ======
 
@@ -361,3 +372,20 @@ class ggml_opt_optimizer_params(ctypes.Structure):
 ggml_backend_sched_eval_callback = ctypes.CFUNCTYPE(
     ctypes.c_bool, ctypes.c_void_p, ctypes.c_bool, ctypes.c_void_p
 )
+
+# //
+# // Backend registry
+# //
+
+# // Load all known backends from dynamic libraries
+# GGML_API void               ggml_backend_load_all(void);
+@ggml_function("ggml_backend_load_all", [], None)
+def ggml_backend_load_all():
+    """Load all known backends from dynamic libraries"""
+    ...
+
+# GGML_API void               ggml_backend_load_all_from_path(const char * dir_path);
+@ggml_function("ggml_backend_load_all_from_path", [ctypes.c_char_p], None)
+def ggml_backend_load_all_from_path(dir_path: ctypes.c_char_p):
+    """Load all known backends from path"""
+    ...