Skip to content

CLAUDE.md

Latest commit

 

History

History
793 lines (619 loc) · 39.7 KB

CLAUDE.md

File metadata and controls

793 lines (619 loc) · 39.7 KB
Raw

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

Java bindings for llama.cpp via JNI, providing a high-level API for LLM inference in Java. The Java layer communicates with a native C++ library through JNI.

Current llama.cpp pinned version: b9682

Upgrading CUDA Version

Current CUDA version: 13.2

To change the CUDA version, update the following three places:

  1. .github/build_cuda_linux.sh — Line 10: sudo dnf install -y cuda-toolkit-13-2
  2. .github/build_cuda_linux.sh — Line 12: -DCMAKE_CUDA_COMPILER=/usr/local/cuda-13.2/bin/nvcc
  3. pom.xml — The <classifier> tag in the cuda jar execution: cuda13-linux-x86-64

Also update the header comment in build_cuda_linux.sh and the job name in .github/workflows/release.yaml for clarity.

Available CUDA versions for RHEL8/Manylinux_2_28 can be browsed at:

https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/

Note: Each CUDA version supports only certain GCC versions. If the dockcross container uses a newer GCC than CUDA supports, the build will fail with unsupported GNU version. Check NVIDIA's compatibility table before downgrading CUDA.

Example: To upgrade from 13.2 to a hypothetical 13.3:

# Edit .github/build_cuda_linux.sh:
#   line 10: cuda-toolkit-13-2 -> cuda-toolkit-13-3
#   line 12: /usr/local/cuda-13.2/bin/nvcc -> /usr/local/cuda-13.3/bin/nvcc
# Edit pom.xml classifier: cuda13-linux-x86-64 (major version only, no need to change for minor bumps)
# Edit CLAUDE.md line: Current CUDA version: **13.2** -> **13.3**
git add .github/build_cuda_linux.sh pom.xml CLAUDE.md
git commit -m "Upgrade CUDA from 13.2 to 13.3"

Android minimum API level

Current Android minimum API level: 28 (Android 9.0 Pie)

This is enforced through bionic's weak-symbol mechanism, not by bumping __ANDROID_API__ or passing -DANDROID_PLATFORM. See "How the API gate is satisfied" below for why. To change anything here, update:

  1. CMakeLists.txt — the add_compile_definitions(__ANDROID_UNAVAILABLE_SYMBOLS_ARE_WEAK__) block and its Android-detection guard (OS_NAME MATCHES "Android" etc.).
  2. CLAUDE.md (this file) — the "Current Android minimum API level" line above.
  3. README.md — the minimum-API note (the [!NOTE] block near the Android classifier entries and the "Importing in Android" section).

Why API 28? mtmd-helper.cpp (part of the upstream llama.cpp mtmd multimodal library) includes vendor/sheredom/subprocess.h, which calls posix_spawn, posix_spawnp, and posix_spawn_file_actions_*. Bionic only exposes those <spawn.h> declarations once the minimum SDK is ≥ 28 (and getifaddrs/freeifaddrs in <ifaddrs.h>, used by cpp-httplib, at ≥ 24). The symbols exist in libc.so at all API levels; bionic only hides the declarations below the introducing API.

How the API gate is satisfied (important — the obvious fixes do not work). The CI cross-compiler is the dockcross-android-arm64 image, which is not the Google NDK CMake toolchain — it is a Debian-style cross-clang at /usr/aarch64-linux-android/bin/clang. Consequently:

  • It never sets the ANDROID / ANDROID_ABI CMake variables, so any if(ANDROID_ABI)-guarded logic silently does nothing.
  • It ignores -DANDROID_PLATFORM=android-28 (CMake prints it as a "Manually-specified variables were not used by the project" warning).
  • clang predefines __ANDROID_API__ from its baked-in target triple, so -D__ANDROID_API__=28 would only clash with the builtin (-Wmacro-redefined) and would not move __ANDROID_MIN_SDK_VERSION__, which is what bionic's __BIONIC_AVAILABILITY_GUARD(api) actually tests.

The working fix is add_compile_definitions(__ANDROID_UNAVAILABLE_SYMBOLS_ARE_WEAK__) for the Android build. That macro forces __BIONIC_AVAILABILITY_GUARD(api) to 1 for every API level (declarations always visible) and makes any symbol newer than the toolchain's baked-in min-SDK a weak reference resolved by the dynamic linker at load time — present on every API-28+ device the artifact targets. It is never compiler-predefined, so defining it is clean. The guard detects Android via OS_NAME MATCHES "Android" (CI passes -DOS_NAME=Linux-Android) and the compiler path, not ANDROID_ABI.

OpenCL / Adreno backend on Android

A second Android arm64 artifact is built with the OpenCL backend enabled and Adreno-tuned kernels embedded. It ships under the Maven classifier opencl-android-aarch64 and is consumed only when callers explicitly request it. The default Android arm64 JAR remains CPU-only.

Three places wire it together (mirrors the CUDA classifier pattern):

  1. CMakeLists.txt — elseif(GGML_OPENCL) branch routes artifacts to src/main/resources_android_opencl/net/ladenthin/llama/${OS_NAME}/${OS_ARCH}/.
  2. .github/workflows/publish.yml — crosscompile-android-aarch64-opencl job runs the dockcross-android-arm64 build with -DGGML_OPENCL=ON -DGGML_OPENCL_EMBED_KERNELS=ON -DGGML_OPENCL_USE_ADRENO_KERNELS=ON and uploads as artifact android-libraries-opencl. The package, publish-snapshot, and publish-release jobs download it into resources_android_opencl/ and activate the opencl-android Maven profile.
  3. pom.xml — the opencl-android profile produces a second JAR with <classifier>opencl-android-aarch64</classifier> from the ${project.build.outputDirectory}_opencl_android tree.

Local sanity build:

.github/dockcross/dockcross-android-arm64 .github/build_opencl_android.sh \
  "-DOS_NAME=Linux-Android -DOS_ARCH=aarch64 \
   -DGGML_OPENCL=ON -DGGML_OPENCL_EMBED_KERNELS=ON \
   -DGGML_OPENCL_USE_ADRENO_KERNELS=ON"

Artifacts land in src/main/resources_android_opencl/net/ladenthin/llama/Linux-Android/aarch64/.

The dockcross image does not ship OpenCL headers or a stub libOpenCL.so, so build_opencl_android.sh first stages Khronos OpenCL-Headers and cross-builds OpenCL-ICD-Loader into /tmp/opencl-stage/ before invoking the main project cmake with -DOpenCL_INCLUDE_DIR=... and -DOpenCL_LIBRARY=.... At runtime the device must provide its own OpenCL ICD (libOpenCL.so); Qualcomm Adreno drivers do. Devices without an ICD should use the default CPU-only Android JAR.

Upgrading/Downgrading llama.cpp Version

To change the llama.cpp version, update the following three files:

  1. CMakeLists.txt — the GIT_TAG line for llama.cpp: GIT_TAG b8831
  2. README.md — the badge and link line with the version number
  3. CLAUDE.md — the "Current llama.cpp pinned version" line

Example: To upgrade from b8808 to b8831:

# Edit CMakeLists.txt: change GIT_TAG b8808 to b8831
# Edit README.md: change b8808 to b8831 (in both badge and link)
# Edit CLAUDE.md: change b8808 to b8831
git add CMakeLists.txt README.md CLAUDE.md
git commit -m "Upgrade llama.cpp from b8808 to b8831"
git push -u origin <your-branch>

Note: Always test the build with cmake -B build && cmake --build build --config Release after version changes to catch compatibility issues early.

Inspecting API changes between versions

Use the GitHub compare URL to diff any two llama.cpp builds:

https://github.com/ggml-org/llama.cpp/compare/b<FROM>...b<TO>

Example — what changed between b6721 and b6732:

https://github.com/ggml-org/llama.cpp/compare/b6721...b6732

The GitHub HTML page may time out for large ranges; fall back to the API:

https://api.github.com/repos/ggml-org/llama.cpp/compare/b<FROM>...b<TO>

For individual file content at a specific build:

https://raw.githubusercontent.com/ggerganov/llama.cpp/b<VERSION>/common/chat.h

Files to check for API compatibility

The three project C++ files (jllama.cpp, server.hpp, utils.hpp) pull in the following llama.cpp headers. Any of these can introduce breaking changes on upgrade.

Include dependency graph:

jllama.cpp / server.hpp / utils.hpp
│
├── arg.h ──────────────────────────► common.h ─┐
├── common.h ──────────────────────────────────►├── ggml-opt.h ──► ggml.h
├── chat.h ─────────────► common.h, peg-parser.h └── ggml-backend.h ──► ggml-alloc.h
├── speculative.h ──────► llama.h, common.h
├── sampling.h ─────────► llama.h, common.h
├── download.h ─────────► (stdlib only, no deps)
├── log.h ──────────────► ggml.h
├── llama.h ────────────────────────────────────► ggml.h, ggml-cpu.h, ggml-backend.h, ggml-opt.h
│                                                  └── llama-cpp.h ──► llama.h
├── json-schema-to-grammar.h
├── base64.hpp
├── mtmd.h
└── mtmd-helper.h

Priority-ordered review list for upgrade diffs (highest break risk first)

The top 8 rows cover all known API-level breaking changes from b5022 → b8831. For future upgrades, provide diffs for at least these 8 files rather than the full patch. Also review the project CMakeLists.txt for build-system-level breaks (e.g. renamed link targets, new required headers) — those are not visible in header file diffs alone.

File What to watch for
common/common.h common_params/common_params_speculative struct fields, model_alias container type, common_init_result shape, build_info symbol (removed in b8831 — now llama_build_info() from build-info.h)
common/chat.h common_chat_parser_params (was common_chat_syntax), to_json_oaicompat, common_chat_msg_diff_to_json_oaicompat, set_tool_call_ids
common/speculative.h common_speculative_init, common_speculative_draft, common_speculative_accept signatures, struct names
tools/mtmd/mtmd.h mtmd_context_params fields, image_marker/media_marker API, deprecated symbols (was common/mtmd.h before ~b8190)
include/llama-cpp.h common_init_result_ptr type, access pattern changes (.get() vs ->method())
common/arg.h n_parallel sentinel value, what moved to download.h across versions
include/llama.h Core llama_ function signatures, token types, llama_model_ptr, renamed structs
common/download.h common_remote_params struct, headers field format (string vs key-value pair)
common/common.cpp Implementation of any inline API used directly
common/speculative.cpp Speculative decoding implementation details
common/chat.cpp Chat parsing implementation
common/sampling.h Sampler API, common_sampler_* functions
common/log.h Log macro signatures
tools/mtmd/mtmd-helper.h Multimodal helper functions
common/json-schema-to-grammar.h Grammar API
ggml/include/ggml.h ggml_type enum values (e.g. GGML_TYPE_F16), tensor primitives
ggml/include/ggml-backend.h Backend/device abstraction types
ggml/include/ggml-opt.h Optimizer params pulled in via common.h

Safe to skip (have never caused a break; not used directly by project code): common/sampling.h, common/log.h, tools/mtmd/mtmd-helper.h, common/json-schema-to-grammar.h, ggml/include/ggml.h, ggml/include/ggml-backend.h, ggml/include/ggml-opt.h, ggml-alloc.h, ggml-cpu.h, peg-parser.h, base64.hpp

For the full record of upstream API breaks across version ranges (b5022 → current), including which rows required project source changes vs. which stayed inside upstream-compiled translation units, see docs/history/llama-cpp-breaking-changes.md. When bumping the llama.cpp version, append a new row to that file covering the upgrade range.

Build Commands

Java (Maven)

mvn compile          # Compiles Java and generates JNI headers
mvn test             # Run all tests (requires native library and model files)
mvn package          # Build JAR
mvn test -Dtest=LlamaModelTest#testGenerate  # Run a single test method

Native Library (CMake)

Must run mvn compile first to generate JNI headers, then:

# CPU only
cmake -B build
cmake --build build --config Release

# CUDA (Linux)
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release

# Metal (macOS)
cmake -B build -DLLAMA_METAL=ON
cmake --build build --config Release

# Optional: enable model downloading via URL
cmake -B build -DLLAMA_CURL=ON

Built libraries are placed in src/main/resources/net/ladenthin/llama/{OS}/{ARCH}/.

Building the native library for local Java tests

mvn test does not build the native library — Maven only compiles Java and runs surefire. The shared library must already exist on disk under the platform-specific resource path that LlamaLoader resolves at runtime. Without it the JVM throws UnsatisfiedLinkError and every Java test fails immediately (it does not auto-skip).

The output path is derived by CMakeLists.txt from OS_NAME and OS_ARCH detected by the helper script .github/dockcross/dockcross-resolve-host (falls back to uname on hosts where the script is absent). The mapping mirrors OSInfo.translateOSNameToFolderName on the Java side, so the same folder name is produced on both ends.

Host Library file Resource path produced by cmake --build
Linux x86_64 libjllama.so src/main/resources/net/ladenthin/llama/Linux/x86_64/
Linux aarch64 libjllama.so src/main/resources/net/ladenthin/llama/Linux/aarch64/
macOS Apple Silicon libjllama.dylib src/main/resources/net/ladenthin/llama/Mac/aarch64/
macOS Intel libjllama.dylib src/main/resources/net/ladenthin/llama/Mac/x86_64/
Windows x86_64 jllama.dll (+ llama.dll, ggml.dll) src/main/resources/net/ladenthin/llama/Windows/x86_64/

The Windows RUNTIME_OUTPUT_DIRECTORY_* properties (CMakeLists.txt:266-269) deposit jllama.dll alongside the upstream llama.dll / ggml.dll; all three must remain co-located so the loader can resolve transitive imports.

End-to-end local workflow for running Java tests:

# 1. Generate JNI headers (one-time per Java API change)
mvn -q compile

# 2. Configure + build the native library for the current host
cmake -B build
cmake --build build --config Release -j$(nproc)
# The shared lib lands directly in src/main/resources/.../{OS}/{ARCH}/ —
# no separate install step is needed.

# 3. Ensure model files referenced by tests are present under models/.
#    The default test models (downloaded by CI in publish.yml) are:
curl -L --fail "$MODEL_URL"          --create-dirs -o models/codellama-7b.Q2_K.gguf
curl -L --fail "$RERANKING_MODEL_URL" --create-dirs -o models/jina-reranker-v1-tiny-en-Q4_0.gguf
curl -L --fail "$DRAFT_MODEL_URL"     --create-dirs -o models/AMD-Llama-135m-code.Q2_K.gguf
curl -L --fail "$REASONING_MODEL_URL" --create-dirs -o models/Qwen3-0.6B-Q4_K_M.gguf

# 4. Run tests. Tests that need a model file self-skip via Assume.assumeTrue()
#    when their GGUF is absent, so partial model availability is OK.
mvn test
# CPU-only host (no GPU): pin GPU layers to 0
mvn test -Dnet.ladenthin.llama.test.ngl=0
# Run a single test class or method
mvn test -Dtest=MemoryManagementTest
mvn test -Dtest=LlamaModelTest#testGenerateAnswer

Optional models referenced by individual tests are gated on a system property so CI can skip them cleanly when the GGUF is not downloaded. The full property → consumer → default table for every net.ladenthin.llama.* property the library understands (runtime + test) is the user-facing System Properties Reference in the README. The summary below covers only the optional-model bindings:

Property Default test that uses it Model
net.ladenthin.llama.nomic.path LlamaEmbeddingsTest#testNomicEmbedLoads nomic-embed-text-v1.5.f16.gguf (issue #98 regression)
net.ladenthin.llama.vision.model MultimodalIntegrationTest (closes #103 / #34) SmolVLM-500M-Instruct-Q8_0.gguf (any vision-capable GGUF works)
net.ladenthin.llama.vision.mmproj MultimodalIntegrationTest matching mmproj for the vision model, e.g. mmproj-SmolVLM-500M-Instruct-Q8_0.gguf
net.ladenthin.llama.vision.image MultimodalIntegrationTest committed default src/test/resources/images/test-image.jpg; override to any png/jpeg/webp/gif on disk

Run those tests by setting the property:

mvn test -Dtest=LlamaEmbeddingsTest#testNomicEmbedLoads \
         -Dnet.ladenthin.llama.nomic.path=models/nomic-embed-text-v1.5.f16.gguf
mvn test -Dtest=MultimodalIntegrationTest \
         -Dnet.ladenthin.llama.vision.model=models/SmolVLM-500M-Instruct-Q8_0.gguf \
         -Dnet.ladenthin.llama.vision.mmproj=models/mmproj-SmolVLM-500M-Instruct-Q8_0.gguf
# The vision.image property defaults to src/test/resources/images/test-image.jpg
# (a CC-BY-4.0 / MIT-granted photo of flowers and bees by the project author);
# override only if you want to test a different image.

MultimodalIntegrationTest self-skips when any of the three vision properties points at a missing path, so a partial setup (just the vision model + the committed image, no mmproj) lets the test class load without erroring.

Restricted-network environments. Some hosts (e.g. ephemeral remote execution sandboxes) block outbound traffic to huggingface.co. In that case downloading models for the Java tests is not possible from the host itself; the native library can still be built and the C++ test suite (ctest --test-dir build) still runs because it depends only on the upstream sources fetched at CMake configure time. Java tests should then be exercised either in CI (via .github/workflows/publish.yml) or on a developer machine with HF access; pre-staged models can also be uploaded into models/ out-of-band.

Verifying the native library loads without models (model-free smoke). Even with HuggingFace blocked you can still do the one piece of real native verification that does not need a GGUF: confirm the library loads and its JNI_OnLoad resolves every Java class it looks up by name. The model-gated tests cannot do this in a restricted sandbox — they self-skip via Assume.assumeTrue(model present) before the lib is ever loaded, so a plain mvn test is silent on load-time breakage. The full local recipe:

# 1. Build the native lib locally (FetchContent pulls llama.cpp from GitHub,
#    which is reachable even when huggingface.co is not):
mvn -q compile
cmake -B build -DBUILD_TESTING=ON
cmake --build build --config Release -j$(nproc)   # -> src/main/resources/.../<os>/<arch>/libjllama.so
# 2. Force LlamaModel.<clinit> (System.load -> JNI_OnLoad) with no model:
mvn test -Dtest=NativeLibraryLoadSmokeTest

NativeLibraryLoadSmokeTest (in the loader package) calls Class.forName("net.ladenthin.llama.LlamaModel"), which runs LlamaLoader.initialize() -> System.load() -> JNI_OnLoad, which in turn calls FindClass(...) for every JNI-referenced Java class. It passes when the lib loads cleanly, fails if the native-resource path in LlamaLoader is wrong (lib not found) or a FindClass/field-signature FQN in src/main/cpp/jllama.cpp is stale after a Java package move (lib loads but JNI_OnLoad throws NoClassDefFoundError: net/ladenthin/llama/...), and self-skips when libjllama is not on the classpath (pure-Java checkout, no CMake build) so it never breaks a build-less mvn test.

Both of those failure modes shipped on a branch once — the layered-package restructure left (a) LlamaLoader.getNativeResourcePath() deriving the resource root from the loader's own package (which moved to …loader) and (b) jllama.cpp still FindClass-ing the old flat paths — and neither was visible to a local mvn test (model tests skipped) or to the pure-Java unit tests. When you move a Java class the JNI layer references by name (LlamaModel [root], exception.LlamaException, value.LogLevel, args.LogFormat, callback.LoadProgressCallback), update the matching FindClass / "L…;" signature string in src/main/cpp/jllama.cpp and keep the native-resource root anchored at net/ladenthin/llama/ in LlamaLoader.NATIVE_RESOURCE_BASE (it must not track the loader's own Java package). This is the same "FQN/path not updated after a package move" class as the stale spotbugs-exclude.xml, PIT targetClasses, and CMakeLists.txt OSInfo repairs.

Code Formatting

C++ formatting is enforced in CI (.github/workflows/clang-format.yml) with a pinned clang-format — currently 22.1.5, installed via pip install clang-format==22.1.5. Format with that exact version before committing; a different clang-format version reflows code differently and will fail the check.

pip install "clang-format==22.1.5"
clang-format -i src/main/cpp/*.cpp src/main/cpp/*.hpp src/test/cpp/*.cpp   # Format C++ code

The generated JNI header src/main/cpp/jllama.h (produced by javac -h) is intentionally excluded. To bump the enforced version, update the pin in both the workflow (CLANG_FORMAT_VERSION) and this line, then reformat the whole tree with the new version in the same commit.

Javadoc — must build cleanly before mvn package

The release packaging job runs mvn package with the release profile, which attaches a javadoc jar via maven-javadoc-plugin. The plugin treats Javadoc tool errors as build failures (warnings are tolerated). After changing any public/protected Java API, verify the javadoc build succeeds locally:

mvn clean javadoc:jar -DskipTests=true -Dgpg.skip=true
# expected: BUILD SUCCESS

Common Javadoc errors that fail the build (not warnings):

  • Unbalanced HTML: </p> without a matching <p>, mismatched <ul>/<li>, stray closing tags. Symptom: error: unexpected end tag: </p>.
  • Invalid {@link …} targets: typo'd class, method, or parameter name.
  • Self-closing void HTML elements written as <br> inside <pre> blocks in HTML5 mode (rare but seen).

Common Javadoc warnings (do not fail the build, but should be cleaned up on new code):

  • no main description — a doc comment containing only @param/@return/@throws tags with no leading prose. Fix: add a one-line description before the tags.
  • no @return / no @param — public method missing the tag. Fix: add it.
  • no comment — public method/field/enum constant has no doc comment at all.
  • use of default constructor, which does not provide a comment — public class with no explicit constructor (the synthetic default has no Javadoc). Fix: add an explicit no-arg constructor with a Javadoc comment.

Preferred doc-comment shapes for getters and small value types:

/**
 * Brief one-line description of the value.
 *
 * @return the value
 */
public T getThing() { ... }

A bare /** @return … */ triggers no main description; add a leading sentence.

If the local check passes (BUILD SUCCESS), the mvn package job in .github/workflows/publish.yml will pass the attach-javadocs step.

Architecture

Two-Layer Design

Java layer (src/main/java/net/ladenthin/llama/):

  • LlamaModel — Main API class (AutoCloseable). Wraps native context for inference, embeddings, re-ranking, and tokenization.
  • ModelParameters / InferenceParameters — Builder-pattern parameter classes that serialize to JSON (extend JsonParameters) for passing to native code.
  • LlamaIterator / LlamaIterable — Streaming generation via Java Iterator/Iterable.
  • LlamaLoader — Extracts the platform-specific native library from the JAR to a temp directory, or finds it on java.library.path.
  • OSInfo — Detects OS and architecture for library resolution.
  • server.OpenAiCompatServer — Optional OpenAI-compatible HTTP endpoint built on the JDK's com.sun.net.httpserver (no new dependency). Serves POST /v1/chat/completions (streaming via SSE + non-streaming) and GET /v1/models by delegating to LlamaModel.chatComplete / LlamaModel.streamChatCompletion, so editors that speak the OpenAI protocol (e.g. VS Code Copilot "Custom Endpoint") can drive a local model. Streaming uses the native OAI chunk path (requestChatCompletionStream / receiveChatCompletionChunk), preserving delta.tool_calls.

Native layer (src/main/cpp/):

  • jllama.cpp — JNI implementation bridging Java calls to llama.cpp. ~1,215 lines; 17 native methods.
  • utils.hpp — Helper utilities (format helpers, argv stripping, token-piece serialisation).
  • json_helpers.hpp — Pure JSON transformation helpers (no JNI, no llama state). Independently unit-testable.
  • jni_helpers.hpp — JNI bridge helpers (handle management + server orchestration). Includes json_helpers.hpp.
  • Uses nlohmann/json for JSON deserialization of parameters.
  • The upstream server library (server-context.cpp, server-queue.cpp, server-task.cpp, server-models.cpp) is compiled directly into jllama via CMake — there is no hand-ported server.hpp fork.

Native Helper Architecture

The project C++ helpers follow a strict semantic split:

json_helpers.hpp — Pure data transforms.

  • Input: nlohmann::json, server_task_result_ptr, plain C++ types.
  • Output: json, std::vector, std::optional, plain C++ types.
  • Zero JNI calls (JNIEnv* never appears).
  • Zero llama state (llama_context*, llama_vocab*, server_context* never appear).
  • Functions are named without _impl suffix — they are the canonical implementation.
  • Testable with JSON literals and fake result objects; no JVM and no loaded model required.
  • Upstream server headers must be included by the translation unit first (they define server_task_result_ptr, json, etc.).

Functions: get_result_error_message, results_to_json, rerank_results_to_json, parse_encoding_format, extract_embedding_prompt, is_infill_request, parse_slot_prompt_similarity, parse_positive_int_config, wrap_stream_chunk.

log_helpers.hpp — Pure log-formatting transforms.

  • Input: ggml_log_level, message text (const char*), an explicit std::time_t timestamp.
  • Output: const char* level label / std::string JSON.
  • Zero JNI calls (JNIEnv* never appears).
  • Zero llama/server state — depends only on the ggml_log_level enum (from ggml.h) and nlohmann/json; no upstream server headers required (more standalone than json_helpers.hpp).
  • Functions are [[nodiscard]] inline, named without an _impl suffix — the canonical implementation.
  • Testable with literal levels/strings and a fixed timestamp; no JVM and no loaded model required.

Functions: log_level_name, format_log_as_json.

jni_helpers.hpp — JNI bridge helpers, split into two layers:

Layer A (no server headers required): handle management.

  • jllama_context struct — owns server_context (value member, pimpl inside), background worker thread, cached vocab, saved params, and a readers map for streaming tasks.
  • get_jllama_context_impl — reads Java ctx handle, returns the jllama_context* wrapper. Does NOT throw on zero handle (valid no-op for destructor-style calls).
  • require_json_field_impl — throws "<field> is required" if key is absent.
  • jint_array_to_tokens_impl — reads a Java int[] into std::vector<int32_t>.

Layer B (requires upstream server headers in the TU before jni_helpers.hpp): orchestration. Includes json_helpers.hpp so all bridge helpers can call transforms directly.

  • json_to_jstring_impl — serialises any json value to a JNI string via dump().
  • results_to_jstring_impl — delegates to results_to_json then json_to_jstring_impl.
  • vec_to_jarray_impl<JArray,JElem,CppElem> — generic C++ vector → JNI primitive array.
  • embedding_to_jfloat_array_impl — converts std::vector<float> to jfloatArray.
  • tokens_to_jint_array_impl — converts std::vector<int32_t> to jintArray.

Functions with _impl suffix are called directly from jllama.cpp.

Include order rule:

// In jllama.cpp and any TU that uses Layer B helpers:
#include "server-context.h"   // upstream server headers must come first
#include "server-queue.h"
#include "server-task.h"
#include "server-common.h"
#include "server-chat.h"
#include "jni_helpers.hpp"    // includes json_helpers.hpp internally

Adding a new pure transform (e.g. a new JSON field parser):

  • Add it to json_helpers.hpp. No JNI, no llama types.
  • Add tests to src/test/cpp/test_json_helpers.cpp.

Adding a new JNI bridge helper:

  • Add it to jni_helpers.hpp in the appropriate layer.
  • If it needs upstream server types, put it in Layer B (after the json_helpers.hpp include).
  • Add tests to src/test/cpp/test_jni_helpers.cpp.

Parameter Flow

Java parameters are serialized to JSON strings and passed to native code, which deserializes them using nlohmann/json. This avoids complex JNI field mapping for the many llama.cpp parameters.

Native Library Resolution

LlamaLoader tries in order:

  1. System property net.ladenthin.llama.lib.path
  2. java.library.path
  3. Extracts from JAR resources at net/ladenthin/llama/{os}/{arch}/

Cross-compilation

Docker-based cross-compilation scripts are in .github/dockcross/ for ARM/Android targets. CI workflows use these for non-x86 Linux builds.

Testing

Java tests

Require a model file. The CI downloads models from HuggingFace:

  • LlamaModel tests: CodeLlama-7B-GGUF (codellama-7b.Q2_K.gguf)
  • RerankingModel tests: Jina-Reranker model

Set the model path via system property or environment variable (see test files for exact property names).

Test files are in src/test/java/net/ladenthin/llama/ and src/test/java/examples/.

C++ unit tests

No JVM and no model file required. All tests run on pure data structures using mock objects. The binary is named jllama_test and is built by CMake when BUILD_TESTING=ON.

Commands

# 1. Configure (once per fresh clone or after CMakeLists.txt changes)
cmake -B build -DBUILD_TESTING=ON

# 2. Build (incremental; -j$(nproc) uses all CPU cores)
cmake --build build --config Release -j$(nproc)

# 3. Run all tests
ctest --test-dir build --output-on-failure

# Count tests across all files
grep -rn "^TEST\b\|^TEST_F\b\|^TEST_P\b" src/test/cpp/ | wc -l

# Run a single named test (GoogleTest filter syntax)
ctest --test-dir build --output-on-failure -R "ResultsToJson"

Test files

File Tests Scope
src/test/cpp/test_utils.cpp 156 Upstream helpers: server_tokens, server_grammar_trigger, gen_tool_call_id, json_value, json_get_nested_values, UTF-8 helpers, format_response_rerank, format_embeddings_response_oaicompat, oaicompat_completion_params_parse, oaicompat_chat_params_parse, are_lora_equal, strip_flag_from_argv, token_piece_value, json_is_array_and_contains_numbers, format_oai_sse, format_oai_resp_sse, format_anthropic_sse
src/test/cpp/test_server.cpp 188 Upstream result types: result_timings, task_params::to_json() (incl. dry_sequence_breakers, preserved_tokens, timings_per_token), completion_token_output, server_task_result_cmpl_partial (non-oaicompat + to_json_oaicompat + logprobs + to_json_oaicompat_chat + to_json_anthropic + dispatcher), server_task_result_cmpl_final (non-oaicompat + to_json_oaicompat + to_json_oaicompat_chat + to_json_oaicompat_chat_stream + to_json_anthropic + to_json_anthropic_stream + tool_calls + dispatcher), server_task_result_embd, server_task_result_rerank, server_task_result_metrics, server_task_result_slot_save_load, server_task_result_slot_erase, server_task_result_apply_lora, server_task_result_error, format_error_response, server_task::need_sampling(), server_task::n_tokens(), server_task::params_from_json_cmpl() (parsing pipeline + grammar routing + error paths), response_fields projection
src/test/cpp/test_json_helpers.cpp 47 All functions in json_helpers.hpp: get_result_error_message, results_to_json, rerank_results_to_json, parse_encoding_format, extract_embedding_prompt, is_infill_request, parse_slot_prompt_similarity, parse_positive_int_config, wrap_stream_chunk
src/test/cpp/test_log_helpers.cpp 13 All functions in log_helpers.hpp: log_level_name, format_log_as_json
src/test/cpp/test_jni_helpers.cpp 41 All functions in jni_helpers.hpp using a zero-filled JNINativeInterface_ mock

Current total: 445 tests (all passing).

Upstream source location (in CMake build tree)

llama.cpp is fetched via CMake FetchContent, pinned to GIT_TAG b9682.

build/_deps/llama.cpp-src/tools/server/   ← server-task.h, server-common.h, etc.
build/_deps/llama.cpp-src/include/        ← llama.h, llama-cpp.h
build/_deps/llama.cpp-src/common/         ← common.h, chat.h, arg.h, etc.

When reading a to_json() implementation to write tests against it, read from: build/_deps/llama.cpp-src/tools/server/server-task.cpp

Mock JNI pattern used in test_jni_helpers.cpp

// Zero-fill the interface so all unpatched fn pointers are nullptr
JNINativeInterface_ iface = {};
// Patch only the stubs this test needs, e.g.:
iface.GetLongField  = [](JNIEnv*, jobject, jfieldID) -> jlong { return some_handle; };
iface.ThrowNew      = [](JNIEnv*, jclass, const char*) -> jint { return 0; };
// Wire up the env
JNIEnv_ fake_env = {};
fake_env.functions = &iface;
JNIEnv *env = &fake_env;

Any stub that is called but not patched will crash (null function pointer) — deliberately, so missing stubs are caught immediately rather than silently.

How to add a new C++ test

  1. Open the appropriate src/test/cpp/test_*.cpp:
    • Pure JSON transform → test_json_helpers.cpp
    • JNI helper → test_jni_helpers.cpp
    • Upstream result type to_json() → test_server.cpp
    • utils.hpp function or upstream utility → test_utils.cpp
  2. Add a TEST(SuiteName, TestName) { ... } block using GoogleTest macros.
  3. Rebuild: cmake --build build --config Release -j$(nproc)
  4. Run: ctest --test-dir build --output-on-failure
  5. Commit with message summarising coverage added and new test total.

Finding untested code paths

# List all functions defined in a header
grep -n "^inline\|^static\|^\[\[nodiscard\]\]" src/main/cpp/utils.hpp

# Check which functions already have tests
grep -n "function_name" src/test/cpp/*.cpp

# Find all fields in an upstream to_json() method
grep -n "\"field_name\"" build/_deps/llama.cpp-src/tools/server/server-task.cpp

# Check which JSON fields Java actually reads (important: must test these)
grep -rn "field_name" src/main/java/net/ladenthin/llama/

Testing complex scenarios — methodology

Simple tests verify individual field values on a default-constructed struct. Complex tests verify control flow: switch dispatchers, cross-cutting flags, and multi-step parameter pipelines. The same build/run/commit loop applies.

1. Dispatcher (switch) coverage

Every to_json() that is a switch on res_type has one test per arm:

// Pattern: set is_updated=true, set res_type, call to_json(), check the
// distinguishing field that differs between arms.
server_task_result_cmpl_final f;
f.is_updated = true;
f.stream     = false;
f.res_type   = TASK_RESPONSE_TYPE_OAI_CMPL;
// ... set required fields ...
const json j = f.to_json();
EXPECT_EQ(j.at("object").get<std::string>(), "text_completion");

The same pattern handles the stream flag fork inside OAI_CHAT: stream=false → single object with "object":"chat.completion"; stream=true → JSON array of chunks with "object":"chat.completion.chunk".

2. Cross-cutting flag interaction

Some flags (verbose, include_usage, timings.prompt_n) cut across multiple formatters. Test each flag in one formatter only — they share the same code path:

// verbose=true must add __verbose to the first chunk/top-level object
f.verbose = true;
EXPECT_TRUE(j.contains("__verbose"));

// timings absent when prompt_n < 0 (default), present when >= 0
f.timings.prompt_n = 5;
EXPECT_TRUE(j.contains("timings"));

3. Parameter parsing (params_from_json_cmpl) without a model

server_task::params_from_json_cmpl(vocab, params_base, n_ctx_slot, logit_bias_eog, data) can be called with nullptr vocab if the JSON does not trigger grammar/preserved_tokens tokenisation (those are the only vocab-dependent paths). This lets us test the full parsing pipeline including error throws:

common_params          params_base;
std::vector<llama_logit_bias> no_bias;
const int n_ctx = 512;

// test: repeat_last_n=-1 is expanded to n_ctx_slot
json data = {{"repeat_last_n", -1}};
auto p = server_task::params_from_json_cmpl(nullptr, params_base, n_ctx, no_bias, data);
EXPECT_EQ(p.sampling.penalty_last_n, n_ctx);

// test: invalid value throws std::runtime_error
json bad = {{"dry_sequence_breakers", json::array()}};  // empty → error
EXPECT_THROW(server_task::params_from_json_cmpl(nullptr, params_base, n_ctx, no_bias, bad),
             std::runtime_error);

4. Array-returning formatters

Some methods (e.g. to_json_oaicompat_chat_stream()) return a JSON array of event objects, not a single object. Check with is_array() first, then iterate or index:

const json j = f.to_json_oaicompat_chat_stream();
ASSERT_TRUE(j.is_array());
ASSERT_GE(j.size(), 1u);
// Last chunk always has a non-null finish_reason
EXPECT_FALSE(j.back().at("choices")[0].at("finish_reason").is_null());

5. response_fields projection

to_json_non_oaicompat() supports a projection list via response_fields. When non-empty, only those dot-separated paths survive:

f.response_fields = {"content", "tokens_predicted"};
const json j = f.to_json_non_oaicompat();
EXPECT_TRUE(j.contains("content"));
EXPECT_FALSE(j.contains("stop_type"));  // filtered out

Key Constraints

  • Java 8+ runtime required. Built with JDK 21 targeting bytecode 1.8 for broad compatibility.
  • Native memory allocated by llama.cpp is not GC-managed — always use LlamaModel in try-with-resources or call close() explicitly.
  • The server.hpp file is adapted from llama.cpp upstream — minimize modifications to ease future upgrades.
  • Platform-specific native libraries must be pre-built and placed under src/main/resources/ before packaging for distribution.

Javadoc Conventions

See ../workspace/policies/javadoc-conventions.md.

SpotBugs Suppressions

See ../workspace/policies/spotbugs-suppressions.md.

Spotless Formatting

See ../workspace/policies/spotless-formatting.md. Run mvn spotless:apply before every commit that touches .java files.

jqwik Policy

See ../workspace/policies/jqwik-prompt-injection.md.

Lombok Config

See ../workspace/policies/lombok-config.md.

CI Test Diagnostics

See ../workspace/policies/ci-test-diagnostics.md.

JPMS Module Descriptor

This repo ships a module-info.java compiled in a separate release 9 execution. Javadoc currently runs in classpath mode (javadoc <source> is 1.8), which is the only thing keeping it clear of the JPMS module-mode javadoc trap that bit BAF. Before raising the Java / javadoc source level to ≥ 9, read ../workspace/policies/jpms-module-descriptor.md.

Open TODOs

Open TODOs for this repo live in TODO.md. Cross-repo status tracking lives in ../workspace/crossrepostatus.md.