llama-cpp-python

[Nwokike]

Hello Flet team!

I am building a local AI agent app for Android/iOS using Flet. I rely on llama-cpp-python for running local GGUF models directly on device.

Because llama-cpp-python relies on building the llama.cpp C++ engine from source, it currently throws a missing wheel error ("No matching distribution found") during the flet build apk process since there are no pre-compiled Android/iOS wheels available on PyPI.

Package Details:

• Name: llama-cpp-python
• PyPI: https://pypi.org/project/llama-cpp-python/
• Use Case: High-performance native bindings for llama.cpp. This is absolutely critical for allowing us to execute LLM inference directly on mobile devices, which is a strict requirement for offline/local-first AI applications.

Could we get llama-cpp-python added to the mobile forge (pypi.flet.dev) so we can run LLMs natively on Flet mobile?

Thank you!

[ndonkoHenri]

We already have it working, and it will be shipped soon: flet-dev/mobile-forge#91
Thanks for your patience.

[Nwokike]

@ndonkoHenri That's amazing news, you guys are on fire! 🔥
Will be watching the PR closely. Thank you!

[2messmer]

This is really exciting news for the community! It’s great to see that the request is being addressed so quickly by the contributors.
​Running LLMs locally via Flet opens up so many possibilities for privacy-focused and offline AI agents. I’m curious now that this is coming, what kind of projects are you all planning to build with it? I’m looking forward to experimenting with some lightweight models myself. Looking forward to seeing what everyone else comes up with!

[Nwokike]

Hi @ndonkoHenri! Thank you so much for putting this recipe together. Having 5 out of the 6 platform matrices green (especially Android 3.12–3.14 and iOS 3.13–3.14) is amazing progress!

I noticed that the iOS build for Python 3.12 is currently the only compilation failing on the CI, along with the zizmor linter.

Since I'm targeting Android first for my app, is there any way I can download and test the compiled Android Python 3.12 wheels locally in my project? Also, let me know if there's anything I can help with to help get the Python 3.12 iOS build resolved or tested!

[ndonkoHenri]

Will be merged in few hours. 🤞🏾

[ndonkoHenri]

llama-cpp-python 0.3.32.1 is now supported on Android and iOS: https://pypi.flet.dev/llama-cpp-python

It's a CPU-only build (all GPU backends — Metal/CUDA/Vulkan/OpenCL — and OpenMP are disabled, since none are portable across mobile), which is plenty for the quantized models that realistically fit on a phone. Everything runs offline; the only network you need is a one-time model download.

Install

Add it to your deps:

# pyproject.toml
dependencies = [
    "flet",
    "llama-cpp-python",
]

[tool.flet.android]
target_arch = ["arm64-v8a", "x86_64", "armeabi-v7a"]  # if on Flet < 0.86

Storage & the model file (the important bit)

GGUF weights are large — a 1–3B Q4 model is ~0.7–2 GB — so don't bundle them in the app (you'll blow past store size limits). Download the .gguf once on first run into Flet's persistent storage (FLET_APP_STORAGE_DATA) and load it from there:

import os, urllib.request

data_dir = os.getenv("FLET_APP_STORAGE_DATA")
model_path = os.path.join(data_dir, "model.gguf")
if not os.path.exists(model_path):
    urllib.request.urlretrieve("https://<your-host>/model.gguf", model_path)

huggingface_hub works too — just point its cache at storage (os.environ["HF_HOME"] = os.getenv("FLET_APP_STORAGE_DATA")) before importing it.

Minimal usage

import os
from llama_cpp import Llama

model_path = os.path.join(os.getenv("FLET_APP_STORAGE_DATA"), "model.gguf")

llm = Llama(
    model_path=model_path,
    n_ctx=2048,        # keep modest on mobile RAM
    n_threads=4,
    n_gpu_layers=0,    # CPU only
    verbose=False,
)
out = llm("The capital of France is", max_tokens=32, stop=["\n"])
print(out["choices"][0]["text"])

Run it off the UI thread

Model load and generation are CPU-bound and take a while — do them in page.run_thread(...) so the UI stays responsive.

What actually runs well

• Sweet spot: 1–3B-parameter models at Q4_K_M. A ~1B model (Gemma 3 1B, Llama 3.2 1B, Qwen2.5 1.5B) is a smooth daily-driver; 3B is usable on a flagship; 7B is loadable on 12 GB+ devices but slow.
• Rough speed: ~8–20 tok/s for 1–3B Q4 on modern phones (faster on recent iPhones).
• Keep n_ctx modest (2–4k) — large contexts spike memory and the OS may kill the app.
• armeabi-v7a (old 32-bit devices) is included but not recommended; prefer arm64.

Let me know if you face any issues!

[Nwokike]

@ndonkoHenri This is incredible, thank you! The detailed recipe with storage tips, threading advice, and model sizing is exactly what we needed. Already adding it to my project. 🙏

Question; I also posted a discussion about a flet-xterm terminal extension (#6669). Would love your thoughts on whether wrapping xterm.dart as a Flet package is something the team might consider, or if you'd recommend we maintain it as a community package.

[ndonkoHenri]

Would love your thoughts on whether wrapping xterm.dart as a Flet package is something the team might consider, or if you'd recommend we maintain it as a community package.

You can build a Flet extension for it: https://flet.dev/docs/extend/user-extensions