|
2 | 2 | --- |
3 | 3 | title: Llama Class |
4 | 4 | class_name: Llama |
5 | | -last_updated: 2026-04-23 |
| 5 | +last_updated: 2026-04-26 |
6 | 6 | version_target: "latest" |
7 | 7 | --- |
8 | 8 | ``` |
@@ -103,6 +103,10 @@ Low-level method to ingest and evaluate a sequence of tokens. Used internally to |
103 | 103 | model.eval(tokens=[1, 453, 234, 987], active_loras=[{"name": "coding_adapter", "scale": 1.0}]) |
104 | 104 | ``` |
105 | 105 |
|
| 106 | +### `abort` |
| 107 | +Immediately halts an active generation loop safely. |
| 108 | +* **Usage**: Typically called from a separate monitoring thread (like a timer). When triggered, the running stream will exit and the final chunk will contain `"finish_reason": "abort"`. |
| 109 | + |
106 | 110 | ### Dynamic LoRA Management |
107 | 111 | The `Llama` class allows you to load multiple LoRAs into VRAM and apply them dynamically per-generation or per-eval. |
108 | 112 | * `load_lora(name: str, path: str)`: Loads an adapter into VRAM (does not apply it yet). |
@@ -193,6 +197,102 @@ The `Llama` class allows you to load multiple LoRAs into VRAM and apply them dyn |
193 | 197 | ctx_checkpoints=0 # <-- SET THIS TO 0 TO ENABLE ZERO-LATENCY FAST PATH |
194 | 198 | ) |
195 | 199 | ``` |
| 200 | +6. **Assistant Prefill**: |
| 201 | + |
| 202 | + `llama-cpp-python` supports native **Assistant Prefill** for seamless message continuation. You can now simply use the `assistant_prefill=True` parameter in the `create_chat_completion` function. |
| 203 | + |
| 204 | + This safely renders the `N-1` conversation history using standard Jinja templates (preserving exact control tokens) and flawlessly appends your partial text directly to the prompt. |
| 205 | + |
| 206 | + ```python |
| 207 | + from llama_cpp import Llama |
| 208 | + |
| 209 | + llm = Llama(model_path="path/to/model.gguf") |
| 210 | + |
| 211 | + # An interrupted/partial conversation |
| 212 | + messages = [ |
| 213 | + {"role": "user", "content": "What are the first 5 planets in the solar system?"}, |
| 214 | + {"role": "assistant", "content": "The first 5 planets in our solar system are:\n1. Mercury\n2."} |
| 215 | + ] |
| 216 | + |
| 217 | + # Seamlessly continue the generation |
| 218 | + response = llm.create_chat_completion( |
| 219 | + messages=messages, |
| 220 | + max_tokens=50, |
| 221 | + assistant_prefill=True # <--- Enables seamless continuation |
| 222 | + ) |
| 223 | + |
| 224 | + prefilled_text = messages[-1]["content"] |
| 225 | + # The model will flawlessly continue from " Venus\n3. Earth..." |
| 226 | + generated_text = response["choices"][0]["message"]["content"] |
| 227 | + |
| 228 | + print(prefilled_text + generated_text) |
| 229 | + ``` |
| 230 | + |
| 231 | +7. **Interrupting Reasoning & Assistant Prefill (Time-boxing)**: |
| 232 | + |
| 233 | + Use the `abort()` method alongside `assistant_prefill=True` to forcefully stop a reasoning model (like Qwen or DeepSeek) if it thinks for too long, inject a bridge text, and force it to output the final answer. |
| 234 | + ```python |
| 235 | + import threading |
| 236 | + from llama_cpp import Llama |
| 237 | + |
| 238 | + llm = Llama(model_path="Qwen3.6-27B.gguf", n_ctx=4096, n_gpu_layers=-1) |
| 239 | + |
| 240 | + def run_controlled_generation(prompt: str, timeout_seconds: int = 10): |
| 241 | + messages = [{"role": "user", "content": prompt}] |
| 242 | + |
| 243 | + # 1. Set a time bomb to interrupt long <think> phases |
| 244 | + def timeout_handler(): |
| 245 | + llm.abort() |
| 246 | + |
| 247 | + timer = threading.Timer(timeout_seconds, timeout_handler) |
| 248 | + timer.start() |
| 249 | + |
| 250 | + stream = llm.create_chat_completion( |
| 251 | + messages=messages, max_tokens=2048, stream=True |
| 252 | + ) |
| 253 | + |
| 254 | + partial_response = "" |
| 255 | + finish_reason = None |
| 256 | + |
| 257 | + for chunk in stream: |
| 258 | + finish_reason = chunk["choices"][0].get("finish_reason") |
| 259 | + |
| 260 | + if finish_reason is not None and finish_reason != "abort": |
| 261 | + timer.cancel() |
| 262 | + break |
| 263 | + |
| 264 | + if finish_reason == "abort": |
| 265 | + break |
| 266 | + |
| 267 | + delta = chunk["choices"][0]["delta"].get("content", "") |
| 268 | + if delta: |
| 269 | + partial_response += delta |
| 270 | + print(delta, end="", flush=True) |
| 271 | + |
| 272 | + # 2. Forced Intervention and Prefill Continuation |
| 273 | + if finish_reason == "abort": |
| 274 | + # Inject bridge text to forcefully close the reasoning tag |
| 275 | + bridge_text = "\n...Wait, I have thought long enough, let's start answering the user.\n</think>\n\n" |
| 276 | + print(bridge_text, end="", flush=True) |
| 277 | + |
| 278 | + prefilled_content = partial_response + bridge_text |
| 279 | + messages.append({"role": "assistant", "content": prefilled_content}) |
| 280 | + |
| 281 | + # Use assistant_prefill=True to seamlessly continue the text block |
| 282 | + stream_part2 = llm.create_chat_completion( |
| 283 | + messages=messages, |
| 284 | + max_tokens=2048, |
| 285 | + stream=True, |
| 286 | + assistant_prefill=True |
| 287 | + ) |
| 288 | + |
| 289 | + for chunk in stream_part2: |
| 290 | + delta = chunk["choices"][0]["delta"].get("content", "") |
| 291 | + if delta: |
| 292 | + print(delta, end="", flush=True) |
| 293 | + |
| 294 | + run_controlled_generation("Explain quantum mechanics in a way that relates to bugs in code.", timeout_seconds=8) |
| 295 | + ``` |
196 | 296 |
|
197 | 297 | --- |
198 | 298 |
|
|
0 commit comments