Skip to content

Commit 7a4dd5b

Browse files
committed
Update /docs/wiki/core/Llama.md for abort() and example code
Signed-off-by: JamePeng <jame_peng@sina.com>
1 parent 2cccd2e commit 7a4dd5b

1 file changed

Lines changed: 101 additions & 1 deletion

File tree

docs/wiki/core/Llama.md

Lines changed: 101 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
---
33
title: Llama Class
44
class_name: Llama
5-
last_updated: 2026-04-23
5+
last_updated: 2026-04-26
66
version_target: "latest"
77
---
88
```
@@ -103,6 +103,10 @@ Low-level method to ingest and evaluate a sequence of tokens. Used internally to
103103
model.eval(tokens=[1, 453, 234, 987], active_loras=[{"name": "coding_adapter", "scale": 1.0}])
104104
```
105105

106+
### `abort`
107+
Immediately halts an active generation loop safely.
108+
* **Usage**: Typically called from a separate monitoring thread (like a timer). When triggered, the running stream will exit and the final chunk will contain `"finish_reason": "abort"`.
109+
106110
### Dynamic LoRA Management
107111
The `Llama` class allows you to load multiple LoRAs into VRAM and apply them dynamically per-generation or per-eval.
108112
* `load_lora(name: str, path: str)`: Loads an adapter into VRAM (does not apply it yet).
@@ -193,6 +197,102 @@ The `Llama` class allows you to load multiple LoRAs into VRAM and apply them dyn
193197
ctx_checkpoints=0 # <-- SET THIS TO 0 TO ENABLE ZERO-LATENCY FAST PATH
194198
)
195199
```
200+
6. **Assistant Prefill**:
201+
202+
`llama-cpp-python` supports native **Assistant Prefill** for seamless message continuation. You can now simply use the `assistant_prefill=True` parameter in the `create_chat_completion` function.
203+
204+
This safely renders the `N-1` conversation history using standard Jinja templates (preserving exact control tokens) and flawlessly appends your partial text directly to the prompt.
205+
206+
```python
207+
from llama_cpp import Llama
208+
209+
llm = Llama(model_path="path/to/model.gguf")
210+
211+
# An interrupted/partial conversation
212+
messages = [
213+
{"role": "user", "content": "What are the first 5 planets in the solar system?"},
214+
{"role": "assistant", "content": "The first 5 planets in our solar system are:\n1. Mercury\n2."}
215+
]
216+
217+
# Seamlessly continue the generation
218+
response = llm.create_chat_completion(
219+
messages=messages,
220+
max_tokens=50,
221+
assistant_prefill=True # <--- Enables seamless continuation
222+
)
223+
224+
prefilled_text = messages[-1]["content"]
225+
# The model will flawlessly continue from " Venus\n3. Earth..."
226+
generated_text = response["choices"][0]["message"]["content"]
227+
228+
print(prefilled_text + generated_text)
229+
```
230+
231+
7. **Interrupting Reasoning & Assistant Prefill (Time-boxing)**:
232+
233+
Use the `abort()` method alongside `assistant_prefill=True` to forcefully stop a reasoning model (like Qwen or DeepSeek) if it thinks for too long, inject a bridge text, and force it to output the final answer.
234+
```python
235+
import threading
236+
from llama_cpp import Llama
237+
238+
llm = Llama(model_path="Qwen3.6-27B.gguf", n_ctx=4096, n_gpu_layers=-1)
239+
240+
def run_controlled_generation(prompt: str, timeout_seconds: int = 10):
241+
messages = [{"role": "user", "content": prompt}]
242+
243+
# 1. Set a time bomb to interrupt long <think> phases
244+
def timeout_handler():
245+
llm.abort()
246+
247+
timer = threading.Timer(timeout_seconds, timeout_handler)
248+
timer.start()
249+
250+
stream = llm.create_chat_completion(
251+
messages=messages, max_tokens=2048, stream=True
252+
)
253+
254+
partial_response = ""
255+
finish_reason = None
256+
257+
for chunk in stream:
258+
finish_reason = chunk["choices"][0].get("finish_reason")
259+
260+
if finish_reason is not None and finish_reason != "abort":
261+
timer.cancel()
262+
break
263+
264+
if finish_reason == "abort":
265+
break
266+
267+
delta = chunk["choices"][0]["delta"].get("content", "")
268+
if delta:
269+
partial_response += delta
270+
print(delta, end="", flush=True)
271+
272+
# 2. Forced Intervention and Prefill Continuation
273+
if finish_reason == "abort":
274+
# Inject bridge text to forcefully close the reasoning tag
275+
bridge_text = "\n...Wait, I have thought long enough, let's start answering the user.\n</think>\n\n"
276+
print(bridge_text, end="", flush=True)
277+
278+
prefilled_content = partial_response + bridge_text
279+
messages.append({"role": "assistant", "content": prefilled_content})
280+
281+
# Use assistant_prefill=True to seamlessly continue the text block
282+
stream_part2 = llm.create_chat_completion(
283+
messages=messages,
284+
max_tokens=2048,
285+
stream=True,
286+
assistant_prefill=True
287+
)
288+
289+
for chunk in stream_part2:
290+
delta = chunk["choices"][0]["delta"].get("content", "")
291+
if delta:
292+
print(delta, end="", flush=True)
293+
294+
run_controlled_generation("Explain quantum mechanics in a way that relates to bugs in code.", timeout_seconds=8)
295+
```
196296

197297
---
198298

0 commit comments

Comments
 (0)