Skip to content

Commit 314e729

Browse files
authored
llama : document that only one on-device state can be saved per sequence (ggml-org#23520)
1 parent d55fb97 commit 314e729

1 file changed

Lines changed: 2 additions & 1 deletion

File tree

include/llama.h

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -874,7 +874,8 @@ extern "C" {
874874
// work only with partial states, such as SWA KV cache or recurrent cache (e.g. Mamba)
875875
#define LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY 1
876876

877-
// keeps the tensor data on device buffers (i.e. not accessible in host memory, but faster save/load)
877+
// Keeps the tensor data on device buffers (i.e. not accessible in host memory, but faster save/load).
878+
// Getting the state for a seq_id with this flag invalidates all prior states gotten for that seq_id with this flag.
878879
#define LLAMA_STATE_SEQ_FLAGS_ON_DEVICE 2
879880

880881
typedef uint32_t llama_state_seq_flags;

0 commit comments

Comments
 (0)