You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
odek supports prompt caching for supported LLM providers. When enabled, the system prompt and first user message are annotated with cache markers, reducing both latency and cost on repeated interactions.
1. Moves the system prompt from the `messages[]` array into a dedicated `system` field with `cache_control: {"type": "ephemeral"}` (Anthropic format — silently ignored by other providers)
16
+
2. Marks the first user message with `cache_control: {"type": "ephemeral"}`
17
+
3. Sends the `anthropic-version: 2023-06-01` header (required by Anthropic for caching; ignored by others)
18
+
19
+
## Enabling
20
+
21
+
### CLI
22
+
```bash
23
+
odek run --prompt-caching "Does this work with caching?"
24
+
```
25
+
26
+
The `--prompt-caching` flag is available on `odek run`, `odek repl`, and `odek serve`.
27
+
28
+
### Config file (`~/.odek/config.json` or `./odek.json`)
29
+
```json
30
+
{
31
+
"prompt_caching": true
32
+
}
33
+
```
34
+
35
+
### Environment variable
36
+
```bash
37
+
export ODEK_PROMPT_CACHING=true
38
+
```
39
+
40
+
### Programmatic API
41
+
```go
42
+
agent, err:= odek.New(odek.Config{
43
+
Model: "claude-sonnet-4",
44
+
APIKey: os.Getenv("ANTHROPIC_API_KEY"),
45
+
BaseURL: "https://api.anthropic.com/v1",
46
+
PromptCaching: true,
47
+
})
48
+
```
49
+
50
+
## When to Enable
51
+
52
+
**Enable for:**
53
+
- Anthropic models (Claude 3.5 Sonnet, Claude 3 Opus, Claude 4 Sonnet) — explicit cache markers provide the largest benefit
54
+
- DeepSeek models — automatic prefix caching works best when the conversation prefix is stable; caching markers don't hurt
55
+
- Any multi-turn session where the system prompt is large (e.g., AGENTS.md files, loaded skills) — the system prompt is cached after the first iteration
56
+
57
+
**Disable for:**
58
+
- One-shot tasks where the agent runs exactly one iteration
59
+
- Providers that don't support caching and have unusual request parsing (safety: unknown fields are ignored by all major providers, but caching is always opt-in via `--prompt-caching`)
60
+
61
+
## Cache Metrics
62
+
63
+
When caching is active, odek tracks cache metrics and exposes them through the agent API:
64
+
65
+
```go
66
+
agent.TotalCacheCreationTokens() // Anthropic: tokens written to cache
67
+
agent.TotalCacheReadTokens() // Anthropic: tokens read from cache hit
These are accumulated across all iterations of the most recent run. Zero values mean the provider didn't return cache metrics (either caching is not enabled, or the provider doesn't report them).
72
+
73
+
## How It Works
74
+
75
+
1.**Before each LLM call**, if `PromptCaching` is enabled, the loop calls `llm.ApplyCacheMarkers(messages)` which:
76
+
- Extracts the first system message and converts it to an Anthropic `SystemBlock` with `cache_control: ephemeral`
77
+
- Marks the first user message with `cache_control: ephemeral`
78
+
79
+
2.**The request is sent** with the system in the `system` field (not `messages[]`) and the cache markers in place. Providers that don't support these fields silently ignore them.
80
+
81
+
3.**The response is parsed** for cache metrics from both Anthropic (`cache_creation_input_tokens`, `cache_read_input_tokens`) and OpenAI (`prompt_tokens_details.cached_tokens`).
82
+
83
+
4.**Metrics are accumulated** across iterations and exposed via the agent API.
84
+
85
+
## Implementation Details
86
+
87
+
- The `anthropic-version: 2023-06-01` header is always sent when caching is enabled. This header is required by Anthropic for prompt caching and is ignored by OpenAI and DeepSeek.
88
+
- Cache markers are applied **per iteration** — the system prompt and first user message are marked on every LLM call. This is safe because the markers reference the same content each time, so the cache is populated on the first iteration and read on subsequent ones.
89
+
- The `max_tokens` field is now included in all requests when set via `odek.Config.MaxTokens` or model profile defaults. Some providers (Anthropic) tie caching behavior to this field being present.
90
+
- The system prompt is moved out of `messages[]` into a separate `system` field only when caching is enabled. When disabled, it stays in `messages[]` for maximum provider compatibility.
0 commit comments