Skip to content

Commit 95fb27f

Browse files
noyitznoyitz
committed
docs: design proposal 044 — plugin body mode capabilities
Signed-off-by: Noy Itzikowitz <nitzikow@redhat.com>
1 parent f9837c9 commit 95fb27f

1 file changed

Lines changed: 261 additions & 0 deletions

File tree

  • docs/proposals/044-plugin-body-mode-capabilities

‎docs/proposals/044-plugin-body-mode-capabilities/README.md‎

Lines changed: 261 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,261 @@
1+
# Plugin Body Mode Capabilities
2+
3+
Author(s): @noyitz
4+
5+
## Proposal Status
6+
***Proposed***
7+
8+
## Summary
9+
10+
This proposal introduces a **body mode capability declaration** mechanism for IPP plugins. Each plugin declares what level of access it needs to request and response bodies, enabling the framework to compute the optimal Envoy ext_proc `processing_mode` and validate it against the actual deployment configuration.
11+
12+
Today, `request_body_mode` and `response_body_mode` are global Envoy settings applied to all plugins uniformly. This forces a "most demanding wins" approach at the infrastructure level — if any plugin needs the full response body, streaming breaks for everyone. This proposal moves the decision to the plugin layer, where the framework can reason about requirements, warn on misconfigurations, and enable streaming-compatible plugins to coexist with body-dependent ones.
13+
14+
## Problem Statement
15+
16+
The ext_proc `response_body_mode` is set to `FULL_DUPLEX_STREAMED` in the default MaaS deployment. The IPP framework accumulates all response body chunks in memory before processing ([server.go#L96-L146](https://github.com/llm-d/llm-d-inference-payload-processor/blob/main/pkg/handlers/server.go#L96)), which means:
17+
18+
1. **Streaming is broken** — clients see all response chunks arrive at once after the full response is collected, resulting in multi-second TTFT for long responses
19+
2. **Memory is unbounded** — each concurrent request accumulates the full response body (`responseBody = append(responseBody, ...)`) with no upper limit. A 200K token streaming response is ~1MB in memory per request
20+
3. **All plugins pay the cost** — even plugins that don't need the response body (e.g., `apikey-injection`) force full buffering because the mode is global
21+
22+
Different plugins have fundamentally different needs:
23+
24+
| Plugin | Request Body | Response Body | Can Process Chunks? |
25+
|--------|-------------|---------------|-------------------|
26+
| `body-field-to-header` | Needs full body (parse JSON field) | Not needed | No |
27+
| `api-translation` | Needs full body (format conversion) | Needs full body (reverse conversion) | No |
28+
| `external-metering` | Not needed | Needs usage from final SSE event | Yes — only reads final chunk |
29+
| `apikey-injection` | Not needed | Not needed | N/A |
30+
| `nemo-guard` | Needs full body (content inspection) | Needs full body (output inspection) | No |
31+
32+
## Design Principles
33+
34+
- Plugins declare capabilities, the framework decides behavior
35+
- Existing plugins that don't declare capabilities continue to work unchanged (backward compatible)
36+
- The framework validates but does not silently override — operators make the final decision
37+
- Chunk processing is opt-in — plugins that can handle streaming declare it explicitly
38+
39+
## Definitions
40+
41+
- **BodyNeed** — what a plugin requires from a body (none, chunked, or buffered)
42+
- **ChunkProcessor** — a plugin that can process individual response body chunks as they stream through
43+
- **Effective mode** — the most demanding body need across all active plugins
44+
- **Mode mismatch** — when the Envoy config doesn't match the computed effective mode
45+
46+
## Proposal
47+
48+
### 1. Body Mode Declaration Interface
49+
50+
Plugins optionally implement `BodyModeCapability` to declare their requirements:
51+
52+
```go
53+
// BodyModeCapability allows plugins to declare what they need from request/response bodies.
54+
// Plugins that don't implement this interface default to BodyFull (backward compatible).
55+
type BodyModeCapability interface {
56+
BodyRequirements() BodyRequirements
57+
}
58+
59+
type BodyRequirements struct {
60+
RequestBody BodyNeed
61+
ResponseBody BodyNeed
62+
}
63+
64+
type BodyNeed int
65+
66+
const (
67+
// BodyNotNeeded — plugin only works with headers, doesn't read or modify the body.
68+
BodyNotNeeded BodyNeed = iota
69+
70+
// BodyChunked — plugin can process individual body chunks as they arrive.
71+
// Plugins declaring this MUST also implement ChunkProcessor.
72+
// Streaming-compatible: client sees chunks in real-time.
73+
BodyChunked
74+
75+
// BodyFull — plugin needs the complete body in memory before processing.
76+
// NOT streaming-compatible: client waits for full response.
77+
BodyFull
78+
)
79+
```
80+
81+
The plugin declares what IT needs — it doesn't know about Envoy modes. The framework maps plugin needs to Envoy configuration:
82+
83+
| Highest plugin need | Envoy `body_mode` |
84+
|--------------------|-------------------|
85+
| All `BodyNotNeeded` | `NONE` |
86+
| Any `BodyChunked` | `STREAMED` |
87+
| Any `BodyFull` | `BUFFERED` |
88+
89+
### 2. Chunk-Aware Response Processing
90+
91+
Plugins that can handle streaming response chunks implement `ChunkProcessor`:
92+
93+
```go
94+
// ChunkProcessor allows a plugin to process individual response body chunks
95+
// without waiting for the full body. The framework calls ProcessResponseChunk
96+
// for each chunk, then calls the standard ProcessResponse with the full
97+
// accumulated body on the final chunk.
98+
type ChunkProcessor interface {
99+
ProcessResponseChunk(ctx context.Context, cycleState *CycleState, chunk []byte, isFinal bool) ([]byte, error)
100+
}
101+
```
102+
103+
Constraints:
104+
- Plugins declaring `BodyChunked` MUST implement `ChunkProcessor` — validated at startup
105+
- `ProcessResponseChunk` returns modified bytes (to mutate the chunk forwarded to the client) or nil (pass through unchanged)
106+
- The standard `ProcessResponse` is still called on the final chunk with the full accumulated body — this is backward compatible and allows the plugin to do aggregated work (e.g., emit a metering event with total token counts)
107+
108+
### 3. Effective Mode Computation
109+
110+
At startup, after all plugins are loaded, the framework computes the effective body mode:
111+
112+
```
113+
Priority: NONE < CHUNKED < BUFFERED
114+
115+
For request body:
116+
If ANY plugin needs BUFFERED → effective = BUFFERED
117+
If ANY plugin needs CHUNKED → effective = CHUNKED (STREAMED in Envoy terms)
118+
If ALL plugins need NONE → effective = NONE
119+
120+
Same logic for response body, independently.
121+
```
122+
123+
Mapping to Envoy ext_proc modes:
124+
125+
| Effective Need | Envoy `body_mode` | Behavior |
126+
|---------------|-------------------|----------|
127+
| `BodyNotNeeded` | `NONE` | Body not sent to ext_proc |
128+
| `BodyChunked` | `STREAMED` | Chunks sent one at a time, acked immediately |
129+
| `BodyFull` | `BUFFERED` | Full body buffered, sent as one message |
130+
131+
### 4. Startup Validation
132+
133+
The framework does **not** auto-override the Envoy configuration. It validates and reports:
134+
135+
```go
136+
type ModeValidation struct {
137+
EffectiveRequestMode BodyNeed
138+
EffectiveResponseMode BodyNeed
139+
Warnings []string
140+
Errors []string
141+
}
142+
```
143+
144+
**Validation rules:**
145+
146+
| Envoy Config | Effective Need | Result | Example |
147+
|-------------|---------------|--------|---------|
148+
| BUFFERED | CHUNKED | **WARNING**: streaming broken unnecessarily | "response_body_mode is BUFFERED but only STREAMED is needed — client streaming is disabled" |
149+
| BUFFERED | NONE | **WARNING**: unnecessary overhead | "response_body_mode is BUFFERED but no plugin needs the response body" |
150+
| NONE | CHUNKED | **ERROR**: plugin will fail | "external-metering requires response body but response_body_mode is NONE — token metering disabled" |
151+
| NONE | BUFFERED | **ERROR**: plugin will fail | "api-translation requires full response body but response_body_mode is NONE" |
152+
| STREAMED | BUFFERED | **ERROR**: plugin will get chunks, not full body | "api-translation requires full response body but response_body_mode is STREAMED" |
153+
| STREAMED | CHUNKED | OK | ✅ |
154+
| Config matches need | — | OK | ✅ |
155+
156+
**Example startup log output:**
157+
158+
```
159+
INFO plugin body mode analysis
160+
computed_request_mode=BUFFERED
161+
computed_response_mode=CHUNKED
162+
request_body_plugins=["body-field-to-header", "api-translation"]
163+
response_body_plugins=["external-metering"]
164+
chunk_capable=["external-metering"]
165+
166+
WARN response_body_mode mismatch
167+
configured=FULL_DUPLEX_STREAMED recommended=STREAMED
168+
detail="Current mode accumulates all response chunks before forwarding.
169+
Use STREAMED for real-time streaming with token metering."
170+
171+
ERROR response_body_mode insufficient
172+
configured=NONE required=STREAMED
173+
plugin=external-metering
174+
detail="Plugin requires response body access for token extraction.
175+
Token metering will not work with current configuration."
176+
```
177+
178+
### 5. Streaming Response Processing Flow
179+
180+
When `response_body_mode` is `STREAMED` and plugins declare `BodyChunked`:
181+
182+
```
183+
Backend streams response
184+
│
185+
▼
186+
Envoy sends chunk₁ ──────────────────────────────────────────────────┐
187+
│ │
188+
Server.Process() │
189+
├── Accumulate: responseBody = append(responseBody, chunk₁) │
190+
├── ChunkProcessor plugins: ProcessResponseChunk(chunk₁) │
191+
├── Send ack to Envoy (BodyResponse{}) ─────────────────────┤
192+
│ Envoy forwards │
193+
│ chunk₁ to client │
194+
▼ │
195+
Envoy sends chunk₂ ──────────────────────────────────────────────────┤
196+
│ │
197+
├── Accumulate: responseBody = append(responseBody, chunk₂) │
198+
├── ChunkProcessor plugins: ProcessResponseChunk(chunk₂) │
199+
├── Send ack ───────────────────────────────────────────────┤
200+
│ → chunk₂ to client
201+
▼ │
202+
Envoy sends chunk₃ (EndOfStream=true) ──────────────────────────────┘
203+
│
204+
├── Accumulate final chunk
205+
├── ChunkProcessor plugins: ProcessResponseChunk(chunk₃, isFinal=true)
206+
├── Parse full accumulated body (JSON or SSE by Content-Type)
207+
├── ResponseProcessor plugins: ProcessResponse(fullBody)
208+
└── Send final response to Envoy
209+
```
210+
211+
Key properties:
212+
- **Client gets real-time streaming** — each chunk is acked and forwarded immediately
213+
- **ChunkProcessor plugins** see each chunk in real-time for lightweight per-chunk work
214+
- **ResponseProcessor plugins** still get the full parsed body on the final chunk (backward compatible)
215+
- **Memory accumulation** still happens (for the final ProcessResponse call) — this is a known trade-off documented in the code. A future enhancement could allow chunk-only plugins to opt out of accumulation entirely
216+
217+
### 6. Migration Path
218+
219+
**Existing plugins** that don't implement `BodyModeCapability`:
220+
- Default to `{RequestBody: BodyFull, ResponseBody: BodyFull}`
221+
- Framework logs: `WARN plugin "X" does not declare body requirements, assuming BUFFERED for both`
222+
- No breaking changes — everything works exactly as before
223+
224+
**New plugins** should implement `BodyModeCapability`. The framework validates `ChunkProcessor` implementation at startup:
225+
- Declares `BodyChunked` but doesn't implement `ChunkProcessor` → startup **ERROR**
226+
- Implements `ChunkProcessor` but declares `BodyFull` → `ChunkProcessor` methods are never called (no harm, but logged as INFO)
227+
228+
### 7. Future: Dynamic Mode Override
229+
230+
Envoy ext_proc supports `mode_override` in responses, allowing the server to change the processing mode per-request. A future enhancement could use this:
231+
232+
1. During request processing, after the ProfilePicker selects a profile, check if the selected profile's response plugins need body access
233+
2. If no response plugin needs the body → override `response_body_mode` to `NONE` for this request
234+
3. This is a per-request optimization, not a global config change
235+
236+
This is explicitly **out of scope** for the initial implementation. It requires careful testing and is opt-in via configuration:
237+
238+
```yaml
239+
framework:
240+
dynamicModeOverride: false # default: validate only, don't override
241+
```
242+
243+
### 8. Implementation Plan
244+
245+
| Phase | Scope | Files |
246+
|-------|-------|-------|
247+
| 1 | Add `BodyModeCapability` and `ChunkProcessor` interfaces | `pkg/framework/interface/requesthandling/plugins.go` |
248+
| 2 | Add `BodyRequirements` types | `pkg/framework/interface/requesthandling/types.go` |
249+
| 3 | Add validation logic at startup | `pkg/config/loader/configloader.go` |
250+
| 4 | Add chunk ack + ChunkProcessor dispatch in server loop | `pkg/handlers/server.go` |
251+
| 5 | Declare capabilities on existing plugins | Each plugin's `plugin.go` |
252+
| 6 | Add startup validation logging | `pkg/handlers/server.go` or `cmd/main.go` |
253+
254+
### 9. Relation to PR #138
255+
256+
[PR #138](https://github.com/llm-d/llm-d-inference-payload-processor/pull/138) (SSE response parsing) addresses a subset of this proposal:
257+
- Adds chunk acks for non-final response body chunks (phase 4 above)
258+
- Adds SSE parsing for token extraction from streaming responses
259+
- Does NOT add the capability declaration or validation — plugins don't declare their needs
260+
261+
This proposal builds on PR #138's streaming infrastructure and adds the framework-level intelligence to reason about plugin requirements.

0 commit comments

Comments
 (0)