Commit 3786fa3
committed
feat: skip response buffering for passthrough and internal model requests
Write SkipResponseBufferingKey to CycleState when response translation
will be skipped, enabling real-time streaming instead of buffering the
full response body:
- Internal models (no ExternalModel match): no translation needed
- Passthrough (inputFormat == outputFormat, not openai-chat): body
passes through unmodified
Depends on: llm-d/llm-d-inference-payload-processor#2701 parent 7889f6a commit 3786fa3
1 file changed
Lines changed: 15 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
163 | 163 | | |
164 | 164 | | |
165 | 165 | | |
| 166 | + | |
166 | 167 | | |
167 | 168 | | |
168 | 169 | | |
| |||
199 | 200 | | |
200 | 201 | | |
201 | 202 | | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
202 | 207 | | |
203 | 208 | | |
204 | 209 | | |
205 | 210 | | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
206 | 221 | | |
207 | 222 | | |
208 | 223 | | |
| |||
0 commit comments