Commit cf8f0b1
feat(opt-b01): Continuous Batching — 2-3x throughput, iteration-level scheduling, 13 tests
continuous_batching.zig (891 lines): Orca/vLLM-style continuous batch scheduler
- BatchScheduler: fixed-pool request management, batch slot lifecycle
- Priority queue: effective priority with wait-time boosting, insertion sort
- Iteration scheduling: completion detection, slot freeing, queue admission
- Continuous admission: new requests join immediately as slots free
- Preemption: lowest-priority eviction, return to queue
- Request lifecycle: queued → prefill → generating → completed/cancelled
- Token budget enforcement: max_tokens_per_iter limits batch expansion
- SchedulerStats: avg batch size, tokens/iter, wait iterations, preemptions
- ThroughputAnalysis: static vs continuous batching comparison
- 13 tests: config, submit, admit, batch cap, completion, continuous flow,
priority ordering, preemption, cancel, stats, throughput, empty batch
- Zig 0.15.2 compatible (zero std.ArrayList, stack-safe fixed arrays)
- build.zig: test-continuous-batching step wired
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>1 parent e1be645 commit cf8f0b1
3 files changed
Lines changed: 970 additions & 6 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| 23 | + | |
| 24 | + | |
23 | 25 | | |
24 | 26 | | |
25 | 27 | | |
| |||
88 | 90 | | |
89 | 91 | | |
90 | 92 | | |
91 | | - | |
92 | 93 | | |
93 | 94 | | |
94 | 95 | | |
| |||
105 | 106 | | |
106 | 107 | | |
107 | 108 | | |
108 | | - | |
| 109 | + | |
109 | 110 | | |
110 | 111 | | |
111 | 112 | | |
112 | 113 | | |
113 | 114 | | |
114 | 115 | | |
115 | 116 | | |
116 | | - | |
| 117 | + | |
117 | 118 | | |
118 | 119 | | |
119 | | - | |
120 | | - | |
121 | | - | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
122 | 123 | | |
123 | 124 | | |
124 | 125 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1843 | 1843 | | |
1844 | 1844 | | |
1845 | 1845 | | |
| 1846 | + | |
| 1847 | + | |
| 1848 | + | |
| 1849 | + | |
| 1850 | + | |
| 1851 | + | |
| 1852 | + | |
| 1853 | + | |
| 1854 | + | |
| 1855 | + | |
| 1856 | + | |
| 1857 | + | |
| 1858 | + | |
1846 | 1859 | | |
0 commit comments