Commit 5b57ac9
committed
feat(pool-manager): rework stuck-runner cleanup to eliminate OOMs and cut DB load
### Why
* A production incident (`System.OutOfMemoryException` during `CheckForStuckRunners`) showed that we were
loading **all** runners and their full `Lifecycle` collections into memory, then letting EF Core’s
change-tracker churn on tens of thousands of entities.
* Each cleanup cycle recreated that pressure, eventually crashing the autoscaler pod and stopping the host.
### What
* **Query trimmed to SQL only**
* Replaced `Include(...).AsEnumerable().Where(...)` with a *pure* LINQ-to-Entities
filter:
`LastState == Created && CreatedTime < now-10min`.
* Removed unconditional `Include(x => x.Lifecycle)`.
* Added `AsNoTracking()` and a **projection** to an anonymous type
(`Select(r ⇒ new { … })`) so no full `Runner` entities are tracked.
* **Context lifetime & tracking**
* Scoped `ActionsRunnerContext` with `await using` – guarantees disposal after
the method exits.
* Disabled auto-detection of changes only for this batch
(`ChangeTracker.AutoDetectChangesEnabled = false`) to minimise
change-tracker work.
* **Batch insert of lifecycle events**
* Accumulate new `RunnerLifecycle` rows in an in-memory list and call
`AddRange` once instead of adding per runner.
* Single `SaveChangesAsync()` at the end → one DB round-trip.
* **Queue deletion without materialising collections**
* For every stuck runner enqueue a `DeleteRunnerTask` directly using the
projected key data (no need for full entity).
* **Logging**
* Added explicit warning log per stuck runner **and** kept the original message
structure for easy grepping.
### Result
* `CheckForStuckRunners` now pulls only the small “actually stuck” set into
memory and never tracks existing `Lifecycle` rows.
**IN CLAUDE WE TRUST**1 parent 14dad2f commit 5b57ac9
1 file changed
Lines changed: 33 additions & 17 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
146 | 146 | | |
147 | 147 | | |
148 | 148 | | |
149 | | - | |
150 | | - | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
151 | 166 | | |
152 | | - | |
153 | | - | |
154 | | - | |
155 | | - | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
156 | 173 | | |
157 | | - | |
158 | | - | |
| 174 | + | |
| 175 | + | |
159 | 176 | | |
| 177 | + | |
160 | 178 | | |
161 | 179 | | |
162 | 180 | | |
163 | 181 | | |
164 | | - | |
165 | | - | |
166 | | - | |
167 | | - | |
168 | | - | |
169 | | - | |
170 | 182 | | |
171 | 183 | | |
172 | | - | |
173 | 184 | | |
174 | 185 | | |
175 | | - | |
176 | | - | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
177 | 193 | | |
178 | 194 | | |
179 | 195 | | |
| |||
0 commit comments