Closes #263.

What

The min-heap priority_queue located elements for deletion with an O(n) linear scan (priority_queue_delete_nolock). This makes delete O(log n): each element stores its own 1-based heap index, the queue keeps that slot in sync on every move, and delete reads it directly and repairs the heap in O(log n). This is the approach from the composite codebase referenced in the issue — and the local_runqueue_mtds.c delete path even carried a TODO: Apply the composite way of PQ deletion O(logn).

Design

• Optional, opt-in per queue. priority_queue_initialize takes a new get_index_fn accessor that returns a pointer to the element's in-struct size_t index slot. Pass NULL to keep the legacy O(n) linear-scan delete — so queues that are never delete()'d from (global_request_scheduler_minheap, tgrq->sandbox_requests) are untouched and behavior-preserving.
• Index maintained on every move (append, swaps in percolate up/down, dequeue, delete) via small record/clear/swap helpers; percolate_up is generalized to percolate_up_from(start).
• Latent bug fixed. Arbitrary-position delete must repair the heap up or down — the old code only percolated down, which left the heap invalid whenever the element moved into the hole was smaller than the hole's parent. The new delete picks the correct direction.
• Added a size_t pq_idx slot + accessor to the four element types that are delete()'d (sandbox, tenant_global_request_queue, perworker_tenant_sandbox_queue, tenant_timeout) and wired the accessor into the 7 queues that call delete. Each tracked element is a member of at most one tracked queue at a time, so a single slot suffices.

Correctness

Deterministic unit test against the real header (heap-invariant assertions, the percolate-up-on-delete case the old code got wrong, absent-element delete returning −1, and 200k random enqueue/delete/dequeue ops proving the tracked O(log n) path is behaviorally identical to the legacy O(n) path). Clean under ASan + UBSan.

Scheduler validation (in the dev container)

traps + stack_overflow sweep MTDS, exercising all four tracked element types' delete paths (sandbox, tgrq, pwt, tenant_timeout).

deadline_description failed in its post-run perf-log analysis ("Failed to get baseline execution for cifar10") — but it fails identically on master (verified by rebuilding the pre-change binary and re-running). All HTTP requests served fine; the panic is in offline bash analysis of perf.log, unrelated to this change. It is not in CI or test.mk all.

Microbenchmark — changed algorithm vs existing O(n)

Standalone harness, container clang-13 -O3, timing delete-all-in-random-order at heap size N (best of 9). old O(n) is the true pre-change header; new O(logn) is this PR.

Indexed delete stays ~flat (~21→40 ns) as the heap grows; the O(n) scan grows linearly (18→441 ns). Crossover ≈ N=128: for tiny heaps the index-maintenance constant makes it marginally slower, but it wins increasingly past that — up to ~11× at 4096.

Mapping to real queue sizes: the MTDS tenant queues are small (RUNTIME_MAX_TENANT_COUNT=32, ~neutral), but the local runqueue (RUNTIME_RUNQUEUE_SIZE=256, grows) and per-tenant sandbox queues (RUNTIME_TENANT_QUEUE_SIZE=4096) get deep under load — exactly where the old O(n) delete hurt. Beyond average throughput, this bounds worst-case delete latency (O(n) on a full queue is ~440 ns and climbs with load; O(log n) stays ~40 ns), which helps scheduler tail-latency/predictability under overload.

🤖 Generated with Claude Code