fix(BA-6927): stop cross-multiplying usage bucket amounts and durations

[jopemachine]

Summary

• FairShareAggregator accumulated the raw resource amounts and the slice durations of a bucket separately and multiplied them afterwards, computing (sum amount_k) * (sum duration_k) instead of sum(amount_k * duration_k). The cross product inflates a bucket by the number of kernel slices folded into it in one observation tick — i.e. by the number of concurrently running kernels. Reported symptom, one user with 4 GPU kernels for a day: kernel_usage_records sums to 345600 fGPU-seconds (correct) while user_usage_buckets reads 1382400 (345600 * 4). The factor compounds up the hierarchy, since each level aggregates the kernels of everything beneath it, which is why domain figures looked most implausible.
• usage_bucket_entries.amount becomes resource_usage and drops its precision limit. resource_usage is what kernel_usage_records, the three bucket tables and the GraphQL/REST responses already call this quantity, so one vocabulary now runs the whole way through and the adapters no longer translate. The column holds the product directly rather than a factor readers had to multiply back out. The typmod is dropped rather than widened because no fixed precision is defensible here: a domain-level daily mem bucket runs to ~1e18 byte-seconds on a large cluster (measured: NUMERIC(24,6) overflows past ~10.5 TB of RAM in scope), while PostgreSQL's unconstrained numeric has no ceiling. Renaming a column and dropping a typmod are both metadata-only — verified on 200k rows: relfilenode unchanged, 15 ms.
• duration_seconds is no longer read anywhere; it stays as provenance for the usage figure beside it.
• The read paths previously summed the stored figure without combining it with duration at all, returning something 300x smaller than the resource-seconds the fair share calculator assumes (it divides by capacity * lookback_days * 86400), so scheduling priorities were computed on the wrong scale — a second defect independent of the cross product.
• Migration c4a91d7e05b2 performs that column change and rebuilds the corrupted values: existing buckets are wrong in both the JSONB mirror and the normalized entries and cannot be corrected in place, so both are rebuilt from kernel_usage_records, which stores per-slice resource-seconds and was never affected. Two deliberate limits: the oldest retained day is excluded, because retention purges kernel records by period_end at a timestamp that falls mid-day and may have truncated it; and buckets older than the kernel-record retention window (90 days by default) keep their inflated values rather than being silently zeroed — they cannot be recovered, and destroying them would lose the only usage history that remains. The fair share lookback (~28 days) sits well inside the rebuildable window, so scheduling is fully corrected.

Test plan

• pants test tests/unit/manager/sokovan/scheduler/fair_share/ — bucket aggregation assertions corrected to resource-seconds, plus a regression class parametrized over 1/2/4/10 concurrent kernels and a full-day 4-kernel case asserting the reported 345600
• Multi-tenant scenario: one tick across two projects, three users and seven kernels, with one user active in both projects. The old code produced 1200 instead of 600 for a user (2x), 3600 instead of 1200 for a project (3x) and 23100 instead of 3300 for the domain (7x). Also pins that user buckets stay keyed by (user, project) so a user active in two projects does not collapse into one bucket
• pants test tests/unit/manager/repositories/resource_usage_history/ tests/unit/manager/repositories/retention/
• pants fmt/fix/lint pass on the changeset; pants check (mypy) clean across changed files
• Data migration verified against a scratch Postgres database seeded with both the reported scenario (4 kernels × 288 slices across two days) and the multi-tenant domain: every level rebuilds to its exact expected value, the user active in two projects rebuilds separately with no join fan-out, the oldest retained day stays untouched, and in-window buckets with no kernel records collapse to {}
• ALTER COLUMN … TYPE numeric measured on a 200k-row table: relfilenode unchanged (no rewrite), 15 ms
• Verify on a live manager that new observation ticks accumulate correctly

Deployment note

The migration rewrites up to 90 days of bucket rows and their entries; on a large installation this takes time. downgrade() restores the column definition but not the inflated values — those are unrecoverable by design. Back up before applying.

Follow-ups (not in this PR)

• The JSONB bucket update is a read-modify-write across a transaction boundary rather than a server-side accumulate, so concurrent observers would lose an update; only leader election protects it today. JSONB has no numeric addition operator, which is why the original code moved it into Python, but jsonb_each + jsonb_object_agg can do it in SQL. The entries path accumulates server-side, so the two representations can drift.
• period_end is not in the on_conflict update set, so the "period_end extension" strategy described in the bucket row docstrings never actually happens.
• usage_bucket_entries.duration_seconds sums across concurrent kernels rather than wall-clock, so it is not a meaningful standalone figure. Nothing reads it now; worth either defining it properly or dropping it.
• calculate_usage_capacity_ratio returns seconds, not a ratio. The GraphQL field descriptions say so ("86400 means full utilization for one day"), so the value is intended — the name is not.
• ResourceSlot is a bare name -> Decimal mapping with no notion of unit, so an allocation and an already-integrated value are the same type. That is what made this defect expressible; distinguishing them properly spans ~15 files and belongs in its own change.

Resolves BA-6927

🤖 Generated with Claude Code