Skip to content

Commit b89eb6e

Browse files
jopemachineclaude
andcommitted
fix(BA-6927): stop cross-multiplying usage bucket amounts and durations
FairShareAggregator accumulated the raw resource amounts and the slice durations of a bucket separately and multiplied them afterwards, yielding (sum amount_k) * (sum duration_k) instead of sum(amount_k * duration_k). The cross product inflates a bucket by the number of kernel slices folded into it, so a user running 4 GPU kernels saw 4x their real fGPU-seconds. The factor compounds up the hierarchy, since each level aggregates the kernels of everything beneath it. The deltas the aggregator hands over now carry the per-slice products already summed, which is the only shape that fixes this: (sum amount, sum duration) does not determine sum(amount * duration), so the pairing has to survive into the delta. BucketDelta held exactly those two sums and had no other purpose, so it collapses to a plain ResourceSlot per bucket. usage_bucket_entries.amount becomes resource_usage, the name kernel_usage_records, the three bucket tables and the GraphQL and REST responses already use for this quantity, so one vocabulary runs the whole way through. The column also drops its precision limit: no fixed precision is defensible here -- a domain-level daily mem bucket runs to ~1e18 byte-seconds on a large cluster -- while PostgreSQL's unconstrained numeric has no ceiling, and dropping a typmod does not rewrite the table. duration_seconds goes too. It only ever existed so readers could reconstitute the product, nothing consulted it on its own, and it counted kernel-seconds rather than wall-clock, so keeping it would leave a column that invites the same misreading. The read paths previously summed the stored figure without combining it with duration at all, returning something 300x smaller than the resource-seconds the fair share calculator assumes (it divides by capacity * lookback_days * 86400), so scheduling priorities were computed on the wrong scale -- a second defect independent of the cross product. Existing buckets cannot be corrected in place, in either the JSONB mirror or the normalized entries, so the migration rebuilds both from kernel_usage_records, which stores per-slice products and was never affected. The oldest retained day is excluded because retention purges kernel records by period_end and may have truncated it; buckets older than the kernel record retention window keep their inflated values rather than being silently zeroed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 9594aab commit b89eb6e

12 files changed

Lines changed: 719 additions & 214 deletions

File tree

changes/12965.fix.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Fix user, project and domain usage buckets being over-reported by the number of concurrently running kernels, and rebuild the affected buckets from the recorded per-kernel usage.

src/ai/backend/manager/data/fair_share/__init__.py

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
from .types import (
2-
BucketDelta,
32
DomainFactorResult,
43
DomainFairShareData,
54
DomainFairShareSearchResult,
@@ -59,6 +58,5 @@
5958
"UserUsageBucketKey",
6059
"ProjectUsageBucketKey",
6160
"DomainUsageBucketKey",
62-
"BucketDelta",
6361
"UsageBucketAggregationResult",
6462
)

src/ai/backend/manager/data/fair_share/types.py

Lines changed: 9 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -419,23 +419,15 @@ class DomainUsageBucketKey:
419419

420420

421421
@dataclass
422-
class BucketDelta:
423-
"""Separated resource amount and duration for a usage bucket.
422+
class UsageBucketAggregationResult:
423+
"""Resource-seconds to add to each bucket, from one observation tick.
424424
425-
Stores raw resource amounts and duration separately instead of
426-
pre-multiplied resource-seconds. The product ``amount * duration_seconds``
427-
is computed at SQL query time where PostgreSQL auto-extends NUMERIC precision,
428-
eliminating overflow risk for large memory values.
425+
Each value is ``sum(amount_k * duration_k)`` over the slices folded into that
426+
bucket. It must be accumulated as a sum of per-slice products: summing the
427+
amounts and the durations separately and multiplying afterwards gives a cross
428+
product inflated by the number of slices.
429429
"""
430430

431-
slots: ResourceSlot = field(default_factory=ResourceSlot)
432-
duration_seconds: int = 0
433-
434-
435-
@dataclass
436-
class UsageBucketAggregationResult:
437-
"""Result of aggregating kernel usage into daily buckets."""
438-
439-
user_usage_deltas: dict[UserUsageBucketKey, BucketDelta] = field(default_factory=dict)
440-
project_usage_deltas: dict[ProjectUsageBucketKey, BucketDelta] = field(default_factory=dict)
441-
domain_usage_deltas: dict[DomainUsageBucketKey, BucketDelta] = field(default_factory=dict)
431+
user_usage_deltas: dict[UserUsageBucketKey, ResourceSlot] = field(default_factory=dict)
432+
project_usage_deltas: dict[ProjectUsageBucketKey, ResourceSlot] = field(default_factory=dict)
433+
domain_usage_deltas: dict[DomainUsageBucketKey, ResourceSlot] = field(default_factory=dict)
Lines changed: 304 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,304 @@
1+
"""rebuild inflated usage buckets
2+
3+
The aggregator summed amounts and durations separately and multiplied them
4+
afterwards, so both the JSONB ``resource_usage`` mirror and the normalized
5+
entries hold a cross product. Neither can be corrected in place, so both are
6+
rebuilt from ``kernel_usage_records``, which was never affected.
7+
8+
``usage_bucket_entries.amount`` becomes ``resource_usage``, matching what the
9+
JSONB mirror on ``kernel_usage_records`` and the three bucket tables already call
10+
this quantity, and drops its precision limit. It now holds the product directly
11+
rather than a factor readers had to multiply back out: a domain-level daily mem
12+
bucket runs past any fixed precision, and unconstrained NUMERIC has no ceiling.
13+
14+
``duration_seconds`` goes with it. It only ever existed to reconstitute that
15+
product, no reader consulted it on its own, and it counted kernel-seconds rather
16+
than wall-clock, so leaving it would leave a column that invites the same
17+
misreading the product form did.
18+
19+
Revision ID: c4a91d7e05b2
20+
Revises: 5405ee0d8eed
21+
Create Date: 2026-07-20 00:00:00.000000
22+
23+
"""
24+
25+
import logging
26+
from datetime import date, timedelta
27+
28+
import sqlalchemy as sa
29+
from alembic import op
30+
31+
# revision identifiers, used by Alembic.
32+
revision = "c4a91d7e05b2"
33+
down_revision = "5405ee0d8eed"
34+
# Part of: NEXT_RELEASE_VERSION
35+
branch_labels = None
36+
depends_on = None
37+
38+
log = logging.getLogger("alembic.runtime.migration")
39+
40+
41+
def upgrade() -> None:
42+
# Schema change: amount -> resource_usage (now the product, so no fixed
43+
# precision), and duration_seconds is no longer needed.
44+
op.alter_column(
45+
"usage_bucket_entries",
46+
"amount",
47+
new_column_name="resource_usage",
48+
existing_type=sa.Numeric(precision=24, scale=6),
49+
type_=sa.Numeric(),
50+
existing_nullable=False,
51+
)
52+
op.drop_column("usage_bucket_entries", "duration_seconds")
53+
54+
# Data change: rebuild the corrupted values from kernel_usage_records.
55+
# Delete every corrupted entry first (all three levels at once), then rebuild
56+
# each level. The three rebuilds share the same shape and differ only in which
57+
# table and key columns they use, so they are written out one by one below.
58+
conn = op.get_bind()
59+
window = _rebuildable_date_range(conn)
60+
if window is None:
61+
# No usage records to rebuild from (fresh install, or everything purged).
62+
return
63+
rebuild_from, rebuild_to = window
64+
_purge_corrupted_usage(conn, rebuild_from, rebuild_to)
65+
_rebuild_user_buckets(conn, rebuild_from, rebuild_to)
66+
_rebuild_project_buckets(conn, rebuild_from, rebuild_to)
67+
_rebuild_domain_buckets(conn, rebuild_from, rebuild_to)
68+
69+
70+
def downgrade() -> None:
71+
op.add_column(
72+
"usage_bucket_entries",
73+
sa.Column("duration_seconds", sa.Integer(), nullable=False, server_default="0"),
74+
)
75+
# The rebuilt values are the correct ones and the inflated originals cannot
76+
# be reconstructed, so only the column definition is reverted. Values that
77+
# exceed the restored precision will fail the cast, which is the honest
78+
# outcome: they do not fit the old column.
79+
op.alter_column(
80+
"usage_bucket_entries",
81+
"resource_usage",
82+
new_column_name="amount",
83+
existing_type=sa.Numeric(),
84+
type_=sa.Numeric(precision=24, scale=6),
85+
existing_nullable=False,
86+
)
87+
log.warning(
88+
"usage_bucket_entries is left corrupt by this downgrade: the column restored to "
89+
"'amount' now holds resource-seconds products, not raw amounts, and "
90+
"duration_seconds is reset to 0. Re-apply revision %s to rebuild the correct "
91+
"values from kernel_usage_records.",
92+
revision,
93+
)
94+
95+
96+
def _rebuildable_date_range(conn: sa.engine.Connection) -> tuple[date, date] | None:
97+
"""Return the date range that kernel_usage_records can faithfully rebuild.
98+
99+
The oldest retained day is excluded because retention purges by ``period_end``
100+
and may have truncated it. Buckets outside the range keep their inflated
101+
values rather than being zeroed: they are unrecoverable, and zeroing them
102+
would destroy the only usage history left.
103+
"""
104+
row = conn.execute(
105+
sa.text(
106+
"SELECT min((period_start AT TIME ZONE 'UTC')::date) AS min_date, "
107+
" max((period_start AT TIME ZONE 'UTC')::date) AS max_date "
108+
"FROM kernel_usage_records"
109+
)
110+
).one()
111+
if row.min_date is None or row.max_date is None:
112+
return None
113+
rebuild_from = row.min_date + timedelta(days=1)
114+
if rebuild_from > row.max_date:
115+
return None
116+
return rebuild_from, row.max_date
117+
118+
119+
def _purge_corrupted_usage(
120+
conn: sa.engine.Connection, rebuild_from: date, rebuild_to: date
121+
) -> None:
122+
"""Delete every corrupted entry the rebuild will replace, across all levels.
123+
124+
Each entry belongs to exactly one bucket via ``bucket_id``, so listing the
125+
in-window buckets of the three tables and deleting entries that point at them
126+
clears user, project and domain in a single statement. The rebuilds then
127+
start from a clean slate and only need to insert.
128+
"""
129+
conn.execute(
130+
sa.text(
131+
"""
132+
DELETE FROM usage_bucket_entries
133+
WHERE (bucket_id, bucket_type) IN (
134+
SELECT id, 'user' FROM user_usage_buckets
135+
WHERE period_start BETWEEN :rebuild_from AND :rebuild_to
136+
UNION ALL
137+
SELECT id, 'project' FROM project_usage_buckets
138+
WHERE period_start BETWEEN :rebuild_from AND :rebuild_to
139+
UNION ALL
140+
SELECT id, 'domain' FROM domain_usage_buckets
141+
WHERE period_start BETWEEN :rebuild_from AND :rebuild_to
142+
)
143+
"""
144+
),
145+
{"rebuild_from": rebuild_from, "rebuild_to": rebuild_to},
146+
)
147+
148+
149+
def _rebuild_user_buckets(conn: sa.engine.Connection, rebuild_from: date, rebuild_to: date) -> None:
150+
"""Recompute user bucket entries and their JSONB mirror from kernel records."""
151+
params = {"rebuild_from": rebuild_from, "rebuild_to": rebuild_to}
152+
153+
# 1. Insert one entry per (bucket, slot), summing the per-slice resource-seconds
154+
# that kernel_usage_records already stores correctly. capacity is refilled by
155+
# the next observation tick, so 0 here is safe.
156+
conn.execute(
157+
sa.text(
158+
"""
159+
INSERT INTO usage_bucket_entries
160+
(bucket_id, bucket_type, slot_name, resource_usage, capacity)
161+
SELECT user_usage_buckets.id, 'user', slot.key, SUM(slot.value::numeric), 0
162+
FROM user_usage_buckets
163+
JOIN kernel_usage_records
164+
ON kernel_usage_records.user_uuid = user_usage_buckets.user_uuid
165+
AND kernel_usage_records.project_id = user_usage_buckets.project_id
166+
AND kernel_usage_records.resource_group_id = user_usage_buckets.resource_group_id
167+
AND (kernel_usage_records.period_start AT TIME ZONE 'UTC')::date
168+
= user_usage_buckets.period_start
169+
CROSS JOIN LATERAL jsonb_each_text(kernel_usage_records.resource_usage) AS slot
170+
WHERE user_usage_buckets.period_start BETWEEN :rebuild_from AND :rebuild_to
171+
GROUP BY user_usage_buckets.id, slot.key
172+
"""
173+
),
174+
params,
175+
)
176+
# 2. The JSONB mirror is just the slot map of the bucket's entries, or {} when
177+
# the bucket has no kernel records left to rebuild from.
178+
conn.execute(
179+
sa.text(
180+
"""
181+
UPDATE user_usage_buckets
182+
SET resource_usage = COALESCE(
183+
(
184+
SELECT jsonb_object_agg(
185+
usage_bucket_entries.slot_name,
186+
usage_bucket_entries.resource_usage
187+
)
188+
FROM usage_bucket_entries
189+
WHERE usage_bucket_entries.bucket_id = user_usage_buckets.id
190+
AND usage_bucket_entries.bucket_type = 'user'
191+
),
192+
'{}'::jsonb
193+
)
194+
WHERE user_usage_buckets.period_start BETWEEN :rebuild_from AND :rebuild_to
195+
"""
196+
),
197+
params,
198+
)
199+
200+
201+
def _rebuild_project_buckets(
202+
conn: sa.engine.Connection, rebuild_from: date, rebuild_to: date
203+
) -> None:
204+
"""Recompute project bucket entries and their JSONB mirror from kernel records."""
205+
params = {"rebuild_from": rebuild_from, "rebuild_to": rebuild_to}
206+
207+
# 1. Insert one entry per (bucket, slot), summing the per-slice resource-seconds
208+
# that kernel_usage_records already stores correctly. capacity is refilled by
209+
# the next observation tick, so 0 here is safe.
210+
conn.execute(
211+
sa.text(
212+
"""
213+
INSERT INTO usage_bucket_entries
214+
(bucket_id, bucket_type, slot_name, resource_usage, capacity)
215+
SELECT project_usage_buckets.id, 'project', slot.key, SUM(slot.value::numeric), 0
216+
FROM project_usage_buckets
217+
JOIN kernel_usage_records
218+
ON kernel_usage_records.project_id = project_usage_buckets.project_id
219+
AND kernel_usage_records.resource_group_id = project_usage_buckets.resource_group_id
220+
AND (kernel_usage_records.period_start AT TIME ZONE 'UTC')::date
221+
= project_usage_buckets.period_start
222+
CROSS JOIN LATERAL jsonb_each_text(kernel_usage_records.resource_usage) AS slot
223+
WHERE project_usage_buckets.period_start BETWEEN :rebuild_from AND :rebuild_to
224+
GROUP BY project_usage_buckets.id, slot.key
225+
"""
226+
),
227+
params,
228+
)
229+
# 2. The JSONB mirror is just the slot map of the bucket's entries, or {} when
230+
# the bucket has no kernel records left to rebuild from.
231+
conn.execute(
232+
sa.text(
233+
"""
234+
UPDATE project_usage_buckets
235+
SET resource_usage = COALESCE(
236+
(
237+
SELECT jsonb_object_agg(
238+
usage_bucket_entries.slot_name,
239+
usage_bucket_entries.resource_usage
240+
)
241+
FROM usage_bucket_entries
242+
WHERE usage_bucket_entries.bucket_id = project_usage_buckets.id
243+
AND usage_bucket_entries.bucket_type = 'project'
244+
),
245+
'{}'::jsonb
246+
)
247+
WHERE project_usage_buckets.period_start BETWEEN :rebuild_from AND :rebuild_to
248+
"""
249+
),
250+
params,
251+
)
252+
253+
254+
def _rebuild_domain_buckets(
255+
conn: sa.engine.Connection, rebuild_from: date, rebuild_to: date
256+
) -> None:
257+
"""Recompute domain bucket entries and their JSONB mirror from kernel records."""
258+
params = {"rebuild_from": rebuild_from, "rebuild_to": rebuild_to}
259+
260+
# 1. Insert one entry per (bucket, slot), summing the per-slice resource-seconds
261+
# that kernel_usage_records already stores correctly. capacity is refilled by
262+
# the next observation tick, so 0 here is safe.
263+
conn.execute(
264+
sa.text(
265+
"""
266+
INSERT INTO usage_bucket_entries
267+
(bucket_id, bucket_type, slot_name, resource_usage, capacity)
268+
SELECT domain_usage_buckets.id, 'domain', slot.key, SUM(slot.value::numeric), 0
269+
FROM domain_usage_buckets
270+
JOIN kernel_usage_records
271+
ON kernel_usage_records.domain_name = domain_usage_buckets.domain_name
272+
AND kernel_usage_records.resource_group_id = domain_usage_buckets.resource_group_id
273+
AND (kernel_usage_records.period_start AT TIME ZONE 'UTC')::date
274+
= domain_usage_buckets.period_start
275+
CROSS JOIN LATERAL jsonb_each_text(kernel_usage_records.resource_usage) AS slot
276+
WHERE domain_usage_buckets.period_start BETWEEN :rebuild_from AND :rebuild_to
277+
GROUP BY domain_usage_buckets.id, slot.key
278+
"""
279+
),
280+
params,
281+
)
282+
# 2. The JSONB mirror is just the slot map of the bucket's entries, or {} when
283+
# the bucket has no kernel records left to rebuild from.
284+
conn.execute(
285+
sa.text(
286+
"""
287+
UPDATE domain_usage_buckets
288+
SET resource_usage = COALESCE(
289+
(
290+
SELECT jsonb_object_agg(
291+
usage_bucket_entries.slot_name,
292+
usage_bucket_entries.resource_usage
293+
)
294+
FROM usage_bucket_entries
295+
WHERE usage_bucket_entries.bucket_id = domain_usage_buckets.id
296+
AND usage_bucket_entries.bucket_type = 'domain'
297+
),
298+
'{}'::jsonb
299+
)
300+
WHERE domain_usage_buckets.period_start BETWEEN :rebuild_from AND :rebuild_to
301+
"""
302+
),
303+
params,
304+
)

src/ai/backend/manager/models/resource_usage_history/row.py

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -443,10 +443,15 @@ class UserUsageBucketRow(LifecycleTimestampsMixin, Base): # type: ignore[misc]
443443
class UsageBucketEntryRow(Base): # type: ignore[misc]
444444
"""Per-slot normalized entry for usage bucket aggregation (Phase 3).
445445
446-
Stores amount and duration separately instead of pre-multiplied resource-seconds,
447-
eliminating overflow risk for large memory values.
448-
The product ``amount * duration_seconds`` is computed at SQL query time
449-
where PostgreSQL auto-extends NUMERIC precision.
446+
``resource_usage`` accumulates a kernel's ``occupied_slots`` integrated
447+
over the time it held them. It records an allocation, not a measurement: a
448+
kernel holding a GPU idle for an hour counts the same as one saturating it.
449+
That is what fair share wants -- holding a resource denies it to others -- but
450+
it means "usage" would overstate what this column knows.
451+
452+
Declared as unconstrained NUMERIC on purpose: a domain-level daily mem bucket
453+
runs to ~1e18 byte-seconds on a large cluster, past any fixed precision worth
454+
writing down, and PostgreSQL's unconstrained numeric has no such ceiling.
450455
451456
One entry per (bucket_id, slot_name). ``bucket_type`` is a discriminator
452457
indicating which parent table (domain/project/user_usage_buckets) owns
@@ -458,10 +463,7 @@ class UsageBucketEntryRow(Base): # type: ignore[misc]
458463
bucket_id: Mapped[uuid.UUID] = mapped_column("bucket_id", GUID(), nullable=False)
459464
bucket_type: Mapped[str] = mapped_column("bucket_type", sa.String(length=16), nullable=False)
460465
slot_name: Mapped[str] = mapped_column("slot_name", sa.String(length=64), nullable=False)
461-
amount: Mapped[Decimal] = mapped_column(
462-
"amount", sa.Numeric(precision=24, scale=6), nullable=False
463-
)
464-
duration_seconds: Mapped[int] = mapped_column("duration_seconds", sa.Integer(), nullable=False)
466+
resource_usage: Mapped[Decimal] = mapped_column("resource_usage", sa.Numeric(), nullable=False)
465467
capacity: Mapped[Decimal] = mapped_column(
466468
"capacity", sa.Numeric(precision=24, scale=6), nullable=False
467469
)

0 commit comments

Comments
 (0)