feat(ai-observability): add time-bucketed breakdown to personal spend endpoint#69891
feat(ai-observability): add time-bucketed breakdown to personal spend endpoint#69891webjunkie wants to merge 8 commits into
Conversation
Opt-in `hourly=true` query param adds a `by_hour` breakdown to GET /api/llm_analytics/@me/spend/: an hour-ascending UTC series with per-hour cost split into uncached input / output / cache read / cache creation components (plus matching token sums), capped to windows of 8 days or less. Enables clients to render a 24h view where prompt-cache cold starts stand out as cache-creation-dominated bars. Generated-By: PostHog Code Task-Id: d7854448-4d18-44e7-b54a-d66e1ccddd0a
|
Hey @webjunkie! 👋 It looks like your git author email on this PR isn't your
You can fix it for this repo with: git config user.email "you@posthog.com"Or set it globally with |
🤖 CI report
|
| File | Size | Δ vs base |
|---|---|---|
posthog-app/src/scenes/instance/QueryPerformance/QueryPerformance.js |
31.8 KiB | 🔺 +2.8 KiB (+9.6%) |
exporter/src/exporter/scenes/ExporterNotebookScene.js |
2.88 MiB | 🔺 +1.3 KiB (+0.0%) |
render-query/src/render-query/render-query.js |
24.17 MiB | 🔺 +1.3 KiB (+0.0%) |
exporter/src/queries/schema.js |
1.00 MiB | 🔺 +1.2 KiB (+0.1%) |
posthog-app/src/queries/schema.js |
1.00 MiB | 🔺 +1.2 KiB (+0.1%) |
Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report
✅ Eager graph — within budget
How much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy import() / React.lazy chunks are not counted.
| Root | Eager (shipped) | Δ vs base | Budget |
|---|---|---|---|
entry (logged-out pages, app bootstrap)src/index.tsx |
1.20 MiB · 22 files | 🔺 +356 B (+0.0%) | ███░░░░░░░ 28.1% of 4.29 MiB |
authenticated shell (every logged-in page)src/scenes/AuthenticatedShell.tsx |
8.08 MiB · 2,957 files | 🔺 +64 B (+0.0%) | █████████░ 87.3% of 9.25 MiB |
🟢 node_modules/monaco-editor/ stays out of src/index.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 node_modules/monaco-editor/ stays out of src/scenes/AuthenticatedShell.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
Largest files eagerly shipped from src/index.tsx
| Size | File |
|---|---|
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 24.6 KiB | ../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js |
| 6.3 KiB | ../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js |
| 4.5 KiB | ../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js |
| 3.9 KiB | ../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js |
| 1.4 KiB | ../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js |
| 1.3 KiB | src/RootErrorBoundary.tsx |
| 912 B | ../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js |
| 789 B | src/scenes/ChunkLoadErrorBoundary.tsx |
| 668 B | src/index.tsx |
Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx
| Size | File |
|---|---|
| 277.3 KiB | ../node_modules/.pnpm/posthog-js@1.399.1/node_modules/posthog-js/dist/rrweb.js |
| 266.9 KiB | ../node_modules/.pnpm/@posthog+icons@0.37.4_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js |
| 224.6 KiB | src/taxonomy/core-filter-definitions-by-group.json |
| 221.5 KiB | ../node_modules/.pnpm/posthog-js@1.399.1/node_modules/posthog-js/dist/module.js |
| 164.0 KiB | src/queries/validators.js |
| 154.0 KiB | ../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js |
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 105.9 KiB | src/lib/api.ts |
| 93.3 KiB | ../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js |
| 90.6 KiB | ../node_modules/.pnpm/@tiptap+core@3.20.6_@tiptap+pm@3.20.6/node_modules/@tiptap/core/dist/index.js |
Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479
⚠️ Dist folder size — 🔺 +180.0 KiB (+0.0%)
Total size of the built frontend/dist folder (all assets), compared against the base branch.
Total: 1435.36 MiB · 🔺 +180.0 KiB (+0.0%)
…inutes Replace the hourly=true boolean with a bucket_minutes param (5, 15, 30, or 60) so clients can pick sub-hour resolution: 5-minute buckets match the prompt-cache TTL, isolating a cold-revival spike that hourly buckets would dilute. The by_hour breakdown becomes by_bucket (row field bucket_start, bucket size echoed) grouped via toStartOfInterval, and the window cap is now a flat 600 buckets of the chosen size. Generated-By: PostHog Code Task-Id: d7854448-4d18-44e7-b54a-d66e1ccddd0a
webjunkie
changed the title
…utes Generated-By: PostHog Code Task-Id: d7854448-4d18-44e7-b54a-d66e1ccddd0a
🦔 Hogbox preview · ✅ ready▶ Open the preview
commit |
|
Reviews (1): Last reviewed commit: "chore(mcp): update personal-spend tool s..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| bucket_minutes = serializers.ChoiceField( | ||
| choices=BUCKET_MINUTES_CHOICES, | ||
| required=False, | ||
| default=None, | ||
| allow_null=True, |
There was a problem hiding this comment.
This field is optional, but marking it nullable makes generated clients expose bucket_minutes: null as a valid value. The frontend URL builder serializes that as bucket_minutes=null on a GET request, while DRF only accepts a real null value, so typed callers can send an allowed client value and receive a 400 instead of the bucketless response.
There was a problem hiding this comment.
Accepted in edbb2d9: dropped allow_null so the schema no longer advertises null (default=None stays for the omitted case), and the EU proxy now omits None-valued params from the signed body so the internal receiver accepts it.
| from_dt, to_dt = _resolve_window(date_from, date_to) | ||
| if bucket_minutes is not None and (to_dt - from_dt).total_seconds() > bucket_minutes * 60 * MAX_TIME_BUCKETS: | ||
| max_hours = bucket_minutes * MAX_TIME_BUCKETS // 60 | ||
| raise exceptions.ValidationError( | ||
| { | ||
| "bucket_minutes": ( | ||
| f"A window this large would exceed {MAX_TIME_BUCKETS} buckets at {bucket_minutes}-minute " | ||
| f"resolution — narrow the window to {max_hours} hours or less, or pick a larger bucket size." | ||
| ) | ||
| } | ||
| ) |
There was a problem hiding this comment.
A window exactly equal to bucket_minutes * 600 passes this check, but partial buckets at both edges can still produce 601 bucket starts. _fetch_by_bucket then returns those 601 rows as untruncated, so clients can receive more buckets than the documented maximum.
There was a problem hiding this comment.
Accepted in edbb2d9: validation now counts the bucket starts the window touches (unaligned edges add partial buckets) instead of comparing durations, and the fetch/truncate limits match, so by_bucket can never exceed 600 rows. Added a boundary test where a 600-bucket-duration window with a half-hour offset (601 starts) is rejected.
There was a problem hiding this comment.
Pull request overview
Adds an opt-in bucket_minutes query param to the personal spend endpoint (GET /api/llm_analytics/@me/spend/) so clients can request a sub-day, UTC time-bucketed spend series (by_bucket) with per-component cost/token breakdowns, enabling higher-resolution usage visualizations (for example, spotting prompt-cache cold-start “revival” spikes).
Changes:
- Add
bucket_minutesrequest validation, cache-key partitioning, EU redirect/proxy forwarding, and a new HogQL aggregation forby_bucketin the personal spend API. - Add ClickHouse-backed tests covering bucket sizing, window caps, cost-component grouping/defaults, and cache-key behavior.
- Regenerate OpenAPI-derived schemas/types and MCP tool wiring to include the new query param and response shape.
Reviewed changes
Copilot reviewed 4 out of 8 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| services/mcp/tests/unit/snapshots/tool-schemas/llma-personal-spend.json | Updates MCP tool JSON schema snapshot to include bucket_minutes. |
| services/mcp/src/tools/generated/ai_observability.ts | Forwards bucket_minutes into the generated MCP tool request query. |
| services/mcp/src/generated/ai_observability/api.ts | Updates generated zod query schema/docstring to include bucket_minutes. |
| services/mcp/src/api/generated.ts | Updates generated TS API types to include by_bucket and the bucket_minutes param enum. |
| products/ai_observability/frontend/generated/api.ts | Updates generated frontend API client docs/types to mention bucket_minutes / by_bucket. |
| products/ai_observability/frontend/generated/api.schemas.ts | Updates generated frontend schema types for the new by_bucket breakdown. |
| products/ai_observability/backend/api/test/test_personal_spend.py | Adds ClickHouse tests for by_bucket, window caps, and cache key behavior. |
| products/ai_observability/backend/api/personal_spend.py | Implements request param validation, new bucketed HogQL query, response serialization, caching, and EU forwarding. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| help_text="If true, bypass the result cache and re-run the underlying queries against ClickHouse.", | ||
| ) | ||
| bucket_minutes = serializers.ChoiceField( | ||
| choices=BUCKET_MINUTES_CHOICES, | ||
| required=False, | ||
| default=None, | ||
| allow_null=True, | ||
| help_text=( | ||
| "When set, additionally return a `by_bucket` breakdown: a time-ascending UTC cost series for " | ||
| "the scoped product at this bucket size in minutes, with per-bucket cost split into uncached " | ||
| "input / output / cache read / cache creation components plus the matching token sums. " | ||
| f"Supported bucket sizes: {', '.join(str(c) for c in BUCKET_MINUTES_CHOICES)}. The window may " | ||
| f"span at most {MAX_TIME_BUCKETS} buckets of the chosen size (e.g. 48 hours at 5-minute " | ||
| "buckets)." | ||
| ), |
There was a problem hiding this comment.
Fixed in edbb2d9: docs and the constants comment now say 50 hours at 5-minute buckets / 25 days hourly. The regenerated OpenAPI/MCP artifacts follow via CI codegen.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
The test used the default 30-day window, which at 60-minute buckets is 720 buckets — rejected by the endpoint's own 600-bucket cap with a 400. Pin the window to a day so the test exercises the cache-key behavior it targets instead of the window validation. Generated-By: PostHog Code Task-Id: d7854448-4d18-44e7-b54a-d66e1ccddd0a
Drop allow_null from bucket_minutes: a nullable query param makes generated clients advertise null, which serializes to the literal string "null" in a GET query and gets rejected. The EU proxy now omits None-valued params from the signed cross-region body accordingly. Cap validation now counts the bucket starts the window touches (unaligned edges add partial buckets) instead of comparing durations, so by_bucket can never return more than 600 rows; fetch and truncate limits match. Also correct the cap arithmetic in docs (600 buckets is 50 hours at 5-minute resolution, not 48) and drop em dashes from the added API text. Generated-By: PostHog Code Task-Id: d7854448-4d18-44e7-b54a-d66e1ccddd0a
![hex-security-app[bot]](https://avatars.githubusercontent.com/in/2665233?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: edbb2d9cfe
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| bucket_minutes = serializers.ChoiceField( | ||
| choices=BUCKET_MINUTES_CHOICES, | ||
| required=False, | ||
| default=None, |
There was a problem hiding this comment.
Although the comment says this should not be nullable, default=None makes the generated OpenAPI type nullable: the committed LlmAnalyticsPersonalSpendListBucketMinutes includes | null, and the generated frontend URL builder serializes null as the literal bucket_minutes=null. A frontend caller that follows the generated type by clearing this optional value to null will get a 400 instead of the parameter being omitted, so this field should avoid a None default or otherwise override the schema while still normalizing missing values server-side.
Useful? React with 👍 / 👎.
Problem
The personal spend endpoint (
GET /api/llm_analytics/@me/spend/) only exposes a daily total-cost series (by_day), so clients like the PostHog Code desktop app can't render a fine-grained usage view. In particular, prompt-cache cold starts — where a stale session is revived and its whole context is re-written to the cache at the cache-write rate instead of being read back cheaply — are invisible: they need sub-hour resolution and a per-component cost split to stand out. Since the prompt-cache TTL is 5 minutes, a 5-minute bucket is roughly one turn, which isolates a revival spike that hourly buckets would dilute.Why: we want the PostHog Code app to show a 24h usage chart granular enough that a sudden burst of full-price tokens (a revived cold session) is visually obvious. A companion PR in the PostHog Code repo (PostHog/code#3329) consumes this.
Changes
bucket_minutesquery param (5, 15, 30, or 60) on/api/llm_analytics/@me/spend/.by_bucket: a time-ascending UTC series (grouped viatoStartOfInterval) with per-bucketcost_usdplus the four cost components forwarded from the LLM gateway's cost breakdown (input_cost_usd,output_cost_usd,cache_read_cost_usd,cache_creation_cost_usd) and the matching token sums. The bucket size is echoed on the breakdown.by_bucketignoreslimitlikeby_daydoes.Note: CI codegen keeps the generated OpenAPI types current (no DB available for
hogli build:openapiin the authoring environment).How did you test this code?
Added tests to
test_personal_spend.py, each catching a distinct regression:test_by_bucket_groups_cost_components_per_utc_hour— wrong column order/mapping or grouping in the new HogQL query, and product-scope leakage.test_by_bucket_five_minute_buckets_split_within_the_hour— the bucket size not actually being applied (two calls 15 minutes apart must split at 5-minute resolution).test_by_bucket_defaults_components_to_zero_when_breakdown_missing— fallback-priced events (total cost only) must produce zero components, not errors.test_by_bucket_absent_unless_requested—by_bucketleaking into default responses.test_cache_key_includes_bucket_minutes— a cached bucketless payload being served to a bucketed request (would silently dropby_bucket).test_bucket_window_cap(parameterized) +test_unsupported_bucket_size_rejected— the 600-bucket cap or the size allowlist being dropped.Could not run ClickHouse-backed tests in this sandbox (no database); verified locally that both files compile, the new HogQL parses via
parse_select(toStartOfInterval/toIntervalMinuteare whitelisted), and the response serializer omitsby_bucketwhen absent. CI runs the full suite.Automatic notifications
Docs update
Endpoint is documented via its OpenAPI schema (
@extend_schemadescription and serializerhelp_textupdated in this PR).🤖 Agent context
Autonomy: Human-driven (agent-assisted)
/improving-drf-endpoints,/writing-tests.$ai_generationevent per LLM call with LiteLLM's per-side cost breakdown, and this endpoint is the sanctioned user-scoped surface over those events — so the view was added here rather than as client-side HogQL (the events live in an internal analytics team, not the user's project).hourly=trueboolean; generalized tobucket_minutesafter deciding hourly resolution dilutes single-turn revival spikes (cache TTL is 5 minutes, so 5-minute buckets ≈ one turn).Created with PostHog Code