feat: freeze product_name instead of line_revenue (+ prod trigger & drop-script fixes)#43
Open
andre-salvati wants to merge 7 commits into
Open
feat: freeze product_name instead of line_revenue (+ prod trigger & drop-script fixes)#43andre-salvati wants to merge 7 commits into
andre-salvati wants to merge 7 commits into
Conversation
Remove the synthetic line_revenue/unit_price_at_sale columns; gold total_value is now SUM(item_total) (the line value the source already freezes on the order). Re-point the silver insert-only-MERGE / streaming- table freeze at the mutable product_name, which the seed now changes by renaming 2 products per run (Product N -> Product N.k); unit_price stays a static attribute. The AI/BI "by product" chart consolidates by product_id labeled with each product's latest name (one line per physical product across renames), while frozen historical names remain in report.order_agg and the Product filter. Tests, schemas, SDP pipeline, and data-model/test-plan docs updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
AI/BI wraps each dataset query as `WITH q AS (<query>) SELECT ... FROM q` and concatenates queryLines with no separator. The multi-line query both mashed tokens at line boundaries (total_orders+FROM) and produced a nested WITH inside the wrapper. Rewrite ds_orders as one line using a derived-table subquery (no leading WITH) for the latest-name lookup; verified the wrapped by-country aggregation parses and runs on the dev warehouse. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…on is sole trigger Remove the daily 0 0 5 CronSchedule from _build_job's prod branch so the batch job1 (job1_prod) no longer self-triggers. In prod, job1_prod_integration (0 0 6 Sao_Paulo) is the single scheduled entry point: it seeds, then triggers the batch job (RunJobTask) and the SDP pipeline (PipelineTask) in parallel. The SDP pipeline already had no schedule. Prod failure/duration alerting on job1 is retained. Documented in architecture.md Jobs DAG. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Some SQL warehouse channels reject the DROP STREAMING TABLE grammar with PARSE_SYNTAX_ERROR (while accepting a plain DROP TABLE on the streaming table), which aborted `make drop` mid-loop and left the catalog half- dropped. Try the kind-specific statement first, then fall back to DROP TABLE. Keeps DROP MATERIALIZED VIEW for MVs (DROP TABLE type-mismatches there). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ta-model.md Add a narrative overview of the medallion flow (external_source -> bronze -> silver -> gold -> dashboard), explaining frozen item_total revenue and how a product rename keeps both old and new names. Document the field naming rules (entity_id suffix, entity-qualified names, item_*/total_* prefixes, no abbreviations, DateType dates, _sdp suffix) that were previously implicit. Point CLAUDE.md at both. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ever edit Add an explicit constraint mirroring the jobs.yml rule: the _deploy copy is gitignored and regenerated from orders_dashboard.lvdash.json on every deploy (DABs can't substitute bundle vars inside .lvdash.json), so edit the source. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… history The filter was bound to the frozen per-row product_name, so selecting a renamed product's current name (e.g. "Product 11.19") matched only the rows physically stamped with it (today's post-rename orders) and dropped the pre-rename history (stamped "Product 11"). Bind the filter to the consolidated latest name (`product`) instead, matching the chart's color encoding: the dropdown lists each product once and selecting it shows full pre-/post-rename history. Old names stay in report.order_agg (frozen product_name) for audit. Verified on prod: filtering "Product 11.19" now returns 1820 rows spanning 2025-06-27..today. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Removes the synthetic
line_revenuerevenue column and re-points the medallion freeze pattern at the product name instead of price, plus two supporting fixes surfaced while migrating prod.Data model
line_revenue/unit_price_at_salefromcurated.order_enriched. Goldtotal_valueis nowSUM(item_total)— the line value the source already freezes on the order at sale time, so revenue is never recomputed from the current price list.product_nameinstead of price: silver's insert-onlyMERGE(batch) and streaming table (SDP) now freeze the product name per row at processing time, so a rename never relabels already-booked orders.unit_pricebecomes a static attribute.Product N→Product N.k, suffix = cumulative rename count) instead of bumping prices.Dashboard
ds_ordersconsolidates the "by product" chart byproduct_id, labeled with each product's latest name (one line per physical product across renames); the Product filter still lists every historical (frozen) name.WITH) to avoid the AI/BIWITH q AS (…)wrapper parse error.Ops fixes
job1's standalone0 0 5schedule sojob1_prod_integrationis the single prod trigger (seed → batch + SDP in parallel). The SDP pipeline already had none. Prod alerting onjob1retained.sdk_drop_tables.py: falls back toDROP TABLEwhen a warehouse's parser rejectsDROP STREAMING TABLE(was abortingmake dropmid-loop and leaving the catalog half-dropped).Docs
data-model.md: plain-words pipeline overview + a written-down field naming conventions section (previously implicit). CHANGELOG#43, CLAUDE.md, README, test-plan updated.Verification
make unit-test— 16/16 pass.SUCCESS;ds_orders+ widget-wrapped by-country query verified on the dev warehouse.job1_prod_integrationranSUCCESS(full backfill);order_enrichedrebuilt without the dropped columns; batch and SDP gold agree (181,500 rows each);total_value = SUM(item_total)confirmed.🤖 Generated with Claude Code