Is there an existing issue for this?
Problem statement
The tpch_sample bundle was started some time ago but never finished — it sat in the repo as an incomplete, work-in-progress sample that didn't reflect the standards since established by feature-samples and pattern-samples. Because it was only ever partially built, it had known medallion gaps (missing facts, deploy bugs, no documentation) and didn't actually demonstrate the framework working end-to-end on a realistic, industry-standard dataset. It needed to be properly revisited and completed so the framework has a single, polished reference that ties its capabilities together — streaming, multi-source ingestion, schema evolution, historization, data quality, a governed star schema, and incremental operation — on one coherent dataset, giving a new user a clear "what good looks like" example across a full medallion build, which will also include a semantic layer (UC Metrics Views) and in future also demostrate Genie and Lakeview end to end.
Proposed Solution
Rebuild tpch_sample on the samples.tpch schema into the most comprehensive end-to-end sample in the framework, covering:
- Multi-source, schema-on-read bronze — Parquet staging from 8 simulated source systems (one bronze schema per system) ingested via Auto Loader with schema inference and evolution (
addNewColumns).
- Conformed, historized silver — SCD2 dimensions, SCD1 reference data, and append-only facts, with data-quality expectations and a quarantine (
quarantineMode: table) demo.
- Governed gold star schema — surrogate keys (
xxhash64) and point-in-time (as-of) joins, plus two metrics approaches side by side: pre-aggregated materialized views and UC metric views. dim_date includes a fiscal calendar; row tracking enabled for incremental MV refresh.
- Template specs — collapse the repetitive bronze flows into one template and the silver flows into per-archetype templates (SCD2 / SCD1 / append), reducing boilerplate.
- Three-run incremental simulation — full refresh (Run 1) then incremental Runs 2–3 exercising SCD2 changes, fact growth, and a backdated out-of-order correction.
- "Real-world complexity" — schema evolution (new
loyalty_tier column auto-picked-up in bronze), CDC deletes/tombstones via apply_as_deletes (a cdc_operation flag; supplier delete), and late-arriving dimensions with an unknown member (-1) on the MV dims + COALESCE in the fact.
- Orchestration + docs — a dedicated
deploy_tpch_and_test.sh / destroy_tpch.sh, and a full README (background, design choices, setup/execution, demo flow, validation queries).
Additional Context
Is there an existing issue for this?
Problem statement
The
tpch_samplebundle was started some time ago but never finished — it sat in the repo as an incomplete, work-in-progress sample that didn't reflect the standards since established byfeature-samplesandpattern-samples. Because it was only ever partially built, it had known medallion gaps (missing facts, deploy bugs, no documentation) and didn't actually demonstrate the framework working end-to-end on a realistic, industry-standard dataset. It needed to be properly revisited and completed so the framework has a single, polished reference that ties its capabilities together — streaming, multi-source ingestion, schema evolution, historization, data quality, a governed star schema, and incremental operation — on one coherent dataset, giving a new user a clear "what good looks like" example across a full medallion build, which will also include a semantic layer (UC Metrics Views) and in future also demostrate Genie and Lakeview end to end.Proposed Solution
Rebuild
tpch_sampleon thesamples.tpchschema into the most comprehensive end-to-end sample in the framework, covering:addNewColumns).quarantineMode: table) demo.xxhash64) and point-in-time (as-of) joins, plus two metrics approaches side by side: pre-aggregated materialized views and UC metric views.dim_dateincludes a fiscal calendar; row tracking enabled for incremental MV refresh.loyalty_tiercolumn auto-picked-up in bronze), CDC deletes/tombstones viaapply_as_deletes(acdc_operationflag; supplier delete), and late-arriving dimensions with an unknown member (-1) on the MV dims +COALESCEin the fact.deploy_tpch_and_test.sh/destroy_tpch.sh, and a full README (background, design choices, setup/execution, demo flow, validation queries).Additional Context