Restructure Week 9 practice exercises & add Exercise 8 (fan-out) (#33)

* fix(data-track/week-9): clarify validation expectations, deterministic sample, assignment cast note

Polish from a Week 9 exercise review (all solutions live-verified against the
shared Azure Postgres: Manhattan top, 0 NULL pickups, 189 duplicate groups,
0 orphans, fact view 56,369 = 56,551 - 182 negatives).

- Ex1: add ORDER BY pickup_datetime before LIMIT 5 so the sample is stable
  between runs (a bare LIMIT returns an arbitrary 5 rows).
- Ex5: tell students 5a returns 0 and 5c returns empty BY DESIGN (pickup IDs
  are complete; orphans do not exist on this load). A 0 is the check passing,
  not a mistake. Prevents the "did I break it?" confusion.
- Ex6: note the practice view is scaled down; the assignment's vw_fact_trips
  also casts pickup_datetime::TIMESTAMP. Flag it so students do not under-build
  the assignment deliverable.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Fix double precision to numeric type casting issue for ROUND() in week 9 exercise solutions

* Restructure Week 9 practice: move EXPLAIN to Ex5, shift subsequent exercises, and add Exercise 8 (Deduplicate Fan-out Join)

---------

Co-authored-by: Lasse Benninga <devops.pipeline@example.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: 3 people

data-track/week-9/README.md

@@ -1,6 +1,6 @@
 # HYF Data Track — Week 9 Practice Exercises
 
-Seven SQL exercises that consolidate Week 9 (SQL for Analytics): joins, CTEs, aggregations, data validation, and building views. They are the runnable counterpart to the in-chapter practice. Run them against your own schema on the shared Azure PostgreSQL instance, not the shared `public` schema.
+Eight SQL exercises that consolidate Week 9 (SQL for Analytics): joins, CTEs, aggregations, data validation, and building views. They are the runnable counterpart to the in-chapter practice. Run them against your own schema on the shared Azure PostgreSQL instance, not the shared `public` schema.
 
 The dataset is the same NYC Taxi data you used in the chapters: `nyc_taxi.raw_trips` (~57K green-taxi rides from January 2024) and `nyc_taxi.raw_zones` (265 location lookups).
 
@@ -12,9 +12,10 @@ The dataset is the same NYC Taxi data you used in the chapters: `nyc_taxi.raw_tr
 | `exercise_2` | Trips and average fare per borough | GROUP BY with COUNT and AVG |
 | `exercise_3` | Find the busiest day with a CTE | WITH block, then query the CTE |
 | `exercise_4` | Refactor a nested subquery into CTEs | Rewrite nested subqueries as named steps |
-| `exercise_5` | Validate the raw data | NULL checks, duplicate detection, orphan detection |
-| `exercise_6` | Build views, then query them | CREATE OR REPLACE VIEW and star-schema queries |
-| `exercise_7` | Compare a cartesian join to a filtered join | EXPLAIN and query plans |
+| `exercise_5` | Compare a cartesian join to a filtered join | EXPLAIN and query plans |
+| `exercise_6` | Validate the raw data | NULL checks, duplicate detection, orphan detection |
+| `exercise_7` | Build views, then query them | CREATE OR REPLACE VIEW and star-schema queries |
+| `exercise_8` | Detect and fix join fan-out | Spot double-counting, unique key checks, deduplication |
 
 ## Folder structure
 
@@ -30,7 +31,7 @@ week-9/
     README.md
     solutions/
       exercise.sql
-  ... (through exercise_7)
+  ... (through exercise_8)
 ```
 
 ## How to run

data-track/week-9/exercise_5/README.md

@@ -1,19 +1,15 @@
-# Exercise 5: Validate the raw data
+# Exercise 5 (stretch): Compare a cartesian join to a filtered join
 
 ## What you do
 
-You write three data-quality checks that the Week 9 assignment's audit task expects, each as its own query:
-
-- 5a: count trips with a NULL `pickup_location_id`.
-- 5b: find duplicate trips, rows that share the same `vendor_id`, `pickup_datetime`, and `dropoff_datetime`.
-- 5c: find orphaned pickup IDs, `pickup_location_id` values that do not exist in `nyc_taxi.raw_zones`.
+A forgotten join condition produces a cartesian product: every trip paired with every zone. You use `EXPLAIN` to see how the query planner treats two versions differently, the cartesian `CROSS JOIN` with no `ON` clause, and the filtered `INNER JOIN` on `pickup_location_id = location_id`, without actually running the expensive one. Then you compare the estimated row counts at the top node of each plan.
 
 ## How to run
 
-Open `exercise.sql` and run each of the three queries against your own schema on the shared Azure PostgreSQL, using psql or any SQL client. These queries only read data, they do not modify anything.
+Open `exercise.sql` and run both `EXPLAIN` queries against your own schema on the shared Azure PostgreSQL, using psql or any SQL client. `EXPLAIN` only prints the plan, it does not execute the query or modify any data.
 
 ## Success criteria
 
-Each check returns a clear answer. For 5a you get a single count, and on this dataset that count is **0**: the pickup location IDs are complete. That is the point of the check, not a sign you wrote it wrong. The real issues live elsewhere: 5b surfaces duplicate trips, and the negative-fare and NULL `payment_type` problems show up in the chapter. For 5b and 5c, an empty result means the check passed: any rows returned are the problems to report. 5c also comes back empty here (every pickup ID resolves to a zone), which is the correct passing outcome, not a missed bug.
+The cartesian plan estimates roughly 57,000 times 265 rows (around 15 million) at its top node, while the filtered plan estimates about one matched zone per trip. That gap is why a forgotten `ON` clause can hang your session.
 
 Stuck? The reference queries are in `solutions/exercise.sql`, try for 10 to 20 minutes first.

data-track/week-9/exercise_5/exercise.sql

@@ -1,29 +1,22 @@
--- Exercise 5: Validate the raw data
+-- Exercise 7 (stretch): Compare a cartesian join to a filtered join
 --
--- Raw data is rarely clean. Write three checks that the assignment's audit task
--- expects you to run:
---   5a. Count trips with a NULL pickup_location_id.
---   5b. Find duplicate trips: rows that share the same vendor_id, pickup_datetime,
---       and dropoff_datetime.
---   5c. Find orphaned pickup IDs: pickup_location_id values in nyc_taxi.raw_trips that do
---       not exist in nyc_taxi.raw_zones.
+-- A missing join condition produces a cartesian product: every trip matched to
+-- every zone. Use EXPLAIN to see how the planner treats the two queries
+-- differently, without actually running the expensive one.
+--
+-- Run EXPLAIN on both versions and compare the estimated row counts at the top node.
 --
 -- Dataset: nyc_taxi.raw_trips (~57K green-taxi rows, Jan 2024) and nyc_taxi.raw_zones (265 rows).
 -- Run these against your OWN schema on the shared Azure PostgreSQL, not public.
 --
--- Hint: For duplicates, GROUP BY the three columns and keep groups with
---       HAVING COUNT(*) > 1. For orphans, a LEFT JOIN to nyc_taxi.raw_zones with
---       WHERE z.location_id IS NULL surfaces the unmatched IDs.
-
--- 5a. Trips with a missing pickup location
--- TODO: count rows where pickup_location_id IS NULL.
---      Expect 0 here: the pickup IDs are complete. A 0 is the check passing,
---      not a mistake. The real dirt is duplicates (5b) and negative fares.
-
+-- Hint: EXPLAIN shows the plan and its cost estimate without executing the query.
+--       The cartesian version has no ON clause; the filtered version joins on
+--       pickup_location_id = location_id. Compare the rows= estimate on the top
+--       line of each plan.
 
--- 5b. Duplicate trips (same vendor + pickup + dropoff time)
--- TODO: group by the three columns and keep groups with more than one row.
+-- Cartesian product: no join condition
+-- TODO: EXPLAIN a CROSS JOIN of nyc_taxi.raw_trips and nyc_taxi.raw_zones (no ON clause).
 
 
--- 5c. Orphaned pickup IDs not present in nyc_taxi.raw_zones
--- TODO: LEFT JOIN nyc_taxi.raw_trips to nyc_taxi.raw_zones and keep rows with no matching zone.
+-- Filtered join: one zone per trip
+-- TODO: EXPLAIN an INNER JOIN of nyc_taxi.raw_trips and nyc_taxi.raw_zones on pickup_location_id = location_id.

data-track/week-9/exercise_5/solutions/exercise.sql

@@ -1,31 +1,18 @@
--- Exercise 5 solution: Validate the raw data
+-- Exercise 7 solution (stretch): Compare a cartesian join to a filtered join
 
--- 5a. Trips with a missing pickup location
-SELECT COUNT(*) AS null_pickup_location
+-- Cartesian product: no join condition, ~57K trips x 265 zones
+EXPLAIN                        -- WHY EXPLAIN (not EXPLAIN ANALYZE): EXPLAIN only estimates and prints the plan; it never runs the query, so the ~15M-row cartesian product never actually materializes and cannot hang your session
+SELECT *
 FROM nyc_taxi.raw_trips t
-WHERE t.pickup_location_id IS NULL;
--- WHY IS NULL (not = NULL): in SQL, NULL is never equal to anything, even NULL.
---   You must test for absence with IS NULL, not with = NULL, or the filter matches nothing.
+CROSS JOIN nyc_taxi.raw_zones z;        -- WHY CROSS JOIN: with no ON clause the planner has no way to pair rows, so it produces every trip x every zone combination
 
 
--- 5b. Duplicate trips (same vendor + pickup + dropoff time)
-SELECT
-    t.vendor_id,
-    t.pickup_datetime,
-    t.dropoff_datetime,
-    COUNT(*) AS copies
+-- Filtered join: one zone per trip
+EXPLAIN
+SELECT *
 FROM nyc_taxi.raw_trips t
-GROUP BY t.vendor_id, t.pickup_datetime, t.dropoff_datetime
-HAVING COUNT(*) > 1            -- WHY HAVING (not WHERE): WHERE filters individual rows before grouping; HAVING filters the GROUPS after aggregation. COUNT(*) only exists per group, so it must be tested in HAVING.
-ORDER BY copies DESC;
+INNER JOIN nyc_taxi.raw_zones z
+    ON t.pickup_location_id = z.location_id;  -- WHY the ON clause matters: it tells the planner each trip matches exactly one zone, so the estimate collapses from millions to roughly the trip count
 
-
--- 5c. Orphaned pickup IDs not present in nyc_taxi.raw_zones
-SELECT DISTINCT t.pickup_location_id
-FROM nyc_taxi.raw_trips t
-LEFT JOIN nyc_taxi.raw_zones z          -- WHY LEFT JOIN: keeps every trip even when no zone matches; an INNER JOIN would silently drop the unmatched (orphan) rows we are hunting for
-    ON t.pickup_location_id = z.location_id
-WHERE z.location_id IS NULL    -- WHY IS NULL here: after a LEFT JOIN, an unmatched trip has NULL zone columns; filtering on z.location_id IS NULL isolates exactly the orphans
-ORDER BY t.pickup_location_id;
-
--- An empty result for 5b or 5c means the check passed. Any rows returned are the problems to report.
+-- WHY this comparison teaches: the cartesian plan estimates roughly 57,000 x 265 (around 15 million),
+--   while the filtered plan estimates about one matched zone per trip. That gap is why a forgotten ON clause can hang your session.

data-track/week-9/exercise_6/README.md

@@ -1,21 +1,19 @@
-# Exercise 6: Build views, then query them
+# Exercise 6: Validate the raw data
 
 ## What you do
 
-You wrap the cleaned-up logic in two views, then query them. This is the star-schema deliverable the Week 9 assignment asks for, scaled down to practice it once:
+You write three data-quality checks that the Week 9 assignment's audit task expects, each as its own query:
 
-- 6a: create `vw_dim_zones` from `nyc_taxi.raw_zones` and `vw_fact_trips` from `nyc_taxi.raw_trips`, excluding rows where `fare_amount` is negative.
-- 6b: using your views, find which borough had the highest total fare revenue.
-- 6c: using your views, find the top 5 pickup zones by trip count.
+- 6a: count trips with a NULL `pickup_location_id`.
+- 6b: find duplicate trips, rows that share the same `vendor_id`, `pickup_datetime`, and `dropoff_datetime`.
+- 6c: find orphaned pickup IDs, `pickup_location_id` values that do not exist in `nyc_taxi.raw_zones`.
 
 ## How to run
 
-Open `exercise.sql` and run it against your own schema on the shared Azure PostgreSQL, using psql or any SQL client. Unlike the other exercises, 6a creates two views in your own schema. It uses `CREATE OR REPLACE VIEW`, so it is safe to re-run while you iterate. Steps 6b and 6c only read from those views.
+Open `exercise.sql` and run each of the three queries against your own schema on the shared Azure PostgreSQL, using psql or any SQL client. These queries only read data, they do not modify anything.
 
 ## Success criteria
 
-After 6a, both views exist in your schema and can be queried. 6b returns the single highest-revenue borough (Manhattan), and 6c returns five pickup zones ranked by trip count.
-
-> This practice view is deliberately scaled down. The assignment's `vw_fact_trips` also casts `pickup_datetime` to a `TIMESTAMP` (`pickup_datetime::TIMESTAMP`). Add that cast when you build the assignment version, or your fact view will not match the deliverable.
+Each check returns a clear answer. For 6a you get a single count, and on this dataset that count is **0**: the pickup location IDs are complete. That is the point of the check, not a sign you wrote it wrong. The real issues live elsewhere: 6b surfaces duplicate trips, and the negative-fare and NULL `payment_type` problems show up in the chapter. For 6b and 6c, an empty result means the check passed: any rows returned are the problems to report. 6c also comes back empty here (every pickup ID resolves to a zone), which is the correct passing outcome, not a missed bug.
 
 Stuck? The reference queries are in `solutions/exercise.sql`, try for 10 to 20 minutes first.

data-track/week-9/exercise_6/exercise.sql

@@ -1,32 +1,29 @@
--- Exercise 6: Build views, then query them
+-- Exercise 5: Validate the raw data
 --
--- Wrap the cleaned-up logic in two views, then query them. This is exactly the
--- star-schema deliverable the assignment asks for, scaled down to practice once.
---   6a. Create vw_dim_zones from nyc_taxi.raw_zones and vw_fact_trips from nyc_taxi.raw_trips,
---       excluding rows where fare_amount is negative.
---   6b. Using your views, find which borough had the highest total fare revenue.
---   6c. Using your views, find the top 5 pickup zones by trip count.
+-- Raw data is rarely clean. Write three checks that the assignment's audit task
+-- expects you to run:
+--   5a. Count trips with a NULL pickup_location_id.
+--   5b. Find duplicate trips: rows that share the same vendor_id, pickup_datetime,
+--       and dropoff_datetime.
+--   5c. Find orphaned pickup IDs: pickup_location_id values in nyc_taxi.raw_trips that do
+--       not exist in nyc_taxi.raw_zones.
 --
 -- Dataset: nyc_taxi.raw_trips (~57K green-taxi rows, Jan 2024) and nyc_taxi.raw_zones (265 rows).
 -- Run these against your OWN schema on the shared Azure PostgreSQL, not public.
--- NOTE: 6a creates views in YOUR OWN schema. CREATE OR REPLACE VIEW is safe to re-run.
--- NOTE: this practice view is scaled down. The Week 9 assignment's vw_fact_trips
---       also casts pickup_datetime to a TIMESTAMP (pickup_datetime::TIMESTAMP).
---       Add that cast when you build the assignment version.
 --
--- Hint: A view is a saved query: CREATE VIEW name AS SELECT ... The borough and
---       zone names live in vw_dim_zones, so join
---       vw_fact_trips.pickup_location_id = vw_dim_zones.location_id for any
---       name-level breakdown.
+-- Hint: For duplicates, GROUP BY the three columns and keep groups with
+--       HAVING COUNT(*) > 1. For orphans, a LEFT JOIN to nyc_taxi.raw_zones with
+--       WHERE z.location_id IS NULL surfaces the unmatched IDs.
 
--- 6a. The two views
--- TODO: CREATE OR REPLACE VIEW vw_dim_zones from nyc_taxi.raw_zones.
--- TODO: CREATE OR REPLACE VIEW vw_fact_trips from nyc_taxi.raw_trips, excluding negative fares.
+-- 5a. Trips with a missing pickup location
+-- TODO: count rows where pickup_location_id IS NULL.
+--      Expect 0 here: the pickup IDs are complete. A 0 is the check passing,
+--      not a mistake. The real dirt is duplicates (5b) and negative fares.
 
 
--- 6b. Highest total fare revenue by borough
--- TODO: join the two views, sum fare per borough, and return the top borough.
+-- 5b. Duplicate trips (same vendor + pickup + dropoff time)
+-- TODO: group by the three columns and keep groups with more than one row.
 
 
--- 6c. Top 5 pickup zones by trip count
--- TODO: join the two views, count trips per zone, and return the top 5 zones.
+-- 5c. Orphaned pickup IDs not present in nyc_taxi.raw_zones
+-- TODO: LEFT JOIN nyc_taxi.raw_trips to nyc_taxi.raw_zones and keep rows with no matching zone.

data-track/week-9/exercise_6/solutions/exercise.sql

@@ -1,40 +1,31 @@
--- Exercise 6 solution: Build views, then query them
+-- Exercise 5 solution: Validate the raw data
 
--- 6a. The two views
-CREATE OR REPLACE VIEW vw_dim_zones AS  -- WHY CREATE OR REPLACE: lets you re-run this script while iterating without first DROPping the view; a plain CREATE VIEW would error if the view already exists
-SELECT
-    location_id,
-    borough,
-    zone
-FROM nyc_taxi.raw_zones;
-
-CREATE OR REPLACE VIEW vw_fact_trips AS
-SELECT *
-FROM nyc_taxi.raw_trips
-WHERE fare_amount >= 0;       -- WHY filter in the view: the cleaning rule (no negative fares) lives in one place, so every query against the view is automatically clean
+-- 5a. Trips with a missing pickup location
+SELECT COUNT(*) AS null_pickup_location
+FROM nyc_taxi.raw_trips t
+WHERE t.pickup_location_id IS NULL;
+-- WHY IS NULL (not = NULL): in SQL, NULL is never equal to anything, even NULL.
+--   You must test for absence with IS NULL, not with = NULL, or the filter matches nothing.
 
 
--- 6b. Highest total fare revenue by borough
+-- 5b. Duplicate trips (same vendor + pickup + dropoff time)
 SELECT
-    d.borough,
-    ROUND(SUM(f.fare_amount)::numeric, 2) AS total_revenue  -- WHY SUM: revenue is the total of all fares in the borough, not an average
-FROM vw_fact_trips f
-INNER JOIN vw_dim_zones d     -- WHY join to the dim view: borough names live in the dimension, not the fact, so we join to get the breakdown label
-    ON f.pickup_location_id = d.location_id
-GROUP BY d.borough
-ORDER BY total_revenue DESC
-LIMIT 1;                      -- WHY LIMIT 1: the question asks for the single highest-revenue borough
+    t.vendor_id,
+    t.pickup_datetime,
+    t.dropoff_datetime,
+    COUNT(*) AS copies
+FROM nyc_taxi.raw_trips t
+GROUP BY t.vendor_id, t.pickup_datetime, t.dropoff_datetime
+HAVING COUNT(*) > 1            -- WHY HAVING (not WHERE): WHERE filters individual rows before grouping; HAVING filters the GROUPS after aggregation. COUNT(*) only exists per group, so it must be tested in HAVING.
+ORDER BY copies DESC;
 
 
--- 6c. Top 5 pickup zones by trip count
-SELECT
-    d.zone,
-    COUNT(*) AS trips
-FROM vw_fact_trips f
-INNER JOIN vw_dim_zones d
-    ON f.pickup_location_id = d.location_id
-GROUP BY d.zone
-ORDER BY trips DESC
-LIMIT 5;                      -- WHY LIMIT 5: the question asks for the top 5 zones by trip count
+-- 5c. Orphaned pickup IDs not present in nyc_taxi.raw_zones
+SELECT DISTINCT t.pickup_location_id
+FROM nyc_taxi.raw_trips t
+LEFT JOIN nyc_taxi.raw_zones z          -- WHY LEFT JOIN: keeps every trip even when no zone matches; an INNER JOIN would silently drop the unmatched (orphan) rows we are hunting for
+    ON t.pickup_location_id = z.location_id
+WHERE z.location_id IS NULL    -- WHY IS NULL here: after a LEFT JOIN, an unmatched trip has NULL zone columns; filtering on z.location_id IS NULL isolates exactly the orphans
+ORDER BY t.pickup_location_id;
 
--- Because these use CREATE OR REPLACE VIEW, you can re-run them safely while you iterate, without dropping the view first.
+-- An empty result for 5b or 5c means the check passed. Any rows returned are the problems to report.