docs: update documentation for Episode feature and scraper improvements

Author: jung-thomas

CLAUDE.md

@@ -34,6 +34,11 @@ npm run hana          # Deploy to HANA HDI container ("starwars")
 npm run load          # Load Star Wars fixture data (hybrid/HANA profile)
 npm run load_sqlite   # Load fixture data into SQLite
 
+# Data scraping
+npm run scrape               # full run, cache-first (fast — uses committed cache)
+npm run scrape:films         # films-only, no episodes (reproduces original dataset)
+npm run scrape:bypass-cache  # fetch fresh from Wookieepedia (requires confirmation)
+
 # Docs generation
 npm run openapi       # Generate OpenAPI docs → docs/
 npm run asyncapi      # Generate AsyncAPI docs
@@ -62,13 +67,17 @@ cap/labs/        Hands-on exercises (lab-01 through lab-05)
 
 ### Domain Model (`cap/db/schema.cds`)
 
-Six core entities: **Film**, **People**, **Planet**, **Species**, **Starship**, **Vehicle**. Many-to-many relationships use explicit junction entities (`Film2People`, `Film2Planets`, etc.) with redirected projections in services. All entities use `managed` + `cuid` from `@sap/cds/common`.
+Core entities: **Film**, **People**, **Planet**, **Species**, **Starship**, **Vehicle**, **Show**, **Episode**. Many-to-many relationships use explicit junction entities (`Film2People`, `Film2Planets`, `Episode2People`, `Episode2Planets`, etc.) with redirected projections in services. All entities use `managed` + `cuid` from `@sap/cds/common`.
+
+`Episode` is a composition child of `Show` — episodes cascade-delete with their parent show. Five `Episode2*` junction tables (`Episode2People`, `Episode2Planets`, `Episode2Starships`, `Episode2Vehicles`, `Episode2Species`) link episodes to the core entity set.
+
+`Show2Planets`, `Show2Starships`, `Show2Vehicles`, and `Show2Species` are CDS `define view` declarations that aggregate over the corresponding `Episode2*` tables rather than being physical tables.
 
 Profile-specific extensions live in `cap/db/hana/`, `cap/db/sqlite/`, `cap/db/postgres/`. Always check these when making cross-profile changes.
 
 ### Service Layer (`cap/srv/`)
 
-Six domain services (one per entity): `StarWarsFilm`, `StarWarsPeople`, `StarWarsPlanet`, `StarWarsSpecies`, `StarWarsStarship`, `StarWarsVehicle`. Protocols: OData v4 (primary), OData v2 (adapter), GraphQL (`/graphql`), REST.
+Six domain services (one per entity): `StarWarsFilm`, `StarWarsPeople`, `StarWarsPlanet`, `StarWarsSpecies`, `StarWarsStarship`, `StarWarsVehicle`. A seventh service, `StarWarsEpisode`, exposes `Episodes` and `Episode2*` junctions as read-only projections (no handler logic required). Protocols: OData v4 (primary), OData v2 (adapter), GraphQL (`/graphql`), REST.
 
 **Critical file separation rule:**
 - Service contracts → `*-service.cds`
@@ -123,6 +132,6 @@ GitHub Actions auto-deploys to GitHub Pages on push to `main` when files under `
 
 - **CDS modeling**: Preserve namespace and `managed`/`cuid` patterns. Prefer explicit many-to-many junction entities. Keep `Common.ValueList` and `UI.*` patterns consistent.
 - **Breaking changes**: Avoid renames/removals without migration intent. See `docs/value-help-migration.md` for a past breaking-change example.
-- **Data loading**: Use `convertDataLite.js` for SQLite (avoids `SQLITE_BUSY`); use `convertData.js` for HANA (parallel chunk loading).
+- **Data loading**: Use `convertData.js` for all profiles (handles parallel chunk loading; SQLite-safe). The `load_sqlite` npm script invokes this directly.
 - **Generated folders**: `cap/gen/` is auto-generated. After HANA deployment, re-run `npm run build` if `gen/srv` is cleared.
 - **cds-mcp**: When editing CDS files, resolve entity/field definitions with `cds-mcp` before modifying models, and check CAP docs via `cds-mcp` before proposing CDS syntax or API usage.

README.md

@@ -10,12 +10,48 @@ The original project was a data set and data model based in Python that exposed
 
 The projects described above have fallen out of maintenance but still offered the opportunity for a fun yet challenging learning experience from a non-trivial data model. The many bi-directional, many-to-many relationships with the data provides a good basis for an SAP Cloud Application Programming Model and Fiori Draft UI sample.
 
+This project extends the original data set with **TV Shows and streaming series** (The Mandalorian, The Clone Wars, Andor, etc.) scraped live from Wookieepedia, and introduces unified `Media` views that query across both films and shows.
+
 ### Data Model
 
 #### Films
 
 ![film diagram](images/Film.png)
 
+#### Shows (TV Series & Streaming)
+
+The `Show` entity represents all canon Star Wars TV and streaming productions. It shares the same relationship structure as `Film` — each show can be linked to characters, planets, starships, vehicles, and species.
+
+| Field | Type | Notes |
+| --- | --- | --- |
+| `title` | String | `@mandatory` |
+| `show_type` | Enum | `LIVE_ACTION_SERIES`, `ANIMATED_SERIES`, `ANIMATED_FILM`, `ANTHOLOGY` |
+| `seasons` | Integer | Number of seasons |
+| `episode_count` | Integer | Total episode count |
+| `network` | String | Broadcaster / streaming platform |
+| `director` | String | Lead director(s) |
+| `producer` | String | Lead producer(s) |
+| `release_date` | Date | First air date |
+
+Each show owns a composition of `Episode` records (cascade-delete). The `Episode` entity captures per-episode metadata: `season_number`, `episode_number`, `title`, `air_date`, `director`, `writer`, `runtime` (minutes), and `timeline` (in-universe date, e.g. "19 BBY").
+
+Junction tables: `Show2People` (physical), `Episode2People`, `Episode2Planets`, `Episode2Starships`, `Episode2Vehicles`, `Episode2Species`
+
+`Show2Planets`, `Show2Starships`, `Show2Vehicles`, and `Show2Species` are **CDS `define view`** declarations that aggregate over the corresponding `Episode2*` tables. They are not physical tables — show-level relationship data is derived from episode-level scraping.
+
+#### Unified Media Views
+
+Six read-only `UNION ALL` views join `Film` and `Show` data across a single query surface. The relationship views each include a branch for `Episode2*` data so that episode-level associations are surfaced at the media level:
+
+| View | Returns |
+| --- | --- |
+| `Media` | All films and shows with a `media_type` discriminator (`'FILM'` / `'SHOW'`) |
+| `MediaCharacters` | All film+show+episode → character relationships |
+| `MediaPlanets` | All film+show+episode → planet relationships |
+| `MediaSpecies` | All film+show+episode → species relationships |
+| `MediaStarships` | All film+show+episode → starship relationships |
+| `MediaVehicles` | All film+show+episode → vehicle relationships |
+
 #### People
 
 ![people diagram](images/People.png)
@@ -43,15 +79,45 @@ The projects described above have fallen out of maintenance but still offered th
 
 ## Download and Installation
 
-The original data model and data source files are in in the [oldPython\resources](./oldPython/resources/) folder.
+The rest of the operations can be performed within the [cap](./cap/) folder and there are scripts in the [package.json](./cap/package.json#L20) file for major operations.
+
+You can use `npm run build` to perform the cds build and should be run before deployment to HANA or whenever you make changes to the data model.
 
-The rest of the operations can be performed within the [cap](./cap/) folder and there are scripts in the [package.json](./cap/package.json#L20) file major operations.
+You can run `npm run hana` to deploy the content to your HANA database. Just be sure from the terminal that you are logged into the cf/xs cli and targeting the Account/Org/Space where you want the content to live. By default this command will create an HDI Container instance named **starwars**. **Note**: due to some strange circumstances in the latest versions of CAP it seems the `/gen/srv` folder is getting cleared after any deployment to HANA. Therefore just execute a `cds build` or `npm run build` after any deployment to restore the `/gen` folder until we find the root cause of this issue.
+
+### Data Pipeline
+
+Star Wars data is sourced from [Wookieepedia](https://starwars.fandom.com/wiki/Wookieepedia) via a rate-limited MediaWiki API scraper. The pipeline is two steps:
+
+**Step 1 — Scrape** (optional — committed cache makes this instantaneous):
+
+```bash
+cd cap
+npm run scrape              # cache-first run — uses committed cache, completes in seconds
+npm run scrape:films        # films-only run (no episodes); reproduces the original SWAPI dataset
+npm run scrape:bypass-cache # fetch fresh data from Wookieepedia (prompts for confirmation)
+```
+
+The scraper crawls three levels deep: Show page → Season page → Episode page. It writes nine JSON files to `scripts/data/raw/`:
+`films.json`, `shows.json`, `episodes.json`, `people.json`, `planets.json`, `species.json`, `starships.json`, `vehicles.json`, `relationships.json`
+
+Cache files (`scripts/data/cache/`) are committed to the repository (~6,469 files), so clones can run `npm run scrape` without any network access. Use `--bypass-cache` only when you need genuinely fresh Wookieepedia data.
+
+**Step 2 — Load** into your target database:
+
+```bash
+npm run load_sqlite    # SQLite (local development)
+npm run load           # SAP HANA Cloud (hybrid profile — requires .cdsrc-private.json)
+npm run load_pg        # PostgreSQL
+```
 
-You can use `npm run build` to perform the cds build and should be ran before deployment to HANA or whenever you make changes to the data model.
+The loading script is [convertData.js](./cap/convertData.js). It reads the raw JSON files and upserts all entities and junction records using CAP CQL.
 
-You can run `npm run hana` to deploy the content to your HANA database.  Just be sure from the terminal that you are logged into the cf/xs cli and targeting the Account/Org/Space where you want the content to live. By default this command will create an HDI Container instance named **starwars**. **Note**: due to some strange circumstances in the latest versions of CAP it seems the `/gen/srv` folder is getting cleared after any deployment to HANA.  Therefore just execute a `cds build` or `npm run build` after any deployment to restore the `/gen` folder until we find the root cause of this issue.
+**Scraped data included:**
 
-You can run the command `npm run load`. This command will read the original JSON data files from the source project and load them into your HANA database using Cloud Application Programming Model [CQL](https://cap.cloud.sap/docs/cds/cql). The loading script is [convertData.js](./cap/convertData.js)
+* **11 canon films** — Episodes I–IX, Rogue One, Solo (with directors, producers, release dates)
+* **15 shows** — 8 live-action series + 6 animated series (The Mandalorian, The Clone Wars, Andor, Rebels, etc.)
+* **772 episodes** — per-episode metadata and entity relationships (characters, planets, starships, vehicles, species)
 
 The command `npm start` or `cds run` will start the service running locally. It will open the standard CAP test page where you can explore the OData Services or the Fiori UI.
 

cap/README.md

@@ -9,21 +9,25 @@ A learning-focused SAP Cloud Application Programming Model (CAP) sample built on
 ## Project Structure
 
 | Path | Purpose |
-|------|---------|
+| --- | --- |
 | `db/schema.cds` | Domain model – entities, types, associations |
 | `db/hana/` `db/sqlite/` `db/postgres/` | Profile-specific model extensions |
 | `srv/*-service.cds` | Service definitions – projections, actions, functions |
 | `srv/*-fiori.cds` | Fiori/UI5 annotations (kept separate from service contracts) |
+| `srv/episode-service.cds` | Read-only `StarWarsEpisode` service (OData v4, GraphQL, REST) |
+| `srv/episode-fiori.cds` | Fiori annotations for the Episodes list |
 | `srv/services-auth.cds` | Centralized authorization – `@requires` / `@restrict` |
 | `srv/*.js` | Service handlers – `before` / `on` / `after` hooks |
 | `test/` | Automated tests by layer (model, service, handler) |
 | `docs/` | Architecture, learning path, cheat sheet, pitfalls |
 | `labs/` | Hands-on exercises with guided steps |
 | `app/` | UI frontends and Fiori Launchpad previews |
+| `../scripts/scraper/` | Wookieepedia scraper — rate-limited MediaWiki API crawler |
+| `../scripts/data/raw/` | Scraped JSON output loaded by `convertData.js` |
 
 ## Architecture at a Glance
 
-```
+```text
 Client (Fiori / REST / GraphQL)
         │
         ▼
@@ -59,32 +63,37 @@ See [docs/cap-architecture.md](docs/cap-architecture.md) for the full annotated
 ## Learning Tracks
 
 ### 🟢 Beginner Track
+
 *Goal: understand CDS modeling and how CAP exposes it automatically.*
 
 | # | Task | Where to look |
-|---|------|--------------|
+| --- | --- | --- |
 | 1 | Read `db/schema.cds` — find entities, types, associations | [db/schema.cds](db/schema.cds) |
 | 2 | Start the app (`npm run sqlite`) and browse `/odata/v4/StarWarsPeople/People` | [people-service.cds](srv/people-service.cds) |
 | 3 | Open Swagger UI at `http://localhost:4004/api-docs` | [server.js](srv/server.js) |
 | 4 | Understand how `@mandatory`, `@assert.range`, `@assert.format` work | [db/schema.cds](db/schema.cds) |
-| 5 | Complete **Lab 01** (add a new entity) | [labs/lab-01-model/](labs/lab-01-model/README.md) |
+| 5 | Find the `Episode` entity and its parent `Show` — notice the `Composition of many` | [db/schema.cds](db/schema.cds) |
+| 6 | Complete **Lab 01** (add a new entity) | [labs/lab-01-model/](labs/lab-01-model/README.md) |
 
 ### 🟡 Intermediate Track
+
 *Goal: understand service handlers, events, and custom operations.*
 
 | # | Task | Where to look |
-|---|------|--------------|
+| --- | --- | --- |
 | 1 | Trace the full `before → on → after` handler lifecycle for People | [people-service.js](srv/people-service.js) |
 | 2 | Call the `rename` bound action via HTTP | [people-service.cds](srv/people-service.cds) |
 | 3 | Call the `countByGender` unbound function | [people-service.cds](srv/people-service.cds) |
 | 4 | Understand how notifications + domain events are emitted | [people-service.js](srv/people-service.js) |
-| 5 | Complete **Lab 03** (add your own hook) | [labs/lab-03-handler/](labs/lab-03-handler/README.md) |
+| 5 | Compare `StarWarsEpisode` (read-only, no `.js` handler) with `StarWarsPeople` (full lifecycle) | [episode-service.cds](srv/episode-service.cds) |
+| 6 | Complete **Lab 03** (add your own hook) | [labs/lab-03-handler/](labs/lab-03-handler/README.md) |
 
 ### 🔴 Advanced Track
+
 *Goal: master authorization, testing by layer, and multi-profile deployment.*
 
 | # | Task | Where to look |
-|---|------|--------------|
+| --- | --- | --- |
 | 1 | Study the role matrix (`Viewer / Editor / Admin`) | [services-auth.cds](srv/services-auth.cds) |
 | 2 | Read handler-level tests and understand what each layer tests | [test/handler.test.js](test/handler.test.js) |
 | 3 | Compare SQLite vs HANA vs PG behavior | [docs/profile-comparison.md](docs/profile-comparison.md) |
@@ -105,7 +114,10 @@ npm run sqlite
 # Run all tests
 npm run test
 
-# Load Star Wars fixture data (SQLite profile)
+# Regenerate scraped Star Wars data (cache-first — fast from committed cache files)
+npm run scrape
+
+# Load Star Wars data into SQLite
 npm run load_sqlite
 ```
 
@@ -116,7 +128,7 @@ Browse to <http://localhost:4004> to see the service index, Swagger UI, and Fior
 ## Runtime Profiles
 
 | Profile | Command | Database | Use when |
-|---------|---------|----------|----------|
+| --- | --- | --- | --- |
 | `sqlite` | `npm run sqlite` | SQLite (file) | Local development, no external services |
 | `hybrid` | `npm run watch` | SAP HANA HDI | Integrated local dev with HANA |
 | `pg` | `npm run pg` | PostgreSQL | PostgreSQL-based environments |
@@ -128,9 +140,9 @@ See [docs/profile-comparison.md](docs/profile-comparison.md) for a full comparis
 
 ## CDS Configuration Notes
 
-- `cds.requires.queue: true` — enables the persistent outbox queue for reliable event delivery.
-- `cds.requires.middlewares: true` — enables the full CAP middleware chain (CORS, authentication, etc.).
-- PostgreSQL credentials are intentionally **not hardcoded**; provide them via `.env` (see `cap/.env`).
+* `cds.requires.queue: true` — enables the persistent outbox queue for reliable event delivery.
+* `cds.requires.middlewares: true` — enables the full CAP middleware chain (CORS, authentication, etc.).
+* PostgreSQL credentials are intentionally **not hardcoded**; provide them via `.env` (see `cap/.env`).
 
 ---
 
@@ -147,7 +159,7 @@ npm run test:migration     # data conversion / migration tests
 ## Learning Resources
 
 | Resource | Link |
-|---------|------|
+| --- | --- |
 | CAP Architecture (this repo) | [docs/cap-architecture.md](docs/cap-architecture.md) |
 | Learning Path | [docs/learning-path.md](docs/learning-path.md) |
 | CAP Cheat Sheet | [docs/cap-cheat-sheet.md](docs/cap-cheat-sheet.md) |
@@ -160,5 +172,5 @@ npm run test:migration     # data conversion / migration tests
 
 ## Upgrade Notes
 
-- **Breaking change**: legacy value-help helper endpoints were removed in favor of `*Values` entities.
+* **Breaking change**: legacy value-help helper endpoints were removed in favor of `*Values` entities.
   Update using [docs/value-help-migration.md](docs/value-help-migration.md).

cap/docs/cap-architecture.md

@@ -78,10 +78,17 @@ db/schema.cds
         ├── Species
         ├── Starship
         ├── Vehicles
+        ├── Show          (draft-enabled)
+        │   └── Episode   (composition child — cascade delete)
         ├── Film2People   (M:N junction with @assert.unique)
-        └── ... (other junctions + views)
+        ├── Episode2People, Episode2Planets, Episode2Starships,
+        │   Episode2Vehicles, Episode2Species  (M:N junctions)
+        └── Show2Planets, Show2Starships, Show2Vehicles, Show2Species
+            (CDS define view over Episode2* — not physical tables)
 ```
 
+`define view` in CDS creates a SQL view rather than a physical table. It is the right choice when the data can be fully derived from another entity — there is no value in storing it redundantly. `Show2Planets` (and its siblings) are a concrete example: Wookieepedia show pages list no per-show relationships, but each episode page lists the planets it features. By defining the show-level view as an aggregation over `Episode2Planets`, show-level data is always correct and requires no separate load step.
+
 ### Service Layer (`srv/*-service.cds`)
 
 Services **project** domain entities into API-facing views. The key insight:

cap/docs/learning-path.md

@@ -22,6 +22,8 @@ Five progressive milestones, each building on the previous one. Every milestone
 
 3. Trace how `Film2People` links `Film` and `People` (M:N relationship).
 
+4. Find the `define view` declarations for `Show2Planets`, `Show2Starships`, `Show2Vehicles`, and `Show2Species`. Explain why they are views instead of physical junction tables. (Hint: where does the underlying data actually come from?)
+
 **Key files:** `db/schema.cds`
 
 **Check your understanding:** Run `npm run test` — all model tests should pass.
@@ -55,7 +57,7 @@ Five progressive milestones, each building on the previous one. Every milestone
 
 **Key files:** `srv/people-service.cds`, `srv/film-service.cds`
 
-**Check your understanding:** Open Swagger UI at `http://localhost:4004/api-docs` and explore the generated spec.
+**Check your understanding:** Open Swagger UI at `http://localhost:4004/api-docs` and explore the generated spec. Then compare `StarWarsEpisode` (`srv/episode-service.cds`) — a read-only service with no `.js` handler at all — with `StarWarsPeople`. Notice that CAP's generic provider handles all CRUD automatically when there is no `on` handler registered.
 
 ---