Skip to content

Commit fe38cda

Browse files
committed
feat: commit Wookieepedia cache and improve scraper CLI
- Remove scripts/data/cache/ from .gitignore — 6469 cached pages now tracked so clones get all data without live Wookieepedia fetches - Remove cache TTL (cache.js): published Star Wars content never changes, speed and reproducibility matter more than freshness - Replace CACHE_ONLY env var with --bypass-cache CLI flag that requires typing "yes" to confirm before hitting the live API - Add --films-only flag for a fast run that scrapes only films and their entities (no shows, seasons, or episodes) — reproduces the original dataset - Update npm scripts: scrape (cache-first), scrape:films (--films-only), scrape:bypass-cache (--bypass-cache with confirmation) - Pass bypassCache flag through mediawiki.fetchWikitext
1 parent afc3e3b commit fe38cda

6,474 files changed

Lines changed: 25996 additions & 98 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.gitignore

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,4 @@
11
docs/superpowers/
22
.superpowers/
33

4-
# Wookieepedia scraper cache (raw API responses)
5-
scripts/data/cache/
64
scripts/scraper/node_modules/

cap/package.json

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"name": "cap-hana-swapi",
3-
"version": "1.8.0",
3+
"version": "2.0.0",
44
"description": "SAP Cloud Application Programming Model fun sample to demonstrate many-to-many relationships.",
55
"repository": "https://github.com/SAP-samples/cloud-cap-hana-swapi",
66
"license": "Apache-2.0",
@@ -38,7 +38,8 @@
3838
"load_sqlite": "cds bind --exec node convertData.js --profile sqlite",
3939
"load_pg": "cds bind --exec node convertData.js --profile pg",
4040
"scrape": "node ../scripts/scraper/index.js",
41-
"scrape:cache": "CACHE_ONLY=true node ../scripts/scraper/index.js",
41+
"scrape:films": "node ../scripts/scraper/index.js --films-only",
42+
"scrape:bypass-cache": "node ../scripts/scraper/index.js --bypass-cache",
4243
"test": "node --env-file=test/test.env --test --test-timeout 60000 test/model.test.js test/handler.test.js",
4344
"test:handler": "node --env-file=test/test.env --test --test-timeout 60000 test/handler.test.js",
4445
"test:profile": "node --env-file=test/test.env --test --test-timeout 60000 --test-name-pattern \"Profile-specific model exposure\" test/model.test.js",

scripts/data/cache/%22Backstabber%22.json

Lines changed: 4 additions & 0 deletions
Large diffs are not rendered by default.
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
{
2+
"cachedAt": 1775142307835,
3+
"data": "{{Top}}\n{{StarshipClass\n|image=[[File:YellowAce-CGSWG.png]]\n|name=\"Changeling\" Mark 71NB\n|designer=\n|manufacturer=[[Ravager Mechanics]]<ref name=\"USW, NE\">''[[Ultimate Star Wars, New Edition]]''</ref>\n|line=\n|model=\"Changeling\" Mark 71NB<ref name=\"USW, NE\" />\n|type=[[Racing starfighter]]<ref name=\"USW, NE\" />\n|cost=\n|modifier=\n|sysmods=\n|length=\n|width=\n|height=\n|mass=\n|max accel=\n|mglt=\n|max speed=\n|maneuverability=\n|engine=\n|hyperdrive=\n|hdrange=\n|hdsystem=\n|poweroutput=\n|power=\n|shield gen=\n|hull=\n|sensor=\n|target=\n|navigation=\n|avionics=\n|maincomp=\n|countermeasures=\n|armament=\n|complement=\n|bays=\n|escapepods=\n|crew=*[[Pilot]] (1)<ref name=\"The Recruit\">{{Resistance|The Recruit}}</ref>\n*[[Astromech droid]] (1)<ref name=\"The Recruit\" />\n|skeleton=\n|passengers=\n|capacity=\n|cargohandling=\n|consumables=\n|lifesupport=\n|communications=\n|othersystems=\n|availability=\n|role=\n|firstuse=\n|retired=\n|affiliation=*[[Ace Squadron]]<ref name=\"Databank\">{{Databank|yellow-ace|Yellow Ace}}</ref>\n*[[Team Colossus|Colossus resistance]]<ref name=\"No Escape 2\">{{Resistance|No Escape: Part 2}}</ref>\n}}\nThe '''\"Changeling\" Mark 71NB''' was a type of [[racing starfighter]]. [[Ace Squadron]] [[pilot]] [[Bo Keevil]] flew one, the ''[[Yellow Ace]]'', during the [[New Republic Era]].<ref name=\"USW, NE\" />\n\n{{Ship-stub}}\n\n==Behind the scenes==\nThe \"Changeling\" Mark 71NB first appeared in part one of the two-part episode \"[[The Recruit]],\"<ref name=\"The Recruit\" /> the first episode of the [[Star Wars Resistance Season One|first season]] of the animated television series ''[[Star Wars Resistance]]'', which aired on [[October 7]], [[2018]].<ref name=\"Disney release\">{{DisneyPress|wdtv=1|url=disneychannel/shows/star-wars-resistance/episodes|text=Star Wars Resistance}}</ref> The model was identified in the [[2019]] reference book ''[[Ultimate Star Wars, New Edition]]''.<ref name=\"USW, NE\" />\n\nThe model was named by [[Adam Bray]] in reference to [[Mark Newbold]], with \"71\" referencing his birth year and \"NB\" referencing \"Newbold.\"<ref name=\"Fantha Tracks\">{{WebCite|url=https://www.fanthatracks.com/news/literature/ultimate-star-wars-find-the-hidden-fantha-tracker|text=Ultimate Star Wars: Find the hidden Fantha Tracker|date=2019-10-14|author=[[Mark Newbold]]|archivedate=20191014193923}}</ref>\n\n==Appearances==\n{{ScrollBox|content=\n*{{Resistance|The Recruit}} {{1st}}\n*{{ResistanceAnimationCite|1|The Recruit - Part 1}}\n*{{Resistance|The Triple Dark}}\n*{{ResistanceAnimationCite|3|The Triple Dark (comic story)}}\n*{{Resistance|Fuel for the Fire}}\n*{{Resistance|The High Tower}}\n*{{Resistance|Synara's Score}}\n*{{Resistance|The Platform Classic}}\n*{{IDWAdventuresCite-2017|issue=16|story=All Aces Battle Royale (comic story)}}\n*{{Resistance|Bibo (episode)}}\n*{{Resistance|The Need for Speed}}\n*{{Resistance|Sixty Seconds to Destruction}}\n*{{Resistance|The Doza Dilemma}}\n*{{Resistance|The Disappeared}}\n*{{Resistance|No Escape: Part 2}}\n*{{Resistance|A Quick Salvage Run}}\n*{{Resistance|Live Fire}}\n*{{Resistance|Rendezvous Point (episode)}}\n*{{Resistance|The Voxx Vortex 5000}}\n*{{Resistance|Kaz's Curse}}\n*{{Resistance|Breakout}}\n*{{Resistance|The New World}}\n*{{Resistance|No Place Safe}}\n*{{Resistance|The Escape (episode)}}\n*[[Star Wars: The Rise of Skywalker Graphic Novel Adaptation|''Star Wars: The Rise of Skywalker'' Graphic Novel Adaptation]]\n}}\n\n==Sources==\n*<!-- 2019-10-04 -->''[[Ultimate Star Wars, New Edition]]'' {{1stID|\"Changeling\" Mark 71NB}}\n*<!-- 2024-11-19 -->''[[Star Wars Encyclopedia: The Comprehensive Guide to the Star Wars Galaxy]]''\n\n==Notes and references==\n{{Reflist}}\n\n{{Interlang\n|fr=\"Changeling\" Mark 71NB\n|ja=“チェンジリング”・マーク71NB\n}}\n\n{{DEFAULTSORT:Changeling Mark 71NB}}\n[[Category:Racers (starship)]]\n[[Category:Starships]]"
4+
}
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
{
2+
"cachedAt": 1775082448650,
3+
"data": "{{Top|legends=Crash site memorial}}\n{{Otheruses|Yard}}\n{{Structure\n|image=[[File:Crash_site_memorial.png]]\n|name=\"Community\" junk yard\n|constructed=\n|destroyed=\n|type=\n|location=[[Wonto District]],<ref name=\"Outlaws\">''[[Star Wars Outlaws]]''</ref> [[Mos Eisley]], [[Tatooine]]<ref name=\"Complete Locations\">[[Star Wars: Complete Locations (2016)|''Star Wars: Complete Locations'']]</ref>\n|interest=\n|rebuilt=\n|builder=\n|height=\n|width=\n|affiliation=\n}}\nThe '''\"community\" junk yard''' of [[Mos Eisley]], [[Tatooine]], was situated directly in front of the entrance to [[Chalmun's Spaceport Cantina]].<ref name=\"Complete Locations\" />\n\n{{Loc-stub}}\n\n==Appearances==\n*[[The Phantom Menace 25th Anniversary Special 1|''The Phantom Menace'' 25th Anniversary Special 1]]\n*''[[Star Wars Battlefront II]]''\n*{{Film|IV}} {{1st}}\n*[[Star Wars (2015) 5|''Star Wars'' (2015) 5]]\n*[[Age of Rebellion - Jabba the Hutt 1|''Age of Rebellion - Jabba the Hutt'' 1]]\n*[[Doctor Aphra Annual 3|''Doctor Aphra Annual'' 3]]\n*''[[Star Wars Outlaws]]''\n*{{TheMandalorian|Chapter 5: The Gunslinger}}\n\n===Non-canon appearances===\n*<!-- 2022-04-05 -->''[[LEGO Star Wars: The Skywalker Saga]]''\n\n==Sources==\n*<!-- 2016-09-27 -->[[Star Wars: Complete Locations (2016)|''Star Wars: Complete Locations'']] {{1stID|\"community\" junk yard}}\n*<!-- 2017-02-XX -->{{BuildR2Cite|7|Building the Galaxy}} {{Po}}\n\n==Notes and references==\n{{Reflist}}\n\n{{Interlang\n|es=Chatarrería «comunal»\n|fr=Ferraillerie \"communautaire\"\n|ja=“共同”ゴミ捨て場\n|ka=მეჯართის წილი\n}}\n\n{{DEFAULTSORT:Community junk yard}}\n[[Category:Junkyards]]\n[[Category:Mos Eisley locations]]"
4+
}
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
{
2+
"cachedAt": 1775138282898,
3+
"data": "==Summary==\n{{Information\n|attention={{Notorphan|Ahsoka Season 2}}\n|description=[[Ahsoka Season 2|''Ahsoka'' Season 2]] mistakenly given a [[2026]] release date\n|source=[[Disney+]] content slider\n|artist=\n|filespecs=\n|licensing={{Web-screenshot}}\n|other versions=\n|cat artist=\n|cat licensee=[[Category:Images from Disney+]]\n|cat subject=[[Category:Images of Ahsoka Tano]]\n|cat type=skip\n}}"
4+
}
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
{
2+
"cachedAt": 1775138281693,
3+
"data": "==Summary==\n{{Information\n|attention={{Notorphan|Ahsoka Season 2}}\n|description=[[Star Wars: Ahsoka|''Ahsoka'']] mistakenly given a Fall [[2025]] release date\n|source=[[Disney+]] content slider\n|artist=\n|filespecs=\n|licensing={{Web-screenshot}}\n|other versions=\n|cat artist=\n|cat licensee=[[Category:Images from Disney+]]\n|cat subject=[[Category:Images of Ahsoka Tano]]\n|cat type=skip\n}}"
4+
}

scripts/data/cache/0G-LC.json

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
{
2+
"cachedAt": 1775077491262,
3+
"data": "{{Top}}\n{{Update|[[Star Wars: Young Jedi Adventures: Galactic Training]], [[The Thieves of Tharnaka]], [[The Team Up]]}}\n{{Droid\n|type=Jedi\n|image=[[File:0GLC-TLJS.png]]\n|name=0G-LC\n|homeworld=\n|birth=\n|death=\n|creator=\n|manufacturer=\n|line=\n|model=\n|class=[[Pilot droid]]<ref name=\"The Lost Jedi Ship\">{{YJA|The Lost Jedi Ship}}</ref>\n|cost=\n|length=\n|width=\n|height=\n|mass=\n|gender=[[Gender|Masculine programming]]<ref name=\"The Lost Jedi Ship\" />\n|pronouns=He/him<ref name=\"The Lost Jedi Ship\" />\n|sensor=[[Color|Blue]]<ref name=\"The Lost Jedi Ship\" />\n|plating=Yellow, rust-colored, and gray<ref name=\"The Lost Jedi Ship\" />\n|armament=\n|equipment=\n|affiliation=*[[Jedi Order]]<ref name=\"The Lost Jedi Ship\" />\n*[[Hyperspace]] [[prospector]]s<ref name=\"Encyclopedia\">''[[Star Wars Encyclopedia: The Comprehensive Guide to the Star Wars Galaxy]]''</ref>\n}}\n{{Quote|Two asteroids! This is exactly how I crashed last time.''\"<br />\"''OG ''[''sic'']'', you've come so far already. You really are brave.|0G-LC and Kai Brightstar|<ref name=\"The Lost Jedi Ship\" />}}\n'''0G-LC''', often simply referred to as '''0G''', was a [[droid]] who [[pilot]]ed the ''[[Star Seeker]]'' during the [[High Republic Era]]. He explored the [[Outer Rim Territories]] for the [[Jedi Order]], but after his [[Starship|ship]] was damaged in an [[asteroid field]], he crash-landed on a [[Fobaris|planet]] while on his way to the [[planet]] [[Tenoo]]. He eventually fixed the ship but was too frightened to pilot it again, opting to remain stranded.\n\nA long [[time]] later, in [[232 BBY]], [[Jedi Initiate|Jedi younglings]] tracked 0G-LC down after hearing stories of him as a brave adventurer. While initially resistant to their visit, he ultimately let them onto the ''Star Seeker'' and told them of his past failures as a [[pilot droid]]. With encouragement and advice from the [[Jedi]], 0G mustered up enough bravery to retake the pilot's seat and fly through the [[asteroid]] field, successfully doing so and returning to Tenoo before deciding to continue his exploration mission.\n\n==Biography==\n===Stranding and searching===\n{{Quote|I used to explore the Outer Rim for the Jedi. My mission was to bring b-b-back what I learned.|0G-LC, to the younglings|<ref name=\"The Lost Jedi Ship\" />}}\n0G-LC was a [[pilot droid]] of the [[Jedi Order]] [[Standard century|centuries]] before<ref name=\"The Lost Jedi Ship\" /> [[233 BBY]].<ref name=\"Date\">{{YJADate|102B}}</ref> He visited over one hundred [[Planet|worlds]] in his [[starship]], the ''[[Star Seeker]]'', exploring the [[Outer Rim Territories]] on behalf of the [[Jedi]] and bringing back countless artifacts. While exploring the [[Bri'ahl sector|sector]] in which the planet [[Tenoo]] was located, however, two [[asteroid]]s within an [[asteroid field]] nearly destroyed his ship and forced him to crash-land on a [[Fobaris|planet]] nearby Tenoo. The ''Star Seeker'' was unable to fly, and 0G-LC was stranded. He eventually fixed the ''Star Seeker'' and returned it to flying condition, but he was too scared of the dangerous asteroids to continue his mission and go back to Tenoo.<ref name=\"The Lost Jedi Ship\" />\n\n[[File:0G-LCHowdyDo.png|thumb|left|300px|0G-LC introduces himself.]]\nAround 233 [['ABY-BBY' dating system|BBY]],<ref name=\"Date\" /> the [[Tenoo Jedi Temple]] [[Jedi Initiate|younglings]] [[Kai Brightstar]], [[Lys Solay]], and [[Nubs]] discovered a [[star map]] that tracked the ''Star Seeker''. Receiving permission to pursue it from [[Jedi Master]] [[Zia Zaldor Zanna|Zia Zanna]], the [[youngling]]s, along with their [[pilot]] friend, [[Nash Durango]], and her [[droid]], [[RJ-83]], traveled to the planet in search of the lost adventurer, about whom they heard stories of bravery. Once they landed, the Jedi went through the thick fog of the planet and found 0G, but he was resistant to visitors and implored them to leave.<ref name=\"The Lost Jedi Ship\" />\n\nThe trio then cleared the fog using [[the Force]], allowing them to see the ''Star Seeker'' and for 0G-LC to identify them as Jedi. They explained that they found him via the [[Starship tracker|tracker]], which the droid remembered he had and for which he was thankful. He shook the [[Limb|hand]] of each youngling and then informed them of his name, which he forgot for a moment. Inviting them onto his ship, 0G showed the Jedi where he kept his discoveries from his voyages. Brightstar, not realizing that 0G-LC was himself the adventurer about whom he had read, asked what had happened to the pilot, so 0G clarified that it was him and of his past Jedi Order missions.<ref name=\"The Lost Jedi Ship\" />\n\n===Return to Tenoo===\n{{Quote|Now that I am not afraid to fly, I must continue my mission. I visited over 100 worlds, but the galaxy contains many, many more.|0G-LC, to the Jedi|<ref name=\"The Lost Jedi Ship\" />}}\nSolay offered to help fix the ''Star Seeker'' and return 0G-LC to Tenoo with them, but the droid told her that he had fixed the ship [[Standard year|years]] prior and was just afraid to try again&mdash;he denied his attributed bravery and slumped off to the [[cockpit]]. The Jedi decided he just needed encouragement and followed him, with Brightstar giving him a pep talk: he told the droid that everyone had fears but that being brave meant facing them. 0G thus resolved to be Jedi-like and face his fear of flying again. Brightstar had Durango meet the group by 0G's ship, where they planned for the younglings to help 0G-LC fly his ship, following a path through the asteroids that Durango would create.<ref name=\"The Lost Jedi Ship\" />\n\n[[File:0G-LCCelebration.png|thumb|right|300px|0G-LC celebrates his success.]]\nThe two starships took off&mdash;0G expressed his fear, but his newfound friends reminded him they were by his side. Near the end of the field, two asteroids threatened to sandwich 0G once again as he had been years prior, but with a nudge from Brightstar to be brave, he found an opening and got through them. He successfully piloted out of the field, much to the squeals of Durango and his passengers. He laughed and exclaimed that he had been brave. When the ''Star Seeker'' landed on Tenoo, 0G brought out a [[Color|purple]] artifact, which Zanna assured Jedi would study for a long [[time]]. 0G then announced that his mission was to continue, for there were many more worlds in [[the galaxy]] for him to explore&mdash;Brightstar told the droid that the Jedi would miss him but that he was always welcome to visit. 0G-LC expressed interest in that option and then bade farewell, taking off in the ''Star Seeker''.<ref name=\"The Lost Jedi Ship\" />\n\n===The Story Stone===\nSome time later, Kai, Lys and Nubs accompny OG to the planet [[Tharnaka]] to see the [[Tharnakan Story Stone]] from within the caves, hoping to shed light on the Tharnakans and to add it the [[Temple Library|Jedi Archives]] at the [[Tenoo Jedi Temple]]. After they find it from the caves, three masked thieves suddenly took the stone and escaped on their [[speeder bike]]s through an electrical storm approching. Seeking shelter for the four, OG recalls the planet to be abandoned after Brightstar spotted a campfire where the thieves originally were.<ref name=\"The Thieves of Tharnaka\">{{YJA|The Thieves of Tharnaka}}</ref>\n\n==Characteristics==\n{{Quote|I'm too afraid to try again. I&mdash;I'm not a brave adventurer. Not anymore.|0G-LC, to the younglings|<ref name=\"The Lost Jedi Ship\" />}}\n0G-LC gained a reputation in the Jedi Order for being a brave, adventurous [[Scout|explorer]]. After crashing the ''Star Seeker'' and fearing to ever fly again, however, he felt that in reality he was not brave. It was only after his newfound Jedi friends offered him words of encouragement that he felt comfortable to try again, though he remained nervous. He was proud of his accomplishment after his flight succeeded. When he first met the Jedi younglings, he took care to introduce himself and use manners. The droid had two arms and two legs and was covered in silver, yellow, gray, and rust-colored plating. He had two blue [[eye]] [[sensor]]s, the left of which was larger than the right, and he had two [[antenna]]e affixed to his head. He had [[Gender|masculine]] programming and spoke with a stutter.<ref name=\"The Lost Jedi Ship\" />\n\n==Behind the scenes==\n0G-LC was created for \"[[The Lost Jedi Ship]],\" the second part of the [[Nash's Race Day / The Lost Jedi Ship|second episode]] of the animated series ''[[Star Wars: Young Jedi Adventures]]''.<ref name=\"The Lost Jedi Ship\" /> The episode was first screened at [[Celebration Europe (2023)|Celebration Europe]] on [[April 8]], [[2023]],<ref name=\"Celebration\">{{SW|url=news/swce-2023-panel-schedule|text=Plan Ahead with the ''Star Wars'' Celebration Europe 2023 Panel Schedule}}</ref> and his voice was provided by [[Michael Sinterniklaas]]. This droid was modeled by Gabriel Lopez.<ref>{{ArtStation|url=artwork/Xg2rgl|username=gabelopez|name=[[Gabriel Lopez]]|text=Star Wars Young Jedi Adventures – Droids 2}}</ref> \n\nWhile the subtitles refer to him as \"OG-LC,\" the credits give his name as \"0G-LC.\"<ref name=\"The Lost Jedi Ship\" /> This article defers to the credits.\n\n==Appearances==\n{{Mediacat|imagecat=Images of 0G-LC}}\n*{{YJA|The Lost Jedi Ship}} {{1st}}\n*{{DisneyNOW|url=games/star-wars-young-jedi-adventures-galactic-training/GM5533269029|text=Star Wars: Young Jedi Adventures: Galactic Training|int=Star Wars: Young Jedi Adventures: Galactic Training}} {{Mo}} {{ID|OG-LC}}\n*{{YJA|The Thieves of Tharnaka}}\n*{{YJA|The Team Up}}\n*{{YJA|The Mission Mixup}} {{Hologram}}\n*{{YJA|Fossil Hunt}}\n*{{YJA|Journey to the Bottom of Naboo}} {{Hologram}}\n\n==Sources==\n*<!-- 2023-04-10 -->{{SWKidsYouTube|otoDnun24zM|Young Jedi Adventures &#124; Official Trailer}} {{1stp}}\n*<!-- 2024-11-19 -->''[[Star Wars Encyclopedia: The Comprehensive Guide to the Star Wars Galaxy]]''\n\n==Notes and references==\n{{Reflist}}\n\n{{Interlang\n|ja=0G-LC\n|nl=OG-LC\n}}\n\n[[Category:Droids of the Jedi Order]]\n[[Category:Droids with he/him pronouns]]\n[[Category:Droids with masculine programming]]\n[[Category:Pilot and navigation droids]]"
4+
}

0 commit comments

Comments
 (0)