Skip to content

feat: create pmtiles for new datasets + update location extraction#1322

Merged
cka-y merged 21 commits into
mainfrom
feat/1290
Aug 28, 2025
Merged

feat: create pmtiles for new datasets + update location extraction#1322
cka-y merged 21 commits into
mainfrom
feat/1290

Conversation

@cka-y

@cka-y cka-y commented Aug 19, 2025

Copy link
Copy Markdown
Contributor

Summary:
This PR introduces a new pipeline task system for GTFS dataset processing, focusing on creating and managing Cloud Tasks for downstream processing (reverse geolocation and PMTiles building). It adds task creation utilities, integrates them into existing batch processing flows, and refactors the PMTiles builder as an HTTP-triggered Cloud Function with updated configuration and documentation.

Expected behavior:
When the batch processing completes, two tasks are created:

  1. The reverse geolocation task is created if stops.txt exists and its hash differs from the previous dataset (i.e., the file was updated).
  2. The PMTiles generation task is created if routes.txt exists, its size is > 0 and < 1,000,000 bytes (~1 MB), and the hash of at least of routes.txt, stops.txt, stop_times.txt, trips.txt or shapes.txt differs from the previous dataset.

Please make sure these boxes are checked before submitting your pull request - thanks!

  • Run the unit tests with ./scripts/api-tests.sh to make sure you didn't break anything
  • Add or update any needed documentation to the repo
  • Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
  • Linked all relevant issues
  • Include screenshot(s) showing how this pull request works and fixes the issue(s)

@cka-y cka-y changed the title Feat/1290 feat: create pmtiles for new datasets + update location extraction Aug 28, 2025
@cka-y cka-y marked this pull request as ready for review August 28, 2025 14:00
@cka-y cka-y requested a review from davidgamez August 28, 2025 15:41
Comment thread functions-python/reverse_geolocation/src/main.py
Comment thread infra/batch/main.tf Outdated
Comment thread functions-python/batch_process_dataset/src/pipeline_tasks.py Outdated
Comment thread functions-python/batch_process_dataset/src/pipeline_tasks.py
Comment thread functions-python/tasks_executor/requirements.txt
@cka-y cka-y merged commit be75b18 into main Aug 28, 2025
4 of 5 checks passed
@cka-y cka-y deleted the feat/1290 branch August 28, 2025 18:21
@cka-y cka-y linked an issue Sep 4, 2025 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stretch goal: Optimize pmtiles creation triggering Create pmtiles for newly downloaded datasets Update extract_location trigger and implementation

2 participants