Skip to content

feat: batch fill pmtiles files#1336

Merged
cka-y merged 35 commits into
mainfrom
feat/1290
Sep 10, 2025
Merged

feat: batch fill pmtiles files#1336
cka-y merged 35 commits into
mainfrom
feat/1290

Conversation

@cka-y

@cka-y cka-y commented Sep 3, 2025

Copy link
Copy Markdown
Contributor

Summary:
This PR introduces improvements and refactorings to dataset processing and pipeline task creation, especially around handling PMTiles builder tasks and visualization files.
The changes include:

  • refactoring the dataset processing pipeline for clarity and maintainability
  • moving the PMTiles builder task creation utility to a shared location
  • adding support for rebuilding missing visualization files.

Testing tips:
Test in dev by calling:

curl -X POST "https://tasks-executor-dev-978785769226.northamerica-northeast1.run.app" \
-H "Authorization: bearer $(gcloud auth print-identity-token)" \
-H "Content-Type: application/json" \
-d '{
  "task": "rebuild_missing_visualization_files",
  "payload": {
    "dry_run": true 
  }
}'

Please make sure these boxes are checked before submitting your pull request - thanks!

  • Run the unit tests with ./scripts/api-tests.sh to make sure you didn't break anything
  • Add or update any needed documentation to the repo
  • Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
  • Linked all relevant issues
  • Include screenshot(s) showing how this pull request works and fixes the issue(s)

@cka-y
cka-y requested a review from Copilot September 3, 2025 20:25
@cka-y cka-y linked an issue Sep 3, 2025 that may be closed by this pull request

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces functionality for batch filling PMTiles visualization files. The changes enable the automated rebuilding of missing visualization files (PMTiles) for GTFS datasets by implementing a new task system and moving PMTiles builder functionality to shared helpers.

Key changes:

  • Added infrastructure for PMTiles builder task queue and IAM permissions
  • Implemented a new task executor function to rebuild missing visualization files with comprehensive testing
  • Refactored PMTiles builder functionality from batch processing to shared helpers for reusability

Reviewed Changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
infra/functions-python/main.tf Added task queue infrastructure and IAM permissions for PMTiles builder batch processing
functions-python/tasks_executor/src/tasks/visualization_files/rebuild_missing_visualization_files.py New task implementation to identify and rebuild missing PMTiles visualization files
functions-python/tasks_executor/tests/tasks/vizualisation_files/test_rebuild_visualization_files.py Comprehensive test suite for the visualization files rebuild functionality
functions-python/tasks_executor/src/tasks/visualization_files/README.md Documentation for the new visualization files rebuild task
functions-python/tasks_executor/src/main.py Added new task handler registration for visualization files rebuilding
functions-python/helpers/utils.py Moved PMTiles builder task creation function to shared helpers
functions-python/helpers/tests/test_helpers.py Added tests for the moved PMTiles builder function
functions-python/batch_process_dataset/tests/test_pipeline_tasks.py Removed PMTiles builder tests (moved to helpers)
functions-python/batch_process_dataset/src/pipeline_tasks.py Refactored to use shared PMTiles builder function
functions-python/batch_process_dataset/src/main.py Updated DatasetProcessor methods and improved error handling
functions-python/batch_process_dataset/tests/test_batch_process_dataset_main.py Updated tests to reflect method name changes

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Comment thread infra/functions-python/main.tf Outdated
cka-y and others added 4 commits September 3, 2025 16:27
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@cka-y
cka-y marked this pull request as ready for review September 3, 2025 20:30

@jcpitre jcpitre left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work!

@davidgamez davidgamez left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@cka-y
cka-y merged commit f971c5a into main Sep 10, 2025
2 of 3 checks passed
@cka-y
cka-y deleted the feat/1290 branch September 10, 2025 17:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Batch filling pmtiles for all feeds

4 participants