feat: create a gcp function to generate the pmtiles of a single dataset by jcpitre · Pull Request #1305 · MobilityData/mobility-feed-api

jcpitre · 2025-07-29T15:24:34Z

Summary:
Closes #1296

Added a function as part of tasks_executor that will attempt to create pmtiles (for displaying routes and stops in the UI)
It will do that only if the dataset already has been unzipped, i.e. the extracted folder is present in the dataset data in the bucket.
It does not verify if the dataset is the latest. It just creates the pmtiles for the dataset passed as argument.

It can be called manually with curl.
The payload is:

{
   "task": "build_pmtiles", 
   "payload": {
      "feed_stable_id": "string",
      "dataset_stable_id": "string" 
   } 
}

Example of call with curl:

curl -X POST https://northamerica-northeast1-mobility-feeds-dev.cloudfunctions.net/tasks_executor -H "Content-Type: application/json" -d '{"task": "build_pmtiles", "payload": {"feed_stable_id": "mdb-1004", "dataset_stable_id": "mdb-1004-202507081807"}}'

After the work is done, it creates a pmtiles folder at the same level as the extracted folder. e.g.:

mobilitydata-datasets-dev/mdb-1004/mdb-1004-202507081807/pmtiles

It should contain 3 files:

routes.json
routes.pmtiles
stops.pmtiles

Please make sure these boxes are checked before submitting your pull request - thanks!

Run the unit tests with ./scripts/api-tests.sh to make sure you didn't break anything
Add or update any needed documentation to the repo
Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
Linked all relevant issues
Include screenshot(s) showing how this pull request works and fixes the issue(s)

Copilot

Pull Request Overview

This PR adds a new Google Cloud Function task to generate PMTiles files from GTFS datasets for route and stop visualization in the UI. The function downloads extracted GTFS data from a GCS bucket, processes it to create GeoJSON files, uses Tippecanoe to generate PMTiles, and uploads the results back to GCS.

Key changes include:

New PmtilesBuilder class that orchestrates the end-to-end PMTiles generation process
Integration with the existing tasks executor framework
Comprehensive test coverage for all components

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
functions-python/tasks_executor/src/tasks/pmtiles_builder/build_pmtiles.py	Core implementation of the PMTiles builder with GTFS processing and Tippecanoe integration
functions-python/tasks_executor/tests/tasks/pmtiles_builder/test_build_pmtiles.py	Comprehensive test suite covering all builder functionality and edge cases
functions-python/tasks_executor/src/main.py	Registration of the new build_pmtiles task with the executor framework
functions-python/tasks_executor/requirements.txt	Added dependencies for GCS storage, Tippecanoe, and cloud services
functions-python/tasks_executor/function_config.json	Updated function configuration to enable HTTP triggers and adjust settings

Comments suppressed due to low confidence (1)

functions-python/tasks_executor/requirements.txt:27

The 'tippecanoe' package is not available as a Python package via pip. Tippecanoe is a command-line tool that needs to be installed separately on the system. This line should be removed from requirements.txt.

tippecanoe

…mtiles-of-a-single-dataset

…ould loop forever.

…mtiles-of-a-single-dataset

davidgamez · 2025-08-07T12:52:03Z

+from shared.helpers.logger import get_logger
+
+# Files are stored locally to be able to run tippecanoe on them. This is the directory
+local_dir = "./workdir"


There is a risk of multiple calls running on the same container. It seems unlikely, but it can happen and have dangerous effects. From calls fail to mix PMTiles for different datasets/feeds. To avoid this scenario, use temporal folders and clean them up at the end of the processing.

Used tempfile to create a temporary directory with a random name

davidgamez · 2025-08-07T13:00:15Z

+            for file_name in file_names:
+                blob_path = f"{unzipped_files_path}/{file_name}"
+                blob = self.bucket.blob(blob_path)
+                local_path = os.path.join(local_dir, file_name)
+                blob.download_to_filename(local_path)
+                self.logger.debug("Downloaded %s to %s", blob_path, local_path)


To reduce noise in the logs, I suggest checking if the files already exist in the bucket before attempting to download them. If the file doesn't exist, an exception here is acceptable as an error in this processing. If the file doesn't exist, it is not an "error" of this process; it is rather a missing file in the feed or an issue with the extraction process; in this case, it should be a warning. Then only errors related to this process will be reflected in the logs

Some of the problems we can encounter (e.g. the dataset is not extracted) are not due to this process, but will still abort the operation. You're saying these cases should be logged as warning? It seems unusual

To reduce noise in the logs, the error about generating files must be in a different function, as this function is not responsible for the generation itself. If we log as an error, the assumption is that this function failed to process due to its "own doing", but in this case, it is a requirement that the files must be in place for the function to process the dataset.

I'm ok with this, but as I said it seems a bit unusual to have for example a warning saying that the extracted directory is missing, then return. Do we usually see warnings leading to early return from a function?

It was a suggestion, not a blocker. The intention of reducing error messages is to make sure those messages are at the error level in the function that we can fix. If a function is the cause of the error, then we have the error "propagated" to other functions; the investigation might be delayed as the developer will be looking at functions that are the root cause. In this case, extracting the files is not a concern of this function.

Modified somehow to issue a warning if a required file is not present.

cka-y · 2025-08-07T16:44:02Z

@@ -0,0 +1,485 @@
+#


While the batch fill for PMTiles could make sense as part of the task executor (since it's a one-time operation), I’m not sure the main PMTiles generation logic belongs here. Unlike the other tasks, this feature isn't a one-time function — it will run continuously for new datasets detected daily and likely needs additional orchestration. Curious if this is planned to be handled in a separate issue?

The original idea is have a cloud run function does just one thing: Generate the pmtiles for a given dataset.
Using tasks_executor was the easiest way to get to that.
Now it's entirely possible that tasks_executor is not the proper way to do that, in particular because the requests for pmtiles might clog the executor if we are limiting the number of instances. Also it's currently limited to receiving http calls.
In #913 (comment) we proposed using a pubsub topic which would be more similar to what is done with batch-datasets and batch-process-datasets.
Now there's the question of who will publish to the pubsub, weather it's to generate pmtiles for newly downloaded datasets, or batchfill for already downloaded datasets. As you can see this question is not entirely answered.

We do have 2 issues for this: #1290 and #1011. I think the detailed design will happen as we work on these issues.

I’d suggest using an HTTP function with a Cloud Task queue to take advantage of the longer execution time (1 hour for HTTP vs. 9 minutes for Pub/Sub). We could set it up similar to how we handle location extraction. That said, I don’t think adding this directly to the task_executor is the best approach.

cka-y · 2025-08-07T16:57:09Z

+                        "routeName": row.get("route_long_name", ""),
+                        "color": f"#{row.get('route_color', '000000')}",
+                        "textColor": f"#{row.get('route_text_color', 'FFFFFF')}",
+                        "routeType": f"{row.get('route_type', '3')}",


[question] why would we default to 3 knowing the route_type is a required field? could we just assume it's a bus route if it's not specified cc @emmambd

One alternative is to "invent" an unknown type instead of using 3
Another is to skip that route so it does not appear in the list.
@emmambd ?

My two cents,

I wouldn't assume that any field should be present if marked as required, as data can be missing regardless.

Having 'unknown', 'not provided', or simply None, it makes sense to me as it reflects the current value(missing in this case).

Used 'unknown' instead of '3'

cka-y · 2025-08-07T17:02:37Z

+                features.append(
+                    {
+                        "type": "Feature",
+                        "properties": {k: stop[k] for k in stop},


we need more properties for stops than what is included in stops.txt like the list of route ids related to the stop and the route colours as described in the requirements in #1296

You're right.
I added an issue for that: #1312
I think we can go ahead with what we have.

…mtiles-of-a-single-dataset

jcpitre added 4 commits July 25, 2025 12:20

First draft

920f412

Reformatted.

d10552a

Correction

db34ac3

Added tests

d88f11e

jcpitre requested a review from Copilot July 29, 2025 15:24

jcpitre linked an issue Jul 29, 2025 that may be closed by this pull request

Create a GCP function to generate the pmtiles of a single dataset #1296

Closed

Copilot AI reviewed Jul 29, 2025

View reviewed changes

Comment thread functions-python/tasks_executor/src/tasks/pmtiles_builder/build_pmtiles.py Outdated

Comment thread functions-python/tasks_executor/src/tasks/pmtiles_builder/build_pmtiles.py Outdated

Comment thread functions-python/tasks_executor/src/tasks/pmtiles_builder/build_pmtiles.py Outdated

jcpitre added 12 commits July 29, 2025 11:34

Corrected the documentation

f29eb4d

Corrected errors found by copilot

c7020ff

Added documentation

6389ec7

Added env variables to the cloud function

cf53a53

Merge branch 'main' into 1296-create-a-gcp-function-to-generate-the-p…

adbdeaa

…mtiles-of-a-single-dataset

Corrected a problem in the pmtiles builder where reading some files w…

56aa6f5

…ould loop forever.

Repaired tests.

d742775

Merge branch 'main' into 1296-create-a-gcp-function-to-generate-the-p…

f12d14c

…mtiles-of-a-single-dataset

Optimized memory.

6cf66b0

Minor corrections.

6c9880f

Modified tests

6ebd55e

Merge branch 'main' into 1296-create-a-gcp-function-to-generate-the-p…

5731778

…mtiles-of-a-single-dataset

davidgamez reviewed Aug 7, 2025

View reviewed changes

cka-y reviewed Aug 7, 2025

View reviewed changes

This was referenced Aug 7, 2025

Further optimize the pmtiles creation algorithm #1311

Closed

Add missing data to the pmtiles. #1312

Closed

jcpitre added 2 commits August 11, 2025 16:58

Modified according to PR comments.

ae947d8

Merge branch 'main' into 1296-create-a-gcp-function-to-generate-the-p…

32ff0e6

…mtiles-of-a-single-dataset

jcpitre requested review from cka-y and davidgamez August 11, 2025 21:19

davidgamez approved these changes Aug 11, 2025

View reviewed changes

jcpitre merged commit 5d2de6e into main Aug 12, 2025
3 checks passed

jcpitre deleted the 1296-create-a-gcp-function-to-generate-the-pmtiles-of-a-single-dataset branch August 12, 2025 15:29

Uh oh!

Conversation

jcpitre commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jcpitre Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jcpitre commented Jul 29, 2025 •

edited

Loading

jcpitre Aug 7, 2025 •

edited

Loading