Skip to content

Update extract_location trigger and implementation #915

Description

@cka-y

Describe the problem
The location extraction Cloud Function currently uses a large amount of memory, due to the extraction and unpacking of GTFS feeds using gtfs_kit within the function itself.

Since the unpacking step will be moved outside the function as part of issue #911, we should revisit the memory allocation settings once that work is completed in order to reduce unnecessary resource allocation.

Proposed solution

  • Re-evaluate memory requirements for the location extraction function after Unzip and store latest GTFS feeds #911 is completed.
  • Adjust the available_memory setting to a more appropriate value based on the new function behavior.
    Remove code that performs the feed extraction, as this step will now be handled upstream.
  • Adjust the location extraction trigger and Pub/Sub and GCS trigger request data to align with the new flow where the feed is already unpacked.
  • Update the trigger of existing cloud functions like extract_location to match only the stop.txt file changes from the previous dataset. Also, consider adding a "tracking" mechanism to determine that the earlier files were changed and processed.
  • Update the logic to avoid unzipping the full GTFS zip

Metadata

Metadata

Assignees

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions