You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the problem
The location extraction Cloud Function currently uses a large amount of memory, due to the extraction and unpacking of GTFS feeds using gtfs_kit within the function itself.
Since the unpacking step will be moved outside the function as part of issue #911, we should revisit the memory allocation settings once that work is completed in order to reduce unnecessary resource allocation.
Adjust the available_memory setting to a more appropriate value based on the new function behavior.
Remove code that performs the feed extraction, as this step will now be handled upstream.
Adjust the location extraction trigger and Pub/Sub and GCS trigger request data to align with the new flow where the feed is already unpacked.
Update the trigger of existing cloud functions like extract_location to match only the stop.txt file changes from the previous dataset. Also, consider adding a "tracking" mechanism to determine that the earlier files were changed and processed.
Update the logic to avoid unzipping the full GTFS zip
Describe the problem
The location extraction Cloud Function currently uses a large amount of memory, due to the extraction and unpacking of GTFS feeds using gtfs_kit within the function itself.
Since the unpacking step will be moved outside the function as part of issue #911, we should revisit the memory allocation settings once that work is completed in order to reduce unnecessary resource allocation.
Proposed solution
Remove code that performs the feed extraction, as this step will now be handled upstream.
extract_locationto match only thestop.txtfile changes from the previous dataset. Also, consider adding a "tracking" mechanism to determine that the earlier files were changed and processed.