Skip to content

feat: batch task to reduce geojson maps precision and remove ids#1353

Merged
davidgamez merged 12 commits into
mainfrom
feat/update_geolocation_precision_batch
Sep 10, 2025
Merged

feat: batch task to reduce geojson maps precision and remove ids#1353
davidgamez merged 12 commits into
mainfrom
feat/update_geolocation_precision_batch

Conversation

@davidgamez

@davidgamez davidgamez commented Sep 9, 2025

Copy link
Copy Markdown
Member

Summary:

Closes MobilityData/product-tasks#152

This pull request introduces a new task for updating the precision of GeoJSON files in Google Cloud Storage and removing OSM-related identifiers from their properties. It also adds the necessary database columns to track the processing of these files and provides comprehensive unit tests for the new functionality. Additionally, there are minor dependency and import improvements to support these changes.

GeoJSON File Processing Task

  • Added a new handler update_geojson_files_precision_handler in tasks/geojson/update_geojson_files_precision.py to round coordinates in GeoJSON files and remove OSM-related keys from feature properties, with support for dry-run mode and configurable precision.
  • Registered the new task in main.py and documented its usage, enabling invocation via the task executor. [1] [2]

Database Schema Updates

  • Added geolocation_file_created_date and geolocation_file_dataset_id columns to the feed table, with a foreign key constraint to gtfsdataset(id), to track processed GeoJSON files. [1] [2]

Testing and Development Environment

  • Implemented comprehensive unit tests for the new task, including mocking GCS interactions and verifying coordinate rounding and property removal.
  • Improved test environment setup to ensure shared imports resolve correctly by updating sys.path in conftest.py.

Dependency Management

  • Added psutil to requirements.txt to support resource tracking for the new handler.

Code Quality and Maintainability

  • Changed import of pycountry in helpers/locations.py to be local within the function, reducing unnecessary global imports. [1] [2]

Expected behavior:

Explain and/or show screenshots for how you expect the pull request to work in your testing (in case other devices exhibit different behavior).

GTFS Feed Map
Screenshot 2025-09-10 at 12 50 21 PM

GBFS Feed Map
Screenshot 2025-09-10 at 1 52 40 PM

Testing tips:

Provide tips, procedures and sample files on how to test the feature.
Testers are invited to follow the tips AND to try anything they deem relevant outside the bounds of the testing tips.

Please make sure these boxes are checked before submitting your pull request - thanks!

  • Run the unit tests with ./scripts/api-tests.sh to make sure you didn't break anything
  • Add or update any needed documentation to the repo
  • Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
  • Linked all relevant issues
  • Include screenshot(s) showing how this pull request works and fixes the issue(s)

@davidgamez
davidgamez marked this pull request as ready for review September 10, 2025 19:00
@davidgamez
davidgamez requested a review from cka-y September 10, 2025 20:14

@cka-y cka-y left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM ! left some minor comments

@davidgamez
davidgamez merged commit b17e4c0 into main Sep 10, 2025
2 of 3 checks passed
@davidgamez
davidgamez deleted the feat/update_geolocation_precision_batch branch September 10, 2025 20:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants