Describe the problem
In the current feed import process, when importing a new feed or updating an existing feed, we need to perform checks on the direct download URL.
Feed Update
In the event that a feed is being updated (deprecated and redirected), the Github issue states both the current URL in MobilityDatabase, and the new incoming URL. We need to:
- Confirm that the current URL in MobilityDatabase does in fact currently exist,
- Confirm that the incoming URL does not yet exist.
Feed Create
In the event that a feed is being created, the Github issue states the new incoming URL. We need to:
- Confirm the incoming URL does not yet exist.
If any of these checks don't pass during the feed update or create flow, it suggests an invalid feed import request because either the new feed already exists in the database, the current feed cannot be found (and therefore cannot be properly deprecated and redirected), or the contributor is importing a feed as brand new when it is actually a feed update.
Proposed solution
We can check for the pre-existence of feeds by checking the incoming URLs against the feeds table in the database. We can check for the normalized url to avoid false negatives on the http protocol (http vs https).
Alternatives you've considered
We should also check for pre-existing feeds through other criteria, such as provider name, feed name, feed bounding box, or location (from reverse geo-location). This would allow us to catch the following scenarios:
Feed submission incorrectly marked as create, when it is update
Contributor may accidentally mark an import as create, when in fact it should be an update. Only the new incoming URL is provided, and rather than simply importing it as a brand new feed, we need to deprecate and redirect the old feed to this new incoming feed.
Performing a check on the provider name, feed name, or location information will reveal the match, whereas simply performing a check between URLs won't because they will be different.
Feed submission marked as update, when the feed to deprecate & redirect cannot be found
Contributor may mark an import as update, but the feed to deprecate and redirect is not present in the MobilityDatabase. Sometimes the provided URL to update is incorrect, and performing a check on the above criteria will reveal the potential intended URL for update. Sometimes the provided URL to update does not exist at all, and this allows us to flag the issue for follow-up, or import as a brand new feed instead.
Additional context
No response
Describe the problem
In the current feed import process, when importing a new feed or updating an existing feed, we need to perform checks on the direct download URL.
Feed Update
In the event that a feed is being updated (deprecated and redirected), the Github issue states both the current URL in MobilityDatabase, and the new incoming URL. We need to:
Feed Create
In the event that a feed is being created, the Github issue states the new incoming URL. We need to:
If any of these checks don't pass during the feed update or create flow, it suggests an invalid feed import request because either the new feed already exists in the database, the current feed cannot be found (and therefore cannot be properly deprecated and redirected), or the contributor is importing a feed as brand new when it is actually a feed update.
Proposed solution
We can check for the pre-existence of feeds by checking the incoming URLs against the feeds table in the database. We can check for the normalized url to avoid false negatives on the http protocol (http vs https).
Alternatives you've considered
We should also check for pre-existing feeds through other criteria, such as provider name, feed name, feed bounding box, or location (from reverse geo-location). This would allow us to catch the following scenarios:
Feed submission incorrectly marked as create, when it is update
Contributor may accidentally mark an import as create, when in fact it should be an update. Only the new incoming URL is provided, and rather than simply importing it as a brand new feed, we need to deprecate and redirect the old feed to this new incoming feed.
Performing a check on the provider name, feed name, or location information will reveal the match, whereas simply performing a check between URLs won't because they will be different.
Feed submission marked as update, when the feed to deprecate & redirect cannot be found
Contributor may mark an import as update, but the feed to deprecate and redirect is not present in the MobilityDatabase. Sometimes the provided URL to update is incorrect, and performing a check on the above criteria will reveal the potential intended URL for update. Sometimes the provided URL to update does not exist at all, and this allows us to flag the issue for follow-up, or import as a brand new feed instead.
Additional context
No response