SDMTS bundle: stop carries unresolvable service-date route-collection → null RouteBean (blanks Blue Line arrivals)

[aaronbrethorst]

Summary

On the SDMTS multi-feed deployment, arrivals-and-departures-for-stop returns the literal body null (HTTP 200) for ~122 of 132 San Diego Trolley Blue Line (route MTS_510) stops. The proximate cause is an uncaught NullPointerException while building the response (fixed separately — see companion PR). This issue tracks the underlying data/bundle root cause: a stop carries a route-collection reference that does not resolve to a RouteBean on the current service date, which injects a null route into the bean graph.

We need a fresh investigation to find the specific corrupt stop→route-collection index entry in the bundle and determine why it was produced, so we can fix it at the source (bundle build) rather than only defending against it in the serializer.

Affected deployment / data

• Service: SDMTS Multi-Feed (Render srv-d8996ejbc2fs73ev4q70), https://sdmts-multi-feed.onrender.com
• Bundle: https://gtfs-bundles-development.onebusawaycloud.com/rule-set-3-merged-gtfs.zip
  • GTFS files dated 2025-05-24; bundle calendar service range 2025-10-12 → 2026-08-02 (covers "today", so not a stale-calendar issue).
• Reproduce (returns the string null, HTTP 200):

  curl "https://sdmts-multi-feed.onrender.com/api/where/arrivals-and-departures-for-stop/MTS_75018.json?key=org.onebusaway.iphone"
  # still null even with a zero-width window:
  curl ".../arrivals-and-departures-for-stop/MTS_75018.json?key=...&minutesBefore=0&minutesAfter=0"
  

What we already know (don't re-derive)

• Mechanism (code): BeanFactoryV2.getStop() (onebusaway-api-core) iterates stop.getRoutes() and calls route.getId() with no null check; a null RouteBean there NPEs. The null enters via StopBeanServiceImpl.fillRoutesForStopBean(), which adds whatever _routeBeanService.getRouteForId(routeCollectionId) returns — including null — with no guard. This fires on the service-date path (RouteService.getRouteCollectionIdsForStopForServiceDate), not the all-routes path. The companion PR adds defensive guards + moves the action's bean conversion inside its try/catch. Those stop the null response but do not fix the bad data.
• Blast radius: the single-stop A&D path builds references for up to 100 nearby stops, so one nearby stop with a phantom route poisons every stop near it. That's why nearly the whole Blue Line fails.
• Evidence it's data, not transient state:
  • 122/132 route-510 stops return null; the only 10 OK are the multi-line interchange stations: Old Town (oldS/75042/75043), Santa Fe Depot (sfdS/75082/75083), UTC (utcS/77787).
  • Fails with a zero-width time window → it's reference/nearby-stop building, not an arrival in the window.
  • Not realtime (route 510 had 0 active vehicles at the time).
  • Not broken static route integrity in the raw GTFS: route 510 exists in routes.txt (MTS,510,Blue,San Ysidro - UTC,0,...); comm -23 of route_ids used by trips.txt vs defined in routes.txt is empty (no dangling refs).
  • Rebuilding + redeploying the Docker image did not clear it → it's baked into the bundle indices, not runtime state.
  • schedule-for-stop/MTS_75018 returns stopRouteSchedules: [], and stop/MTS_75018 (all-routes path) returns fine with routeIds:["MTS_510"] — consistent with the corruption being specific to the service-date route-collection lookup.

What the investigation needs to produce

1. Identify the specific culprit stop(s): the stop(s) whose service-date route-collection id does not resolve to a route bean. It is one of the ~100 stops near Barrio Logan (MTS_75018, lat/lon 32.6981, -117.1467). Likely approach: load the bundle and, for each stop near the Blue Line, call getRouteCollectionIdsForStopForServiceDate and check each id against _routeBeanService.getRouteForId(...) for null; OR add temporary logging in fillRoutesForStopBean to print the stop id + route-collection id when getRouteForId returns null.
2. Determine why the index entry exists with no resolvable collection: inspect how the stop→routeCollection-for-service-date index is built in onebusaway-transit-data-federation-builder (route collection / stop-route index tasks) and why a service-date entry survives without a corresponding RouteCollection on that date. Suspect areas: route-collection merging/consolidation, the multi-feed GTFS merge (gtfs-merge-service), and service-date → routeCollection index generation.
3. Propose a source fix at bundle-build time (drop/repair orphaned service-date stop→route-collection entries, or fix the merge that produces them), plus a regression check.

Notes / environment

• Render CLI: render logs -r srv-d8996ejbc2fs73ev4q70 .... The web tier logs no stack trace for this (the REST plugin swallows it). Realtime logs (null block, no stoptimes) are loud noise on this service and are a separate, unrelated NCTD/agency-list matter — ignore them for this issue.
• Bundle download + analysis can be done locally (the zip is ~11 MB).
• Companion code-fix PR: (link once opened).

[aaronbrethorst]

Companion defensive/serialization fix opened as #462 — that stops one corrupt record from blanking the API (returns a proper error envelope + drops the unresolvable route with a WARN). This issue remains open for the root-cause bundle fix: why a stop carries an unresolvable service-date route collection in the first place.