Skip to content

route_check: fix 'string indices must be integers' crash in FRR route mitigation#4657

Closed
cshivashgit wants to merge 1 commit into
sonic-net:masterfrom
cshivashgit:fix-route-check-mitigate-string-indices
Closed

route_check: fix 'string indices must be integers' crash in FRR route mitigation#4657
cshivashgit wants to merge 1 commit into
sonic-net:masterfrom
cshivashgit:fix-route-check-mitigate-string-indices

Conversation

@cshivashgit

Copy link
Copy Markdown
Contributor

What I did

ixed a route_check.py crash — TypeError: string indices must be integers, not 'str' — that occurs during FRR route mitigation when SUPPRESS_FIB_PENDING is enabled.

The ijson streaming refactor of fetch_routes() changed the data shape it produces: process_route_entry() started appending bare prefix strings to the missed/failed route lists. However, the downstream consumer mitigate_installed_not_offloaded_frr_routes() (and its unit test test_mitigate_routes) still expects dicts with prefix and protocol keys. When there are installed-but-not-offloaded FRR routes and FIB-suppress is enabled, mitigate indexes entry['prefix'] on a str, raising TypeError: string indices must be integers. This makes check_routes() return -1, so route_check.py reports a failure even though routing is healthy.

This is not caught by CI because the unit tests mock check_frr_pending_routes to return strings and patch mitigate_installed_not_offloaded_frr_routes to a no-op, so the real string-vs-dict mismatch never executes.

Crash:

Traceback (most recent call last):
  File "/usr/local/bin/route_check.py", line ..., in check_routes_for_namespace
    mitigate_installed_not_offloaded_frr_routes(namespace, rt_frr_miss, rt_appl)
  File "/usr/local/bin/route_check.py", line ..., in mitigate_installed_not_offloaded_frr_routes
    for entry in [entry for entry in missed_frr_rt if entry['prefix'] in rt_appl]:
TypeError: string indices must be integers, not 'str'

How I did it

  • scripts/route_check.py
    • fetch_routes() / process_route_entry(): collect {'prefix': prefix, 'protocol': route_entry.get('protocol')} dicts instead of bare prefix strings, so mitigate has the protocol it needs for the FieldValuePairs notification it sends to fpmsyncd.
    • check_routes_for_namespace(): extract the prefix when populating results["missed_FRR_routes"] / results["failed_FRR_routes"] and the failed-routes log message, so the reported/result format remains a list of prefix strings (no change to external output or existing test data).
    • mitigate_installed_not_offloaded_frr_routes() is unchanged — it now receives the dict shape it always expected.
  • tests/route_check_test.py
    • Updated the mock_fetch_routes helper to return the same {'prefix', 'protocol'} dict shape as the real fetch_routes, keeping the mock consistent with production behavior.

How to verify it

  1. Run the unit tests:

    python3 -m pytest tests/route_check_test.py
    

    All cases pass, including test_mitigate_routes (which exercises the dict contract directly).

  2. On a device with SUPPRESS_FIB_PENDING enabled, create an installed-but-not-offloaded FRR route condition (e.g. a selected BGP route present in APPL_DB whose offload flag has not been set in zebra) and run:

    route_check.py
    

    Before the fix this aborts with TypeError: string indices must be integers and a non-zero return code; after the fix it mitigates the route (sends the SWSS_RC_SUCCESS response) and returns success.

Previous command output (if the output of a command-line utility has changed)

New command output (if the output of a command-line utility has changed)

Backport: this fix also needs to be ported to the 202605

… mitigation

fetch_routes()/process_route_entry() collected bare prefix strings, but
mitigate_installed_not_offloaded_frr_routes() and its unit test
(test_mitigate_routes) expect dicts with 'prefix' and 'protocol' keys.
When SUPPRESS_FIB_PENDING is enabled and FRR has installed-not-offloaded
routes, mitigate indexed entry['prefix'] on a str -> TypeError
'string indices must be integers', causing route_check.py to return -1.

Collect {'prefix', 'protocol'} dicts in process_route_entry so mitigate
gets the protocol it needs for the FieldValuePairs notification, and
extract prefixes when populating missed_FRR_routes/failed_FRR_routes so
the reported/result format stays unchanged. Update the unit-test mock to
return the same dict shape as the real fetch_routes.

Signed-off-by: Shivashankar CR <shivashankar.c.r@gmail.com>
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@cshivashgit

Copy link
Copy Markdown
Contributor Author

Closing this PR as changes are already merged as part of #4649

@cshivashgit cshivashgit closed this Jul 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants