Skip to content

Fix route_check.py: intersection retry and snapshot reorder#4650

Merged
deepak-singhal0408 merged 1 commit into
sonic-net:masterfrom
mike-dubrovsky:route-check-correctness
Jul 14, 2026
Merged

Fix route_check.py: intersection retry and snapshot reorder#4650
deepak-singhal0408 merged 1 commit into
sonic-net:masterfrom
mike-dubrovsky:route-check-correctness

Conversation

@mike-dubrovsky

Copy link
Copy Markdown
Contributor

Summary

Split from #4576 per @deepak-singhal0408's reviewBucket B (correctness, land after Bucket A).

Depends on #4649 (dict-format bugfix). Includes two improvements:

  1. Intersection-based retry in check_frr_pending_routes(): only routes stuck in every poll window are mitigated, preventing premature mitigation of converging routes.
  2. Snapshot reorder: move ASIC/APPL DB snapshot to after FRR polling completes so rt_appl_miss / rt_asic_miss are temporally aligned with FRR results when the mitigation cross-check fires.

Keeps the ijson JSON parser (no text-streaming change).

Test plan

  • tests/check_frr_pending_routes_test.py — intersection accumulator cases
  • CI unit tests pass
  • Routes that converge between retry iterations are not mitigated
  • Routes finishing ASIC programming during the FRR retry window do not suppress mitigation

@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@mike-dubrovsky mike-dubrovsky force-pushed the route-check-correctness branch from 7905d88 to 3597574 Compare June 29, 2026 19:22
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@mssonicbld

Copy link
Copy Markdown
Collaborator

Hi, there are workflow run(s) waiting for approval, you may be first-time contributor. I will notify maintainers to help approve once PR is approved. Thanks!

---Powered by SONiC BuildBot

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates scripts/route_check.py to improve correctness of suppress-fib-pending mitigation decisions by (1) using an intersection-based retry strategy in check_frr_pending_routes() and (2) reordering FRR polling to complete before taking ASIC/APPL DB snapshots, so cross-check conditions are evaluated against temporally aligned observations. It also updates existing unit tests and adds a new test module to validate the intersection retry behavior.

Changes:

  • Implement intersection-based accumulation across FRR polling retries in check_frr_pending_routes().
  • Reorder check_routes_for_namespace() so FRR polling completes before ASIC/APPL DB route snapshots are taken.
  • Standardize “missed FRR route” entries to dicts ({'prefix', 'protocol'}) and update/add tests accordingly.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
scripts/route_check.py Adds intersection-based retry logic for FRR polling and reorders FRR polling ahead of ASIC/APPL snapshots; updates missing-route representation to dicts.
tests/check_frr_pending_routes_test.py New unit tests validating the intersection accumulator behavior for missed/failed FRR routes.
tests/fetch_routes_chunk_test.py Updates chunked JSON parsing tests to assert on dict-form missing routes (prefix/protocol).
tests/route_check_test.py Updates mocked FRR polling return shape to dict-form missed routes.
tests/route_check_test_data.py Updates expected missed_FRR_routes fixtures to dict-form entries.

Comment thread scripts/route_check.py Outdated
Comment thread scripts/route_check.py
@mike-dubrovsky mike-dubrovsky force-pushed the route-check-correctness branch from 29bb6fd to e86ed34 Compare July 8, 2026 05:42
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@mike-dubrovsky mike-dubrovsky force-pushed the route-check-correctness branch from e86ed34 to e0e4dec Compare July 8, 2026 05:56
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

Comment thread scripts/route_check.py
@mike-dubrovsky mike-dubrovsky force-pushed the route-check-correctness branch from e0e4dec to ef3572f Compare July 8, 2026 06:43
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@mike-dubrovsky mike-dubrovsky force-pushed the route-check-correctness branch from ef3572f to e4f70d1 Compare July 8, 2026 06:44
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

Replace the overwrite-on-each-iteration retry loop with an intersection
accumulator so that only routes stuck in every poll window are mitigated,
avoiding premature mitigation of routes that are still converging.

Add unit tests for the intersection logic with dict-format missing routes.

Signed-off-by: mike-dubrovsky <mdubrovs@cisco.com>
@mike-dubrovsky mike-dubrovsky force-pushed the route-check-correctness branch from e4f70d1 to 1076912 Compare July 8, 2026 06:47
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@deepak-singhal0408 deepak-singhal0408 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All review comments addressed. Changes LGTM.

@deepak-singhal0408 deepak-singhal0408 merged commit 85bba56 into sonic-net:master Jul 14, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants