[frr]: Fix zebra NB RIB route lookup_next route_node lock under-run

[deepak-singhal0408]

Why I did it

Fixes #27788.

When the bgp container restarts (e.g. critical-process SIGKILL / container_autorestart), zebra aborts during the post-restart RIB sweep with an assertion failure (node->lock > 0), wedging vtysh and stalling bgpd.

Root cause is in zebra's northbound oper-state RIB route iterator. lib_vrf_zebra_ribs_rib_route_lookup_next() calls route_table_get_next() (which returns the next node already locked, seeding the get_next iteration) and then erroneously calls route_unlock_node() on it. On resume-after-yield, get_next -> route_next() unlocks its input node again, under-running the route_node lock to zero and tripping the assertion that aborts zebra. The yielding oper walk fires autonomously after restart because, on FRR >= 10.5.2, VRF enable notifies the bare VRF node (/frr-vrf:lib/vrf[name="Vrf1"]) and the resulting walk descends into the whole VRF subtree (including the ~12.9k-route RIB), so it yields and resumes through lookup_next.

The point-lookup sibling (lib_vrf_zebra_ribs_rib_route_lookup_entry) correctly keeps its unlock and is left untouched — this fix is intentionally minimal (delete the single erroneous unlock).

Work item tracking

• Microsoft ADO (number only): 38582781

How I did it

This is the SONiC downstream backport of the upstream FRRouting fix FRRouting/frr#22490 (FRR issue FRRouting/frr#22489).

That upstream PR carries two commits:

1. zebra: keep route_node lock in NB RIB route lookup_next — removes the erroneous route_unlock_node(rn) in lib_vrf_zebra_ribs_rib_route_lookup_next(). This is the crash fix (the node->lock > 0 abort).
2. zebra: skip link-local routes without leaking node lock in NB RIB walk — folds the IPv6 link-local filter in lib_vrf_zebra_ribs_rib_route_get_next() into the empty-node skip loop, fixing a lock leak + walk truncation (missing oper-data routes). This is a correctness fix that cannot crash.

Only commit (1) is backported here, as FRR patch 0116-zebra-keep-route_node-lock-in-NB-RIB-route-lookup_next.patch (registered in src/sonic-frr/patch/series). The patch is the verbatim upstream commit — its commit message and in-code comment are unchanged from upstream commit (1); only the diff hunk offset differs (SONiC pins FRR 10.5.4, so the same change lands at @@ -399 instead of upstream's @@ -466). It removes the erroneous route_unlock_node(rn) from zebra/zebra_nb_state.c with a clarifying comment that the node returned by route_table_get_next() must stay locked because it seeds the get_next iteration.

Why only one commit: SONiC carries FRR as a pinned tag plus a patch/series of downstream patches; per policy we backport only what SONiC needs now to fix the observed defect (issue 27788 — the crash), and drop the patch once the FRR pin bumps to a tag that already contains the upstream commit. Commit (2) does not address the crash (it is a non-crashing oper-data correctness fix), so it is intentionally not carried as a downstream patch; it will arrive automatically via the normal FRR version bump once upstream PR 22490 merges and SONiC re-pins to a newer frr tag. This keeps the downstream patch set minimal.

How to verify it

Reproduce on a t1-lag VS testbed with a populated Vrf1 RIB (both IPv4 and IPv6 converged, ~12.9k routes total), then SIGKILL zebra (docker exec bgp kill -9 $(pidof zebra)) so the kernel RIB is preserved and the restarted zebra reads the full RIB at startup.

• Before (unfixed): zebra aborts (signal 6 ... aborting); core generated; backtrace route_next -> _zlog_assert_failed -> abort via lib_vrf_zebra_ribs_rib_route_get_next, in thread nb_op_walk_continue scheduled from nb_op_yield(); vtysh wedged.
• After (fixed): zebra survives, no core, no assert, vtysh responsive. Matched A/B over identical DUT/trigger/RIB scale: unfixed crashed at trial 2; fixed survived 5/5.
• Integration: tests/autorestart/test_container_autorestart.py::test_containers_autorestart[<dut>-None-bgp] PASSES on the fixed image (all critical processes restart healthy; mandatory post-test core-dump check passes), whereas the unfixed image would core during the restart sweep and fail that check.

Which release branch to backport (provide reason below if selected)

• 202305
• 202311
• 202405
• 202411
• 202505
• 202511
• 202605

Reason: 202605 only. 202605 ships FRR 10.5.4 (same as master), where the crash self-fires during the post-restart RIB sweep — the same nightly autorestart signature this PR fixes on master. The buggy route_unlock_node(rn) line is byte-identical at the 202605 FRR pin, so the patch applies as-is.

Not backported to 202511 / 202505 / 202411. The autonomous self-fire requires the bare-VRF oper-notification walk introduced in FRR 10.5.2, which these pins predate (202511 = 10.4.1, 202505 = 10.3, 202411 = 10.0.1). On those branches the lock bug is present but only reachable via an explicit oper-state GET of the RIB — and no SONiC component issues such a GET (only vtysh can, and show ip route json does not drive the northbound oper path), so the crash does not manifest in SONiC there. Verified empirically for 202511 (recent nightly autorestart runs: zebra healthy, 0 cores / 0 rc=124). Per the minimal/need-driven backport policy, the fix is not carried where the defect is unreachable; it will still arrive on those pins automatically once the FRR version bumps to a tag containing upstream PR 22490.

Tested branch (Please provide the tested image version)

• master (sonic-vs, FRR 10.5.4 + patch 0116; bgp container image docker-fpm-frr 7d46bd9139a5)

Description for the changelog

zebra: keep route_node lock in NB RIB route lookup_next (fix lock under-run abort on restart)

Link to config_db schema for YANG module changes

N/A — no CONFIG_DB schema or YANG changes.

A picture of a cute animal (not mandatory but encouraged)

🦦

[Copilot]

Pull request overview

This PR fixes an FRR zebra crash triggered during northbound oper-state RIB iteration after a bgp container restart by correcting the route_node lock/unlock contract in lookup_next via a new downstream patch in the sonic-frr patch stack.

Changes:

• Append patch 0115-zebra-keep-route_node-lock-in-NB-RIB-route-lookup_next.patch to the src/sonic-frr/patch/series stack.
• Add a new FRR patch that removes an erroneous route_unlock_node(rn) in lib_vrf_zebra_ribs_rib_route_lookup_next() and documents why the node must remain locked.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
src/sonic-frr/patch/series	Registers the new FRR patch in the applied patch series.
src/sonic-frr/patch/0115-zebra-keep-route_node-lock-in-NB-RIB-route-lookup_next.patch	Removes the incorrect unlock in zebra’s NB RIB lookup_next and adds an explanatory comment.

[mssonicbld]

/azp run Azure.sonic-buildimage

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[tirupatihemanth]

@deepak-singhal0408 can you please help fix conflicts?

[mssonicbld]

/azp run Azure.sonic-buildimage

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[deepak-singhal0408]

@deepak-singhal0408 can you please help fix conflicts?

@tirupatihemanth fixed. could you help confirm if the issue

@deepak-singhal0408 can you please help fix conflicts?

@tirupatihemanth fixed. Could you or @yizhenzha confirm if this fixes the issue as well from your side.
cc: @dgsudharsan as well..

[yizhenzha]

@deepak-singhal0408 I applied this patch and tested. I confirmed this fixes the issue.
cc @tirupatihemanth @dgsudharsan

[mssonicbld]

/azp run Azure.sonic-buildimage

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

[mssonicbld]

/azp run Azure.sonic-buildimage

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[deepak-singhal0408]

@deepak-singhal0408 I applied this patch and tested. I confirmed this fixes the issue. cc @tirupatihemanth @dgsudharsan

Thanks @yizhenzha..

@dgsudharsan, @tirupatihemanth @yizhenzha could you help review/sign-off the PR as well. FYI, The fix has been upstreamed and merged there as well.

[deepak-singhal0408]

Hi @eddieruan-alibaba @lguohan @yxieca could you help merge this PR?
JFYI, the fix has been unstreamed as well.. thanks,

[eddieruan-alibaba]

@GaladrielZhao will review it on Alibaba's behave.

[GaladrielZhao]

Thanks, LGTM.

[mssonicbld]

The change is not in 202605 yet. @deepak-singhal0408, please manually create the cherry pick PR for branch 202605.
You can ping the release branch owner(github account: vaibhavhd) to approve your cherry pick PR.
If this change is already in 202605, please comment "already in 202605". Thanks!

_{---Powered by SONiC BuildBot}