Skip to content

gnoi-shutdown: use per-thread DB connections to avoid crossed DPU reads#408

Open
gpunathilell wants to merge 1 commit into
sonic-net:masterfrom
gpunathilell:gnoi-shutdown-per-thread-db-connections
Open

gnoi-shutdown: use per-thread DB connections to avoid crossed DPU reads#408
gpunathilell wants to merge 1 commit into
sonic-net:masterfrom
gpunathilell:gnoi-shutdown-per-thread-db-connections

Conversation

@gpunathilell

Copy link
Copy Markdown
Contributor

What I did

gnoi-shutdown-daemon spawns one worker thread per DPU on shutdown, but all threads shared a single CONFIG_DB/STATE_DB redis connection created once in main(). A redis DBConnector is a single, non-thread-safe socket, so when multiple DPUs shut down in parallel their hget/Table reads interleave on the wire and replies get matched to the wrong request — producing crossed IP/port values between DPUs.

This gave gnoi_client a bad target like 169.254.200.1:169.254.200.2 (dpu0's IP as host, dpu1's IP in the port slot), causing:

ERR gnoi-shutdown-daemon: DPU1: Reboot command failed (rc=2, target=169.254.200.1:169.254.200.2):
panic: rpc error: ... dial tcp: lookup tcp/169.254.200.2: unknown port

Fix

Give each per-DPU worker thread its own CONFIG_DB/STATE_DB connection so reads can never cross:

  • handle_and_cleanup() opens fresh db_connect("CONFIG_DB") / db_connect("STATE_DB") per thread and passes them into _handle_transition().
  • _handle_transition() / _wait_for_gnoi_halt_in_progress() take optional per-thread connections, falling back to the shared ones for direct/unit-test callers (backward compatible).

Testing

  • py_compile clean.
  • Smoke-tested both paths: per-thread connections are the ones actually used (correct 169.254.200.2:50052), and fallback-to-shared preserves existing unit-test behavior.

Signed-off-by: gpunathilell <gpunathilell@nvidia.com>
@mssonicbld

Copy link
Copy Markdown

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants