[clear] Add 'sonic-clear pfcwdstats' to clear PFC watchdog counters by pinky-nexthop · Pull Request #4634 · sonic-net/sonic-utilities

pinky-nexthop · 2026-06-22T05:08:00Z

What I did

There is currently no CLI to clear PFC watchdog statistics. Clearing them is needed when troubleshooting PFC storms — stale counts from a prior event make it hard to tell whether a storm is currently ongoing. show pfcwd stats displays the counters, but there is no corresponding clear command. This PR adds sonic-clear pfcwdstats.

Addresses the PFC-watchdog-counter-clear item in #4595.

How I did it

scripts/pfcwdstat: new tool that zeroes the PFC_WD_QUEUE_STATS_* fields in COUNTERS_DB, following the standard counter-clear pattern (pfcstat -c, queuestat -c, …). It enumerates queues via COUNTERS_QUEUE_NAME_MAP — the same source show pfcwd stats uses — so stale stats from queues that are no longer actively monitored (STATUS=N/A) are also cleared.
Queues with an active storm (PFC_WD_STATUS == 'stormed') are skipped, so the in-flight DEADLOCK_DETECTED/DEADLOCK_RESTORED pair is not split (which would otherwise leave RESTORED > DETECTED once the storm ends).
clear/main.py: adds sonic-clear pfcwdstats, invoking pfcwdstat -c.
setup.py: installs scripts/pfcwdstat.
Unit tests for the clear command and the pfcwdstat clear logic (including the stale-stats and active-storm cases).

How to verify it

pytest tests/pfcwd_test.py::TestPfcwdstatClear tests/clear_test.py -k pfcwd

Manually: show pfcwd stats to see counts, sonic-clear pfcwdstats to clear them; an actively storming queue is left untouched (and reported as skipped) until the storm ends.

Previous command output (if the output of a command-line utility has changed)

admin@sonic:~$ show pfcwd stats
        QUEUE       STATUS    STORM DETECTED/RESTORED    TX OK/DROP    RX OK/DROP    TX LAST OK/DROP    RX LAST OK/DROP
-------------  -----------  -------------------------  ------------  ------------  -----------------  -----------------
Ethernet192:3  operational                        1/1           0/0           0/0                0/0                0/0

New command output (if the output of a command-line utility has changed)

admin@sonic:~$ sonic-clear pfcwdstats
PFC Watchdog counters cleared.

admin@sonic:~$ show pfcwd stats
  QUEUE    STATUS    STORM DETECTED/RESTORED    TX OK/DROP    RX OK/DROP    TX LAST OK/DROP    RX LAST OK/DROP
-------  --------  -------------------------  ------------  ------------  -----------------  -----------------

Which release branch to backport (provide reason below if selected)

There is currently no CLI to clear PFC watchdog statistics. Clearing them is needed when troubleshooting PFC storms, where stale counts from a prior event make it hard to tell whether a storm is ongoing. This adds: - scripts/pfcwdstat: a new tool that zeroes the PFC_WD_QUEUE_STATS_* fields in COUNTERS_DB, following the standard counter-clear pattern (pfcstat -c, queuestat -c, ...). It enumerates queues via COUNTERS_QUEUE_NAME_MAP -- the same source 'show pfcwd stats' uses -- so stale stats from queues that are no longer actively monitored (STATUS=N/A) are also cleared. - Queues with an active storm (PFC_WD_STATUS == 'stormed') are skipped, so the in-flight DEADLOCK_DETECTED/RESTORED pair is not split (which would otherwise leave RESTORED > DETECTED once the storm ends). - clear/main.py: 'sonic-clear pfcwdstats', invoking 'pfcwdstat -c'. - setup.py: install scripts/pfcwdstat. - tests for the clear command and the pfcwdstat clear logic. Signed-off-by: Pinky Agrawal <pinky@nexthop.ai>

mssonicbld · 2026-06-22T05:08:08Z

/azp run

azure-pipelines · 2026-06-22T05:08:19Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-06-22T11:18:10Z

Hi, there are workflow run(s) waiting for approval, you may be first-time contributor. I will notify maintainers to help approve once PR is approved. Thanks!

_{---Powered by SONiC BuildBot}

mssonicbld · 2026-07-01T11:17:03Z

This PR has backport request for branch(es): 202605.
Added label(s) for branch(es) 202605.

_{---Powered by SONiC BuildBot}

saksarav-nokia · 2026-07-01T21:38:57Z

+
+    Queues with an active storm (PFC_WD_STATUS == 'stormed') are skipped.
+    """
+    db = SonicV2Connector()


sonicV2connector() handles only global namespace and does not work in multi asic. Please add support for multi-asic also

saksarav-nokia · 2026-07-01T22:03:57Z

+    db = SonicV2Connector()
+    db.connect(db.COUNTERS_DB)
+
+    queue_name_map = db.get_all(db.COUNTERS_DB, 'COUNTERS_QUEUE_NAME_MAP')


Usually all the clear commands take the snapshot of the stats from COUNTER_DB and saves it in /tmp/cache . Then show commands shows the diff between stats COUNTER_DB and /tmp/cache. Why are you clearing it from COUNTER_DB. Is this the requirement?

saksarav-nokia · 2026-07-01T22:06:20Z

+
+@cli.command()
+def pfcwdstats():
+    """Clear PFC Watchdog stats counters"""


Please update doc/Command-Reference.md

saksarav-nokia · 2026-07-01T22:07:01Z

+            skipped.append(queue_name)
+            continue
+        for field in PFC_WD_STATS_FIELDS:
+            db.set(db.COUNTERS_DB, counter_key, field, '0')


Even if you have to clear from COUNTERS_DB, can't you use hmset and clear all stats at once instead of clearing 1 by 1

vmittal-msft

Thanks for the PR! I reviewed the change end-to-end against the current sources. It works on a single-ASIC box, but two items I'd treat as blocking:

No multi-ASIC / chassis support — SonicV2Connector() only touches the global DB, so nothing is cleared on multi-ASIC while show pfcwd stats still shows counts.
Destructive COUNTERS_DB mutation — every sibling clear command uses the non-destructive UserCache snapshot+diff pattern instead of writing into the shared counters DB.

Plus a doc/Command-Reference.md update. Details inline — these corroborate and extend @saksarav-nokia's review. Nice touch skipping actively-stormed queues to keep the DEADLOCK_DETECTED/RESTORED pair intact.

vmittal-msft · 2026-07-08T21:28:47Z

+
+    Queues with an active storm (PFC_WD_STATUS == 'stormed') are skipped.
+    """
+    db = SonicV2Connector()


Multi-ASIC: this won't clear anything on a multi-ASIC/VOQ chassis (+1 to @saksarav-nokia, confirming it's a functional gap, not style).

SonicV2Connector() with no namespace binds only to the global DB. But the reader show pfcwd stats (pfcwd/main.py) walks each ASIC via the @multi_asic_util.run_on_multi_asic decorator, and the sibling clear tools (pfcstat, queuestat) and even pg-drop -c clear all iterate multi_asic.get_namespace_list().

So on a multi-ASIC box, COUNTERS_QUEUE_NAME_MAP in the global DB is empty → this prints No PFC Watchdog stats found. and clears nothing, while show pfcwd stats still shows per-ASIC counts. Please wrap clear_stats() in a namespace loop (MultiAsic / run_on_multi_asic) or add -n/--namespace like pg-drop.

vmittal-msft · 2026-07-08T21:28:47Z

+            skipped.append(queue_name)
+            continue
+        for field in PFC_WD_STATS_FIELDS:
+            db.set(db.COUNTERS_DB, counter_key, field, '0')


Directly zeroing COUNTERS_DB diverges from the standard clear pattern (this is the crux of @saksarav-nokia's question).

Every other stats-clear in this repo is non-destructive: pfcstat -c, queuestat -c, portstat -c, and pg-drop -c clear snapshot COUNTERS_DB into UserCache (/tmp/...) and the show command renders current - cached. They never mutate COUNTERS_DB.

Writing '0' straight into the shared COUNTERS_DB is a different contract: any other consumer of these counters (telemetry / gNMI streaming, rate calculators) will observe a reset / negative delta, and the clear is global + irreversible rather than per-user. It only works here because you skip stormed queues (no active producer to overwrite the zeros).

Preferred: the cache+diff approach (which also means teaching show pfcwd stats to subtract the baseline). If a hard reset is genuinely required, please call that out and justify it in the PR description.

Minor (also @saksarav-nokia): if you keep the DB write, one hmset/pipeline per queue avoids the per-field round-trips.

vmittal-msft · 2026-07-08T21:28:47Z

+"""
+
+import argparse
+import sys


Nit: sys is imported but unused (F401). Also clear_stats (L27), main (L69), and the __main__ guard (L82) are separated by a single blank line — PEP8 wants two (E302/E305).

Heads-up: the repo's pre-commit runs flake8==4.0.1 in --diff mode, which does not flag these on a brand-new file (I reproduced it locally — that's why the Static Analysis check is green), so they'll silently slip through unless cleaned up by hand.

vmittal-msft · 2026-07-08T21:28:47Z


+
+@cli.command()
+def pfcwdstats():


+1 to @saksarav-nokia on docs. Please add sonic-clear pfcwdstats to doc/Command-Reference.md — there's already a show pfcwd stats section and a NOTE: PFC counters ... sonic-clear pfccounters blurb, so a parallel note under the PFC watchdog section keeps the new command discoverable.

vmittal-msft · 2026-07-08T21:28:47Z

        importlib.reload(pfcwd.main)
+
+
+class TestPfcwdstatClear(object):


These tests fully mock SonicV2Connector, so they pass regardless of the multi-ASIC gap flagged in scripts/pfcwdstat — a single mocked DB hides the fact that only the global namespace is cleared.

Once namespace handling is added, please add a multi-ASIC case (e.g. UTILITIES_UNIT_TESTING_TOPOLOGY=multi_asic with per-namespace mock DBs) asserting each namespace's counters are cleared. Tests for the -c arg path in main() and for the Skipped queue(s) ... / No PFC Watchdog stats found. output would also lock in the behavior.

mssonicbld added the Request for 202605 Branch label Jul 1, 2026

vmittal-msft requested review from kperumalbfn, saksarav-nokia, vmittal-msft and ysmanman July 1, 2026 17:27

saksarav-nokia reviewed Jul 1, 2026

View reviewed changes

vmittal-msft reviewed Jul 8, 2026

View reviewed changes

		importlib.reload(pfcwd.main)


		class TestPfcwdstatClear(object):



		@cli.command()
		def pfcwdstats():

Uh oh!

Conversation

pinky-nexthop commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What I did

How I did it

How to verify it

Previous command output (if the output of a command-line utility has changed)

New command output (if the output of a command-line utility has changed)

Which release branch to backport (provide reason below if selected)

Uh oh!

mssonicbld commented Jun 22, 2026

Uh oh!

azure-pipelines Bot commented Jun 22, 2026

Uh oh!

mssonicbld commented Jun 22, 2026

Uh oh!

mssonicbld commented Jul 1, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vmittal-msft left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pinky-nexthop commented Jun 22, 2026 •

edited

Loading