Added pre-upgrade check for defect CSCwt69100 by Priyanka-Patil14 · Pull Request #385 · datacenter/ACI-Pre-Upgrade-Validation-Script

Priyanka-Patil14 · 2026-05-15T13:02:02Z

Summary:
-This PR adds a new validation check: Stale dbgacEpgSummaryTask Objects.

-The check detects stale dbgacEpgSummaryTask objects stuck in processing state that can cause policymgr to crash on all APICs during upgrade (CSCwt69100).

What Changed:

Added stale_epg_summary_task_check in aci-preupgrade-validation-script.py
Added validation documentation in docs/docs/validations.md
Added dedicated unit tests and test data under:
tests/checks/stale_epg_summary_task_check/

Check Behavior:

Returns `MANUAL' if target version is missing
Returns N/A if target version is not in affected range (<= 6.1(5e) or <= 6.2(1g))
Returns PASS if no dbgacEpgSummaryTask objects found in processing state
Returns PASS if objects found but startTs is within 24 hours
Returns FAIL_O if any dbgacEpgSummaryTask object has startTs older than 24 hours

Test Results:

Pytest Logs: Full Run:
CSCwt69100_Pytest_FullRun_Logs.txt
Pytest Logs: stale_epg_summary_task_check (8 test cases)
CSCwt69100_pytest.txt
Full run (tversion 6.1(5e) and 6.2(2a)):
CSCwt69100_FullRun_logs.txt
Script Run Logs (PASS / NA / FAIL):
CSCwt69100_PASS:FAIL:NA_logs.txt

Note: Added FAIL log from CU as lab repro is not feasible.

Harinadh-Saladi

I could see you have added only full script run logs. Can you attach your function alone logs. Please add the logs for PASS,FAIL and NA scenarios.

As Lovkesh has confirmed that attaching FAIL logs is not required (Issue is not recreatable, since repro factor is very less), pls attach cu log as a fail evidence.

Harinadh-Saladi · 2026-05-18T10:39:28Z

+Due to [CSCwt69100][68], a stale `dbgacEpgSummaryTask` object stuck in `processing` state with empty content can cause the policymgr process to crash on all APICs during an upgrade or process restart.

-Impact:
+Affected versions: version <= 6.1(5e) or version <= 6.2(1g).


Pls remove the affected versions, no need to mention here

Harinadh-Saladi · 2026-05-18T10:40:59Z

+Affected versions: version <= 6.1(5e) or version <= 6.2(1g).

-When upgrading Apic from versions prior to 6.0(4c) to versions 6.0(4c) or later, if there is a misconfiguration in the inband management policies (mgmtRsInBStNode) with invalid values, the re-processing triggered by [CSCwh80837][67] will expose the underlying [CSCwd40071][68] defect. This results in continuous policyelem core dumps and switch reboot if Switch are running impacted version of [CSCwd40071][68].
+The check queries for `dbgacEpgSummaryTask` objects with `operSt="processing"` and `startTs` older than 24 hours. Such objects are considered stale and unexpected. If found, delete them before proceeding with the upgrade to prevent policymgr from crashing on restart.


No need to mention about the check and what it does. Just provide appropriate recommended action.

Harinadh-Saladi · 2026-05-19T05:43:41Z

-The [CSCwd40071][68] defect affects versions 5.2(5c) and later with a fix available in 6.0(1g). However, the issue will only be triggered during Apic upgrades crossing 6.0(4c) due to [CSCwh80837][67].

 [0]: https://github.com/datacenter/ACI-Pre-Upgrade-Validation-Script
+[68]: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwt69100


Pls remove this line and add at the end and also correct the number from 68 to 67

The reference 68, CSCwt69100 near 0, was already removed. It is now placed at the end as 70 (after 69). The number is 70 and not 67 because after rebasing to v4.2.0-dev, references 67, 68, 69 are already used by other checks in the base branch.

Harinadh-Saladi · 2026-05-27T09:50:45Z

 The svccoreCtrlr and svccoreNode objects represent core files related to Apic and Leaf/Spines process respectively.

-Due to [CSCws84232][67], the APIC GUI may become unresponsive after login, with dashboards stuck in a continuous “Loading…”state.
+Due to [CSCws84232][67], the APIC GUI may become unresponsive after login, with dashboards stuck in a continuous "Loading…"state.


Pls revert this if it's done by mistake.

Harinadh-Saladi · 2026-05-27T09:52:57Z

+

 [0]: https://github.com/datacenter/ACI-Pre-Upgrade-Validation-Script
+[70]: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwt69100


Pls remove it from here as it needs to be added at the end after [69]

Harinadh-Saladi · 2026-05-27T12:09:12Z

+
+
+@pytest.mark.parametrize(
+    "tversion, icurl_outputs, expected_result, expected_data",


Pls add the test case for tversion missing.

Added the test case.

Harinadh-Saladi · 2026-05-27T12:14:30Z

+        # Case 1: Target version 6.2(2a) is beyond both affected ranges (6.1(5e) and 6.2(1g)).
+        # The target binary has the fix so version gate fails. Expected: NA without any API calls.
+        (
+            "6.2(2a)",


Pls change the version 6.2(2a), it doesn't exist. Update with existing CCO version.

Updated to 6.2(1h)

Harinadh-Saladi · 2026-05-27T12:35:34Z

+            ],
+        ),
+    ],
+)


Pls add the test cases for the following
Stale exist for exactly 24hrs
Stale exists for more than 24hrs(25hrs) and less than 24hrs(like 23hrs 59mins) combo

Added two new test cases: one for exactly 24h (PASS) and one combo of 25h + 23h59m (FAIL_O with only the 25h task reported).

Harinadh-Saladi · 2026-05-27T12:55:31Z

+        except ValueError:
+            continue
+        if task_dt < threshold:
+            data.append([dn, start_ts])


I don't see node_id in the output. Pls add it to know on which node issue is encountered.

node_id is not available in the object's attributes or DN. The DN is already unique enough to identify and delete the specific object.

Harinadh-Saladi

Address the comments

Harinadh-Saladi · 2026-06-03T02:50:09Z

+
+Due to [CSCwt69100][70], a stale `dbgacEpgSummaryTask` object stuck in `processing` state with empty content can cause the policymgr process to crash on all APICs during an upgrade or process restart.
+
+Delete any stale `dbgacEpgSummaryTask` objects before proceeding to prevent policymgr from crashing on restart.


Pls update the recommended action with 'upgrade'.

Harinadh-Saladi · 2026-06-03T04:01:28Z

+    result = PASS
+    headers = ["DN", "Start Time"]
+    data = []
+    recommended_action = "Delete the listed stale dbgacEpgSummaryTask objects to prevent policymgr crash."


Pls update recommended action appropriately.

Harinadh-Saladi · 2026-06-03T06:28:01Z

+
+Due to [CSCwt69100][70], a stale `dbgacEpgSummaryTask` object stuck in `processing` state with empty content can cause the policymgr process to crash on all APICs during an upgrade or process restart.
+
+Delete any stale `dbgacEpgSummaryTask` objects before proceeding with the upgrade to prevent policymgr from crashing on restart.


Pls remove restart here. We will see issue either through upgrade or process restart. Pls don't include both together.

Updated the recommended action, removed "on restart" and kept only "upgrade"

…s.md

Harinadh-Saladi · 2026-06-03T07:33:28Z

Pls add pytest logs for your check as you added additional test cases.

Priyanka-Patil14 · 2026-06-03T09:23:48Z

Pls add pytest logs for your check as you added additional test cases.

Attached the pytest logs for the check.
CSCwt69100_pytest.txt

lovkeshsharma702 · 2026-06-03T09:57:10Z

+    result = PASS
+    headers = ["DN", "Start Time"]
+    data = []
+    recommended_action = "Delete the listed stale dbgacEpgSummaryTask objects before the upgrade to prevent policymgr from crashing."


is user allowed to perform the Deletion. does delete need root access? if yes. please suggest cu to follow workaround. and share steps in workaround in the bug, .

lovkeshsharma702 · 2026-06-03T10:00:13Z

        rogue_ep_coop_exception_mac_check,
        n9k_c9408_model_lem_count_check,
        inband_management_policy_misconfig_check,
+        stale_epg_summary_task_check,


name should be stale_dbgacEpgSummaryTask_check

lovkeshsharma702 · 2026-06-03T10:00:35Z



+@check_wrapper(check_title="Stale dbgacEpgSummaryTask Objects")
+def stale_epg_summary_task_check(tversion, **kwargs):


chaneg the name to stale_dbgacEpgSummaryTask_check

Priyanka-Patil14 changed the base branch from v4.1.0-dev to v4.2.0-dev May 25, 2026 04:53

Added pre-upgrade check for defect CSCwt69100

a66b043

Priyanka-Patil14 force-pushed the CSCwt69100-fix-clean branch from 5ba0fb4 to a66b043 Compare May 25, 2026 08:30

Harinadh-Saladi reviewed May 27, 2026

View reviewed changes

Addressed PR comments

1b64143

Harinadh-Saladi reviewed Jun 3, 2026

View reviewed changes

Updated aci-preupgrade-validation-script.py & validations.md

300a70f

Harinadh-Saladi reviewed Jun 3, 2026

View reviewed changes

Updated both aci-preupgrade-validation-script.py docs/docs/validation…

c7b7ff9

…s.md

lovkeshsharma702 reviewed Jun 3, 2026

View reviewed changes

lovkeshsharma702 suggested changes Jun 3, 2026

View reviewed changes



		[0]: https://github.com/datacenter/ACI-Pre-Upgrade-Validation-Script
		[70]: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwt69100



		@pytest.mark.parametrize(
		"tversion, icurl_outputs, expected_result, expected_data",


		Due to [CSCwt69100][70], a stale `dbgacEpgSummaryTask` object stuck in `processing` state with empty content can cause the policymgr process to crash on all APICs during an upgrade or process restart.

		Delete any stale `dbgacEpgSummaryTask` objects before proceeding to prevent policymgr from crashing on restart.



		@check_wrapper(check_title="Stale dbgacEpgSummaryTask Objects")
		def stale_epg_summary_task_check(tversion, **kwargs):

Conversation

Priyanka-Patil14 commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Harinadh-Saladi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Harinadh-Saladi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Harinadh-Saladi commented Jun 3, 2026

Uh oh!

Priyanka-Patil14 commented Jun 3, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Priyanka-Patil14 commented May 15, 2026 •

edited

Loading