Added WRED with affected Leaf/LC/FM model check#379
Open
Priyanka-Patil14 wants to merge 17 commits into
Open
Conversation
Author
|
WredCheck_APIC_Output_logs.txt Uploaded the test logs. |
Harinadh-Saladi
left a comment
Harinadh-Saladi
left a comment
Contributor
There was a problem hiding this comment.
Pls address the comments given and also Pls add the bug details in validations.md file. It's missing.
Pls execute the script on Fab3 and share PASS, FAIL and NA logs. Will review it.
Author
WRED_PASS:FAIL:NA_APIC_Logs.txt Please find the attached logs. Executed on fab3 for PASS, FAIL and NA scenario. |
lovkeshsharma702
left a comment
lovkeshsharma702
left a comment
There was a problem hiding this comment.
please work on all comments.
muthu-ku
reviewed
Apr 16, 2026
Harinadh-Saladi
left a comment
Harinadh-Saladi
left a comment
Contributor
There was a problem hiding this comment.
Pls address the comments. If there is any different understanding with me in the test results or technical aspects, will discuss with team and address after getting the confirmation.
lovkeshsharma702
left a comment
lovkeshsharma702
left a comment
There was a problem hiding this comment.
please sort the other comments and we are good.
lovkeshsharma702
approved these changes
Apr 21, 2026
Author
|
Attached the pytest and full script run logs as requested. |
… cscwh68103 invalid fabricpathep targets (datacenter#357) * specific testing for known failure conditions of cscwh68103 as to not catch valid scenarios
Added rogue ep/coop exception mac check for the CFD CSCwp64296
Priyanka-Patil14
force-pushed
the
bugfix/CSCwt50713
branch
from
0eee803 to
d299ca5
Compare
lovkeshsharma702
suggested changes
May 25, 2026
muthu-ku
reviewed
May 26, 2026
muthu-ku
reviewed
May 26, 2026
muthu-ku
reviewed
May 26, 2026
muthu-ku
reviewed
May 26, 2026
muthu-ku
reviewed
May 26, 2026
Author
|
Attaching the full run logs: CSCwt50713_Pytest_FullRun_Logs.txt |
lovkeshsharma702
approved these changes
May 26, 2026
lovkeshsharma702
left a comment
lovkeshsharma702
left a comment
There was a problem hiding this comment.
intigration testing completed. Py test also completed.
monrog2
requested changes
Jun 11, 2026
monrog2
left a comment
monrog2
left a comment
Collaborator
There was a problem hiding this comment.
We need to get this bug updated with fixed release versions:
CSCwt50713
regardless of its dup nature, as its a different set of conditions to trigger the same RCA underlying crash.
Also include a known fixed version in those details, we want the bug to be the source of truth on affected/fixed versions, not this script.
lovkeshsharma702
approved these changes
Jun 12, 2026
There was a problem hiding this comment.
integrated test passed .
[2026-06-12 10:10:45 UTC INFO root:81] (tac_bdsol_pod83) No errors.
[2026-06-12 10:10:47 UTC INFO root:182] (tme_sj_fab1) Completed. Checking errors...
[2026-06-12 10:10:47 UTC INFO root:81] (tme_sj_fab1) No errors.
[2026-06-12 10:10:47 UTC INFO root:182] (tme_sj_fab2) Completed. Checking errors...
[2026-06-12 10:10:47 UTC INFO root:81] (tme_sj_fab2) No errors.
[2026-06-12 10:10:49 UTC INFO root:182] (tac_rtp_pod1) Completed. Checking errors...
[2026-06-12 10:10:49 UTC INFO root:81] (tac_rtp_pod1) No errors.
[2026-06-12 10:10:58 UTC INFO root:182] (tac_rtp_calo_a) Completed. Checking errors...
[2026-06-12 10:10:58 UTC INFO root:81] (tac_rtp_calo_a) No errors.
[2026-06-12 10:11:03 UTC INFO root:182] (tac_bdsol_pod12) Completed. Checking errors...
[2026-06-12 10:11:03 UTC INFO root:81] (tac_bdsol_pod12) No errors.
[2026-06-12 10:11:18 UTC INFO root:182] (tac_bdsol_pod19) Completed. Checking errors...
[2026-06-12 10:11:18 UTC INFO root:81] (tac_bdsol_pod19) No errors.
[2026-06-12 10:12:11 UTC INFO root:182] (bu_qa_scale1) Completed. Checking errors...
[2026-06-12 10:12:11 UTC INFO root:81] (bu_qa_scale1) No errors.
[2026-06-12 10:12:12 UTC INFO root:182] (tac_bdsol_pod20) Completed. Checking errors...
[2026-06-12 10:12:12 UTC INFO root:83] (tac_bdsol_pod20) Found 1 errors.
[2026-06-12 10:12:43 UTC INFO root:182] (tac_rtp_calo_b) Completed. Checking errors...
[2026-06-12 10:12:43 UTC INFO root:83] (tac_rtp_calo_b) Found 2 errors.
[2026-06-12 10:12:43 UTC ERROR root:248] (MainThread) ####################################
[2026-06-12 10:12:43 UTC ERROR root:249] (MainThread) tac_rtp_pod8
[2026-06-12 10:12:43 UTC ERROR root:250] (MainThread) ####################################
[2026-06-12 10:12:43 UTC ERROR root:251] (MainThread) Message: Uncaught exception found
[2026-06-12 10:12:43 UTC ERROR root:253] (MainThread) Output: PermissionError(13, 'Permission denied')
[2026-06-12 10:12:43 UTC ERROR root:248] (MainThread) ####################################
[2026-06-12 10:12:43 UTC INFO root:257] (MainThread) Fabrics with errors - 6 / 22 fabrics
[2026-06-12 10:12:43 UTC WARNING root:263] (MainThread) Fabrics below seem to be unreachable, not counting as a script error.
[2026-06-12 10:12:43 UTC WARNING root:267] (MainThread) bu_trunk10
[2026-06-12 10:12:43 UTC WARNING root:267] (MainThread) bu_qa_scale6
[2026-06-12 10:12:43 UTC WARNING root:267] (MainThread) bu_ifav30
Uploading artifacts for failed job
00:02
Uploading artifacts...
/builds/aci-pre-upgrade-validation/aci-pre-upgrade-validation-script/tests/*.log: found 17 matching artifact files and directories
Uploading artifacts as "archive" to coordinator... 201 Created id=448 responseStatus=201 Created token=glcbt-64
ERROR: Job failed: exit code 1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new pre-upgrade validation check to detect fabric nodes at risk due to CSCwt50713, where WRED-enabled QoS combined with specific Leaf/LC/FM hardware models can cause N9504 spine crashes after upgrading to affected ACI releases.
Detection Logic
Three gates must all be true to trigger a FAIL:
Version Gate – Target version is in the affected range:
Feature Gate – WRED is enabled (
qosCong.algo = wred)Hardware Gate – Any of the following affected models are present:
Testing
tests/checks/wred_affected_model_check/