Skip to content

Fix #813: continue create pipeline on config-save HTTP 500#819

Merged
mikewiebe merged 2 commits into
netascode:developfrom
dacasti2mx:fix/813-configsave-500-nonfatal
Jul 9, 2026
Merged

Fix #813: continue create pipeline on config-save HTTP 500#819
mikewiebe merged 2 commits into
netascode:developfrom
dacasti2mx:fix/813-configsave-500-nonfatal

Conversation

@dacasti2mx

Copy link
Copy Markdown
Contributor

Related Issue(s)

Fixes #813

Related Collection Role

  • cisco.nac_dc_vxlan.validate
  • cisco.nac_dc_vxlan.dtc.create
  • cisco.nac_dc_vxlan.dtc.deploy
  • cisco.nac_dc_vxlan.dtc.remove
  • other

Related Data Model Element

  • vxlan.topology

Proposed Changes

On a pre-provisioned fabric the switches can still be reloading or
unreachable when the create pipeline runs the intermediate config-save.
NDFC then returns HTTP 500. 0.7.2 logged this and continued; 0.8.0 aborts
at inventory_config_save, so the rest of the fabric intent is never
applied. This is a regression.

Root cause: pipeline_base._config_save already meant to treat a 500 as
non-fatal, but the check could never fire. fabric_deploy_manager discarded
the NDFC response on a failed config-save and returned failed=True with no
msg, so the RETURN_CODE was lost before the caller could look at it.

Fix:

  • fabric_config_save() returns the NDFC response.
  • manage_fabrics() propagates it into results['msg'] when config-save
    fails, so the RETURN_CODE survives.
  • _config_save() treats RETURN_CODE == 500 as non-fatal and emits a clear
    warning (full detail under -v). Any other failure — 400, auth, or a
    malformed/missing response — still aborts the run. This is stricter and
    safer than the old 0.7.2 rescue, which swallowed every error.

Only the create pipeline registers _config_save (create_resources.yml);
remove_resources.yml does not, and the deploy role uses operation: all,
so deploy and remove behavior are unchanged.

Test Notes

The collection has no unit-test harness today, and CI runs only build,
sanity, and lint — so a proper unit test needs new infrastructure. To avoid
blocking the 0.8.1 fix on that project-structure decision, tests are
deferred and tracked in a follow-up: #818.

Verified locally with a temporary mock harness (not committed) that drives
the real _config_save:

  • HTTP 500 -> run continues, warning emitted
  • HTTP 400 -> run aborts (fatal)
  • missing or malformed msg -> run aborts (fatal)
  • HTTP 200 -> success

Deferred unit-test matrix (in the follow-up):

  • 500 -> manager returns failed=True with the response intact
  • 500 -> _config_save returns non-fatal
  • 400 -> fatal
  • missing/malformed msg -> fatal
  • 200 -> ok

Cisco Nexus Dashboard Version

4.1 (per issue #813)

Checklist

  • Latest commit is rebased from develop with merge conflicts resolved
  • New or updates to documentation has been made accordingly
  • Assigned the proper reviewers

@dacasti2mx
dacasti2mx requested a review from a team as a code owner June 23, 2026 07:18
@dacasti2mx dacasti2mx self-assigned this Jun 23, 2026
@dacasti2mx dacasti2mx added bug Something isn't working wip work-in-progress 0.8.1 Release 0.8.1 labels Jun 23, 2026
@dacasti2mx

Copy link
Copy Markdown
Contributor Author

Update: the 500 bypass is now gated on the data model.

Before, this PR treated any config-save HTTP 500 as non-fatal — so a real 500 on a healthy fabric would also be swallowed.

The latest commit only tolerates the 500 when the data model actually has pre-provisioned switches (poap.preprovision) — i.e. the case where NDFC returns 500 because the switches aren't reachable yet. A 500 on a fabric without pre-provisioned switches stays fatal (as do 400 / auth / malformed).

Verified live against a pre-provisioned fabric: config-save returns 500, the run continues, and create completes 21/21. The new check was also validated deterministically (preprov → tolerate, non-preprov / empty → fatal).

When a fabric is pre-provisioned, its switches may still be reloading or
not yet reachable. In that state NDFC returns HTTP 500 to the intermediate
config-save (recalculate). In 0.7.2 the run logged this and kept going; in
0.8.0 it aborts at the inventory_config_save step, so the rest of the
fabric intent is never applied.

The "treat 500 as non-fatal" logic in pipeline_base._config_save never ran,
because the response that carries the 500 was thrown away earlier:
fabric_deploy_manager set results['failed']=True but no 'msg', so the
RETURN_CODE never reached the check.

Changes:
- fabric_config_save() now returns the NDFC response.
- manage_fabrics() puts that response in results['msg'] when config-save
  fails, so callers can read the RETURN_CODE.
- _config_save() treats RETURN_CODE 500 as non-fatal and logs a clear
  warning. Other failures (400, auth, malformed responses) still abort.

Unit tests are deferred to a follow-up: the collection has no unit-test
harness today and CI only runs build, sanity, and lint. Verified locally
with a temporary mock harness (not committed): 500 -> continues with a
warning, 400 -> aborts, missing/malformed msg -> aborts, 200 -> ok.
The netascode#813 fix treated any HTTP 500 from config-save as non-fatal, which
could also swallow an unrelated 500 on a healthy fabric. Restrict the
non-fatal path to fabrics whose data model declares pre-provisioned
switches (poap.preprovision) -- the documented condition where NDFC
legitimately returns 500 while switches are still reloading or not yet
reachable. A 500 on a fabric without pre-provisioned switches, and any
other failure (400, auth, malformed), remain fatal.

Adds PipelineRunnerBase._fabric_has_preprovisioned_switches() and makes
the warning message explicit about why the run continues.
@dacasti2mx
dacasti2mx force-pushed the fix/813-configsave-500-nonfatal branch from 528c340 to c1eaa1c Compare July 7, 2026 15:43

@ccoueffe ccoueffe left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@juburnet
juburnet requested a review from mthurstocisco July 9, 2026 14:07

@mthurstocisco mthurstocisco left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested and works for a full pre provisioned fabric for me

@juburnet juburnet left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve

@mikewiebe
mikewiebe merged commit 620d153 into netascode:develop Jul 9, 2026
23 checks passed
@juburnet juburnet mentioned this pull request Jul 15, 2026
@dacasti2mx
dacasti2mx deleted the fix/813-configsave-500-nonfatal branch July 19, 2026 02:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

0.8.1 Release 0.8.1 bug Something isn't working ready for review PR Ready for Review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[0.8.0] Pre-provisioned fabric run aborts on config-save HTTP 500 instead of continuing

5 participants