Fix #813: continue create pipeline on config-save HTTP 500#819
Conversation
|
Update: the 500 bypass is now gated on the data model. Before, this PR treated any config-save HTTP 500 as non-fatal — so a real 500 on a healthy fabric would also be swallowed. The latest commit only tolerates the 500 when the data model actually has pre-provisioned switches ( Verified live against a pre-provisioned fabric: config-save returns 500, the run continues, and create completes 21/21. The new check was also validated deterministically (preprov → tolerate, non-preprov / empty → fatal). |
When a fabric is pre-provisioned, its switches may still be reloading or not yet reachable. In that state NDFC returns HTTP 500 to the intermediate config-save (recalculate). In 0.7.2 the run logged this and kept going; in 0.8.0 it aborts at the inventory_config_save step, so the rest of the fabric intent is never applied. The "treat 500 as non-fatal" logic in pipeline_base._config_save never ran, because the response that carries the 500 was thrown away earlier: fabric_deploy_manager set results['failed']=True but no 'msg', so the RETURN_CODE never reached the check. Changes: - fabric_config_save() now returns the NDFC response. - manage_fabrics() puts that response in results['msg'] when config-save fails, so callers can read the RETURN_CODE. - _config_save() treats RETURN_CODE 500 as non-fatal and logs a clear warning. Other failures (400, auth, malformed responses) still abort. Unit tests are deferred to a follow-up: the collection has no unit-test harness today and CI only runs build, sanity, and lint. Verified locally with a temporary mock harness (not committed): 500 -> continues with a warning, 400 -> aborts, missing/malformed msg -> aborts, 200 -> ok.
The netascode#813 fix treated any HTTP 500 from config-save as non-fatal, which could also swallow an unrelated 500 on a healthy fabric. Restrict the non-fatal path to fabrics whose data model declares pre-provisioned switches (poap.preprovision) -- the documented condition where NDFC legitimately returns 500 while switches are still reloading or not yet reachable. A 500 on a fabric without pre-provisioned switches, and any other failure (400, auth, malformed), remain fatal. Adds PipelineRunnerBase._fabric_has_preprovisioned_switches() and makes the warning message explicit about why the run continues.
528c340 to
c1eaa1c
Compare
mthurstocisco
left a comment
There was a problem hiding this comment.
Tested and works for a full pre provisioned fabric for me
Related Issue(s)
Fixes #813
Related Collection Role
Related Data Model Element
Proposed Changes
On a pre-provisioned fabric the switches can still be reloading or
unreachable when the create pipeline runs the intermediate config-save.
NDFC then returns HTTP 500. 0.7.2 logged this and continued; 0.8.0 aborts
at
inventory_config_save, so the rest of the fabric intent is neverapplied. This is a regression.
Root cause:
pipeline_base._config_savealready meant to treat a 500 asnon-fatal, but the check could never fire.
fabric_deploy_managerdiscardedthe NDFC response on a failed config-save and returned
failed=Truewith nomsg, so theRETURN_CODEwas lost before the caller could look at it.Fix:
fabric_config_save()returns the NDFC response.manage_fabrics()propagates it intoresults['msg']when config-savefails, so the
RETURN_CODEsurvives._config_save()treatsRETURN_CODE == 500as non-fatal and emits a clearwarning (full detail under
-v). Any other failure — 400, auth, or amalformed/missing response — still aborts the run. This is stricter and
safer than the old 0.7.2 rescue, which swallowed every error.
Only the create pipeline registers
_config_save(create_resources.yml);remove_resources.ymldoes not, and the deploy role usesoperation: all,so deploy and remove behavior are unchanged.
Test Notes
The collection has no unit-test harness today, and CI runs only build,
sanity, and lint — so a proper unit test needs new infrastructure. To avoid
blocking the 0.8.1 fix on that project-structure decision, tests are
deferred and tracked in a follow-up: #818.
Verified locally with a temporary mock harness (not committed) that drives
the real
_config_save:msg-> run aborts (fatal)Deferred unit-test matrix (in the follow-up):
failed=Truewith the response intact_config_savereturns non-fatalmsg-> fatalCisco Nexus Dashboard Version
4.1 (per issue #813)
Checklist