Rollback on failed OXP POST by congwang09 · Pull Request #524 · atlanticwave-sdx/sdx-controller

congwang09 · 2026-04-09T15:07:39Z

Resolves: #519
This PR should merge after atlanticwave-sdx/sdx-lc#204 and atlanticwave-sdx/pce#304

If OXP POST request failed, or timeout, roll back partially provisioned connection request

coveralls · 2026-04-09T15:10:41Z

Coverage Report for CI Build 25450424367

Coverage decreased (-3.3%) to 49.499%

Details

Coverage decreased (-3.3%) from the base build.
Patch coverage: 303 uncovered changes across 5 files (109 of 412 lines covered, 26.46%).
16 coverage regressions across 4 files.

Uncovered Changes

File	Changed	Covered	%
sdx_controller/handlers/lc_message_handler.py	128	7	5.47%
sdx_controller/handlers/connection_handler.py	158	62	39.24%
sdx_controller/controllers/l2vpn_controller.py	78	9	11.54%
sdx_controller/init.py	47	31	65.96%
sdx_controller/messaging/rpc_queue_consumer.py	1	0	0.0%

Coverage Regressions

16 previously-covered lines in 4 files lost coverage.

File	Lines Losing Coverage	Coverage
sdx_controller/controllers/l2vpn_controller.py	5	39.61%
sdx_controller/handlers/connection_handler.py	5	57.98%
sdx_controller/utils/db_utils.py	4	77.27%
sdx_controller/handlers/lc_message_handler.py	2	9.55%

Coverage Stats


Relevant Lines:	2792
Covered Lines:	1382
Line Coverage:	49.5%
Coverage Strength:	0.99 hits per line

💛 - Coveralls

italovalcy · 2026-04-29T19:39:49Z

As discussed on slack and on the weekly meeting, after reviewing this PR and their dependencies, the end-to-end tests are passing. However the mentioned issue #519 (#519) seems still unfixed: we are still seeing the breakdowns on the OXPs after the failure. The expected behavior would be an empty reply from OXP's L2VPN. Cong mentioned that it ca be because of the merging with main branch.

italovalcy · 2026-05-04T17:32:05Z

Hello Cong, I've re-tested the new code you added and here are the results:

End-to-end tests: OK passing
initial test to verify if creating one L2VPN with a breakdown failing and get them removed: OK working
creating a L2VPN successfully and changint/PATCHing it to a value that will fail on purpose: not working, it keeps down

here are the details:

~/sdx-end-to-end-tests$ docker compose exec -it mininet curl -s -X POST -H 'Content-type: application/json' http://sdx-controller:8080/SDX-Controller/l2vpn/1.0 -d '{"name": "L2VPN-use-case4", "endpoints": [{"port_id": "urn:sdx:port:tenet.ac.za:Tenet03:50", "vlan": "666"}, {"port_id": "urn:sdx:port:ampath.net:Ampath3:50", "vlan": "666"}]}'
{
  "reason": "Connection published",
  "service_id": "88a0bf35-4386-4a64-83fd-9ea20b8fd8d9",
  "status": "under provisioning"
}
~/sdx-end-to-end-tests$ ./scripts/show-sdx-controller.sh l2vpn
ID                                    STATUS  ENDPOINT-1                           VLAN-1  ENDPOINT-2                          VLAN-2
--                                    ------  ----------                           ------  ----------                          ------
88a0bf35-4386-4a64-83fd-9ea20b8fd8d9  up      urn:sdx:port:tenet.ac.za:Tenet03:50  666     urn:sdx:port:ampath.net:Ampath3:50  666
~/sdx-end-to-end-tests$ for oxp in ampath sax tenet; do echo $oxp; docker compose exec -it mininet curl -s http://$oxp:8181/api/kytos/mef_eline/v2/evc/ | jq -r '.[] | .id + " " + .name + " " + (.enabled|tostring) + " " + .status + " " +(.active|tostring) + " " + .uni_a.interface_id + " " + (.uni_a.tag.value|tostring) + " " + .uni_z.interface_id + " " + (.uni_z.tag.value|tostring)'; done
ampath
e0dcc35a59cc4b SDX-L2VPN-AMPATH_vlan_666_666 true  true aa:00:00:00:00:00:00:01:40 666 aa:00:00:00:00:00:00:03:50 666
sax
3e5f5cfd1ef14b SDX-L2VPN-SAX_vlan_666_666 true  true dd:00:00:00:00:00:00:04:41 666 dd:00:00:00:00:00:00:04:40 666
tenet
94e94aefc74246 SDX-L2VPN-TENET_vlan_666_666 true  true cc:00:00:00:00:00:00:08:50 666 cc:00:00:00:00:00:00:06:41 666

~/sdx-end-to-end-tests$ docker compose exec -it mininet curl -s -X PATCH -H 'Content-type: application/json' http://sdx-controller:8080/SDX-Controller/l2vpn/1.0/88a0bf35-4386-4a64-83fd-9ea20b8fd8d9 -d '{"name": "L2VPN-use-case4", "endpoints": [{"port_id": "urn:sdx:port:tenet.ac.za:Tenet03:50", "vlan": "777"}, {"port_id": "urn:sdx:port:ampath.net:Ampath3:50", "vlan": "777"}]}'
{
  "reason": "Failure, rolled back to last successful L2VPN: Patched connection provisioning failed: DOWN",
  "service_id": "88a0bf35-4386-4a64-83fd-9ea20b8fd8d9",
  "status": "under provisioning"
}

it says it will rollback to the previous state, however I waited more 5 minutes and no updates:

~/sdx-end-to-end-tests$ ./scripts/show-sdx-controller.sh l2vpn
ID                                    STATUS  ENDPOINT-1                           VLAN-1  ENDPOINT-2                          VLAN-2
--                                    ------  ----------                           ------  ----------                          ------
88a0bf35-4386-4a64-83fd-9ea20b8fd8d9  down    urn:sdx:port:tenet.ac.za:Tenet03:50  666     urn:sdx:port:ampath.net:Ampath3:50  666
~/sdx-end-to-end-tests$ date
Mon May  4 05:25:40 PM UTC 2026

~/sdx-end-to-end-tests$ date
Mon May  4 05:31:45 PM UTC 2026
~/sdx-end-to-end-tests$ ./scripts/show-sdx-controller.sh l2vpn
ID                                    STATUS  ENDPOINT-1                           VLAN-1  ENDPOINT-2                          VLAN-2
--                                    ------  ----------                           ------  ----------                          ------
88a0bf35-4386-4a64-83fd-9ea20b8fd8d9  down    urn:sdx:port:tenet.ac.za:Tenet03:50  666     urn:sdx:port:ampath.net:Ampath3:50  666

# the breakdown does not seems to be re-created:
~/sdx-end-to-end-tests$ for oxp in ampath sax tenet; do echo $oxp; docker compose exec -it mininet curl -s http://$oxp:8181/api/kytos/mef_eline/v2/evc/ | jq -r '.[] | .id + " " + .name + " " + (.enabled|tostring) + " " + .status + " " +(.active|tostring) + " " + .uni_a.interface_id + " " + (.uni_a.tag.value|tostring) + " " + .uni_z.interface_id + " " + (.uni_z.tag.value|tostring)'; done
ampath
sax
tenet

congwang09 · 2026-05-05T19:00:29Z

Hello Cong, I've re-tested the new code you added and here are the results:

End-to-end tests: OK passing
initial test to verify if creating one L2VPN with a breakdown failing and get them removed: OK working
creating a L2VPN successfully and changint/PATCHing it to a value that will fail on purpose: not working, it keeps down

here are the details:

~/sdx-end-to-end-tests$ docker compose exec -it mininet curl -s -X POST -H 'Content-type: application/json' http://sdx-controller:8080/SDX-Controller/l2vpn/1.0 -d '{"name": "L2VPN-use-case4", "endpoints": [{"port_id": "urn:sdx:port:tenet.ac.za:Tenet03:50", "vlan": "666"}, {"port_id": "urn:sdx:port:ampath.net:Ampath3:50", "vlan": "666"}]}'
{
  "reason": "Connection published",
  "service_id": "88a0bf35-4386-4a64-83fd-9ea20b8fd8d9",
  "status": "under provisioning"
}
~/sdx-end-to-end-tests$ ./scripts/show-sdx-controller.sh l2vpn
ID                                    STATUS  ENDPOINT-1                           VLAN-1  ENDPOINT-2                          VLAN-2
--                                    ------  ----------                           ------  ----------                          ------
88a0bf35-4386-4a64-83fd-9ea20b8fd8d9  up      urn:sdx:port:tenet.ac.za:Tenet03:50  666     urn:sdx:port:ampath.net:Ampath3:50  666
~/sdx-end-to-end-tests$ for oxp in ampath sax tenet; do echo $oxp; docker compose exec -it mininet curl -s http://$oxp:8181/api/kytos/mef_eline/v2/evc/ | jq -r '.[] | .id + " " + .name + " " + (.enabled|tostring) + " " + .status + " " +(.active|tostring) + " " + .uni_a.interface_id + " " + (.uni_a.tag.value|tostring) + " " + .uni_z.interface_id + " " + (.uni_z.tag.value|tostring)'; done
ampath
e0dcc35a59cc4b SDX-L2VPN-AMPATH_vlan_666_666 true  true aa:00:00:00:00:00:00:01:40 666 aa:00:00:00:00:00:00:03:50 666
sax
3e5f5cfd1ef14b SDX-L2VPN-SAX_vlan_666_666 true  true dd:00:00:00:00:00:00:04:41 666 dd:00:00:00:00:00:00:04:40 666
tenet
94e94aefc74246 SDX-L2VPN-TENET_vlan_666_666 true  true cc:00:00:00:00:00:00:08:50 666 cc:00:00:00:00:00:00:06:41 666

~/sdx-end-to-end-tests$ docker compose exec -it mininet curl -s -X PATCH -H 'Content-type: application/json' http://sdx-controller:8080/SDX-Controller/l2vpn/1.0/88a0bf35-4386-4a64-83fd-9ea20b8fd8d9 -d '{"name": "L2VPN-use-case4", "endpoints": [{"port_id": "urn:sdx:port:tenet.ac.za:Tenet03:50", "vlan": "777"}, {"port_id": "urn:sdx:port:ampath.net:Ampath3:50", "vlan": "777"}]}'
{
  "reason": "Failure, rolled back to last successful L2VPN: Patched connection provisioning failed: DOWN",
  "service_id": "88a0bf35-4386-4a64-83fd-9ea20b8fd8d9",
  "status": "under provisioning"
}

it says it will rollback to the previous state, however I waited more 5 minutes and no updates:

~/sdx-end-to-end-tests$ ./scripts/show-sdx-controller.sh l2vpn
ID                                    STATUS  ENDPOINT-1                           VLAN-1  ENDPOINT-2                          VLAN-2
--                                    ------  ----------                           ------  ----------                          ------
88a0bf35-4386-4a64-83fd-9ea20b8fd8d9  down    urn:sdx:port:tenet.ac.za:Tenet03:50  666     urn:sdx:port:ampath.net:Ampath3:50  666
~/sdx-end-to-end-tests$ date
Mon May  4 05:25:40 PM UTC 2026

~/sdx-end-to-end-tests$ date
Mon May  4 05:31:45 PM UTC 2026
~/sdx-end-to-end-tests$ ./scripts/show-sdx-controller.sh l2vpn
ID                                    STATUS  ENDPOINT-1                           VLAN-1  ENDPOINT-2                          VLAN-2
--                                    ------  ----------                           ------  ----------                          ------
88a0bf35-4386-4a64-83fd-9ea20b8fd8d9  down    urn:sdx:port:tenet.ac.za:Tenet03:50  666     urn:sdx:port:ampath.net:Ampath3:50  666

# the breakdown does not seems to be re-created:
~/sdx-end-to-end-tests$ for oxp in ampath sax tenet; do echo $oxp; docker compose exec -it mininet curl -s http://$oxp:8181/api/kytos/mef_eline/v2/evc/ | jq -r '.[] | .id + " " + .name + " " + (.enabled|tostring) + " " + .status + " " +(.active|tostring) + " " + .uni_a.interface_id + " " + (.uni_a.tag.value|tostring) + " " + .uni_z.interface_id + " " + (.uni_z.tag.value|tostring)'; done
ampath
sax
tenet

Hi Italo, I tried your steps, it seems to be working fine for me. Can you give it another try? If it's not working, we can debug this together.

[cwang@aw-sdx-lc-2 sdx-end-to-end-tests]$ docker compose exec -it mininet curl -s -X POST -H 'Content-type: application/json' http://sdx-controller:8080/SDX-Controller/l2vpn/1.0 -d '{"name": "L2VPN-use-case4", "endpoints": [{"port_id": "urn:sdx:port:tenet.ac.za:Tenet03:50", "vlan": "666"}, {"port_id": "urn:sdx:port:ampath.net:Ampath3:50", "vlan": "666"}]}'
{
  "reason": "Connection published",
  "service_id": "7a70fecb-69b3-4eba-a659-7dba3d62d94e",
  "status": "under provisioning"
}
[cwang@aw-sdx-lc-2 sdx-end-to-end-tests]$ ./scripts/show-sdx-controller.sh l2vpn
ID                                    STATUS  ENDPOINT-1                           VLAN-1  ENDPOINT-2                          VLAN-2
--                                    ------  ----------                           ------  ----------                          ------
7a70fecb-69b3-4eba-a659-7dba3d62d94e  up      urn:sdx:port:tenet.ac.za:Tenet03:50  666     urn:sdx:port:ampath.net:Ampath3:50  666
[cwang@aw-sdx-lc-2 sdx-end-to-end-tests]$ docker compose exec -it mininet curl -s -X PATCH -H 'Content-type: application/json' http://sdx-controller:8080/SDX-Controller/l2vpn/1.0/7a70fecb-69b3-4eba-a659-7dba3d62d94e -d '{"name": "L2VPN-use-case4", "endpoints": [{"port_id": "urn:sdx:port:tenet.ac.za:Tenet03:50", "vlan": "777"}, {"port_id": "urn:sdx:port:ampath.net:Ampath3:50", "vlan": "777"}]}'
{
  "reason": "Failure, rolled back to last successful L2VPN: Patched connection provisioning failed: DOWN",
  "service_id": "7a70fecb-69b3-4eba-a659-7dba3d62d94e",
  "status": "up"
}
[cwang@aw-sdx-lc-2 sdx-end-to-end-tests]$ ./scripts/show-sdx-controller.sh l2vpn
ID                                    STATUS  ENDPOINT-1                           VLAN-1  ENDPOINT-2                          VLAN-2
--                                    ------  ----------                           ------  ----------                          ------
7a70fecb-69b3-4eba-a659-7dba3d62d94e  up      urn:sdx:port:tenet.ac.za:Tenet03:50  666     urn:sdx:port:ampath.net:Ampath3:50  666
[cwang@aw-sdx-lc-2 sdx-end-to-end-tests]$ for oxp in ampath sax tenet; do echo $oxp; docker compose exec -it mininet curl -s http://$oxp:8181/api/kytos/mef_eline/v2/evc/ | jq -r '.[] | .id + " " + .name + " " + (.enabled|tostring) + " " + .status + " " +(.active|tostring) + " " + .uni_a.interface_id + " " + (.uni_a.tag.value|tostring) + " " + .uni_z.interface_id + " " + (.uni_z.tag.value|tostring)'; done
ampath
72faa5db2f0b4b SDX-L2VPN-AMPATH_vlan_666_666 true  true aa:00:00:00:00:00:00:01:40 666 aa:00:00:00:00:00:00:03:50 666
sax
9057e53391fc4a SDX-L2VPN-SAX_vlan_666_666 true  true dd:00:00:00:00:00:00:04:41 666 dd:00:00:00:00:00:00:04:40 666
tenet
5279228f5e6642 SDX-L2VPN-TENET_vlan_666_666 true  true cc:00:00:00:00:00:00:08:50 666 cc:00:00:00:00:00:00:06:41 666

italovalcy · 2026-05-06T14:55:47Z

Hi @congwang09, thanks for testing and reporting the reporting your results! You are right, with the steps to reproduce I provided on the issue, your result is consistent!

I forgot to mention earlier that I extended the test to include a scenario where the OXP failed but it took some time to send the failure reply. The idea here was to simulate an OXP that was overloaded or something like that.

Thus, in my tests I have a slightly different failure mechanism on TENET:

diff --git a/main.py b/main.py
index d207f64..5ecafd8 100644
--- a/main.py
+++ b/main.py
@@ -435,6 +435,10 @@ class Main(KytosNApp):  # pylint: disable=R0904
             log.warning(f"EVC creation failed: {msg}. request={content}")
             return JSONResponse({"description": msg}, code)

+        if evc_dict["uni_a"].get("tag", {}).get("value") == 777:
+            time.sleep(60)
+            raise ValueError("error on purpose")
+
         try:
             response = requests.post(KYTOS_EVC_URL, json=evc_dict, timeout=30)
             assert response.status_code == 201, response.text

Notice the time.sleep(60). I used 60sec just as a random value, but it could be anytime, assuming we cannot predict how overloaded the OXP can be.

In the scenario above, I got the behavior I mentioned to you earlier:

## L2VPN is created successfully:

$ docker compose exec -it mininet curl -s -X POST -H 'Content-type: application/json' http://sdx-controller:8080/SDX-Controller/l2vpn/1.0 -d '{"name": "L2VPN-use-case4", "endpoints": [{"port_id": "urn:sdx:port:tenet.ac.za:Tenet03:50", "vlan": "666"}, {"port_id": "urn:sdx:port:ampath.net:Ampath3:50", "vlan": "666"}]}'
{
  "reason": "Connection published",
  "service_id": "f6fb9f54-5af1-4be5-b94a-97e02e8217fb",
  "status": "under provisioning"
}
$ ./scripts/show-sdx-controller.sh l2vpn
ID                                    STATUS  ENDPOINT-1                           VLAN-1  ENDPOINT-2                          VLAN-2
--                                    ------  ----------                           ------  ----------                          ------
f6fb9f54-5af1-4be5-b94a-97e02e8217fb  up      urn:sdx:port:tenet.ac.za:Tenet03:50  666     urn:sdx:port:ampath.net:Ampath3:50  666


## Then PATCH behaves unexpectedly by leaving the L2VPN down, even after 5min or more:

$ ID=f6fb9f54-5af1-4be5-b94a-97e02e8217fb
$ date; docker compose exec -it mininet curl -s -X PATCH -H 'Content-type: application/json' http://sdx-controller:8080/SDX-Controller/l2vpn/1.0/$ID -d '{"name": "L2VPN-use-case4", "endpoints": [{"port_id": "urn:sdx:port:tenet.ac.za:Tenet03:50", "vlan": "777"}, {"port_id": "urn:sdx:port:ampath.net:Ampath3:50", "vlan": "777"}]}'
Wed May  6 02:47:53 PM UTC 2026
{
  "reason": "Failure, rolled back to last successful L2VPN: Patched connection provisioning failed: DOWN",
  "service_id": "f6fb9f54-5af1-4be5-b94a-97e02e8217fb",
  "status": "under provisioning"
}
$ date
Wed May  6 02:48:40 PM UTC 2026
$ ./scripts/show-sdx-controller.sh l2vpn
ID                                    STATUS  ENDPOINT-1    VLAN-1                               ENDPOINT-2  VLAN-2
--                                    ------  ----------    ------                               ----------  ------
f6fb9f54-5af1-4be5-b94a-97e02e8217fb  under   provisioning  urn:sdx:port:tenet.ac.za:Tenet03:50  666         urn:sdx:port:ampath.net:Ampath3:50  666
$ date
Wed May  6 02:54:42 PM UTC 2026

$ ./scripts/show-sdx-controller.sh l2vpn
ID                                    STATUS  ENDPOINT-1                           VLAN-1  ENDPOINT-2                          VLAN-2
--                                    ------  ----------                           ------  ----------                          ------
f6fb9f54-5af1-4be5-b94a-97e02e8217fb  down    urn:sdx:port:tenet.ac.za:Tenet03:50  666     urn:sdx:port:ampath.net:Ampath3:50  666

YufengXin · 2026-05-06T15:54:31Z

The extended test case deserves a new issue/PR: timeout is always a problem anyway.

I recommend to close this PR to have a working tagged version.

italovalcy · 2026-05-06T20:16:49Z

Hi Cong, tested here and it all worked just fine! Excellent progress! thank you!

congwang09 added 3 commits April 8, 2026 00:49

Rollback when a OXP post fails

d4078d8

Refine error handling

fe5b729

Fix e2e test

8065eaf

congwang09 marked this pull request as draft April 9, 2026 15:07

lmarinve assigned congwang09 Apr 13, 2026

congwang09 added 9 commits April 15, 2026 01:47

Merge main

c1500c6

Format

b528e43

Fix tests

d3e83ef

Format

62b56f5

Bug fix

26c0afe

Fix use_case_topology errors

01daffd

Sanitize topology updated before TE parsing

d4db040

Format

0cb2b07

Add import

dcd02c0

This was referenced Apr 22, 2026

Send breakdown domain to sdx controller atlanticwave-sdx/sdx-lc#204

Merged

Use list instead of dict for breakdown atlanticwave-sdx/pce#304

Merged

congwang09 requested review from YufengXin and italovalcy April 22, 2026 15:05

congwang09 marked this pull request as ready for review April 22, 2026 17:03

congwang09 added 2 commits April 25, 2026 01:50

Merge main

9e60d33

Fix e2e test

dc1d081

congwang09 added 3 commits April 30, 2026 00:15

Solve use case 4

35f1a4b

Fix PATCH race condition

f972207

Format

f52141f

later OXP failure response can still trigger rollback

f97dea5

italovalcy approved these changes May 6, 2026

View reviewed changes

congwang09 merged commit 5a34ae5 into main May 7, 2026
4 checks passed

congwang09 deleted the rollback-on-failed-oxp-post branch May 7, 2026 14:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Rollback on failed OXP POST#524

Rollback on failed OXP POST#524
congwang09 merged 18 commits into
mainfrom
rollback-on-failed-oxp-post

congwang09 commented Apr 9, 2026 •

edited

Loading

Uh oh!

coveralls commented Apr 9, 2026 •

edited

Loading

Uh oh!

italovalcy commented Apr 29, 2026

Uh oh!

italovalcy commented May 4, 2026 •

edited

Loading

Uh oh!

congwang09 commented May 5, 2026

Uh oh!

italovalcy commented May 6, 2026

Uh oh!

YufengXin commented May 6, 2026

Uh oh!

italovalcy commented May 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

congwang09 commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coveralls commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage Report for CI Build 25450424367

Coverage decreased (-3.3%) to 49.499%

Details

Uncovered Changes

Coverage Regressions

Coverage Stats

💛 - Coveralls

Uh oh!

italovalcy commented Apr 29, 2026

Uh oh!

italovalcy commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

congwang09 commented May 5, 2026

Uh oh!

italovalcy commented May 6, 2026

Uh oh!

YufengXin commented May 6, 2026

Uh oh!

italovalcy commented May 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

congwang09 commented Apr 9, 2026 •

edited

Loading

coveralls commented Apr 9, 2026 •

edited

Loading

italovalcy commented May 4, 2026 •

edited

Loading