CI/FT: Adds fault tolerance CI test by william-gallagher-nv · Pull Request #11570 · openucx/ucx

william-gallagher-nv · 2026-06-23T03:14:19Z

What?

Adds Fault Tolerance test using ucx_perftest
Adds 'failover' as endpoint failure mode in ucx_perftest

Why?

Allows testing of UCX handling of lane failures.

william-gallagher-nv · 2026-06-24T02:30:39Z

/run

- Adds Fault Tolerance CI test using ucx_perftest - Adds 'failover' as endpoint failure mode in ucx_perftest

william-gallagher-nv · 2026-07-02T04:32:44Z

@svc-nvidia-pr-review

svc-nvidia-pr-review · 2026-07-02T04:32:48Z

🚫 @william-gallagher-nv — not authorized to trigger a review on this repo. Contact a repo admin to be added to the allowed-users list.

Alexey-Rivkin · 2026-07-02T15:04:58Z


 		# Run AMO tests
-		echo -e "4 -s 4\n8 -s 8" > "$ucx_inst_ptest/msg_atomic"
+		echo -e peer "4 -s 4\n8 -s 8" > "$ucx_inst_ptest/msg_atomic"


Does not look like a perftest option

gleon99 · 2026-07-05T10:38:41Z

+        if [[ $word == */*:* ]]; then
+            device=${word#*/}
+            device=${device%%:*}
+            echo "$device"


Can we fail if cfg isn’t found? Missing cfg parses as empty, so the final check can pass without seeing any reconfiguration

I have added a check so that a missing cfg print will fail.

gleon99 · 2026-07-05T10:56:52Z


 #define TL_RESOURCE_NAME_NONE   "<none>"
-#define TEST_PARAMS_ARGS        "t:n:s:W:O:w:D:i:H:oSCIqM:r:E:T:d:x:A:BUem:a:R:lyzL:F:Y:"
+#define TEST_PARAMS_ARGS \


Requiring an argument for -e breaks existing perftest command lines..

Yes. I have updated the places that run ucx_perftest with -e to use -e peer in test_jenkins.sh. Are there other places that needed to be updated? Or would you prefer an entirely new optional to set error mode failover?

-e still requires an argument, breaking existing ucx_perftest -e command lines. Updating the in-tree call site does not preserve compatibility for external users.

maybe implement like this:

no -e -> none

-e [none|peer|failover] where "empty" == "peer" for compatibility

no -e -> none

-e [none|peer|failover] where "empty" == "peer" for compatibility

This approach makes sense and allows perftest to stay compatible with external users. But it doesn't map well to get_opt() handling which seems to either always expects an argument or never does. I'll think about how this approach can be implemented.

We can use :: to allow an optional argument argument for the -e flag.

#define TEST_PARAMS_ARGS \ "t:n:s:W:O:w:D:i:H:oSCIqM:r:E:T:d:x:A:BUe::m:a:R:lyzL:F:Y:"

The :: implementation is a gnu libc extension that is not required by POSIX. It has the advantage of allowing -e with no argument to map to error handling mode peer. However, it requires merging the flag and the optional argument when an optional argument is used: -epeer or -efailover.

This seems a little awkward and different from the other argument parsing in ucx_perftest. If you think this is OK, let me know.

gleon99 · 2026-07-05T10:59:20Z

            ep_params.err_handler.cb  = ucp_perf_test_err_handler;
            ep_params.err_handler.arg = NULL;
-            ep_params.err_mode        = UCP_ERR_HANDLING_MODE_PEER;
+            ep_params.err_mode        = perf->params.ucp.err_mode;


Existing libperf callers set ERR_HANDLING without initializing the new field, so they now get NONE instead of PEER. Can we preserve PEER as the default?

I think if existing libperf callers set ERR_HANDLING via -e flag with no following value, they will likely get a parser error. I am happy to update any tests that currently use -e to e peer. In order to preserve complete backwards compatibility, we could leave -e for error handling mode peer and use -f for failover mode. It seem more intituitive to me to use -e failover and -e peer instead of -e and -f. But let me know what you prefer.

Existing direct libperf callers using UCX_PERF_TEST_FLAG_ERR_HANDLING still receive UCP_ERR_HANDLING_MODE_NONE when the new field is zero-initialized. Initializing it in the CLI path does not fix library callers..

Adds handling for detecting a missing protocol reconfiguration print in perftest stdout. Also removes the use of tee in when piping perftest stdout to a file.

gleon99

See last comments

gleon99 · 2026-07-06T06:35:20Z

+parse_active_devices() {
+    file=$1
+    config_cnt=$2
+    found=0


It is never set to 1.

gleon99 · 2026-07-06T07:19:11Z


 #define TL_RESOURCE_NAME_NONE   "<none>"
-#define TEST_PARAMS_ARGS        "t:n:s:W:O:w:D:i:H:oSCIqM:r:E:T:d:x:A:BUem:a:R:lyzL:F:Y:"
+#define TEST_PARAMS_ARGS \


-e still requires an argument, breaking existing ucx_perftest -e command lines. Updating the in-tree call site does not preserve compatibility for external users.

gleon99 · 2026-07-06T07:20:05Z

            ep_params.err_handler.cb  = ucp_perf_test_err_handler;
            ep_params.err_handler.arg = NULL;
-            ep_params.err_mode        = UCP_ERR_HANDLING_MODE_PEER;
+            ep_params.err_mode        = perf->params.ucp.err_mode;


Existing direct libperf callers using UCX_PERF_TEST_FLAG_ERR_HANDLING still receive UCP_ERR_HANDLING_MODE_NONE when the new field is zero-initialized. Initializing it in the CLI path does not fix library callers..

evgeny-leksikov · 2026-07-06T11:49:27Z

        ucp_perf_datatype_t     recv_datatype;
        size_t                  am_hdr_size; /* UCP Active Message header size
                                                (not included in message size) */
+        ucp_err_handling_mode_t err_mode; /* UCP endpoint error handling


minor: comment alignment

evgeny-leksikov · 2026-07-06T11:57:30Z


 #define TL_RESOURCE_NAME_NONE   "<none>"
-#define TEST_PARAMS_ARGS        "t:n:s:W:O:w:D:i:H:oSCIqM:r:E:T:d:x:A:BUem:a:R:lyzL:F:Y:"
+#define TEST_PARAMS_ARGS \


maybe implement like this:

no -e -> none

-e [none|peer|failover] where "empty" == "peer" for compatibility

evgeny-leksikov · 2026-07-06T11:57:49Z

+    printf("                        peer     - peer failure error handling\n");
+    printf("                        failover - lane failover error handling\n");


I would extend with "none"

william-gallagher-nv marked this pull request as draft June 24, 2026 02:31

william-gallagher-nv force-pushed the fault_tolerance_perftest_ci branch 3 times, most recently from d5cd60e to 2675908 Compare June 24, 2026 20:20

CI/FT: Add Fault Tolerance CI test

806418f

- Adds Fault Tolerance CI test using ucx_perftest - Adds 'failover' as endpoint failure mode in ucx_perftest

william-gallagher-nv changed the title ~~CI/PERFTEST Add Fault Tolerance CI test~~ CI/FT: Adds fault tolerance CI test Jul 2, 2026

william-gallagher-nv force-pushed the fault_tolerance_perftest_ci branch from afec5e0 to 806418f Compare July 2, 2026 05:30

william-gallagher-nv requested review from evgeny-leksikov and gleon99 July 2, 2026 05:31

william-gallagher-nv marked this pull request as ready for review July 2, 2026 10:55

gleon99 requested a review from Alexey-Rivkin July 2, 2026 12:33

Alexey-Rivkin requested changes Jul 2, 2026

View reviewed changes

CI/FT: Removes incorrect added perftest optional

029337a

Alexey-Rivkin approved these changes Jul 2, 2026

View reviewed changes

Merge branch 'master' into fault_tolerance_perftest_ci

76e09d3

gleon99 enabled auto-merge (squash) July 5, 2026 08:16

gleon99 reviewed Jul 5, 2026

View reviewed changes

Alexey-Rivkin requested changes Jul 5, 2026

View reviewed changes

Comment thread contrib/test_jenkins.sh Outdated

Comment thread contrib/test_jenkins.sh Outdated

CI/FT: Fault Tolerance CI edge case handling

246df6b

Adds handling for detecting a missing protocol reconfiguration print in perftest stdout. Also removes the use of tee in when piping perftest stdout to a file.

gleon99 reviewed Jul 6, 2026

View reviewed changes

evgeny-leksikov reviewed Jul 6, 2026

View reviewed changes

william-gallagher-nv added 2 commits July 7, 2026 07:24

CI/FT: Fix device parsing

c1e1d85

CI/FT: Preserve legacy handling of -e flag in perftest

8e6e131

		printf(" peer - peer failure error handling\n");
		printf(" failover - lane failover error handling\n");

Uh oh!

Conversation

william-gallagher-nv commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What?

Why?

Uh oh!

william-gallagher-nv commented Jun 24, 2026

Uh oh!

william-gallagher-nv commented Jul 2, 2026

Uh oh!

svc-nvidia-pr-review commented Jul 2, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

william-gallagher-nv Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

william-gallagher-nv Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

gleon99 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

william-gallagher-nv commented Jun 23, 2026 •

edited

Loading

william-gallagher-nv Jul 7, 2026 •

edited

Loading

william-gallagher-nv Jul 13, 2026 •

edited

Loading