Fix Distributed Initialization by Validating Legacy tcp:// and Updating Tests by TahaZahid05 · Pull Request #3758 · pytorch/ignite

TahaZahid05 · 2026-05-25T15:03:49Z

Problem

Using tcp:// directly in PyTorch's init_method causes multi-process communication to hang and fail in modern PyTorch versions.

Solution

Validation & User Error: Added explicit validation in Parallel and _NativeDistModel that detects legacy tcp:// initialization methods and raises a helpful, actionable ValueError guiding the user on how to use PyTorch's native TCPStore instead.
Updated Tests: Cleaned up the distributed test suite to remove legacy tcp:// connections from test parameterizations and updated low-level context tests to use modern dist.TCPStore objects directly.

Check list:

New tests are added (if a new feature is added)
New doc strings: description and/or example code are in RST format
Documentation is updated (if required)

Copilot

Pull request overview

This PR updates Ignite’s PyTorch-native distributed initialization to proactively reject legacy init_method="tcp://..." (which can hang on newer PyTorch versions) and introduces/validates TCPStore-based initialization paths, with accompanying test suite updates.

Changes:

Added tcp:// init_method validation with actionable ValueError guidance in idist.Parallel and _NativeDistModel.
Plumbed store through idist.Parallel → idist.initialize → native dist model initialization.
Refactored distributed tests/fixtures to avoid legacy tcp:// and to use env:// or TCPStore.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
`ignite/distributed/launcher.py`	Adds `store` support to `Parallel` and blocks legacy `tcp://` init methods.
`ignite/distributed/comp_models/native.py`	Adds `store` support and blocks legacy `tcp://` init methods in native model initialization.
`ignite/distributed/utils.py`	Updates `initialize()` API docs to include `store` usage.
`tests/ignite/conftest.py`	Switches single-node test contexts from `tcp://` to `env://` and sets `MASTER_ADDR/PORT`.
`tests/ignite/distributed/test_launcher.py`	Removes `tcp://` parameterizations; adds regression tests for tcp validation and `TCPStore` init.
`tests/ignite/distributed/comp_models/test_native.py`	Replaces `tcp://` init with `TCPStore` in low-level/native context tests; adds new regression tests.
`tests/ignite/distributed/utils/test_native.py`	Removes legacy `tcp://` from parameterized init_method test cases.

Comments suppressed due to low confidence (1)

ignite/distributed/launcher.py:261

When nproc_per_node is provided, Parallel uses idist.spawn() and does not pass the store anywhere (store is only forwarded to idist.initialize() in the non-spawn path). Accepting a non-None store in this mode is misleading; it should be rejected to avoid silently ignoring user configuration.

        if self.backend is not None:
            if nproc_per_node is not None:
                self._spawn_params = self._setup_spawn_params(
                    nproc_per_node, nnodes, node_rank, master_addr, master_port, init_method, **spawn_kwargs
                )

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

        else:
            arg_names = ["nproc_per_node", "nnodes", "node_rank", "master_addr", "master_port"]
            arg_values = [nproc_per_node, nnodes, node_rank, master_addr, master_port]
            for name, value in zip(arg_names, arg_values):
                if value is not None:
                    raise ValueError(f"If backend is None, argument '{name}' should be also None, but given {value}")


+        if init_method is not None and init_method.startswith("tcp://"):
+            raise ValueError(
+                f"TCP initialization via init_method='{init_method}' will hang. "
+                "To fix this, please configure MASTER_ADDR and MASTER_PORT in the environment and "
+                "use 'env://' (or omit init_method). Alternatively, you can configure a TCPStore and "
+                "pass it using the 'store' argument. For example:\n\n"
+                "    import torch.distributed as dist\n"
+                '    store = dist.TCPStore("<master_addr>", <master_port>, world_size, is_master)\n'
+                "    # Then pass the store to idist.Parallel:\n"
+                "    # idist.Parallel(backend=backend, store=store, ...)"
+            )
+


    free_port = _setup_free_port(local_rank)
+    os.environ["MASTER_ADDR"] = "localhost"
+    os.environ["MASTER_PORT"] = str(free_port)



        free_port = _setup_free_port(local_rank)
-        init_method = f"tcp://localhost:{free_port}"
-        temp_file = None
+        os.environ["MASTER_ADDR"] = "localhost"
+        os.environ["MASTER_PORT"] = str(free_port)


            free_port = _setup_free_port(local_rank)
-            init_method = f"tcp://localhost:{free_port}"
-
-        dist_info = {
-            "world_size": world_size,
-            "rank": local_rank,
-            "init_method": init_method,
-        }
+            os.environ["MASTER_ADDR"] = "localhost"
+            os.environ["MASTER_PORT"] = str(free_port)
+            dist_info = {
+                "world_size": world_size,
+                "rank": local_rank,
+                "init_method": "env://",
+            }


+    is_master = rank == 0
+    store = dist.TCPStore("0.0.0.0", 2222, world_size=world_size, is_master=is_master)
+    dist.init_process_group(true_backend, store=store, world_size=world_size, rank=rank)


+    is_master = local_rank == 0
+    store = dist.TCPStore("0.0.0.0", 2222, world_size=world_size, is_master=is_master)
+    dist.init_process_group("nccl", store=store, world_size=world_size, rank=local_rank)


+    try:
+        store = dist.TCPStore("0.0.0.0", 2222, world_size=1, is_master=True, use_libuv=False)
+    except TypeError:
+        # PyTorch < 2.4 doesn't support use_libuv
+        store = dist.TCPStore("0.0.0.0", 2222, world_size=1, is_master=True)


        if temp_file:
            temp_file.close()
+        else:
+            os.environ.pop("MASTER_ADDR", None)
+            os.environ.pop("MASTER_PORT", None)


vfdev-5 · 2026-06-02T20:11:09Z

            if backend is not None:
                self._create_from_backend(
-                    backend, timeout=timeout, init_method=init_method, world_size=world_size, rank=rank, **kwargs
+                    backend,


There is no change in this, let's revert the code formatting.

vfdev-5 · 2026-06-02T20:14:23Z

    assert _NativeDistModel.create_from_context() is None

-    dist.init_process_group(true_backend, "tcp://0.0.0.0:2222", world_size=1, rank=0)
+    store = dist.TCPStore("0.0.0.0", 2222, world_size=1, is_master=True)


We had a fixture to get a freeport in the conf.py, can we use it here?

vfdev-5 · 2026-06-02T20:15:03Z


 @pytest.mark.distributed
-@pytest.mark.parametrize("init_method", [None, "tcp://0.0.0.0:22334", "FILE"])
+@pytest.mark.parametrize("init_method", [None, "FILE"])


Instead of removing a test, let's do store arg instead

github-actions Bot added the module: distributed Distributed module label May 25, 2026

Reverted to warn against tcp ip use

1127fd7

TahaZahid05 force-pushed the fix-torchrun-ci branch from aff62c9 to 1127fd7 Compare May 26, 2026 18:32

TahaZahid05 requested a review from vfdev-5 May 26, 2026 19:38

TahaZahid05 changed the title ~~parse tcp:// init_method into TCPStore and improve keyword forwardingFix torchrun ci~~ Fix Distributed Initialization by Validating Legacy tcp:// and Updating Tests May 26, 2026

aaishwarymishra reviewed May 27, 2026

View reviewed changes

Comment thread ignite/distributed/comp_models/native.py Outdated

Comment thread ignite/distributed/launcher.py Outdated

TahaZahid05 and others added 2 commits May 29, 2026 20:41

Fixed error message and refactored code to avoid using None.

ec61bd4

Merge branch 'master' into fix-torchrun-ci

65794e1

vfdev-5 reviewed May 29, 2026

View reviewed changes

Comment thread ignite/distributed/comp_models/native.py Outdated

vfdev-5 reviewed May 29, 2026

View reviewed changes

Comment thread ignite/distributed/comp_models/native.py Outdated

vfdev-5 reviewed May 29, 2026

View reviewed changes

Comment thread ignite/distributed/launcher.py Outdated

TahaZahid05 added 3 commits May 29, 2026 22:53

Added store support for distributed, fixed messages, and added tests

53aa731

Set USE_LIBUV=0 for windows

bf344c1

Used argument use_libuv instead

33770d9

TahaZahid05 requested a review from Copilot May 30, 2026 20:04

Copilot started reviewing on behalf of TahaZahid05 May 30, 2026 20:04 View session

Copilot AI reviewed May 30, 2026

View reviewed changes

Trying workflow solution

963b079

github-actions Bot added the ci CI label May 30, 2026

TahaZahid05 and others added 3 commits June 2, 2026 17:50

Added store to kwargs

55d5ba7

Fixed test

d08cd26

Merge branch 'master' into fix-torchrun-ci

a0fe019

TahaZahid05 marked this pull request as draft June 2, 2026 17:21

vfdev-5 reviewed Jun 2, 2026

View reviewed changes

Skipped test, and reverted wrong distributed tag

ffcefb4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix Distributed Initialization by Validating Legacy tcp:// and Updating Tests#3758

Fix Distributed Initialization by Validating Legacy tcp:// and Updating Tests#3758
TahaZahid05 wants to merge 11 commits into
masterfrom
fix-torchrun-ci

TahaZahid05 commented May 25, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

vfdev-5 Jun 2, 2026

Uh oh!

vfdev-5 Jun 2, 2026

Uh oh!

vfdev-5 Jun 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

TahaZahid05 commented May 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

vfdev-5 Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

vfdev-5 Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

vfdev-5 Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

TahaZahid05 commented May 25, 2026 •

edited

Loading