multiGPU infer hanging.

[yufang67]

i use 0.15.0.dev version. The model initialized by
llm=tensorrt_llm.LLM( path, tokenizer, dtype="float16", build_config=build_config, tensor_parallel_size=2, pipeline_parallel_size=1, )
Then, the infer run on a 4xa100 node with llm.generate(input, sampling_params).
The infer run through all samples and reach the end of the main program, but hang there.
Did i miss any config ?

Thanks

[Advait251206]

A hang at program exit usually indicates that one or more TensorRT-LLM worker processes (or the underlying NCCL communicators) have not been cleanly shut down, rather than an issue with llm.generate() itself.

A few things to check:

• Are you using mpirun, torchrun, or launching a single Python process? Multi-GPU Tensor Parallel requires all ranks to participate correctly. An imbalance between launched processes and tensor_parallel_size can result in a deadlock during cleanup.

• Explicitly destroy the LLM instance before exiting. For example:

del llm
import gc
gc.collect()

If CUDA contexts or communicators are still alive when Python exits, the destructor may block waiting for other ranks.

• Synchronize CUDA before exit to ensure all kernels have completed:

import torch

torch.cuda.synchronize()

• Check for lingering background threads (e.g., tokenizer workers or executor threads) that may still be waiting on outstanding requests.

Since the inference itself completes successfully and the hang only occurs after processing all samples, it sounds more like a shutdown/finalization issue than an inference issue.

A few questions that would help narrow it down:

1. Are you launching with one process or two processes for tensor_parallel_size=2?
2. Does the hang occur before or after del llm?
3. Does it still hang if you run with tensor_parallel_size=1?
4. Are there any NCCL messages (e.g., NCCL WARN or NCCL INFO) printed when the program is about to exit?

If tensor_parallel_size=1 exits normally but tensor_parallel_size=2 hangs, that would strongly suggest the issue is in the distributed cleanup (NCCL/MPI/executor shutdown) rather than the generation pipeline itself.

---

If this answer helped or pointed you in the right direction, I'd appreciate it if you could mark it as the accepted answer so it's easier for others with the same issue to find.

Also, if you found my contribution useful, I'd appreciate it if you could check out my GitHub profile, follow me, and star any repositories you find interesting.

GitHub: https://github.com/Advait251206