Repository navigation
Conversation
`ddp_init_group` sized its rank -> device guard with `torch.cuda.device_count()` and
only accepted the nccl/gloo/mpi backends, so on a host whose accelerator is not CUDA
(e.g. Ascend NPU through torch_npu, which ships the 'hccl' collective library) every
rank aborted with
ValueError: Killing process
Not enough GPUs available!
before the process group could be created, and the accelerator's own backend was
rejected even when PyTorch reports it as available.
Size the guard and bind the rank's device through `torch.accelerator`, and accept any
backend `torch.distributed.is_backend_available` knows about. CUDA hosts keep the
previous behaviour (nccl plus `torch.cuda.set_device`).
Signed-off-by: li-lizhe <147392333@qq.com>
ruff-format (pre-commit) rewrites this unconditional raise into one line. Signed-off-by: li-lizhe <147392333@qq.com>
The repository formats with ruff-format (line-length 80); wrap the guard expression, the backend availability call and the new tests accordingly. Signed-off-by: li-lizhe <147392333@qq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ddp_init_group()assumed the accelerator is CUDA:torch.cuda.device_count(), andOn a host whose accelerator is not CUDA (Ascend NPU through
torch_npu, whose collective library ishccl),torch.cuda.device_count()is 0, so every rank aborted withbefore
init_process_groupwas ever reached — even with 4 NPUs visible — andhcclwas rejected as abackend that "doesn't exist" even though PyTorch reports it as available. This is the same device-agnostic
migration that
infer_device()went through in #3080; the distributed launch path was left behind.What
torch.accelerator(current_accelerator().type,device_count());torch.distributed.is_backend_available()reports as available is accepted (accelerators other thanCUDA ship their own collective library, e.g.
hcclfor Ascend NPU).CUDA hosts keep the previous behaviour: backend
ncclandtorch.cuda.set_device(device).Verification
Real hardware: Ascend 910B host (aarch64, 4 NPUs,
torch 2.15.0.dev,torch_npu 2.15.0.dev,torch.accelerator.is_available() == True,current_accelerator() == npu,device_count() == 4,torch.cuda.is_available() == False).speechbrain/utils/distributed.pyfromdevelopand from thisbranch was loaded by file path on that host, and
ddp_init_group({"distributed_backend": "hccl"})was runthrough
torchrun --nproc_per_node=2:The repository's own test file was run on the same host against the real module (
pytest8.3.2, with thepackage skeleton in place):
8 passedwith this change. With thedevelopmodule and the three new tests,2 failed, 1 passed— the failure is the "Killing process" guard above.Not tested
torch.cuda.set_device) but was not executedon a GPU machine.
mpibackend.dependencies, so the real module and the repository's own tests were driven instead.