Pewarta Proxy

📋 Server Request & Proxy Logs

Loading active proxy settings...
Developed by Khairul Adha (adhexa.id)
Mengambil log server...

⚙️ Pengaturan Proxy Target

Skip to content

fix(distributed): initialise the DDP group on the active accelerator - #3116

Open
li-lizhe wants to merge 3 commits into
speechbrain:developfrom
li-lizhe:fix/ddp-init-active-accelerator
Open

li-lizhe wants to merge 3 commits into
speechbrain:developfrom
li-lizhe:fix/ddp-init-active-accelerator

Conversation

@li-lizhe

Copy link
Copy Markdown
Contributor

Problem

ddp_init_group() assumed the accelerator is CUDA:

  • the rank -> device guard was sized with torch.cuda.device_count(), and
  • the backend whitelist was nccl/gloo/mpi.

On a host whose accelerator is not CUDA (Ascend NPU through torch_npu, whose collective library is
hccl), torch.cuda.device_count() is 0, so every rank aborted with

ValueError: Killing process
Not enough GPUs available!

before init_process_group was ever reached — even with 4 NPUs visible — and hccl was rejected as a
backend that "doesn't exist" even though PyTorch reports it as available. This is the same device-agnostic
migration that infer_device() went through in #3080; the distributed launch path was left behind.

What

  • the device count for the guard, and the per-rank device binding, now come from
    torch.accelerator (current_accelerator().type, device_count());
  • the nccl/gloo/mpi checks stay as they are, and any other backend that
    torch.distributed.is_backend_available() reports as available is accepted (accelerators other than
    CUDA ship their own collective library, e.g. hccl for Ascend NPU).

CUDA hosts keep the previous behaviour: backend nccl and torch.cuda.set_device(device).

Verification

Real hardware: Ascend 910B host (aarch64, 4 NPUs, torch 2.15.0.dev, torch_npu 2.15.0.dev,
torch.accelerator.is_available() == True, current_accelerator() == npu, device_count() == 4,
torch.cuda.is_available() == False). speechbrain/utils/distributed.py from develop and from this
branch was loaded by file path on that host, and ddp_init_group({"distributed_backend": "hccl"}) was run
through torchrun --nproc_per_node=2:

develop: [rank 0] FAILED ValueError: Killing process / Not enough GPUs available!
         [rank 1] FAILED ValueError: Killing process / Not enough GPUs available!
branch : [rank 0] ddp_init_group OK | pg backend=hccl world_size=2
         [rank 1] ddp_init_group OK | pg backend=hccl world_size=2
         [rank 0] all_reduce sum=6.0 on npu:0   (two elements per rank, values rank+1)
         [rank 1] all_reduce sum=6.0 on npu:1

The repository's own test file was run on the same host against the real module (pytest 8.3.2, with the
package skeleton in place): 8 passed with this change. With the develop module and the three new tests,
2 failed, 1 passed — the failure is the "Killing process" guard above.

Not tested

  • A real CUDA/nccl host — the code path is written as before (torch.cuda.set_device) but was not executed
    on a GPU machine.
  • The mpi backend.
  • End-to-end speechbrain recipe training: that host has torch/torch_npu but not speechbrain's runtime
    dependencies, so the real module and the repository's own tests were driven instead.

`ddp_init_group` sized its rank -> device guard with `torch.cuda.device_count()` and
only accepted the nccl/gloo/mpi backends, so on a host whose accelerator is not CUDA
(e.g. Ascend NPU through torch_npu, which ships the 'hccl' collective library) every
rank aborted with

    ValueError: Killing process
    Not enough GPUs available!

before the process group could be created, and the accelerator's own backend was
rejected even when PyTorch reports it as available.

Size the guard and bind the rank's device through `torch.accelerator`, and accept any
backend `torch.distributed.is_backend_available` knows about. CUDA hosts keep the
previous behaviour (nccl plus `torch.cuda.set_device`).

Signed-off-by: li-lizhe <147392333@qq.com>
ruff-format (pre-commit) rewrites this unconditional raise into one line.

Signed-off-by: li-lizhe <147392333@qq.com>
The repository formats with ruff-format (line-length 80); wrap the guard expression,
the backend availability call and the new tests accordingly.

Signed-off-by: li-lizhe <147392333@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant