A PyTorch backend lets Mac GPUs train together over Ethernet
MCCL: Training Pytorch models across a fleet of Macs at Skywalker Sound · arXiv:2610.39029
In plain words
PyTorch lacked a way to share Apple GPU training work across Macs without copying gradients to CPU memory and interrupting the GPU. MCCL adds a PyTorch backend that lets CPU threads average gradients in memory shared with the GPU while GPU work continues.
In one test, a Vision Transformer trained on 24 Macs processed 4,568 samples per second, reaching 95.9% of the throughput expected from scaling up from one Mac. That result is for this model and setup.
The research
Could Macs running Apple GPUs train together through PyTorch without interrupting GPU work to copy gradients to CPU memory? MCCL addresses this gap with a backend that averages gradients in shared memory while GPU computation continues.
Its tests measure network use, communication overlap and training performance on clusters of up to 24 Macs.
MCCL adds a PyTorch backend that averages Mac GPU gradients in shared memory while training continues, enabling distributed training over Ethernet.
MCCL averages gradient buckets in shared memory while the GPU continues computing, with Metal events ordering each handoff. On a 113.5-million-parameter Vision Transformer at batch 256 per Mac, throughput reached 4,568 samples per second on 24 Macs, or 95.9% of linear scaling.
On the same 24-Mac setup, MCCL reached higher throughput than a manual Gloo reducer that copied gradients to CPU memory. Together, these results show the backend can support efficient multi-Mac training in the tested setting.
These results come from tests of up to 24 Macs on one 10 Gb Ethernet switch. Communication is harder to hide when a model has a short backward pass or sends small, frequent messages; the reported training results also cover specific models and batch settings.
What the evidence shows
- A missing PyTorch backend
PyTorch did not have a distributed backend for Apple GPU tensors. Its available Gloo backend operates on CPU memory, so manually copying gradients to use it interrupts GPU work during each exchange.
The Metal Communication Collective Library (MCCL) is an open-source PyTorch backend built to let unmodified distributed training run across Macs over switched Ethernet.
- A shared-memory handoff
MCCL’s central change is where and when gradients are averaged. CPU threads work directly on gradient buckets in memory shared with the GPU, while the GPU continues its backward computation.
Metal shared events order the handoff so the CPU reads a bucket only after the GPU has written it, and the optimizer waits for averaging to finish.
- Large exchanges fill the link
The network measurements test whether switched Ethernet could carry large exchanges efficiently as more Macs joined. For 16–256 MB all-reduce buckets, measured bus bandwidth was 98–99.7% of the TCP payload ceiling across 4–24 Macs.
The result applies to those large buckets; smaller messages face a different constraint because switch latency dominates.
- Overlap depends on the handoff
A synthetic training step tested whether the in-place handoff kept communication hidden behind GPU computation. About 94% of communication time overlapped with backward computation from 4 to 24 Macs.
When the test instead copied buckets through a staging buffer and waited, overlap dropped to zero. This isolates overlap in that synthetic setup, rather than measuring full training performance.
- Training results have bounds
In a 113.5-million-parameter Vision Transformer trained with batch 256 per Mac, throughput reached 4,568 samples per second on 24 Macs, or 95.9% of linear scaling. On that same setup, MCCL also exceeded the throughput of a manual Gloo reducer.
Separate ResNet runs converged across tested cluster sizes, but both global batch size and learning rate increased with Mac count, so that comparison does not isolate the effect of adding Macs.
Deep
- Original paper authors
- Kevin Sasso, Chris Fiegel, Francis Aitken, Nicolas Tsingos, Nicholas Tong
- Submitted to arXiv
- License
- arXiv non-exclusive