One GPU cluster,three kinds of traffic
Training, inference and developer work each need something different from the network. We design and configure it so each one gets its own path, from the switches up to the pods.
Each kind of traffic, and the path it takes
Training traffic runs straight on the switch fabric and never enters the pod network. Inference and developer traffic stay on the pod network.
Training
Gradients between GPUs on different servers, at every step of a job.
RDMA on a lossless path, at the full speed of the link.
Inference
Requests from users and applications, and the responses streamed back.
Low, steady latency, and one stable address in front of every replica.
Developer work
Notebooks, terminals and dashboards: many small sessions through the day.
Access for each person, and a boundary between teams.
Storage and out-of-band management run on networks of their own.
What we configure, from the switch to the pod
Switches and links
- Leaf and spine layout, link speeds and oversubscription
- A lossless class for RDMA, with PFC and ECN set the same way on switches and NICs
- MTU and DSCP matched end to end, ECMP across the spines
- On InfiniBand: subnet manager, partition keys and adaptive routing
Tenant and pod networks
- VXLAN segments and VRFs on the switches, where tenants share a fabric
- The pod network, with encapsulation that fits inside the underlay's MTU
- A load balancer and a gateway for traffic from outside
- A namespace and a network policy for each team
The second interface in a pod
- Host-device when one pod owns the whole NIC
- SR-IOV when several pods share a NIC, each isolated in hardware
- IPoIB on InfiniBand fabrics
- Macvlan for development and single-NIC servers
The GPU Cluster Audit
Two weeks on a cluster you already run. We test each path under load, fix what is safe to fix, and hand you the rest as a plan.
You get
- Test results from before and after
- What is wrong, in order of priority
- A fix plan, handed over to your team
Write-ups and open source
- network-operatorOne global configuration for the NIC cluster policy. MTU for IPoIB networks. Access mode for NIC firmware storage. Pod restarts on RDMA, SR-IOV and Multus changes.
- ipoib-cniMTU support.
- doca-driver-buildNFS tools on the host, for NFS over RDMA.
Changes we contributed to NVIDIA's open-source networking software.See the full list