Skip to main content
GPU cluster networking on Kubernetes

One GPU cluster,three kinds of traffic

Training, inference and developer work each need something different from the network. We design and configure it so each one gets its own path, from the switches up to the pods.

Book a GPU Cluster Audit

Each kind of traffic, and the path it takes

Training traffic runs straight on the switch fabric and never enters the pod network. Inference and developer traffic stay on the pod network.

USERS AND APPS DEVELOPERS REQUESTSSESSIONS GATEWAY KUBERNETES CLUSTER GPU NODE GPU NODE eth0 eth0 net1 net1 POD NETWORK · OVERLAY MODEL SERVER TRAINING POD TRAINING POD NOTEBOOK NIC NIC SWITCH FABRIC · UNDERLAY INFINIBAND OR ROCE LEAFSPINELEAF RDMA 123
USERS AND APPS DEVELOPERS REQUESTSSESSIONS GATEWAY KUBERNETES CLUSTER GPU NODE GPU NODE eth0 net1 eth0 net1 POD NETWORK · OVERLAY MODEL SERVER TRAINING POD NIC TRAINING POD NIC NOTEBOOK UNDERLAY SWITCHES LEAFSPINELEAF 123
1East-west

Training

What it carries

Gradients between GPUs on different servers, at every step of a job.

What it needs

RDMA on a lossless path, at the full speed of the link.

2North-south

Inference

What it carries

Requests from users and applications, and the responses streamed back.

What it needs

Low, steady latency, and one stable address in front of every replica.

3North-south

Developer work

What it carries

Notebooks, terminals and dashboards: many small sessions through the day.

What it needs

Access for each person, and a boundary between teams.

Storage and out-of-band management run on networks of their own.

What we configure, from the switch to the pod

Underlay

Switches and links

  • Leaf and spine layout, link speeds and oversubscription
  • A lossless class for RDMA, with PFC and ECN set the same way on switches and NICs
  • MTU and DSCP matched end to end, ECMP across the spines
  • On InfiniBand: subnet manager, partition keys and adaptive routing
Overlay

Tenant and pod networks

  • VXLAN segments and VRFs on the switches, where tenants share a fabric
  • The pod network, with encapsulation that fits inside the underlay's MTU
  • A load balancer and a gateway for traffic from outside
  • A namespace and a network policy for each team
Secondary networks

The second interface in a pod

  • Host-device when one pod owns the whole NIC
  • SR-IOV when several pods share a NIC, each isolated in hardware
  • IPoIB on InfiniBand fabrics
  • Macvlan for development and single-NIC servers
The offer

The GPU Cluster Audit

Two weeks on a cluster you already run. We test each path under load, fix what is safe to fix, and hand you the rest as a plan.

You get

  • Test results from before and after
  • What is wrong, in order of priority
  • A fix plan, handed over to your team
Book a GPU Cluster Audit