8 posts tagged with "networking"

The fabric looks non-blocking on paper. A k=8 fat-tree, full bisection bandwidth, sixteen equal-cost paths between any two pods, ECMP enabled everywhere to use them. Then the first real training job lands and AllReduce sustains something like 60% of line rate, with a handful of spine uplinks pinned and others barely warm.

The usual explanation is that all-to-all traffic floods the fabric with so many flows that some are bound to collide. That has the mechanism backwards. ECMP copes well with many flows. It falls apart precisely because a collective gives it almost nothing to hash on.

Every GPU cluster design review reaches the same slide: the nodes and the leaf switch are in one rack, nothing is further apart than about three metres, and somebody has to decide what goes in the cable order. The textbook answer is passive Direct Attach Copper, and the textbook answer is usually right - lowest power, lowest cost, one fewer active component per link.

The part that bites is the number people carry around in their head. "DAC does 5 metres" was true at EDR. At NDR it is not, and a rack of 800G links planned on that assumption produces a cable order that does not physically reach.

How to set up a management network and a dedicated RDMA training network inside Kubernetes pods - from hardware discovery to working RDMA verbs, on a bare-metal GH200 Grace Hopper superchip.

A practical guide to understanding why your multi-node GPU training might be slower than expected.

Macvlan, host-device, SR-IOV, and IPoIB - what they are, how they differ, and when to use each for RDMA and NCCL traffic in GPU training clusters.

If you've ever written a NicClusterPolicy manifest for the NVIDIA Network Operator, you know the pain: the same repository, version, and imagePullSecrets copied and pasted across every single sub-component. OFED driver, RDMA shared device plugin, SR-IOV device plugin, Multus, CNI plugins, IPAM plugin, NV-IPAM - each one needs its own repository: nvcr.io/nvidia/mellanox and version: network-operator-v25.7.0. Change the version during an upgrade, and you're editing 8+ places in the same YAML. Miss one, and you get a partially upgraded cluster with mismatched component versions.

We recently contributed a fix for this: global config support for NicClusterPolicy (PR #2070). It's now merged into the NVIDIA Network Operator, and this post explains the problem, the implementation, and why it matters for anyone operating RDMA-capable GPU clusters on Kubernetes.

If you're building a multi-node GPU cluster for distributed training, you've probably run into a confusing mess of terminology - NVLink, NVSwitch, InfiniBand, RoCE, GPUDirect. Half the blog posts out there mix these up, and vendor documentation assumes you already know what you're doing.

So let's sort this out.

I was lucky to work on a computer vision setup which involved NVIDIA RTX 6000 GPUs, Mellanox ONYX switches, and high-resolution cameras. The system was designed for real-time video capture and processing, pushing massive amounts of data through our network infrastructure. Here's a gist of debugging Mellanox switch metrics that led to some surprising discoveries about network traffic flow.

When monitoring network equipment like switches, routers, or network cards, you'll constantly encounter two metrics: RX and TX. These simple abbreviations are fundamental to understanding how data flows through your network, yet they often cause confusion. Let's demystify them with real-world examples from our production Mellanox ONYX switch. If you're operating GPU infrastructure, this is exactly the kind of low-level visibility you need - it's a recurring theme in our GPU monitoring and observability engagements.