9 posts tagged with "gpu"

The fabric looks non-blocking on paper. A k=8 fat-tree, full bisection bandwidth, sixteen equal-cost paths between any two pods, ECMP enabled everywhere to use them. Then the first real training job lands and AllReduce sustains something like 60% of line rate, with a handful of spine uplinks pinned and others barely warm.

The usual explanation is that all-to-all traffic floods the fabric with so many flows that some are bound to collide. That has the mechanism backwards. ECMP copes well with many flows. It falls apart precisely because a collective gives it almost nothing to hash on.

Every GPU cluster design review reaches the same slide: the nodes and the leaf switch are in one rack, nothing is further apart than about three metres, and somebody has to decide what goes in the cable order. The textbook answer is passive Direct Attach Copper, and the textbook answer is usually right - lowest power, lowest cost, one fewer active component per link.

The part that bites is the number people carry around in their head. "DAC does 5 metres" was true at EDR. At NDR it is not, and a rack of 800G links planned on that assumption produces a cable order that does not physically reach.

If you run GPU workloads on Kubernetes at any meaningful scale, you've probably hit a point where the default scheduler isn't enough. Fractional GPU requests, quota enforcement, gang scheduling, preemption - none of that comes out of the box. That's the gap KAI Scheduler fills.

KAI Scheduler is NVIDIA's open-source Kubernetes-native GPU scheduler, originally built inside the Run:ai platform and released under the Apache 2.0 license in April 2025. It's now a CNCF Sandbox project with over 1,200 GitHub stars, and it's quickly becoming the go-to scheduler for teams running AI workloads on Kubernetes - whether on-prem, colo, or cloud.

At BaaZ, we work with KAI Scheduler in production GPU clusters. We recently contributed a queue validation webhook (PR #857) that prevents a class of misconfiguration bugs in hierarchical queue setups. This post explains the problem, the fix, and why it matters for anyone operating multi-tenant GPU infrastructure.

How to set up a management network and a dedicated RDMA training network inside Kubernetes pods - from hardware discovery to working RDMA verbs, on a bare-metal GH200 Grace Hopper superchip.

A practical guide to understanding why your multi-node GPU training might be slower than expected.

Macvlan, host-device, SR-IOV, and IPoIB - what they are, how they differ, and when to use each for RDMA and NCCL traffic in GPU training clusters.

If you've ever written a NicClusterPolicy manifest for the NVIDIA Network Operator, you know the pain: the same repository, version, and imagePullSecrets copied and pasted across every single sub-component. OFED driver, RDMA shared device plugin, SR-IOV device plugin, Multus, CNI plugins, IPAM plugin, NV-IPAM - each one needs its own repository: nvcr.io/nvidia/mellanox and version: network-operator-v25.7.0. Change the version during an upgrade, and you're editing 8+ places in the same YAML. Miss one, and you get a partially upgraded cluster with mismatched component versions.

We recently contributed a fix for this: global config support for NicClusterPolicy (PR #2070). It's now merged into the NVIDIA Network Operator, and this post explains the problem, the implementation, and why it matters for anyone operating RDMA-capable GPU clusters on Kubernetes.

If you're building a multi-node GPU cluster for distributed training, you've probably run into a confusing mess of terminology - NVLink, NVSwitch, InfiniBand, RoCE, GPUDirect. Half the blog posts out there mix these up, and vendor documentation assumes you already know what you're doing.

So let's sort this out.

When monitoring GPU infrastructure, you might occasionally notice something peculiar: your high-end GPUs connected to PCIe 5.0 slots are operating at PCIe Gen 1 speeds. Before raising any alarms, let's walk through a systematic debugging approach that reveals this is often expected behavior rather than a configuration issue.