3 posts tagged with "gpu"

A practical guide to understanding why your multi-node GPU training might be slower than expected.

If you're building a multi-node GPU cluster for distributed training, you've probably run into a confusing mess of terminology — NVLink, NVSwitch, InfiniBand, RoCE, GPUDirect. Half the blog posts out there mix these up, and vendor documentation assumes you already know what you're doing.

So let's sort this out.

When monitoring GPU infrastructure, you might occasionally notice something peculiar: your high-end GPUs connected to PCIe 5.0 slots are operating at PCIe Gen 1 speeds. Before raising any alarms, let's walk through a systematic debugging approach that reveals this is often expected behavior rather than a configuration issue.