A practical guide to understanding why your multi-node GPU training might be slower than expected.
How to Calculate if Your Network is Bottlenecking Distributed Training
· 8 min read
3 posts tagged with "networking"
View All TagsGPU-to-GPU Communication Across Nodes: What Actually Works
· 6 min read
If you're building a multi-node GPU cluster for distributed training, you've probably run into a confusing mess of terminology — NVLink, NVSwitch, InfiniBand, RoCE, GPUDirect. Half the blog posts out there mix these up, and vendor documentation assumes you already know what you're doing.
So let's sort this out.
Understanding RX vs TX - Making Sense of Network Traffic Direction with Real Examples
· 5 min read
I was lucky to work on a computer vision setup which involved NVIDIA RTX 6000 GPUs, Mellanox ONYX switches, and high-resolution cameras. The system was designed for real-time video capture and processing, pushing massive amounts of data through our network infrastructure. Here's a gist of debugging Mellanox switch metrics that led to some surprising discoveries about network traffic flow.
When monitoring network equipment like switches, routers, or network cards, you'll constantly encounter two metrics: RX and TX. These simple abbreviations are fundamental to understanding how data flows through your network, yet they often cause confusion. Let's demystify them with real-world examples from our production Mellanox ONYX switch.