The Monopoly Tax is Getting Old

I spent three hours yesterday trying to provision a single H100 instance on AWS. Three hours. For one node. When I finally got it, the hourly rate made me physically wince.

This is the reality of machine learning right now. We are all trapped in Jensen Huang’s walled garden, paying whatever Nvidia decides to charge because CUDA is the only game in town. Or at least, it was.

When the news dropped recently about Meta signing a $100 billion AI chip deal with AMD, my feed filled up with finance bros talking about market caps and supply chains. I don’t care about any of that. I care about what happens when I type import torch.

And honestly? This deal is the best thing to happen to ML engineers in five years.

The Hardware Was Never the Problem

Let’s get one thing straight. AMD hardware has been fast for a while. The Instinct MI300X accelerators are absolute monsters on paper. But nobody wanted to use them.

Why? Because ROCm—AMD’s answer to CUDA—used to trigger PTSD in anyone who tried to deploy models in production. Two years ago, trying to run a complex training loop on AMD hardware meant fighting obscure C++ compilation errors at 2 AM. It was a mess. You’d spend more time debugging memory allocation faults than actually training your model.

But Meta isn’t just buying chips. They are the primary maintainers of PyTorch. When a company drops $100 billion on AMD silicon to train Llama 4 and power their internal recommendation engines, they are going to make damn sure the software works.

And it’s already happening.

Benchmarking the Reality

I managed to get access to a staging cluster with 8x MI300X cards last Tuesday. I wanted to see if the software stack had actually caught up to the hype. I pulled down a Llama-3-70B-Instruct model, loaded up PyTorch 2.5.1, and ran a standard inference benchmark.

My expectations were low. I figured I’d hit a wall with custom kernels.

I was wrong. It just ran.

Here are the actual numbers. On our standard H100 nodes, we typically see about 34 tokens per second per user at batch size 16. On the MI300X cluster? 32 tokens per second. The performance difference was barely statistical noise.

But

Common questions

How does AMD MI300X performance compare to Nvidia H100 for Llama 3 inference?

Benchmarking Llama-3-70B-Instruct on an 8x MI300X cluster with PyTorch 2.5.1 produced roughly 32 tokens per second per user at batch size 16. Standard H100 nodes delivered about 34 tokens per second under the same conditions. The gap is barely statistical noise, indicating the MI300X now offers real inference performance parity with Nvidia’s flagship training accelerator for large language model workloads.

Why did ML engineers historically avoid AMD GPUs despite strong hardware specs?

AMD’s Instinct accelerators looked powerful on paper, but the ROCm software stack made production deployment miserable. Two years ago, running a complex training loop on AMD hardware meant fighting obscure C++ compilation errors at 2 AM and debugging memory allocation faults instead of training models. The hardware was never the bottleneck—the toolchain was—so teams stayed locked inside Nvidia’s CUDA ecosystem despite the cost.

Why does Meta’s $100 billion AMD deal matter for PyTorch users specifically?

Meta is the primary maintainer of PyTorch, so a $100 billion commitment to AMD silicon for training Llama 4 and powering internal recommendation engines forces them to make the software stack genuinely work on AMD. That investment pressure translates directly into better ROCm integration, smoother kernels, and reliable AMD support inside PyTorch itself—benefits every ML engineer importing torch inherits automatically.

Can you run Llama 3 on AMD MI300X without writing custom kernels?

According to hands-on testing on a staging cluster with 8x MI300X cards, Llama-3-70B-Instruct loaded and ran under PyTorch 2.5.1 without hitting a wall on custom kernels. The author expected compatibility problems and was surprised when the standard inference benchmark simply executed. The software stack has caught up enough that typical PyTorch workflows now function on MI300X out of the box.