Two years ago, an H100 cost $8 an hour, everywhere, take it or leave it. In October 2026, the same GPU — same silicon, same 80GB of HBM3, same CUDA cores — rents for $2.64 an hour on Spheron, $6.88 on AWS, $12.29 on Azure, and as little as $1.49 on the Vast.ai marketplace. The spread between the cheapest and most expensive H100 on the planet is roughly 8x.
That is not a functioning commodity market. It is a market in the middle of a violent repricing — and the force driving it is inference.
Training built the market. Inference is eating it.
For the first three years of the AI boom, GPU pricing was a training story. Labs bought H100s in 8-GPU nodes to train models, and the hyperscalers rationed supply at premium prices. Then the models shipped, and the workload flipped. Inference — serving models to users, millions of times a day — now accounts for roughly two-thirds of all AI compute in 2026, per Spheron’s market analysis. Models are trained once. They serve requests millions of times.
That shift broke the old pricing logic. Training is a batch workload: you need the GPUs for weeks, you can plan capacity, and reserved instances make sense. Inference is spiky, latency-sensitive, and brutally cost-per-token driven. A team serving a chatbot does not care about dollars per GPU-hour in the abstract. It cares about dollars per million tokens served — and that metric rewards throughput, not just cheap hours.
The numbers tell the story. An H100 on a hyperscaler runs $3.00 to $6.98 an hour; on a specialist GPU cloud, $1.49 to $2.99. At 22,290 tokens per second on Llama 70B, the price per million tokens lands around $0.019 to $0.037 on the cheap clouds versus $0.037 to $0.087 on hyperscalers. Same model, same speed, half the cost — before you factor in egress fees, which add roughly $2,600 to $3,600 a month per terabyte per day on the hyperscalers and nothing on several specialists.
The three tiers of the market
The GPU cloud market has stratified into three distinct layers, and buyers who shop in only one are overpaying.
Tier one: the hyperscalers. AWS p5.48xlarge at roughly $6.88 per H100-hour, Google Cloud A3 at $3.00 to $9.80 depending on region, Azure ND H100 v5 at $12.29. You are paying for compliance certifications, private networking, managed Kubernetes, and the procurement department’s comfort. For regulated industries and enterprises with existing cloud commitments, the premium buys something real. For a startup serving an API, it mostly buys a logo on the invoice.
Tier two: the GPU specialists. CoreWeave at $4.76 to $6.16, Lambda Labs at $2.99, Spheron at $2.64 on-demand and $2.20 spot. Same datacenter-grade hardware, dramatically lower prices, because these companies do one thing — run GPUs — instead of amortizing a global cloud empire. CloudZero’s tracking puts the median on-demand H100 rate at approximately $4.17 an hour on dedicated GPU clouds versus $7.89 on hyperscalers.
Tier three: the marketplaces. Vast.ai from $1.49, with a median around $2.00, auctioning spare capacity from independent datacenters. The cheapest silicon on earth, with availability that varies by the hour and support that varies by the host. Fine for batch jobs and experimentation; a gamble for production.
Two years of supply catching up with demand did this. The $8-an-hour H100 was a shortage price. The $2.64 H100 is much closer to the true cost of the hardware plus a reasonable margin — which tells you how much margin the shortage was hiding.
The sleeper pick and the real metric
Here is the detail most pricing guides bury: the H100 is no longer the obvious answer for inference. The L40S — 48GB of GDDR6, a fraction of the H100’s mystique — delivers 1,466 FP8 TFLOPS and rents for around $0.96 an hour on Spheron. For inference workloads that can use FP8 precision, it is surprisingly competitive at roughly a third of the H100’s price. The H200, at $5.51 an hour, serves larger models per card with 141GB of HBM3e; its throughput advantage often pulls the price-per-token back in line with the cheaper H100.
The durable lesson: judge on cost per token, not cost per hour. A pricier card that serves more tokens usually wins. The teams optimizing GPU spend in 2026 think in tokens per dollar, route queries across model tiers dynamically, and treat the hourly rate as an input — not the answer.
The counterargument: why anyone still pays $12
If the market is so inefficient, why does Azure’s $12.29 H100 have customers? Because the hourly rate is not the total cost for everyone. Enterprises with committed cloud spend get effective rates far below list. Regulated workloads need the compliance posture only hyperscalers offer. And multi-year reserved instances — 1 to 3 year commitments — open access to the hyperscalers’ genuinely competitive pricing, just not at the on-demand rates that make headlines.
There is also the switching cost nobody models: rearchitecting around a new provider, requalifying security, retraining the platform team. For a team spending $50,000 a month on GPUs, a 40% saving is $20,000 a month — worth a migration. For a team spending $3,000, it is $1,200 — possibly not worth the engineering time. The rational buyer is not always the cheapest buyer.
What this means for infrastructure budgets
The inference price war has one clear implication: GPU spend is now an optimization problem, not a procurement problem. The winners run a portfolio — reserved capacity for the steady-state baseline, spot and marketplace for batch and burst, per-token serverless APIs for the long tail. They benchmark quarterly, because the provider that was cheapest in April may not be cheapest in October. In a market moving this fast, the pricing page you bookmarked six months ago is a historical document.
Two years ago, the question was whether you could get GPUs at all. Today it is whether you are paying double for the same silicon. In 2026, nobody should be paying $8 an hour for an H100. The market has spoken — loudly, in about six different prices.
The definitive source for data center pricing intelligence is still up for grabs. DataCenterPrices.com is available for acquisition by the company that will own this category — inquiries to info@primedomaincapital.com.