The Total Cost of Ownership (TCO) Formula
Evaluating AI hardware purely on chip acquisition price (CAPEX) is an engineering trap. Across a 3-year amortization window, facility power, cooling, network switches, and rack space (OPEX) account for 45% to 60% of total expenditure.
$$\text{TCO} = \text{CAPEX}_{\text{Hardware}} + (\text{Power}_{\text{IT}} \times \text{PUE} \times \text{Electricity Rate} \times 8760 \times 3) + \text{Facility Maintenance}$$
| Accelerator Platform | 8-GPU System CAPEX | Full Rack Power | 3-Year Amortized TCO | Approx. Cost / 1M Tokens (Llama 70B) |
|---|---|---|---|---|
| NVIDIA GB200 NVL72 | ~$3,000,000 / rack | ~120 kW | ~$4,850,000 | $0.12 ~ $0.16 (Lowest at high concurrency) |
| NVIDIA H100 HGX (8x) | ~$300,000 | ~10.2 kW | ~$480,000 | $0.45 ~ $0.55 |
| NVIDIA H200 HGX (8x) | ~$360,000 | ~10.2 kW | ~$540,000 | $0.28 ~ $0.35 |
| AMD MI300X (8x) | ~$220,000 | ~11.0 kW | ~$395,000 | $0.24 ~ $0.30 |
| Groq LPU Cluster (Rack) | ~$450,000 | ~14.0 kW | ~$660,000 | $0.35 (Lowest instantaneous latency) |
Key Infrastructure Takeaways
- Memory Density Dictates Cost: Chips with 141GB+ HBM (H200, MI300X, B200) achieve dramatically lower token cost because they can batch multiple user queries simultaneously without running out of VRAM.
- Liquid Cooling Lowers PUE: Moving from PUE 1.35 (air) to PUE 1.08 (liquid) on a 1 MW AI cluster saves over $240,000 annually in pure electricity costs.