The SRAM vs HBM Paradigm Shift
While GPUs rely on high-bandwidth external memory (HBM) with hundreds of nanoseconds of access latency, Groq's Language Processing Unit (LPU) discards external DRAM entirely. Built on a Tensor Streaming Processor (TSP) design, each Groq chip contains 230 MB of ultra-fast on-chip SRAM providing an astonishing 80 TB/s aggregate memory bandwidth.
| Architecture Attribute | Groq LPU (First Gen) | NVIDIA H100 (SXM5) | Architectural Tradeoff |
|---|---|---|---|
| Memory Technology | 230 MB On-Chip SRAM | 80 GB External HBM3 | Groq: 10x faster memory, limited capacity |
| Memory Bandwidth | 80 TB/s (Local SRAM) | 3.35 TB/s (HBM3) | Zero memory access latency overhead |
| Execution Model | Software-Scheduled Deterministic | Hardware Warp Dynamic Scheduling | Groq knows exact instruction clock cycle |
| Llama 3 8B Output Speed | 500 ~ 750 tokens/sec | 120 ~ 180 tokens/sec | Groq delivers instantaneous human response |
| Board TDP | 185W (Standard PCIe) | 700W (SXM5) | Air cooled, fits standard 1U/2U servers |
| Cluster Scaling Requirement | ~576 chips for 70B model | 8 chips for 70B model | High chip count networking required |
Software-Defined Determinism
Groq eliminates hardware instruction caches, branch prediction, and memory controllers. The compiler determines the physical cycle-by-cycle movement of data across the silicon matrix. This creates deterministic latency with zero jitter, making it the premier architecture for voice AI, real-time code completion, and high-frequency agentic loops.