3 hours ago
5
This Tom's Hardware Premium article is free to read with a Tom's Hardware account; no payment necessary. We're offering free access from August 23 to 26 so you can read all of our reporting from Hot Chips.
Groq's former chief architect stood on stage at Hot Chips 2026 and presented his former company's inference chip as Nvidia silicon. Igor Arsovski, now Nvidia's VP of hardware, presented the Groq 3 LPX rack's architecture and published the first third-party benchmark of the hardware: Artificial Analysis measured it at 3,431 output tokens per second on a 100K-context Gemma 4 31B reasoning workload, roughly four times the 870 tokens per second of the next-fastest public endpoint. Arsovski said the rack is already in production, built on the LP30 chip Nvidia obtained through its $20 billion Groq deal in December 2025, the same deal that pushed the Rubin CPX it replaced off Nvidia's roadmap.
SRAM without HBM
Artificial Analysis ran the comparison on a private, pre-release Gemma 4 31B endpoint served through Google Cloud, taking the median of 50 sequential client requests at a concurrency of one, while the public providers it measured against ran shared production serverless endpoints. Serving one request at a time produces the highest per-user token rate the hardware can post, and it's not directly comparable to the multi-tenant conditions the other endpoints run under.
Nvidia's on-stage demo showed a higher figure still, 10,996 tokens per second on the same 31B model, which Igor Arsovski, Nvidia's VP of hardware, flagged on stage as "self-reported" before telling the audience the aim was "third-party verified independent benchmarks that you guys can trust." Gemma 4 31B is also a dense model small enough to sit inside a single LPX rack, and the picture at trillion-parameter mixture-of-experts scale, where memory capacity becomes the main constraint, went unaddressed.
Each LP30 carries roughly 500MB of on-die SRAM and no HBM, so a full LPX rack of 256 chips holds 128GB of memory delivering 40 PB/s of aggregate bandwidth against 315 PFLOPS of FP8 compute, with 350 ns of chip-to-chip latency in a Vera Rubin-compatible, MGX liquid-cooled rack that scales past 1,000 LPUs.
Keeping model weights resident in SRAM rather than streaming them from HBM removes the memory-access latency that dominates single-token decode, and the design drops caches, branch prediction, and out-of-order execution in favor of a fully deterministic pipeline that the compiler schedules at clock-cycle granularity. The architecture descends directly from the Tensor Streaming Processor that Groq, founded by ex-Google TPU engineer Jonathan Ross, described in a 2020 ISCA paper titled Think Fast, the same title Arsovski and Raghavan reused at Hot Chips.
A Rubin GPU carries 288GB of HBM4, roughly 576 times the memory of a single LP30, so a 31-billion-parameter model at FP8 needs on the order of 62 LPUs to hold its weights, and a large mixture-of-experts model runs into four figures of chips across several racks. Capacity is the cost of the SRAM-only design, and it's why Nvidia is describing the LPU as for decode rather than as a general-purpose replacement for its GPUs.
Determinism lets the compiler predict power draw cycle by cycle, which Nvidia uses to pre-order current from the rack's regulators ahead of demand, cutting voltage droop by more than 60% and overshoot by more than 70% against an uncompensated load. The same per-block scheduling lets the hardware equalize heat instead of throttling to the hottest tile, which Arsovski put at roughly 10% to 11% additional performance under a fixed thermal limit. "By doing this, we can actually get more utilization of the chip under the same thermal limit, basically. So we can actually get, again, about 10 to 11% more performance under the same thermal limit. So this is another benefit of deterministic execution."
Across racks, Nvidia synchronizes chips to a single virtual clock in what it calls a plesiosynchronous network, with each chip acting as both processor and router so the fabric needs no adaptive routing or congestion sensing, and clock drift between chips is compensated at the chip-to-chip links. Asked during Q&A about the blast radius of a chip that fails mid-workload, Arsovski said users "would experience the exact same as any other hardware in the industry" and would "just checkpoint it or reconfigure the hardware."
Splitting inference with Rubin
Nvidia is pitching the LPX rack as a decode co-processor bolted onto Vera Rubin NVL72, with Rubin GPUs handling the compute-heavy prefill phase and building the KV cache while the LPUs generate output tokens. Nvidia showed three ways to divide the work: disaggregated prefill and decode; attention-FFN disaggregation, which keeps attention and its cache on GPU HBM while the LPU runs the feed-forward layers; and external-draft speculative decoding, where a small model on the LPU proposes tokens that the GPU verifies in parallel, with only draft tokens crossing the link.
An FPGA bridges the synchronous LPU domain and the asynchronous world of host I/O and GPU hand-offs, and Nvidia's Dynamo runtime, together with an LPU extension to CUDA, orchestrates the split. The company put the gains from these modes at roughly three-to-five-times over Rubin alone on a two-trillion-parameter workload with a 400K-token cached context, all Nvidia-measured.
Cerebras CS4
Cerebras used the same Hot Chips session to present its CS4 wafer-scale system, which chief system architect Jean-Philippe Fricker said runs up to 30 times faster than GPUs and doubles the token rate of the CS3 while carrying 10 times the token capacity. Each CS4 rack packs three wafer-scale engines into a new modular platform Cerebras calls Nexus, built around pluggable compute "backpacks" that separate power, compute, and I/O, and Fricker put its memory bandwidth at 43 PB/s, which he told the audience was "2,000 times higher memory bandwidth than Nvidia's next-generation Rubin chip." Cerebras also has a partner for the prefill side of the same problem: it agreed in July to pair AMD Helios GPUs for prefill with its wafer-scale engines for decode, the same division of labor Nvidia now builds in-house with Groq.
Nvidia pulled the Rubin CPX, its own GDDR7-based long-context accelerator, to focus on shipping the LPU this year, a decision VP Ian Buck laid out at GTC 2026. The $20 billion deal that produced the LP30 was structured as a non-exclusive IP license plus the hiring of Ross, president Sunny Madra, and most of Groq's engineers, a form that avoided a formal merger review. Arsovski opened the Hot Chips talk by calling it "a pinch me moment for the Groq team that's now integrated into the Nvidia group."
Senators Elizabeth Warren and Richard Blumenthal wrote to the FTC and to Nvidia in early 2026, arguing the arrangement acquired Groq "in all but name," and no formal, deal-specific investigation has been confirmed as of late August.
Full Nvidia Groq Hot Chips 2026 presentation
Image 1 of 44
English (US) ·