Hot Chips 2026: OpenAI's Jalapeño AI ASIC unpacked — accelerator developed using AI achieves efficiency and throughput gains against power-hungry Blackwell

5 hours ago 6

OpenAI made quite a splash back in June, when it unveiled its 'Jalapeño' AI accelerator and revealed that the chip reached tape-out in just nine months. At the Hot Chips conference, OpenAI disclosed more details about the architecture of its Jalapeño inference processor as well as shared its target and real-world performance numbers. The company claims its NUMA-style spatial architecture enables Jalapeño to outperform Nvidia's GB200 and GB300 in low-latency inference and in terms of performance-per-watt, while a 2,048-processor system scales to 27 exaFLOPS and 32 PB/s of aggregate memory bandwidth.

OpenAI's Jalapeño is a massive AI inference accelerator co-developed with Broadcom, with 216 GB of HBM4 memory and up to 15.4 TB/s of bandwidth. The processor delivers up to 3.4 MXFP8 PFLOPS as well as up to 13.4 MXFP4 PFLOPS at 700W, which makes it suitable for inference, though the MXFP4 format may not be enough for training. The ASIC has a 700W power rating and operates at 1.70 GHz on silicon already running in OpenAI's labs, though OpenAI's engineers said at Hot Chips that the plan is to increase clocks up to 1.80 GHz, perhaps to get higher peak performance.

On the scalability side of matters, Jalapeño can scale to 128 accelerators in a local rack interconnected using Ethernet at 600 GB/s and to 2,048 ASICs in a 16-rack pod configuration at 200 GB/s per processor. The complete system offers 27 EFLOPS of MXFP4 performance, 432 TB of HBM4 memory, and 32 PB/s of aggregate memory bandwidth. For connectivity, OpenAI uses Broadcom Tomahawk 6 Ethernet switches and what it calls a 'half-flattened' two-level Clos topology that provides higher bandwidth for tensor-parallel traffic, lower bandwidth for expert-parallel communication, and prioritizes low latency for both. During the Q&A session, OpenAI confirmed that the scale-up network uses Ethernet with 200-Gb/s links. Jalapeño's physical hardware around the Broadcom-made chip is set to be made by Celestica.

The company says a 128-ASIC Jalapeño domain has more than 1 PB/s of aggregate HBM4 bandwidth, which is significantly higher compared to GB300 NVL72 (576 TB/s). A one-trillion-parameter model using FP4 weights requires about 0.5 TB. So, purely from a bandwidth perspective, the system could read the entire model more than 2,000 times per second. Meanwhile, actual inference performance comes nowhere near that theoretical ceiling, which is why hardware developers do not tend to add HBM bandwidth infinitely.

Instead, Jalapeño uses what OpenAI describes as a memory-sliced, or NUMA-style, architecture. The chip has 64 core slices, and each of them is paired with its own HBM slice to guarantee predictable latency and bandwidth. OpenAI says this arrangement avoids conflicts associated with a unified memory subsystem and lets frequently used operands remain close to the compute resources that need them. OpenAI does not explain why it chose exactly 64 slices, which likely means it was a sweet spot for the current architecture.

To connect the 64 core slices, OpenAI uses a specialized high-bandwidth, low-latency collective network that moves data coupled to compute operations 'register-to-register, with zero conflicts' in a bid to eliminate several performance bottlenecks. In addition, Jalapeño has a separate general-purpose network-on-chip (NoC) that handles less common communication (e.g., remote/global memory accesses) and provides access to the external scale-up network.

The distinction between these fabrics is substantial. OpenAI describes the general NoC as deliberately more 'anemic than you would expect from a normal chip architecture' as it does not want performance-critical traffic to use it typically. By contrast, the ultra-fast collective network organizes data movement, so operands arrive in registers when needed rather than leaving compute engines and then waiting for memory, network traffic, or global synchronization, which creates internal performance bottlenecks.

To explain how it works, let us compare OpenAI's and Nvidia's approaches. Nvidia's standard system-level decomposition is primarily two phases: prefill and decode. Prefill is generally compute-bound (which is why Nvidia tried to assign Rubin CPX with GDDR7 for this one), while decode is generally memory-bound (which is why Nvidia wants to keep GPUs with HBM for this). By contrast, OpenAI breaks the process into three phases: prefill (compute-bound), draft (latency-bound), and verification (bandwidth-bound).

In theory, OpenAI could have assigned each phase to a different type of specialized accelerator. However, the amount of prefill, drafting, and verification changes depending on the model, context length, token efficiency, and software algorithms. As a result, it is hard to predict the number of processors that must be deployed, so it's inevitable that some phases would sit idle. To that end, it makes more sense to develop one balanced ASIC that can do everything.

Despite Jalapeño's exceptionally short development cycle, OpenAI says that most of the processor was designed from scratch rather than assembled from existing Broadcom accelerator IP. OpenAI's Richard Ho said the compute die reuses some interface IP, but most of its Register Transfer Level (RTL) was newly written using XLS and Verilog. Meanwhile, development moved remarkably quickly: initial RTL work began in February 2025, the design taped out in November, first silicon arrived in May 2026, and OpenAI had Codex running on Jalapeño that same month.

One reason OpenAI was able to move so fast with its development was its extensive use of AI to assist and optimize the design. More than half of the core was written using the XLS hardware language and compiler infrastructure, while OpenAI's own AI models searched for ways to improve power, performance, and area (PPA). Compared with human 'baseline' designs, OpenAI reports improvements of 56% for a BF16 multiplier, 21% for an FP4 dot-product block, and 10% for an FP32 accumulator, along with 10% and 8% area reductions for the matrix and SIMD units. The company says that AI-assisted optimization even helped squeeze circuitry into a floorplan block that otherwise would not have fit, though OpenAI remains tight-lipped about what circuitry it was.

Because Jalapeño features its unique spatial and sliced architecture, it is programmed differently from Nvidia's CUDA GPUs. OpenAI says Jalapeño can be programmed using a low-level programming environment in the open-source Triton ecosystem. Unlike the latest versions of CUDA, which ensure that its code runs on all Nvidia GPUs and aligns with the tensor-heavy execution model of Blackwell processors and their successors, Jalapeño gives software more explicit control over where data and computation are placed.

Each core has fast access to its local portion of HBM; these cores are interconnected using an ultra-fast network, so the software must be able to determine where tensors are physically located and how they are distributed across the chip. This makes programming the spatial architecture more complicated, so OpenAI also uses AI to find efficient data placement, scheduling, and communication patterns and to optimize kernels for the hardware.

Meanwhile, optimal mapping is architecture-dependent, so once OpenAI changes the number of cores, local-memory organization, collective-network topology/bandwidth, or compute resources in next generations of its accelerators, the old placement and scheduling may no longer be optimal and will require AI tools to perform hardware-specific optimizations again. By contrast, software written for Blackwell will work on Rubin and then Feynman without modifications.

Instead of comparing peak performance numbers, OpenAI used SemiAnalysis' InferenceX benchmark to compare Jalapeño and Nvidia's GB200/GB300 across their complete latency-versus-throughput curves. The company measured how many tokens each system could deliver at comparable user-perceived latency and normalized the results by package power — 700W for Jalapeño, 1,200W for GB200, and 1,400W for GB300 — meaning that while Nvidia's hardware can lead in terms of absolute performance, OpenAI's accelerator leads in efficiency.

Image 1 of 9

Roughly 1.5X – 1.9X higher peak throughput per watt
1.7X – 3.6X lower end-to-end latency
2.1X – 4.1X lower minimum time between tokens (TBT)
At Nvidia's minimum-TBT operating points, Jalapeño can deliver up to 104.3X higher throughput.
This methodology particularly favors Jalapeño's strength at low latency, so the result is significantly inflated. The 104.3X figure means that at GB300's lowest-latency operating point, Jalapeño can maintain 104.3X higher throughput, not that Jalapeño is 104.3X faster overall.
OpenAI also noted that its internal models show an even larger advantage for the Jalapeño platform and says applying multi-token prediction to Jalapeño can improve latency by another 3X to 5X at equivalent efficiency.

OpenAI used Hot Chips 2026 to fully detail Jalapeño, its first custom inference AI accelerator co-developed with Broadcom. The ASIC relies on a NUMA-style architecture built around 64 memory/core slices, carries 216 GB of HBM4 memory, and has compute performance of up 13.4 MXFP4 PFLOPS at 700W. Being aimed at AI data centers, Jalapeño scales from 128 accelerators per rack to 2,048 inference processor per pod and can provided up to 27 EFLOPS of MXFP4 compute, 432 TB of HBM4, and 32 PB/s of aggregate memory bandwidth per cluster.

While absolute performance of Jalapeño may fall short of what Nvidia's Blackwell or Rubin offer, the company argues that the main advantage of its AI inference accelerator platform is its performance-per-watt achievements as well low latency. OpenAI says its Jalapeño delivers roughly 1.5X – 1.9X higher peak peak-performance-per watt and 1.7X – 3.6X lower end-to-end latency than Nvidia GB200/GB300 in its InferenceX comparisons.

Perhaps the most impressive, or maybe even terrifying, though certainly not unexpected thing that OpenAI revealed is that AI played a major role in Jalapeño's unusually fast nine-month RTL-to-tapeout development cycle and also enabled the company to improve performance, power, and area, of the design. Jalapeño's successor is already heading toward tapeout and OpenAI's 3^rd Generation AI accelerator is already in development.

Image 1 of 36

Anton Shilov is a contributing writer at Tom’s Hardware. Over the past couple of decades, he has covered everything from CPUs and GPUs to supercomputers and from modern process technologies and latest fab tools to high-tech industry trends.

Read Entire Article