The open-source benchmark scores AI agents on real Stripe tasks, and the results show plenty of room for improvement
Stripe has released a public benchmark that grades AI coding agents on one specific job: building Stripe integrations that actually work.
The project, called Stripe evals, launched publicly on November 19, 2026. The headline finding is humbling for the machines. Agents averaged roughly 78% of the required criteria, but only 29% of runs fully met all of them.
What Stripe is actually measuring
The benchmark spans 11 reproducible environments split into three buckets. The first covers backend-only tasks. The second covers full-stack workflows, where results are checked in a real browser. The third bucket is a set of specialized “gym” problem sets. These zero in on specific Stripe features, including Checkout and subscription billing.
Stripe compares performance across three variables: the harness, the model, and the integration scenario. A harness is the scaffolding that wraps a model and gives it tools.
The company says it plans to publish fresh results periodically. The stated goal is to help businesses pick the agent configuration that best fits their needs.
The tooling and the scoreboard
Under the hood, the framework runs on a Goose-based agent harness. It pairs that with Model Context Protocol (MCP) tools, which let agents reach outside the chat window.
Through MCP, agents get terminal access, browser automation, and documentation lookup.
The most interesting number may belong to Stripe’s own plugin. Across the tested model and harness combinations, it lifted scores by 3 to 8 percentage points. It also cut the need for web searches by 96%. Extraneous test code fell by approximately 40%.
On the model leaderboard, Claude Opus 4.5 posted a 92% average on full-stack tasks. GPT-5.2 reached 73% on the gym sets, using a best-of-three approach.
The tasks were not trivial, either. Agents averaged 63 turns in their best runs, meaning dozens of back-and-forth steps to finish a single integration.
From blog post to public benchmark
Stripe did not build this overnight. The underlying evaluation framework was first described in a company blog post on March 2, 2026.
The November launch turns that internal methodology into something the public can inspect and run. The benchmark is open source and hosted on GitHub, with methods and results available for review.
The focus is on realistic, end-to-end payment scenarios rather than toy coding puzzles.
What this means for developers and the AI agent race
The most practical takeaway is the gap between 78% and 29%. Agents are good at checking most boxes and much worse at checking every box.
The plugin results also hint at where the competitive action may move. If a platform-specific tool can add 3 to 8 points regardless of the model, better context could matter as much as a smarter model.
For businesses picking an AI coding setup, periodic results could serve as a buyer’s guide. Instead of trusting marketing claims, teams can check how a specific model and harness pair actually performed on Stripe tasks.
The caveat is that any benchmark measures what it measures. Eleven environments cover a lot of ground, but they cannot capture every company’s quirks, legacy systems, or unusual pricing models.
Disclosure: This article was edited by Diego Almada Lopez. For more information on how we create and review content, see our Editorial Policy.

10 hours ago
4








English (US) ·