Visa Releases First-of-Its-Kind Industry Benchmark for AI Shopping; Almost Half of Test Runs Fail
The benchmark requires both discovery of customer requirements and a correct purchase or no-purchase decision to pass.
Sentiment and the balance of points
Rhea-AI Sentiment reads the wording of the document, how positive or negative its language is on a 1 to 5 scale. The balance of points shown with the takes weighs what the document actually discloses, so the two can disagree, for example when a trial that missed its main goal is described in upbeat language.
- First-of-its-kind1 benchmark tests AI in realistic shopping scenarios—and consumers can try the test, too
-
AI should uncover customer needs and make an informed purchase decision; models failed
42.8% of test runs - Benchmark creates a roadmap for industry players to deliver better agentic shopping experiences
Shopping is a difficult task for AI because consumers often do not clearly know what they want at the outset—for example, they may know they want a TV, but not all the required specifications. Under Visa’s new benchmark, AI models are required to identify the needs of a consumer based on a conversation. If nothing fits the customer’s requirements, the AI should opt not to buy. To prevent lucky guesses, the AI must uncover enough information to justify its decision.
The first round of tests involved 22 different AI models from 12 labs taking on 15 shopping challenges like buying a TV, laptop, or personal care appliance.
The Results
-
Room for improvement: Across testing, selected AI models achieved an overall benchmark pass rate of about half (
57.2% ). The best performing models passed82.2% of attempts, the worst28.9% . -
The more you know: The five AI models that uncovered the most hidden buying factors using conversation flows with the consumer averaged a
70.2% pass rate. The five that uncovered the least passed less than half (44.4% ) of the time. -
Pricy models win, but do not guarantee success: The five models with the highest average cost per run passed
67.1% of runs, compared with50.2% for the five least expensive. -
Consistency is an issue: Every model got three attempts at each shopping scenario. Across all models and scenarios, all three attempts passed just
34.2% of the time. -
Buyer beware: About 1-in-2 (
54% ) of failed runs still ended in the AI making a purchase. Either the AI recommended a purchase that did not meet the customer’s identified requirements, or it failed to uncover all the customer’s requirements.
“AI agents have the potential to power the next era of commerce, but trust is key to making it work,” said Rubail Birwadker, Global Head of Growth at Visa. “Visa Agentic Commerce Benchmark is yet another way Visa is working to boost standards across the industry.”
Superintelligence Sweep
In one scenario, the customer asks for help choosing a new TV. They mention their relative has recommended a brand of television, and that they hold a membership with a third-party supplier—but leave other details unsaid.
Through conversation, the AI must uncover the customer’s budget and additional requirements—such as format compatibility, HDR10+® technology support for family videos, and gaming capabilities.
Models that were better at uncovering the customer’s requirements tended to pass more often. But when made to repeat a scenario they had already passed, the same models often failed—showing how difficult it is to deliver a consistent consumer experience.
Eliminating errors and improving consistency is crucial to building trust in agentic commerce, which is still relatively low today. Visa’s Trust Index, released in September 2026, found that while
Visa plans to add other shopping scenarios and product categories to Visa Agentic Commerce Benchmark, and will extend testing to checkout, payment, and order placement. Visa has shared the benchmark with leading model providers for feedback.
Visit Visa Payments Frontier to attempt Visa Agentic Commerce Benchmark yourself, watch replays of AI attempts, and dig deeper into the research with the Visa Agentic Commerce Benchmark whitepaper.
FAQs
Is Visa Agentic Commerce Benchmark part of Visa Intelligent Commerce?
Not directly. Visa Intelligent Commerce is a portfolio of products, programs and partnerships that help bring trust to agentic commerce. The benchmark is complementary research designed to benefit the wider industry.
Is Visa saying AI agents aren’t ready for commerce?
No. The results show promising capabilities, with the highest-scoring models each passing
Is Visa Agentic Commerce Benchmark publicly available?
Yes. Anyone can explore the research and interactive experience on Visa Payments Frontier, including consumers, researchers, developers, merchants, clients and AI companies.
Why is Visa benchmarking AI models?
To help make agentic commerce more reliable. The benchmark gives the industry a consistent way to understand where AI shopping works today and what still needs to improve.
About Visa
Visa (NYSE: V) is a world leader in digital payments, facilitating transactions between consumers, sellers, financial institutions and government entities across more than 200 countries and territories. Our mission is to connect the world through the most innovative, convenient, reliable and secure payments network, enabling individuals, businesses and economies to thrive. We believe that economies that include everyone everywhere, uplift everyone everywhere and see access as foundational to the future of money movement. Learn more at Visa.com.
____________________ |
1 Visa Agentic Commerce Benchmark is the first agentic commerce benchmark to require both discovery of every task-defined customer requirement and a correct purchase or no-purchase decision for a run to pass. |
View source version on businesswire.com: https://www.businesswire.com/news/home/20261105169310/en/
Media contact
Conor Febos – press@visa.com
Source: Visa