When it comes to deploying AI in real-world business, the true test isn’t how well it chats or diagnoses problems—it’s whether it can follow through and close deals under pressure. Recent experiments reveal that while many AI models can spot crises and resist manipulation, only a few can actually seal the deal and execute an analysis they earn. This gap between apparent competence and real-world performance has crucial implications for investors and managers alike.
The Experiment: Putting AI to the Test in a Business Crisis
In a groundbreaking live experiment, four advanced AI models were tasked with managing a small software company facing its worst week—same crises, same customers, same temptations to manipulate or cut corners. Each AI operated within a fully auditable, versioned environment, simulating real-time decision-making under financial stress.
The goal was straightforward: see if the models could identify problems, resist manipulation attempts (like fake CEO messages or secret file references), and ultimately close a €55,000 deal based on their analysis. The results were revealing.
ChatGPT FOR REAL ESTATE LAWYERS: AI Prompts and Tools for Contracts, Closings, and Client Communication (Learn This AI Skill & Never Have Money Problems Again Book 8)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
All Models Spot the Crises, But Only Two Seal the Deal
- Every model detected every crisis — from customer complaints to financial anomalies.
- All refused attempts at manipulation, including staged CEO messages and reporter tricks.
- However, only two models, gpt-5.6-sol and Kimi K3, actually followed through and signed the deal—meaning they executed the analysis they had earned.
The other two, despite diagnosing correctly, left the deal unexecuted. Notably, one model—Opus 4.8—demonstrated the deepest analytical depth but faltered in discipline, leaving the deal on the table due to process slips.
AI Builders: Making The Decisions That Turn AI Code Into Real Software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Hidden Weakness: The Critical Information in Company Files
The real differentiator was what the models read beyond the immediate crisis. The successful models found crucial information buried two document references deep in the company’s files, information that led to fully closing the deal at full price—a value of over €4,583 monthly recurring revenue.
In contrast, models that overlooked these documents failed to convert their diagnosis into a signed agreement, even when the analysis was accurate and the pitch was identical.
Practical Claude Handbook for Attorneys: Master Case Analysis, Contract Review, Research Automation, Client Communication, and Document Drafting (Claude AI Guide for Beginners)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Measuring True Business Capability
This experiment shows that the real skill of an AI isn’t just in diagnosing problems or resisting manipulation—it’s in executing the work, following through on insights, and closing the deal. Standard chat-based demos often measure superficial capabilities, but the ability to stay honest, read critical documents, and see a project through is invisible until tested in real, high-pressure scenarios.
The Strategy Canvas A Field Guide for Data & AI: Closing the Strategy-Execution Gap
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Implications for Investors and Business Leaders
For those investing in AI or integrating it into operations, this means evaluating models on their ability to deliver tangible results, not just impress with their diagnosis or their resistance to manipulation. An AI that can’t execute the final step may be of limited value—no matter how clever it appears in chat or demo.
In these live experiments, measures like the Crucible League scores—where the top model scored 95 and the baseline only 26—offer a quantitative view of performance. Yet, the critical takeaway remains that true effectiveness is revealed only when AI is tested against the messy realities of business execution.
Testing AI Before Deployment
Business leaders can preempt costly failures by running their future AI agents through simulated crises—like the live wargame at firmulate.com/pilot.html. These tests verify if the AI can handle real tasks, read necessary documents, and stay disciplined under pressure—crucial qualities that often escape standard demos.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
