AI’s Hidden Weaknesses: Why Success in Chat Doesn’t Guarantee Closing Deals

When it comes to deploying AI in real-world business, the true test isn’t how well it chats or diagnoses problems—it’s whether it can follow through and close deals under pressure. Recent experiments reveal that while many AI models can spot crises and resist manipulation, only a few can actually seal the deal and execute an analysis they earn. This gap between apparent competence and real-world performance has crucial implications for investors and managers alike.

The Experiment: Putting AI to the Test in a Business Crisis

In a groundbreaking live experiment, four advanced AI models were tasked with managing a small software company facing its worst week—same crises, same customers, same temptations to manipulate or cut corners. Each AI operated within a fully auditable, versioned environment, simulating real-time decision-making under financial stress.

The goal was straightforward: see if the models could identify problems, resist manipulation attempts (like fake CEO messages or secret file references), and ultimately close a €55,000 deal based on their analysis. The results were revealing.

All Models Spot the Crises, But Only Two Seal the Deal

• Every model detected every crisis — from customer complaints to financial anomalies.
• All refused attempts at manipulation, including staged CEO messages and reporter tricks.
• However, only two models, gpt-5.6-sol and Kimi K3, actually followed through and signed the deal—meaning they executed the analysis they had earned.

The other two, despite diagnosing correctly, left the deal unexecuted. Notably, one model—Opus 4.8—demonstrated the deepest analytical depth but faltered in discipline, leaving the deal on the table due to process slips.

The Hidden Weakness: The Critical Information in Company Files

The real differentiator was what the models read beyond the immediate crisis. The successful models found crucial information buried two document references deep in the company’s files, information that led to fully closing the deal at full price—a value of over €4,583 monthly recurring revenue.

In contrast, models that overlooked these documents failed to convert their diagnosis into a signed agreement, even when the analysis was accurate and the pitch was identical.

Measuring True Business Capability

This experiment shows that the real skill of an AI isn’t just in diagnosing problems or resisting manipulation—it’s in executing the work, following through on insights, and closing the deal. Standard chat-based demos often measure superficial capabilities, but the ability to stay honest, read critical documents, and see a project through is invisible until tested in real, high-pressure scenarios.

Implications for Investors and Business Leaders

For those investing in AI or integrating it into operations, this means evaluating models on their ability to deliver tangible results, not just impress with their diagnosis or their resistance to manipulation. An AI that can’t execute the final step may be of limited value—no matter how clever it appears in chat or demo.

In these live experiments, measures like the Crucible League scores—where the top model scored 95 and the baseline only 26—offer a quantitative view of performance. Yet, the critical takeaway remains that true effectiveness is revealed only when AI is tested against the messy realities of business execution.

Testing AI Before Deployment

Business leaders can preempt costly failures by running their future AI agents through simulated crises—like the live wargame at firmulate.com/pilot.html. These tests verify if the AI can handle real tasks, read necessary documents, and stay disciplined under pressure—crucial qualities that often escape standard demos.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI