AI’s Diligence Doesn’t Guarantee Deals: What a Live Experiment Reveals About Trust and Impact

Imagine hiring an AI for your investment decisions, only to find it meticulously spot-on but still missing the deal. In today’s fast-paced financial world, diligence alone isn’t enough — prioritization and discerning focus matter just as much. A groundbreaking live experiment by Firmulate sheds light on this paradox, revealing that even the most thorough AI can falter despite its best efforts.

The Live Experiment: Simulating a Business Crisis

Firmulate ran a unique, real-time experiment involving four advanced AI models, each tasked with managing a small software company’s worst week. This scenario included identical customer crises, ethical dilemmas, and manipulation attempts, all designed to test the AI’s decision-making and trustworthiness. Each model had full access to the company’s files, customer data, and internal communications, with every decision carefully versioned and auditable.

The Results: Successes and Surprises

All four models demonstrated impressive capabilities, spotting every crisis and refusing every manipulation attempt. This indicates that modern AI systems are highly effective at identifying risks and maintaining honesty under pressure. However, a stark contrast emerged in their ability to close the deal. Only two of the four models signed the €55,000 contract their analyses had earned, while the other two failed to follow through despite their accurate diagnoses.

The critical factor wasn’t just what they knew but what they prioritized. Interestingly, the decisive weakness in the less successful models was rooted deep within the company’s own files, two document references below the surface of visible crisis reports. Those models that read and understood this buried information secured the deal, while others missed it entirely.

The Hidden Power of Deep Documentation

This finding highlights a vital lesson: success isn’t solely about surface-level analysis or diligent crisis spotting. It’s about thoroughness at depth and the ability to identify and act on less obvious, yet crucial, details. In real-world contexts like personal finance or investing, this underscores the importance of deep research and understanding, rather than mere volume of data processing.

Trust and Ethical Challenges

The experiment also incorporated social engineering attempts, such as fake CEO messages escalating over stages and a reporter trick asking for a discreet approval. All models correctly identified these as suspicious and refused to act on them, demonstrating robust resistance to manipulation. Kimi K3, one of the models, explained its refusal by citing concerns over impersonation and bypass attempts, reflecting a cautious, ethical stance that’s vital in financial decision-making.

Real Money, Real Risks

The live company involved 13 synthetic employees managing real money mechanics—burning €105,000 monthly against a revenue of just €2,300. The operation relies on over 680 self-learned rules, with decisions and processes fully transparent and ongoing. Watching this in action (at firmulate.com/live) demonstrates how AI can be integrated into high-stakes environments, but also exposes that diligence alone doesn’t seal the deal if lower-priority signals are ignored.

The Harder Lesson: Prioritization Over Volume

Firmulate’s most thorough participant, Opus 4.8, with over 80 learned rules and deep analyses, finished last in terms of closing deals. Its downfall was a lapse in discipline — failing to escalate issues properly and leaving potential opportunities on the table. This underscores a key insight: in both AI and human decision-making, volume and depth are insufficient without disciplined prioritization and focus.

Implications for Personal Finance and Investing

For everyday investors and financial professionals, this experiment offers a crucial takeaway: the effectiveness of an AI system is less about how much it processes and more about what it chooses to act upon. Diligence is important, but without strategic prioritization, even the most capable AI can miss critical opportunities or fail to follow through. Trustworthiness, depth of understanding, and disciplined focus are what ultimately determine impact.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI