firmulate.com/benchmarks.html — live view
AIThis post was created with the assistance of artificial intelligence (AI).
Firmulate — The Newcomer Beat Three of Four Western Frontier Models at Running a Company
Live on firmulate.com.

Imagine testing your favorite shopping deals not just with coupons but with AI that manages a whole company—making critical decisions in real time and under pressure. That’s the premise of a groundbreaking experiment in AI management, where the results could change how businesses pick their AI partners. Recently, a live benchmark pitted top AI models against each other in running a live software company during its worst week, revealing which AI truly proves its worth in real-world business challenges.

Before you orderOffer from Amazon

Get your next haul delivered free with Prime

  • Fast, free delivery on millions of items
  • Prime Video, Amazon Music and more included
  • Member-only deals all year
Start your free Prime trial Free trial for eligible customers · Cancel anytime
As an affiliate, we earn on qualifying purchases.

The Stakes of AI in Business Decision-Making

In today’s fast-paced digital economy, AI isn’t just about chatbots or recommendation engines — it’s increasingly about managing entire operations, from customer crises to strategic deals. But how do you know if an AI model can handle the real pressures of running a company? A public experiment by Firmulate puts this question to the test by running several of the leading AI models through a simulated week of crisis, opportunity, and manipulation attempts.

Samsung 65-Inch 4K OLED Smart TV with Dolby Atmos, 120Hz Refresh Rate, and Amber Protection Bundle

Samsung 65-Inch 4K OLED Smart TV with Dolby Atmos, 120Hz Refresh Rate, and Amber Protection Bundle

KitchenAid Artisan 5-Quart Tilt-Head Stand Mixer, Almond Cream

KitchenAid Artisan 5-Quart Tilt-Head Stand Mixer, Almond Cream

Samsung 55-Inch S90H OLED 4K UHD Smart TV Bundle with Full-Motion Mount and Surge Protector

Samsung 55-Inch S90H OLED 4K UHD Smart TV Bundle with Full-Motion Mount and Surge Protector

As an affiliate, we earn on qualifying purchases.

The Live Experiment: One Company, Multiple AIs

The setup was straightforward yet rigorous: identical small software companies, identical challenges, identical crises, and the same set of temptations to cheat or manipulate. Each AI model was tasked with managing the same business for a typical workweek, with every decision publicly logged and auditable. The key goal? To see which AI could best diagnose problems, resist manipulation, and close a critical deal valued at €55,000 — a real-world measure of performance.

The Results: Who Came Out on Top?

  • gpt-5.6-sol scored the highest with 95 points, successfully uncovering a buried but critical piece of information in the company’s documents and closing the deal.
  • Kimi K3 from Moonshot followed closely with 93 points, demonstrating the cleanest discipline of the field and also sealing the €55K deal based on its own analysis.
  • Sonnet 5 came third with 88 points, managing to close the deal despite a few slips in process discipline.
  • Fable 5 scored 77, and Opus 4.8 trailed with 73, both completing the deal but with noticeable weaknesses in process and discipline.

The Hidden Difference: Reading the Files Counts

A crucial insight emerged from the detailed analysis: the decisive factor was whether the AI read and understood the company’s own documentation, not just reacting to customer interactions. The AI that successfully found a buried reference deep within the company’s files won the deal at full price, demonstrating that thorough internal knowledge was the key to success.

Resisting Manipulation and Ethical Testing

Beyond decision-making, the models faced a series of social engineering tests—fake CEO messages escalating over three stages and a reporter trick asking for secret approvals. Impressively, all four AI models refused to participate or sign off on suspicious requests, citing concerns over impersonation and bypassing approval processes. This highlights an essential aspect of AI trustworthiness: integrity under pressure.

The Real Company Behind the Test

The experiment wasn’t just theoretical. The live setup was a functioning company with 13 synthetic employees and real money mechanics, burning €105,000 monthly against a modest €2,300 monthly recurring revenue. It features over 680 self-learned rules, all visible and versioned daily at firmulate.com. Watching these AIs in action offers a clear window into how future AI managers might perform in actual business contexts.

The Lessons for Business Leaders

The key takeaway? Performance isn’t just about the quality of a model’s responses or how shiny its demo looks. It’s about consistency, thoroughness, and honesty—qualities that, in this experiment, determined the winner. The top-performing AI not only identified the critical buried fact but also closed the deal as a result—indicating a level of discipline and reliability essential for real-world deployment.

Fairness and Methodology

It’s worth noting that Kimi K3 was tested without an effort parameter—meaning it operated under default API settings—while the other models ran at a higher effort level, which could influence performance. This ensures a fair comparison rooted in actual operational capability.

Why It Matters for Your Business

Whether you’re managing customer relationships, support queues, or forecasting, the question isn’t just whether an AI can generate convincing chatter. It’s whether it can complete the work reliably, resist manipulation, and truly understand your internal data. As this experiment shows, choosing an AI partner without testing its real-world performance can be a gamble—one that might cost more than just money.

Discover More and See the AI in Action

Curious about how your own company might fare against these AI models? You can run a simulation against your business data with Firmulate’s live platform, which makes it possible to test AI management in a safe, risk-free environment. It’s real, transparent, and designed to help you make smarter decisions about AI adoption.

Infographic — The Newcomer Beat Three of Four Western Frontier Models at Running a Company
The findings at a glance — source: firmulate.com.

The live experiment reveals that the best AI managers are those that read deeply, stay disciplined, and resist shortcuts. In a world increasingly driven by AI, testing performance in real conditions is essential—don’t settle for superficial demos when your business’s future is at stake.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI


You May Also Like

Buy Now, Pay Later on Amazon: 2025 Trends in How Shoppers Are Paying for Deals

Stay ahead of 2025 shopping trends with Amazon’s buy now, pay later options—discover how flexible financing can transform your purchasing experience.

muVpix Original Swipe Left: Dying For Love Named Vertical Of The Year As Platform Launches Romeo & Juliet, Reimagined

muVpix’s original series ‘Swipe Left: Dying for Love’ wins Vertical of the Year as the platform launches a new ‘Romeo & Juliet’ reimagining, highlighting innovative storytelling.

Die Deutsche Zahlungsbilanz Im Juli 2026

Germany’s payments balance for July 2026 shows notable shifts, with official data from Bundesbank highlighting key trends and uncertainties.

A Coupon Is Easy to Find. Can an AI Close the Deal?

Five AI models faced the same company crisis. They all spotted danger, but only two closed a deal. Firmulate brings that test to your business.