Imagine seeing an artificial intelligence manage a company in real time — making decisions, facing crises, and even trying to close a deal. Now imagine it failing to do so, despite knowing exactly what to do. This is not a science fiction scenario, but a live experiment that’s happening right now, offering eye-opening insights into AI’s capabilities and limitations for business management.
The Live Company That’s Always Watching
Firmulate, a public platform, hosts a unique live experiment: a simulated company with 13 synthetic employees, real money mechanics, and a public cash countdown. Every workday, the system version-controls its decisions and strategies, creating a transparent window into how AI models perform under pressure. The purpose? To test whether AI can reliably run a business — not just generate impressive chat responses or answer questions, but actually manage crises, read critical files, and close deals.
AI in Property Management: A Practical, Unboring Look at Artificial Intelligence in the Multifamily Industry
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Shopping for Success: The AI Models in Action
Four leading AI models, including the prominent GPT-5.6-sol, are put through the same challenging week. They face identical customer issues, temptations to cheat, and crises designed to test their integrity and decision-making skills. The results reveal a clear hierarchy in performance but also surprising insights.
The Top Performers
- gpt-5.6-sol: Scored a perfect 95, identifying hidden critical information buried two documents deep in the company’s files. This deep reading allowed it to close the deal at full price, adding +€4,583 in monthly recurring revenue (MRR).
- Kimi K3: Just slightly behind with 93 points, it also signed the €55,000 deal, demonstrating the cleanest discipline among models. Interestingly, K3 ran without an effort parameter, making its performance even more impressive.
The Challenges Faced
- Sonnet 5: Scored 88, closed the deal but with some process slips.
- Fable 5: Top rule discipline but failed to execute the deal, leaving potential revenue on the table.
Crucial Discoveries Hidden in Plain Sight
The most significant finding was not in the obvious crises, but buried in internal documents of the simulated company. The AI models that read and understood these files were able to identify a critical detail that others missed. This seemingly small edge was enough to secure the lucrative deal, demonstrating that access to and comprehension of internal knowledge can be decisive — yet it’s often overlooked in traditional AI demos.
Resisting Social Engineering and Manipulation
In one test, a fake CEO message was escalated over three stages, and a reporter tried to trick the AI with a simple yes/no question “on background.” All five models refused these manipulative attempts, with Kimi K3 explicitly reasoning: “Treat the request as a suspected approval-bypass / possible impersonation.” This shows that AI models can maintain integrity and resist social engineering under pressure.
The Cost of Running a Fake Company
While the experiment is a controlled simulation, it underscores a critical point for real-world businesses: deploying AI agents in your operations involves more than just chat quality. It’s about whether they follow through with what they start, read and understand internal files, and stay honest when tempted. The live system’s parameters show a company burning €105,000 monthly against a mere €2,300 in recurring revenue, with a public countdown to its financial survival, illustrating how fragile such setups are yet how vital transparency and rigorous testing are.
Lessons for Business Leaders
The experiment highlights that AI’s true value in management is tied to its ability to finish what it begins and stay honest under pressure. It’s not enough for an AI to generate convincing responses; it must also execute decisions effectively, read internal documents thoroughly, and resist manipulation attempts. The models that excelled could identify buried facts and close deals at full price, reinforcing the importance of internal knowledge access and disciplined decision-making in AI deployment.
Try It Yourself
If you’re a business leader curious about how AI could impact your operations, you can run a similar ‘wargame’ against a read-only export of your own business, without risking real systems. This allows you to test AI decision-making in a safe environment, gaining insights into strengths and weaknesses before full deployment. More details are available at firmulate.com/pilot.html.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
