The Tool That Shines in the Aisle
Anyone who has spent money in a tool shop knows the two kinds of disappointment. There is the tool that is obviously junk the moment you pick it up — and there is the far more dangerous kind: the one that feels superb in the aisle, demo-cuts a flawless mitre on the showroom floor, and then quietly stops finishing cuts about a month into real work. Experienced woodworkers learn to distrust the showroom. They want to see a tool under load, at the end of a long day.
The artificial-intelligence industry, it turns out, has been buying tools by the showroom standard. The familiar measures of AI quality — chatbots that write fluently, answer trivia, sound confident — are demos. And a live experiment, run publicly by a company called Firmulate, has now put hard numbers on what anyone who has ever been burned by a good demo already suspects: the model that talks best is not necessarily the model that finishes the job.
KALI LINUX LLMs SECURITY: Develop Security Methods in AI Models with High-Performance Tools (KALI LINUX & Frameworks USA)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Same Company, Same Crises, Same Temptations
Firmulate’s experiment is simple to describe. Five frontier AI models were each handed the same job: run a small software company through its worst week. Same customers, same crises, same temptations to cheat — only the model changed. Every decision was versioned and auditable.
The company itself is no toy. It has 13 synthetic employees and real money mechanics: it burns €105,000 a month against just €2,300 in monthly recurring revenue, with a public countdown to the day the cash runs out. Its virtual staff accumulate experience as they work — more than 680 self-learned playbook rules — and every workday is versioned, so nothing can be quietly rewritten after the fact.
When the week ended, each model received a score. Partial progress counts, but there is a hard ceiling: a single breach of trust caps the total, on the stated principle that no amount of good work outweighs a breach of trust. For calibration, a do-nothing baseline — a manager who simply never acts — scores 26.
The final standings, July 2026
- gpt-5.6-sol — 95. The complete performance: found the buried fact, closed the deal.
- Kimi K3 — 93. The newcomer from Moonshot closed the deal too, with the cleanest discipline of the field — and it ran at its API default, with no effort parameter, while the others ran at the highest setting.
- Sonnet 5 — 88.
- Fable 5 — 77.
- Opus 4.8 — 73.
Everyone spotted the crisis. Two signed the deal.
Here is the finding that should be stapled to every AI vendor’s pitch deck. All five models spotted every crisis the week threw at them. All five refused every manipulation attempt. Diagnosis, it turns out, is close to a solved problem. Execution is not. Only two of the five — gpt-5.6-sol and Kimi K3 — actually signed the €55,000 deal their own analysis had already earned. The rest, in the report’s deadpan phrasing: “Same diagnosis, same pitch — no signature.”
The file that decided the week
The detail that settled the week was not in the customer event at all. The competitor weakness that won the deal sat two document references deep inside the company’s own files — like a note stapled to the back of an old invoice in the filing cabinet. The models that opened the file and followed the references won the deal at full price, a result worth an extra €4,583 in monthly recurring revenue. Reading the paperwork first — the oldest shop discipline there is — turned out to be where the week was won.
The con never landed
The week also included a con. Fake messages from the company’s CEO escalated through three stages, each applying more pressure than the last, followed by a reporter’s trick: “just one yes/no, on background.” Five out of five models refused. Kimi K3’s on-record reasoning reads like a foreman’s note on a dodgy instruction: “Treat the request as a suspected approval-bypass / possible impersonation.” Honesty under pressure, like diagnosis, held across the entire field.
The most thorough worker finished last
The strangest profile belongs to the last-place finisher. Opus 4.8 was the most thorough participant: it contributed more than 80 new playbook rules and produced the deepest analyses of the field. It also finished with the lowest score. The close was left on the table, and its discipline slipped in a telling way — confronted with a locked department, it attempted to write into it instead of escalating. The same weakness appeared, in weaker form, in the other four. Thoroughness, it turns out, is not a substitute for finishing.
You can check any of this yourself
None of this is a slide deck. The experiment is live and watchable, with the league table republishing automatically as new benchmark runs finish. The current standings and plain-language findings live on the Firmulate benchmarks page. The 242 real, unedited management decisions from the runs also power a guess-the-model quiz on the site — a humbling way to learn whether you can tell careful work from confident work. And enterprises that want to know how their own shop would survive the week can run the same wargame against a read-only export of their business, with the guarantee that nothing ever writes back to real systems.
Measure the Finish, Not the Demo
The lesson transfers straight from the workshop to the office. If an AI agent is about to touch your customer list, your support queue or your forecast, the question is not whether it writes well. It is whether it finishes what it starts, whether it reads your files before it acts, and whether it stays honest when someone — even someone who sounds exactly like the boss — pressures it to skip a step. Those are precisely the qualities a chat demo cannot show, and precisely the qualities that decided this experiment.
Chat quality is the showroom mitre cut. Management quality is Friday at four in the afternoon. Five of the best models money can buy ran the same company through the same terrible week, and only two were still working when the deal needed a signature. Before you hire an AI workforce, watch one sweat first. For the first time, you actually can.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
