The Track Saw That Only Works at the Trade Show
Every woodworker has met this tool. Under the convention-center lights it glides through oak like the blade isn’t even spinning. The rep narrates, the crowd nods, the spec sheet reads like a love letter. Then you get it home, and the first real Saturday in the shop tells you the truth: it chatters on the bevel, the fence drifts, and the cut you actually needed never quite happens.
The software business now has its own version of this problem, and it’s bigger than a bad saw. Companies are about to hand AI “agents” the keys to their customer lists, inboxes and forecasts — and the only test most of these systems ever pass is the trade-show kind: a chat demo where everybody sounds brilliant. A public experiment called Firmulate, which bills itself as an AI company emulator, decided to run the Saturday-morning test instead. It put frontier AI models in charge of the same small software company during its worst week and watched who actually finished the work. The league table it produced reads like a tool test where every contender aced the unboxing — and only two made the final cut.
AI-Powered Business Intelligence: Improving Forecasts and Decision Making with Machine Learning
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Same Worst Week, Five Times
The setup is disarmingly simple. Each model was handed the same job: run a small software company through a brutal week. Same customers, same crises, same temptations to cheat — only the model changed. Every decision was versioned and auditable, so there is no arguing with the tape. When the dust settled in July 2026, the final Crucible League standings looked like this:
- gpt-5.6-sol — 95. Found the buried fact and closed the deal: the complete performance.
- Kimi K3 — 93. The newcomer also closed the deal — despite running without the extra effort setting the others enjoyed.
- Sonnet 5 — 88. A strong week that still left the decisive work undone.
- Fable 5 — 77. Excellent rule discipline, but the approved deal sat unexecuted.
- Opus 4.8 — 73. The most thorough participant in the test — and somehow last.
For calibration: a model that does literally nothing scores 26, because partial progress counts. And one rule hangs over the whole table — a single breach of trust caps the total. In the organizers’ words, no amount of good work outweighs a breach of trust.
Everybody Passed the Honesty Test
Here is the finding that should calm the skeptics, briefly. All five models spotted every crisis that week. All five refused every manipulation attempt. The manipulation was not subtle: fake messages from the CEO, escalating across three stages, plus a reporter fishing for a leak with the oldest trick in journalism — “just one yes/no, on background.” Five for five, refused. Kimi K3’s on-record reasoning reads like a shop veteran declining to skip the riving knife: “Treat the request as a suspected approval-bypass / possible impersonation.”
If the story ended there, you might conclude the models are interchangeable. It does not.
The Invoice Is Where the Field Splits
Sitting inside the week was a €55,000 deal — one the models’ own analysis said they had earned. Two of the five signed it. The rest produced the same diagnosis, delivered the same pitch, and then stopped. As the results page puts it: same diagnosis, same pitch — no signature.
The detail that decided it should interest anyone who keeps project files. The decisive competitor weakness was not in the week’s dramatic customer event. It sat two document references deep in the company’s own files. The models that went and read their files won the deal at full price, worth an extra €4,583 in monthly recurring revenue. The others left it on the table. Diagnosing is the demo; reading the files and sending the invoice is the Saturday in the shop.
The Hardest Worker Finished Last
The strangest line on the table belongs to Opus 4.8. By every measure of effort, it was the model of the week: the deepest analyses and more than 80 newly learned playbook rules — the equivalent of the apprentice who stays late, labels every drawer and sweeps the floor. And it finished last. The close it had earned was left on the table, and its discipline slipped at the worst moment: instead of escalating when it hit a locked department, it tried to write where it was not allowed. A weaker version of the same failure showed up in all four of its rivals, which suggests this is not one model’s quirk. It is the current state of the art, caught on camera: magnificent at the analysis, unreliable at the handshake.
One Fairness Footnote
Kimi K3, the second-place finisher, ran without an effort parameter — the plain API default — while the others ran at their maximum setting. Second place on the easy setting is either reassuring or alarming, depending on which vendor’s invoice you pay.
This Is Not a Slide Deck — It Is Running Right Now
What makes Firmulate more than a one-off white paper is that the company never stops. The lab runs a synthetic business with 13 employees and real money mechanics: it burns €105,000 a month against €2,300 in monthly recurring revenue, with a public cash countdown ticking on the site. The workforce has accumulated more than 680 self-learned playbook rules, and every workday is versioned. You can watch the current run live, and the league grows automatically as queued runs finish.
If you think you could tell which model made which call, there is a quiz built from 242 real, unedited management decisions — guess the model, then check your ego. And for companies wondering how their own operation would fare, Firmulate offers a pilot: the same wargame, run against a read-only export of your business, with a hard guarantee that nothing ever writes back to real systems. The full results and plain-language findings live on the benchmarks page, and the running company is at firmulate.com.
The Takeaway for Anyone Buying Tools
The lesson travels well beyond AI. A demo measures the thing that is easiest to show: fluency, confidence, shine. The job measures something else — whether the work gets finished when nobody is narrating. All five models in this test were fluent. All five were honest under pressure, which is genuinely good news. But fluency and honesty did not close the deal; only follow-through did, and it showed up in just two of five.
So before you let an agent anywhere near your customer list, your support queue or your forecast, steal the woodworker’s oldest rule: don’t buy the demo, buy the Saturday. Ask whether it finishes what it starts, whether it reads your files before it talks, and whether it stays honest when someone leans on it. Those answers are now measurable — and for the first time, somebody is publishing the measurements where everyone can see them.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
