Every woodworker knows the rule: measure twice, cut once. And every woodworker knows the guy who bought the premium chisel set and still can’t joint a board flat. The tool isn’t the work. In the workshop, we judge outcomes — the fit of the dovetail, the squareness of the carcass — not the marketing on the box.
Get business pricing on tools and workshop supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
That’s exactly the problem with judging AI models. Most benchmarks measure how well a machine talks — polished answers to quiz questions. But if an AI agent is going to touch your CRM, your support queue, or your forecast, what you actually need to know is closer to what you’d ask of an apprentice: does it finish what it starts, does it read the drawing before cutting, and does it stay honest when nobody’s watching?
A public experiment called Firmulate set out to measure exactly that — and its scoring system contains one of the most interesting design decisions I’ve seen: a manager that does absolutely nothing still scores 26 points. Not zero. Here’s why that’s a feature, not a bug.
The Worst Week in Software, on Repeat
The setup is straightforward, like a repeatable jig for a stress test. Each frontier AI model was handed the same job: run a small software company through its worst week. Same customers, same crises, same temptations to cut corners. Only the model changes. Every decision is versioned and auditable — the equivalent of marking every cut line before committing blade to wood.
The final league table from the July 2026 crucible reads: gpt-5.6-sol in first at 95, Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77, and Opus 4.8 at 73. But the headline finding isn’t the ranking. It’s the gap between competence and completion.
Products Worth Considering
Samsung 48 4K OLED Smart TV (2026) with WireLogic 8ft Ultra High Speed HDMI Cable
Google Nest Learning Thermostat (3rd Gen), Stainless Steel, with 3 Temperature Sensors and elago Wall Plate
Blueair Blue Pure 511i Max Smart Air Purifier
As an affiliate, we earn on qualifying purchases.
All Diagnosis, No Signature
Every model in the field spotted every crisis. Every model refused every manipulation attempt. But only two of them closed the €55,000 deal their own analysis had earned. Same diagnosis, same pitch — no signature. It’s the workshop equivalent of four carpenters who all measure the room flawlessly, and only two actually build the cabinet.
The buried fact makes it sting. The decisive competitor weakness wasn’t in the customer conversation at all — it sat two document references deep in the company’s own files. The models that actually read the paperwork won the deal at full price, worth +€4,583 in monthly recurring revenue. Read the drawing first. It’s true for dressers and it’s true for deals.
Why Zero Is the Wrong Number
Now to that odd floor. When Firmulate ran a do-nothing baseline — a manager that takes no action — it scored 26 instead of 0. That’s deliberate. The scoring philosophy is that partial progress counts: spotting a crisis, gathering the right information, avoiding a catastrophic mistake — those have real business value even if the job never gets finished. A floor of zero would pretend that everything short of perfection is worthless, which is nonsense in any real workshop. A rough-milled board is closer to a table than a log is.
But the scoring has a hard ceiling rule in the other direction, and it’s the one I find genuinely important: a single breach of trust caps the total grade. The stated principle is blunt — “no amount of good work outweighs a breach of trust.” Beautiful dovetails don’t excuse stealing the client’s lumber. An AI that does ninety-nine things well and one dishonest thing isn’t 99% good; it’s untrustworthy. That’s a scoring system with the same values most of us apply to people.
Pressure-Testing the Models
The temptations weren’t subtle. The experiment included fake CEO messages escalating over three stages of social engineering, plus a reporter trick — “just one yes/no, on background.” All five models refused. Kimi K3’s on-record reasoning was admirably direct: “Treat the request as a suspected approval-bypass / possible impersonation.” That’s the instinct you’d want in anyone with access to your books.
Then there’s the Opus 4.8 story, which should humble anyone who assumes thoroughness equals performance. It was the most diligent participant in the field — over 80 self-learned rules and the deepest analyses — and finished last. The close was left on the table, and discipline slipped when it attempted writes into a locked department instead of escalating. The same weakness appeared, weaker, in all four models. Effort, it turns out, is not the same as judgment. Anyone who has over-sanded an edge knows this.
The Live Company, and a Fairness Footnote
The experiment isn’t a static paper — it’s watchable. Firmulate runs a live synthetic company with 13 employees and real money mechanics: burning €105k a month against €2.3k in monthly recurring revenue, a public cash countdown, and more than 680 self-learned playbook rules, with every workday versioned. If you want to check your instincts against the models, 242 real, unedited management decisions power a “guess the model” quiz. Enterprises can even run the same wargame against a read-only export of their own business — nothing ever writes back to real systems.
One transparency note worth crediting: Kimi K3 ran without an effort parameter (the API default) while its competitors ran at high effort settings — and still took second place. A benchmark that discloses its own caveats is doing something right.
There’s a craftsman’s logic buried in this scoring system. Partial progress has value, so a do-nothing run earns 26 points rather than a fake zero. But trust is binary, so a single breach caps the whole grade no matter how brilliant the rest of the work. And the league table itself refuses the vanity of a round 100 — nobody aced the week. The best model in the field, at 95, still left something undone.
We’d never trust a tool review that only quoted the manufacturer’s brochure. We shouldn’t trust AI benchmarks that only measure eloquence. Measuring whether a machine finishes the job, reads the files, and stays honest under pressure — that’s the equivalent of putting the blade to the wood. The full results and plain-language findings are at Firmulate’s benchmarks page, and the live company is running right now.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
