Model benchmarks are no longer sufficient.

A new model may surpass its predecessor on a benchmark testing the model alone. In fact, new models are not released until the developers make sure they do. However, changes in the model's behaviour from release to release may cause the new model to exhibit lower performance than the expectation set by the model benchmark.

I have seen this first-hand when switching from GPT 5.6 SOL to Astra 6. Whereas the SOL model was operating just fine in its Hermes harness, Astra fought against it, as I reported here.

So what is to be done?

For me, I am developing a suite of harness-specific benchmarks for both Hermes and Pi that will smoke test typical agent tasks for new models against the current production model. It's not exhaustive, I can't personally fund hours of agentic benchmarking. But it's enough to give me an idea of where failure modes might crop up, and I can take it from there.

But that's just me, with time on my hands and a long-standing interest in the nuts and bolts of the technology.

To me, it's unreasonable that the burden of agentic testing should fall to the customer, when AI developers are specifically engineering their models for agentic work.

There is probably an argument to be made that the developers' own proprietary harnesses are updated hand in hand with new models. But that fails in real-world practice.

First, has the harness really been upgraded, for a particular customer's specific production environment? Possibly "close enough" is the answer, with the model improvising some behind-the-scenes patching to make it all work. Until it doesn't.

Second, it's another step down the path of locking customers into a model platform through the proprietary harness ransom. Not that that is anything new, anyone remember the saying "DOS ain't done till Lotus won't run"? Corporate self-interest is, in fact, often a necessity of a competitive market. But as users, we should be wise enough to push against it.

So what is really to be done?

It seems to me that harness developers are the ones that need to pick up the slack in this case. Maybe reduce the 1,000 commits a day for cosmetic fixes and switch some resource to benchmark advisories for model changes. An advisory can't cover every customer's production environment, so ship a harness-specific skill with it that tells an agent to find the production tasks, benchmark them for the current model, establish a baseline, and then run them against a proposed new model.

That is something harness developers are better placed to do, rather than leaving it up to every end user. If I can do it, it should be a walk in the park for the developers of leading harnesses like OpenClaw, Hermes, and Pi. I mean, it's exactly the sort of thing their own agents can be tasked to do.

In the meantime, if you do want your own model tester for your environment, feel free to drop me a line. We can no doubt work something out fairly quickly.