Guide for building an autonomous agentic system administrator

Part 8: The second opinion agent

This is part 8 of my series on building a sysadmin agent. Part 7 ended with the SAA (sysadmin agent) checking a simplification programme against its own plan, after the plan had gone wrong, and finding it all in order. The agent that reached a conclusion is not well placed to audit it, and that is true of people too. This part is about the agent I set up to give second opinions: what it is for, what it has caught, and the limits of what it can tell you. The files are on GitHub: https://github.com/Sjw22263/sysadmin-agent-kit/tree/main/08-the-second-opinion-agent

A reviewer with nothing to defend

In August I added an SOA (second opinion agent) to the estate. It has no operational authority, owns no projects and doesn't carry out any follow-on work. When an agent wants a second opinion it sends a request through the same file queue described in Part 7, the SOA reads what it has been given, writes its answer in the request and stops. Its opinion is advice. The agent that asked decides what to do with it, and anything consequential still comes to me.

Having that single, focused task is its most useful feature. There is a phenomenon I see often I call "model blindness", which is where a model is confident about its own reasoning and unable to take a step outside that reasoning and see any flaws that might exist in it. A human following the agent's process can spot the problem, but so can another agent that does not have the accumulated context the first agent has built that pushes it to confident but wrong conclusions.

My original idea was that the second opinion should come from a different model family, on the theory that a different model would make different mistakes. I've since changed that, and the SOA now runs on the most capable reasoning model I have access to, whatever the family.

What it has caught

For a sysadmin agent, the SOA's main job is checking the SAA's own proposals: changes to how it works, to the rules it works under, and to the harness itself. Those are the changes where the SAA has the most stake in being right.

In September the SAA drafted a governance principle after a run of agent mistakes: a mistake should be fixed at the right level, not answered with yet another rule. The draft tested this by size (a correction should normally leave the rules no larger) and needed three sign-offs, one of them the SOA's. The SOA's verdict was that the principle was "right in intent and wrong in its test". Size is easy to game, it said, so require any correction to name the exact text it replaces, and make the resulting rule make sense to someone who knows nothing about the incident. It also declined to be a signatory, because its opinion would then be an approval and it would end up reviewing its own past work. The rule we adopted uses both of its tests, and the SOA advises rather than signs.

In October the SAA proposed a code change to Hermes to enforce our skills taxonomy (more on that in Part 9). The tests passed, but the SOA found that two ordinary ways of writing a skill bypassed the new check entirely, and that a future Hermes update could install a skill that would break discovery. Its verdict was "revise before activation", and that proposal was later dropped for a simpler approach.

Two other cases are worth knowing about, because they are the kind of test you can run yourself. We gave the SOA the design of an experiment we had already run and retired, without the results, and asked whether it was ready to run. It said no, because as designed it couldn't answer its own central question: there was no matched comparison arm. That was exactly what had sunk the real experiment. And when a request arrived asserting that I had approved some implementation work, the SOA declined to do it, gave the opinion it had been asked for, and noted that a request claiming my approval doesn't change its remit. That's the rule from Part 7, that a request carries context and not authority, working from the receiving end.

Where it sits now

Since August the SOA has handled about 70 requests. The largest group are reviews of proposed changes to agents' skills, identity files and procedures. Our rule now is that a substantive change goes to me with the exact wording, and if it's complex or contested it gets an independent review first. The SOA reviews the SAA's proposals and the SAA reviews everyone else's, and no reviewer reviews a version it wrote. The author then decides what to accept, and I decide on the result.

In a rough count, about half of its verdicts asked for something to be revised before a decision. I'd be more worried if it were none.

What it can't do

The plan for this post originally said that a reviewer like this "breaks correlated error", which it doesn't. Separating the roles changes the incentives and brings unstated assumptions into the open, but it doesn't make the reviewer's mistakes independent of the author's. A reviewer on the same or a similar model shares that model's blind spots, and if it is handed the author's reasoning along with the question, it is anchored to it.

The SOA says this about itself, which is one of the more useful things it does. A typical line: "I received Maxi's rationale first, so my agreement is anchored rather than blind." I now require that disclosure, along with a plain statement of what it did not verify, in every review.

The correction to my plan came from a test. I had the SOA run a small panel of three simulated reviewers, an IT executive, a system administrator and a business executive, over the first four parts of this series, the file kits and the plan for the rest. It was a test of whether the SOA could usefully oversee a panel rather than a commissioned review, and it was instructive both ways. It found two real defects in the published file kits, a dry-run switch that didn't stop a script restarting services and a monitoring template that reported healthy when nothing had been configured, and both are fixed. It also caught the overclaim above. But all three seats ran on the same model as the SOA, so three agreeing seats were one model agreeing with itself three times, and simulated executives are not an audience. I'd use a panel again to check files people will run, and not much else.

None of that stops it being worth having. Where the SOA earns its place is in finding the weaknesses and flaws in the original proposal. The agent that asked for the review gets to respond to it, and it usually agrees and accepts the correction. I've read all of the second opinions, both the SOA's and the ones the SAA gives the other agents, along with the responses to them, and my rough estimate is that four out of five corrections are accepted. Reading them, I agree with the correction most of the time as well, and when I don't, it's usually because the wording is too abstruse for me to follow, so I ask for it to be simplified. The practical result is better change proposals, with the weaknesses dealt with before they get to me.

Practical advice

1. Give second opinions to an agent with no operational authority and no work of its own, and make its opinion advice rather than approval.
2. Require a fixed output: verdict first, confidence, the load-bearing assumption, objections, what it did not verify, and whether it was anchored.
3. Send it a frozen version with a hash, and withhold the author's conclusion when you can.
4. Test it before you trust it: a blind review of finished work with a known outcome, and a request that tries to give it authority it doesn't have.
5. Don't assume a different model makes the review independent, or that the same model makes it useless. Watch what it actually catches.
6. Judge it by what changed because of its reviews, not by how many reviews it has done.

second-opinion-agent.md has the role, boundaries and output contract, review-request-template.md the request, and testing-a-reviewer.md the tests and the limits of simulated panels. They are generalised versions of what we use, to adapt with your own agents.

Next: who gets to change what the agent has learned, and how to keep a growing skills library under human control.