Guide for building an autonomous agentic system administrator

Part 10: The cost of customising a fast-moving harness

This is part 10 of my series on building a sysadmin agent. Part 9 covered who may change what the agent learns. This part is about changing the harness itself, the agent platform's own code, and what that costs every time the platform moves. If you run an agent on an open-source harness, sooner or later you'll want it to behave in a way the stock version doesn't, and editing the code is the quickest way to get there. Each edit is also a commitment to carry it through every upgrade, against a project that in our case changes thousands of times a month. The files are on GitHub: https://github.com/Sjw22263/sysadmin-agent-kit/tree/main/10-the-cost-of-customising-a-fast-moving-harness

Why we changed the harness at all

Most of what makes the SAA (sysadmin agent) do its job lives outside Hermes's own code, in its identity file, mandate, skills and runbooks, which upgrades leave alone. The changes to the code itself are there because we couldn't get the behaviour any other way. The first came in July, when the SAA found that a stock tool instruction told it to patch a skill whenever it hit a problem, which contradicted our rule that a request for information doesn't authorise changes. I asked it to "run it as a local patch first, to test as a production trial, with a safe roll back path", and that patch, which keeps information requests and execution apart, is still the most important change we carry. Others followed: write-safety checks on files, an authorisation check on skill changes, model catalogue entries the releases hadn't caught up with, and corrections to a few skills that ship with Hermes.

Measured on 30 October against the release we run, Hermes v0.22.5, that comes to 47 files, 22 of them tests, with 1,585 lines added and 263 removed. That isn't much code, and the cost was never in writing it but in carrying it from one release to the next.

How fast is fast

The upgrade we prepared on 5 October was 7,240 commits behind its target. The next, on 25 October, moved us another 8,991, from a commit dated 5 October to a release dated 22 October. Those are commits in the project history, not features or releases, but the point stands: every few weeks most of the files we have changed have also been changed by someone else, and a few of them conflict on nearly every upgrade because upstream is busy in the same places we are.

The upgrade that worked and lost the safeguards

In August an upgrade went through cleanly, the services came back healthy and the conversation I was in carried on behaving exactly as before. When I asked the SAA how it handled our changes, "Do you quarantine them from the updates or do you update the modified files and then re-apply the modifications?", it went to check and found its own report was wrong: "my statement yesterday that upstream had absorbed these changes was wrong."

The cause was mechanical. Reapplying our changes with a three-way merge leaves them staged in Git, and the step that saved them used plain git diff, which shows only unstaged changes. The saved patch held a single line in a test file. The authority safeguards were gone, and nothing looked wrong because the session I was talking to had started before the upgrade and still carried the old instructions. A fresh session would have shown it straight away.

That one was our mistake, not upstream's, and the fix was small: capture with git diff HEAD, which includes staged changes, and check the result in a fresh session rather than the old conversation.

The upgrade that couldn't stop checking

October went the other way. After that August scare the upgrade procedure had grown, and on 25 October the overnight preparation ran into blockers and the morning disappeared into more of them: safe shutdown of the web interface and dashboard, an independent executor, a final launch specification. By midday the SAA was still telling me cutover wasn't safe, so I wrote:

"You are treating this as if it is a 24x7 global operation with multiple critical dependencies. That is not the case. If you upgrade and something breaks, it breaks. It can always be fixed later, or, worse case, rolled back. So far over 6 hours has been spent doing checks and tests, and there is no sign of that ever ending. Hermes commits will outpace your ability to update them."

and gave it the only test that mattered: "The only thing you really need to do for a Hermes update is make sure the custom changes made to Hermes files are preserved, or, if superseded, retired." Four minutes later the upgrade was running.

And it broke, though not anywhere the six hours had been looking. Our changes went in correctly. The executor then compared a checksum of the patch text, and because Git abbreviated object IDs to eight characters in the small test copy and ten in the much larger production repository, the text differed though the files didn't. It decided the patch was corrupt and started a rollback, which failed straight away trying to save the failed state into a directory whose parent didn't exist. The service definitions also pointed at a Python environment the upgrade had removed, so the agents' gateways couldn't start. Another agent repaired the services, the SAA finished the job, and by mid-afternoon v0.22.5 was running with all 47 changed files verified intact.

The obvious objection is that I rushed it and got what I asked for. I don't think the evidence supports that, and neither did the SAA's own review: "I had substantial preparation and test evidence, but it did not adequately cover the exact executor and recovery path I ultimately ran." None of the three faults was in anything the morning's checks had covered, and all three were in steps that had never been run the way they were run that day. More testing would have helped only if it had been testing of the rollback actually used, which is a much smaller job than the one being done.

Preserved, superseded or retired

Upstream doesn't stand still while you preserve your changes. In October four of our 51 changed files were dependency versions where upstream had since shipped later fixes than ours, so we took theirs and dropped ours. Upstream had also removed a command-security integration we used, having moved its key checks into the built-in detector, and a configuration migration would have deleted our settings for it. I made that call knowingly before the upgrade, and that's the point, because finding it missing afterwards would have been a very different conversation.

You can't do any of that without knowing what you changed and why, so the kit includes a register: one entry per change, saying why it has to be in core code, who approved it, how you prove it still works and when you'd retire it. Hermes, meanwhile, now reports that it is "Up to date", which is true of the release and says nothing about the 47 files we've changed.

In A Tale of Two Harnesses I argued that frontier models can now write and maintain a bespoke harness. Customising someone else's is the halfway house, and this is its maintenance bill. We still carry our changes, so for now the bill is worth paying, but each one has to justify itself at every upgrade.

Practical advice

1. Put customisation outside the harness's own code wherever you can: configuration, skills, identity files, plugins. Change core files only when nothing else gives you the behaviour.
2. Keep a register of every core change, with its reason, approval, proof and retirement condition.
3. Capture local changes with git diff HEAD, or against the upstream base, never plain git diff, and compare checkouts by content, not by checksums of patch text.
4. Check the result in a fresh session. The conversation you're in may still be running on the old instructions.
5. Make the upgrade's acceptance test that every local change is preserved, superseded or retired. Exercise the rollback through the path you'll actually run, then stop testing.
6. Size the caution to the estate. If an outage is tolerable, say so, and be prepared to fix forward.

local-change-register.md, upgrade-reconciliation.md, guarded-update-routine.md and capture-effective-patch.py are generalised versions of what we use. The script runs its own tests, including both of the mistakes above.

This concludes the SAA series. My practical advice to use the SAA repo is to point your favoured capable agent at it and tell it to stage it on your server, then have it conduct a Q&A with you to establish its 'identity' on your estate. I plan an addendum to this series that will cover agentic failure modes for persistent long running tasks in a changing environment. I am still collating data on that. If you want advice or have any comments or corrections on this series, feel free to email me.