Guide for building an autonomous agentic system administrator

Part 9: Who may change what the agent learns

This is part 9 of my series on building a sysadmin agent. Part 8 covered the second opinion agent. This part is about skills, the procedures an agent loads when a task calls for them, and who gets to change them once there are several agents learning at once. A sysadmin agent's skills are its operating procedures on your systems. If they change without anyone reading the change, or drift apart between agents, or grow until you can't say what's in them, you no longer know what your agents are doing. The files are on GitHub: https://github.com/Sjw22263/sysadmin-agent-kit/tree/main/09-who-may-change-what-the-agent-learns

Skills need review, memories are the agent's

Part 1 warned that a Hermes agent will, by default, write its own skills and memory without asking, and that we found this out in September when one agent's background review rewrote its procedures and its reading of what it was allowed to publish. I wrote about that at the time in Nous, what were you thinking?, so I won't retell it.

The emergency fix was to gate both so two days later I put it to the SAA (sysadmin agent) more simply: "Skills need review, memories are yours. Agree or not?" Skills are procedures the agent follows, and that other agents may share, so a change to one needs someone to read it. Memory is different, because an agent needs to carry context from one session to the next, or every conversation starts with fifteen minutes of rediscovering and re-explaining everything, and gating every memory write gets in the way of that.

That doesn't mean agents are good at it out of the box. By default they are terrible at deciding what's worth keeping, and when we started on Hermes the SAA's memory quickly bloated with trivia while important durable facts were lost. What fixed it was careful wording in the skills about what belongs in memory, rather than a gate, along with the runbook system from Part 6, which gave detail somewhere else to go. With those in place, now I can trust agents to remember what is important and discard the trivial. So skill changes are staged for review and memory writes go straight in. You may draw the line elsewhere, but draw it on purpose, and if you leave memory open, give the agent clear guidance on what to keep before you trust it.

One shared method, local practice kept separate

With several agents, the same procedure ends up in several places. Each agent learns its own way of sending mail or checking a service, and the copies drift. We fixed that in September with one shared library maintained by the SAA, each agent selecting only the shared skills its job needs, and anything genuinely specific to one agent kept in a small local addendum with its own name. My instruction when we set it up was to start from the stock Hermes skills and "do not conflate changes specific to your use with an improvement to be shared."

The Hermes gotcha here is worth knowing before you start. If an agent has a local skill with the same name as a shared one, the local one wins. Fix the shared skill and that agent never sees the fix. Give local addendums distinct names and get rid of stale local copies.

A review route that doesn't drown you

Gating skill writes only moves the problem if nobody looks at what's queued. Our route for a substantive change is the exact wording, its effect and the author's recommendation, then my decision. If it's straightforward I approve it directly. If it's complex or contested it gets an independent review first, by an agent that didn't write it, and the author decides what to take from the review before it comes back to me.

The automatic proposals get a weekly triage. Each agent reviews every unresolved pending change to its own skills and emails me, one numbered entry per change, with a recommendation based on three questions: is a durable change actually needed, will it materially improve the work, and will it make the work simpler once you count the extra instruction? More rules aren't automatically better, and the agents are expected to say so. Nothing is approved, rejected or archived by a scheduled job, and every unresolved item comes back every week until I decide it.

A procedure nobody can find doesn't exist

We had all of that written down and still had an agent fail to follow it, because nothing in its own instructions said where the procedure was. The SAA looked and found no route at all from that agent's identity file or tasks to the procedure. The fix wasn't to load the whole procedure into every agent's context. As I said at the time, "The desired outcome is just that the agent knows where to find the information if they need it." The procedure now lives in one file every agent can read, and each identity file has a short pointer to it that says when to read it.

Test that from the agent's side. Start a fresh session and ask it how it would propose a change to one of its own skills.

A library you can hold in your head

By late September the estate had 242 skill packages and 45 pending proposals, and I could no longer say what was in it. My brief to the SAA was "one, understandable, taxonomy for all skills with the underlying principle of simplicity and necessity", because "once it gets too complicated that the parts no longer cohere in my head, then effective control is lost."

The result was eight categories based on kinds of work, a rule that a new lesson goes into an existing skill unless it really is a new job, and an actual move of every package rather than a registry laid over the old folders. The pilot was honest about the trade-off. The new structure loaded 18.3% less skill text but needed more retrievals and more model calls, and the SAA's verdict was "That is not a clean efficiency win." We went ahead because the point was a library I could understand and govern. The full account, and the three approaches that didn't work first, is in Human in Command.

Tidier folders aren't proof of better behaviour, and I haven't measured whether the agents work better for it. What I can say is that I can now find things, and so can they.

Practical advice

1. Decide which of the agent's stores need review and which are its own. For a sysadmin agent, review skills.
2. Keep one shared source for common methods with one owner, let each agent pick what it needs, and keep local detail in distinctly named addendums.
3. Write down one route for skill changes: exact proposal, your decision, independent review when it's complicated, then verify in a fresh session.
4. Triage automatic proposals regularly, ask whether each one is needed and makes the work simpler, and keep every unresolved item visible until you decide.
5. Put a pointer to the procedure where each agent looks, and test that it finds it.
6. When the library outgrows your head, reorganise it for human understanding first, and don't expect it to save tokens.

shared-and-local-skills.md, skill-placement.md, skill-change-review.md and skills-taxonomy.md are generalised versions of what we use. The Part 1 file on how a Hermes agent changes its own instructions has the settings. For when a mistake should and shouldn't produce a new rule, see Corrections Consolidate.

Next: the cost of customising a harness that changes thousands of times a month.