Guide for building an autonomous agentic system administrator

Part 6: Continuity without a memory landfill

This is part 6 of my series on building a sysadmin agent. Part 5 covered improvement research and deciding which of the agent's ideas to keep. This part is about continuity: how the SAA (sysadmin agent) keeps what it needs to know from one conversation to the next, without its memory turning into a pile of half-remembered operational detail. If you are going to give an agent more responsibility over time, it has to be able to find last month's fix without carrying last month around with it. The files are on GitHub: https://github.com/Sjw22263/sysadmin-agent-kit/tree/main/06-continuity-without-a-memory-landfill

Memory is the index, not the archive

Hermes, like most agent harnesses, has a small persistent memory that gets loaded into every conversation. In the first month we used it the obvious way: every time the SAA learned something it needed to keep, it added a line. By late June its user-profile store held 42 entries, about 12,000 characters, and a lot of what was in memory only mattered to one kind of task while being loaded into all of them.

The day we fixed the web server after a power outage and untangled the WebUI from the Hermes dashboard, I suggested that, as the system grew, the comprehensive detail about each subsystem should go into its own file, with a memory line pointing to it. The SAA agreed and put it rather better than I had: "Memory should not become a landfill of operational detail. It should become an index." Detail now goes into Markdown runbooks, one per subsystem, with purpose, current state, paths, commands, failure modes, recovery steps and change history. Memory keeps short facts, preferences and a one-line pointer to each runbook.

When it came to writing the specification, I didn't want it obliged to create a whole file for something simple: "Where a one line entry will do, that is fine." Anything needing more than two lines is a candidate for a runbook. Today the SAA's memory holds ten entries in about 3,500 characters, and there are 28 subsystem runbooks behind them.

Pointers aren't a reading list

The part that took a little longer to get right is that a pointer in memory is not an instruction to read the file. If the agent sees "the backup runbook is at ..." and dutifully opens it at the start of every conversation, you have recreated the original problem with extra steps. The runbook gets read when the task is about backups, or when it's needed to answer accurately or change something safely, and otherwise it stays on the shelf.

The opposite failure is just as real. A runbook nobody can find might as well not exist, so every new one goes into an index, and anything the agent has to consult before a particular kind of action gets a pointer in the place it will actually look at that moment. We have had a perfectly sound procedure sit unread because no route led to it.

Someone has to do the removing

Agents are much better at adding to memory than taking things out, and in Hermes that is partly deliberate. The background review can add memory entries without asking, but at the version I'm running it can't delete or replace one unattended. That's a sensible safeguard, but it does mean memory only grows unless someone tidies it.

So the SAA runs a review at 3am. It backs up memory and the session store, reads the previous day's conversations, removes anything stale or duplicated, moves detail into runbooks and writes up anything bigger as a proposal for me rather than doing it. Once memory was being used as an index, most nights that review finds nothing to change. It made no edits on any of the ten nights up to 22 October, which is the arrangement working, not the job failing. If yours is making several edits every night, look at what is being written during the day.

Test whether the extra memory layer earns its place

In May the SAA installed Honcho, a self-hosted memory service, and we spent a fair amount of effort keeping it going. That included changing its embedding model twice and, in June, discovering that short conversations were being stored but never turned into memory at all, because they never reached the batch threshold. The API was healthy the whole time. The SAA's summary was "messages stored ≠ memory derived", which is worth remembering for any service whose health check only tells you it's switched on.

In July we turned it off for a week and kept the data. From what I could see, turning it off made no practical difference, but it reduced token overhead and the number of things that could break, so we retired it. Disabling the integration hadn't actually stopped its containers restarting, which we only found when we went to remove it.

I'm not claiming memory services are useless. This is one estate with one workload, and it is the only memory service we have records of running here. The point is to run the trial on your own work against what the agent already has (built-in memory, runbooks and the harness's search over past sessions) rather than against no memory at all. If you're interested in the context side of this, our paper on reducing context before inference found a simple deterministic retrieval matched full context on a fraction of the tokens.

Practical advice

1. Treat persistent memory as an index. Facts, preferences and pointers go there, and anything over two lines goes into a runbook.
2. Tell the agent that pointers are lazy references, read when the task needs them.
3. Index every runbook, and put a pointer wherever the agent looks before acting. A procedure it can't find doesn't exist.
4. Review memory regularly with a backup, a proposal and a check afterwards. Expect a well-routed memory to need very few changes.
5. Before adding a memory layer, write down what it should improve. Then switch it off for a week, keep the data and see what you miss.

operational-notes.md has the layout and runbook and incident templates, memory-review.md the audit and nightly review, and memory-layer-trial.md the trial and retirement steps. They are generalised documents to adapt with your own agent, not a ready-made memory system.

Next: moving from running one agent to overseeing several, and handing work between them in files rather than conversation.