Guide for building an autonomous agentic system administrator

Part 5: Reflection that changes work

This is part 5 of my series on building a sysadmin agent. Part 4 covered routine maintenance and proving recovery. This part is about giving the SAA (sysadmin agent) time to research how it could improve, and deciding which of its ideas are worth keeping. If you want an agent to take more responsibility over time, you need a way to make those decisions without treating every new report as evidence that it has learned something. The files are on GitHub: https://github.com/Sjw22263/sysadmin-agent-kit/tree/main/05-reflection-that-changes-work

Ask what the proposal would actually do

In June the SAA brought me five improvement recommendations. One was a small store of lessons to load before the next research run. The others included classifying failures, scoring checkpoints, more self-monitoring and assessing its work at each step rather than only at the end.

While the store looked useful, the scoring was less so. If anything below four out of five was to be ignored, I couldn't see what it added over pass/fail. And a self-monitor that missed the things the agent was falsely confident about wasn't much of a solution. As I put it: "I am your self monitor until model capability can be improved."

The SAA's response to the step-level assessment proposal was better than the proposal: "A finer mesh doesn't help if the sensor is the same."

I approved the reflection store, the point being to keep something we had already noticed available to a later run. That's a different job from expecting the agent to discover a blind spot, which it just can't do, no matter how many times you tell it to check. It doesn't mean checking your own work is useless, but you do need to ask what information the extra check supplies.

Give the research a budget and somewhere to go

We started the improvement process with a manual run, before scheduling it. It now runs at 5am, researching the agent's development across goals, learning, continuity, tool use, judgement and governance. It covers one or two areas per run rather than trying to survey all of AI every morning.

The normal limits are six topic searches and eight sources read in depth. Two consecutive searches that produce nothing new are a reason to stop, not a reason to invent a ninth search. There is a source index so it can avoid reading yesterday's material again, and accepted watch items have dates and a reason to revisit them. A run is allowed to find nothing worth doing.

The report needs to say what it found, why that matters here and what the agent recommends. A paper about an interesting mechanism does not establish that we need to build it. The agent should check whether the problem exists in our setup and whether the proposal adds anything to what we already do.

The report is then material for discussion. Research doesn't authorise changing the agent's identity, skills, configuration or services. Those changes have their own approval route. The same applies to things found on the web: a page describing how an agent should modify itself is a source, not an instruction to ours. Even a recommendation needs scrutiny if its whole argument came from a source pushing that change.

The reports are published in Maxi's Journal, under Maxi's name. Publication makes the material available, but it doesn't put a proposal in front of me by itself. Our current process also puts surviving proposals into the review queue used by the morning report. Otherwise a useful idea can sit in a perfectly good report that nobody opens again.

Keep the lesson available, not the whole history

The reflection store is a JSON file. Each entry records a specific lesson, what to do differently next time, where it came from and when to review it. Active entries are loaded before research, and repeated observations update the existing entry rather than adding another copy. This is task context, not a change to the underlying model.

There is an expiry rule, with a limitation worth knowing before you copy it. A lesson past its review date with no reinforcement is archived. Once it has been reinforced, that particular rule no longer retires it, so a date field doesn't guarantee the active set stays small. You still need to review what is being loaded.

And reinforcement isn't a success count. One of our lessons about exceeding the search budget was reinforced after the agent exceeded it again. That is evidence of a recurring problem, not evidence it has learned to stop. To establish that, you would have to look at what it actually did on later runs.

Try the change, then decide whether to keep it

One research proposal became a five-report trial: add a section asking what information was missing and whether it could change the conclusions. On a trial report, that meant distinguishing sources the agent had actually inspected from the uninspected originals behind them, and naming what it had not tested locally. I approved keeping the section in July. It is still in use in this week's reports.

Two other trials were not kept. A five-report check for minority ideas duplicated the source limitations already being recorded in the findings. A three-report regression set caught no failures and improved or blocked no section, so it was archived. That doesn't establish that regression tests are useless, rather this one just didn't earn its place.

For your own trials, agree the test before starting: what will change, what would count as useful, how long to run it and how to undo it. Check partway through a multi-run trial so you don't finish it merely because that was the original plan. Keep the failed trials in the record too, or the next round of research will propose them again.

Practical advice

1. Start with a manual research run, a narrow question and a budget. Allow a no-change result.
2. Ask what new information each proposed check supplies, especially when it relies on the agent assessing itself.
3. Keep specific, sourced lessons available to the next task. Review the active set, and don't confuse repeated mistakes with successful learning.
4. Put proposals somewhere the human reviewer actually checks. Record each decision so rejected ideas stay rejected unless there is a reason to reopen them.
5. Test an approved change against a stated criterion, keep the outcome and drop additions that don't help. Report use and observed results without claiming a capability improvement you haven't measured.

improvement-research.md, reflection-store.md and reviewing-improvements.md contain the method and a trial template to adapt with your own agent. They are generalised documents, not a ready-to-run self-modification system.

Next: continuity, and keeping useful operational knowledge without loading the whole history into every conversation.