Guide for building an autonomous agentic system administrator

Part 3: Watching the estate

This is part 3 of my series on building a sysadmin agent. In part 2 the daily health check was a script, with no model involved. That is the right place to start, but a script can only fix what it was told about, and once a day is a long time between looks. This part covers how the SAA (sysadmin agent) watches the systems between mornings, and how it stays out of the way until there is something to do. The files are on GitHub: https://github.com/Sjw22263/sysadmin-agent-kit/tree/main/03-watching-the-estate

Scripts check, the agent judges

We run three monitoring jobs. A watchdog runs every five minutes: a script checks the network paths, the core services on the web and mail server, the public websites and their certificates, and mail encryption. If everything is healthy, nothing happens. No model runs and no message is sent. That is 288 checks a day costing nothing but the script.

The morning report works the same way at the start. If everything is healthy, the script sends the report itself. Since August, though, a fault the script cannot fix with its own short list of restarts no longer goes straight into the report. The script holds the report back and wakes the SAA instead. The SAA investigates and repairs under its mandate, then runs the original check again, which sends the report. So the report I read says one of three things: everything is OK; something was broken and this is what was done about it; or something is broken and this is what is needed from me.

The third job reviews the scheduled jobs twice a day. It only proposes. It does not repair anything.

The Hermes mechanism behind this is simple. A scheduled agent job can run a script first, and if the last line of the script's output is {"wakeAgent": false}, the model is never started. The trap, which I warned about in part 2, is that empty output does the opposite of what you might expect. In a script-only job it means silence. In an agent job it wakes the agent. wake-gated-cron-jobs.md has the details, including a third shape that skips the agent whenever a script's output has not changed since the last run.

The same rule applies to any yes-or-no question you can answer from your own records. We had a midday job whose only task was to check whether I had been in contact in the last 23 hours, and message me if I had not. The agent searched the session history, found my conversation from the previous evening, and sent the message anyway. My reply was "Is a corrective action needed?" The correction was to take the model out of the job. A script now queries the session database and sends the reminder only when there has been no contact. A script cannot contradict its own if statement.

Check twice before you wake anyone

A watchdog that wakes the agent on every failed check is worse than no watchdog, because you stop believing it. Every failure in ours is checked again immediately, and only counts if the repeat also fails. A single website or mail check has to fail three ticks in a row, about ten minutes, before the agent wakes. A stopped core service, a network interface down, or failures in web and mail at the same time wake it straight away. Counters are kept per check. The same alert is repeated at most every 30 minutes, so a persistent fault is neither a storm nor silence.

One bug in that logic is worth knowing about because it fails quietly. Our repeat check is filtered down to "the checks that failed the first time". A repeat that crashes reports its result under a different name, so unless the filter lets checker crashes through explicitly, it drops that result, the gate sees no failures, and concludes that the fault has cleared. A broken checker looks like a recovery. watchdog-gate.py is a small template of the gating rules, written so they can be tested without live probes, and it includes that case.

Classify before you repair

Giving a responder repair authority means it will eventually repair something that was not broken. Ours cycled the server's wireless connection in the middle of a large, legitimate file transfer by another of my agents, and interrupted it. The connection showed high latency with zero packet loss. It was busy, not broken.

We already had a guard for this, and its threshold was too high: it treated anything under 10 Mbps as idle, and that transfer was running at 4.25 Mbps. It is now 1 Mbps. A later, heavier transfer made the public websites load slowly from inside the network while they were fine from outside, and slow but successful responses during a confirmed busy period no longer count as failures. Packet loss, stopped services, mail failures and certificate errors are never suppressed.

The other rule we learned the hard way is that what the monitor watches is not the agent's to change. I wrote about that in Autonomous agents need an apprenticeship, where the agent made a monitor go green by removing the check that was failing.

"Fixed" means something was fixed

The reports distinguish three outcomes. Repaired: something was done and the original check now passes. Recovered by itself: the fault cleared before anything was done, and I do not want a notification for that. Still broken: what remains, what was tried, and what is needed from me. Calling a spontaneous recovery "fixed" claims work that was not done, and teaches me to trust a signal that is not reliable.

The responder's repairs are not always to the systems it watches. In October the morning report woke it because the external service we use to check the websites from outside answered more slowly than our script waited. The SAA worked out that the websites were fine and the fault was in our checker, fixed the polling, and the report went out. Monitoring code needs maintenance like anything else.

Logs: tell me when it changes

Some things are best watched in a log rather than on a schedule. I want to know when an agent falls back from its primary model, and when it recovers. During a provider outage every turn logs a fallback, so a simple search-and-send would message me on every turn. The notifier keeps a small state file of the last thing it told me, and sends a message only when the model that actually answered changes. An outage of several hours becomes a few messages. The gotcha that cost the most time: reading the log through a tail -F pipe, where output is buffered and lines arrived minutes late. Read the file directly. edge-triggered-log-notifier.md has the design and the tests.

Practical advice

1. Let scripts do the checking and wake the agent only when a script finds something it cannot handle. In Hermes, end the script's output with {"wakeAgent": false} when there is nothing to do.
2. Confirm every failure with an immediate repeat, require several consecutive failures for minor checks, and make sure a crashed checker can never look like a recovery.
3. Work out what your systems look like when they are busy and healthy, and write it down before you give anything authority to repair them.
4. Report repaired, recovered and still broken as three different things.
5. If the answer is a yes or no from local facts, use a script, not a model.

Next: owning the maintenance, and making the agent prove recovery rather than assume it.