Guide for building an autonomous agentic system administrator

Part 4: Owning the maintenance

This is part 4 of my series on building a sysadmin agent. Part 3 covered watching the systems and responding when something breaks. This part covers the rest of the job: the routine work that stops things breaking, the backups for when they do, and making the SAA (sysadmin agent) prove that something has recovered rather than assume it. The files are on GitHub: https://github.com/Sjw22263/sysadmin-agent-kit/tree/main/04-owning-the-maintenance

Give it the job you would give a person

In July I gave the SAA its routine duties in five lines: work out what a competent system administrator does daily and routinely, list it, script what can be scripted, do the rest each day, and take the necessary action, including updates, upgrades and security patches. Then I told it why: "I am giving you the same tasks I have used in job goals for system administrators I have employed in the past. I figure this will be both a good test for really useful agentic work, and hermes as a suitable harness."

Its reply set the standard better than I had: "The honest test is not 'can Maxi administer two Linux hosts?' It is whether, after weeks of ordinary operations and a few real faults, you find yourself trusting that the boring work is genuinely covered without having to supervise every move."

The result is a duty table, daily, weekly and monthly, and each duty has a control. Scripts collect the facts and the SAA handles the exceptions, the same pattern as part 3. The daily maintenance audit runs at 4:15am, after the backups, and checks package updates, pending reboots, failed services, disk and memory, and backup freshness. Most mornings it finds nothing and no model runs. When it does find updates, the SAA applies them and then checks the services and websites that depend on them, because the package manager exiting cleanly doesn't tell you the websites still work.

Routine is not unlimited

The SAA applies ordinary updates without asking, meaning the standard upgrade, which installs nothing new and removes nothing. Full upgrades, removals, driver changes and new software sources need a decision. When I approve one, the approval names the exact transaction, it is checked against the real transaction immediately before installing, and it is used once. An approval that isn't used up stays around, and sooner or later something treats it as standing permission.

A decision not to act gets written down too. One server has a GPU driver update that the package manager holds back, and nothing on that server uses the GPU. Without a record of that decision, and of what would reopen it, the agent would raise the same package every morning, or eventually 'fix' it.

Backups: keep history and fail loudly

An early backup plan used fixed weekly and monthly file names. It looks tidy, but if the data is corrupted while nobody is watching, each tier gets overwritten with the bad copy in turn, and you end up with three copies of the same damage.

The rules we now write into new and changed backup jobs are:

• Keep dated generations.
• Verify a new generation before it counts.
• Re-verify the generations you keep, because an archive can go bad while it sits there.
• Treat an archive without its checksum file as a fault, rather than quietly making a new checksum.
• Fail if the backup drive isn't mounted, rather than filling the system disk and reporting success.

Not every job we already run meets all of them, which is the normal state of an estate that grew before the rules did. Successful backups no longer send a message at all, only failures do.

Both of our backup copies are in the same building, which protects us against a failed drive but not against a fire. It is worth knowing the same about your own.

If you use Hermes, two cron behaviours matter here. When a job's script reaches its timeout, Hermes kills the whole process tree outright, so a shell exit trap never runs. A backup that stops a service and relies on the trap to restart it will leave the service stopped. And check the timeout for your version: it was 120 seconds until upstream raised it to an hour in late June. Our own notes still said 120 seconds when we prepared this post, which is why every file in this series carries a version line. We run long backups on the system scheduler anyway, so they don't depend on the agent platform being up.

Recovered is a claim, so check it

After a power cut in August, the boot times on both servers looked exactly as if they had switched themselves back on when the power returned. They hadn't, I switched them on manually. One server's firmware has no option to power on after a failure at all. A boot time only tells you that the system started, not how it started, and the SAA's records now say so. Automatic power-on stays unverified until it has been tested.

The same goes for the agent's own repairs. After that power cut the SAA ran a missed backup by hand, as root. The backup worked, but it left a lock file owned by root, and the next morning's scheduled run, as the ordinary user, could not open it and failed. The SAA's explanation began, "My post-outage recovery run caused it." It moved the lock, tested the root-then-ordinary-user sequence, and reran the backup. So after any manual recovery, check that the next normal run actually works, because the recovery itself can be what breaks it.

A restore is the real test of a backup. We restore into a scratch directory and compare it with the source when each backup is commissioned, and several of the jobs verify the checksums of the archives they write. While preparing this post we found a gap anyway. Our monthly row asked only that a representative archive be readable with a valid checksum, and a weekly job already does that much. Nothing was scheduled to take a backup apart again and compare it with the source. So the weak version of the duty had a control, and the version we actually wanted had never been written down as a duty at all. It is worth reading your own monthly rows twice, once to check that something runs them and again to check that they ask for the test you think they ask for.

Check before you retry

One night the job that copies our backups to a second device reported failure. Its sender had timed out waiting for the far end to finish verifying, and the far end finished three minutes later and published a good backup. Taken at face value, that was a failed backup and a reason to run it again, and it was neither.

When a change ends ambiguously, the error message and the real effect are separate evidence. Check the real state, retry at most once if it really did not happen, and stop if you cannot tell. Here the fix was a longer deadline. When the job was rerun to test the fix, it found the backup already there and didn't write another, because it checks for that generation before writing. So design a job so that running it again can't leave two copies, and still check the outcome afterwards rather than trusting the design. verify-before-retry.md has the procedure, and it is the shortest one in the kit.

Practical advice

1. Give the agent a system administrator's duties, and give every duty a control. Check both halves: that something actually runs the duty, and that the duty asks for the test you meant.
2. Let it apply routine updates, put everything else behind a decision, and make one-off approvals exact and single-use.
3. Keep dated backup generations, re-verify the old ones, and restore one on a schedule.
4. Treat boot times, exit codes and error messages as claims, and check the real state before acting on them.
5. If you use Hermes, check the cron script timeout for your version, and don't rely on exit traps in jobs that can time out.

Next: reflection, and how to tell whether an agent's daily self-improvement work actually changes anything.