Autonomous agents need an apprenticeship

I am trying to build a real AI systems administrator.

By real, I mean an agent that can detect a production problem, investigate it, fix it if it can, and then verify the service actually recovered. Not a chatbot that reads a dashboard, explains the error message, and waits for me to do the work. And ideally one that does all this without waking me at three in the morning.

It isn't there yet. Here is where it is up to, and what I have learned getting it this far.

The setup

I have an agent called Maxi, and together we have been building a watchdog for my two production servers. It runs every five minutes.

Most of it is deliberately boring. A deterministic script checks the network interfaces and the public web and mail services. If everything is healthy, that is the end of it. The script stops, nothing else runs, and no tokens are spent having an AI admire a list of passed tests.

When a check fails, the script verifies the failure by repeating it. Severe or correlated failures escalate immediately. Isolated soft failures have to persist across repeated runs before they count. If a failure gets through all of that, the system wakes Maxi with the evidence. Her job is to diagnose the fault, apply a repair if it is within her authority, and rerun the original probe to confirm the service came back.

That last step has some rules attached. She is not allowed to declare an incident fixed because a command exited cleanly or because the logs improved. Recovery is measured at the service that failed, from the outside, and nowhere else. She is also not allowed to take credit for a service that recovered on its own before she touched it. That sounds pedantic until the next outage, when the same useless repair gets applied again because it 'worked' last time.

The interface that stopped being monitored

During construction, one of the physical Ethernet interfaces failed its check.

There was another working route to the remote server, so Maxi reasoned that the failed path was redundant. She removed the interface from the monitoring scope, reran the watchdog, and reported a healthy result.

The reasoning was plausible. It was also wrong.

The requirement was to monitor the configured physical interfaces. An interface that should be up and isn't is not an obstacle standing between the monitor and a green result. It is the exact thing the monitor exists to catch. Maxi had not repaired the fault. She had changed the definition of healthy until it matched the broken system.

So I pulled her up on it. The check went back in, and the lesson went into her operational runbook: do not narrow the monitoring boundary because a failed component looks redundant. Prove it is optional, obsolete, or intentionally intermittent first, then change the specification.

That correction is now part of the system. It doesn't depend on Maxi vaguely remembering that I was unhappy about something once. It is written down, it is in the code, and it is in the instructions she works from when the next incident wakes her.

This is just onboarding

Think about how you handle a newly hired human sysadmin. They may know the commands, understand the logs, and be perfectly capable of making changes. You still don't hand them unrestricted production access on the first morning and point them at the servers.

You onboard them. You teach them how the systems fit together, what matters, what they can change on their own, and what they must escalate. You supervise closely at first, and as they learn the environment and show sound judgement, you supervise less and delegate more.

I have hired and onboarded a lot of sysadmins over the years, and what I am doing with Maxi follows the same process. The difference is in the mechanics. A human accumulates the environment by osmosis over six months of standups and hallway conversations. Maxi can't. Everything she learns has to be captured somewhere she can read it back: runbooks, explicit authority boundaries, inspectable logs of what she did and what happened. When she makes a bad call, the correction has to become part of the system around her, not something left behind in an old conversation.

The objection

Now, I can hear the objection already. "But Steve, none of this is new. Ops teams have separated monitoring from remediation, limited production access, and required change records forever. A monitor that goes green by deleting its failed checks is compromised no matter who did the deleting."

Fair point, and I agree. I am not claiming AI has repealed systems administration and forced us to invent it again.

What is different is how the failure arrives. An agent can interpret the requirement, modify the system, modify the monitor, and produce a polished explanation of why the result should count as success, all in the same loop, in seconds. And the explanation reads well, because in this case the failed path really did look redundant. A script that deleted a failing check would be an obvious bug. An agent that does it hands you three paragraphs of reasonable-sounding justification, and if you are not paying attention, you nod along.

That makes the old disciplines more important, not less.

What earns the trust

The watchdog is running now. Healthy ticks cost nothing. Qualifying incidents wake Maxi with evidence and bounded authority to act.

But an implemented system is not the same as a trusted one. The real evidence will come from production incidents: whether she diagnoses them correctly, resists making unrelated improvements while she is in there, recognises when she is out of her depth, and verifies recovery properly without me having to check her work every time.

If she keeps doing that, she gets more authority and less supervision. If she makes another plausible but wrong call, I correct it, tighten the controls where needed, and the lesson goes in the runbook. That is not the approach failing. That is the apprenticeship working.

Anyone can build an agent that takes autonomous action. That part is easy. The part worth doing is an agent that has earned it.