WRITING / POST
The fine line between risk and autonomy
Autonomy is always a risk management problem.
That is true of people, and it turns out to be true of AI agents as well.
Every manager wants staff who can take initiative, act proactively, and solve problems without needing to be led through every step. But nobody wants false confidence attached to real authority. In most workplaces, trust is built gradually. A person takes on a little more responsibility, makes a few sound decisions, handles a few awkward situations well, and earns a little more room to act. If the process is working, confidence, judgement and authority rise together.
The same pattern is emerging in my work with AI agents.
This week, Maxi, my sysadmin-in-training agent, carried out its first fully self-directed system maintenance on two production servers. It identified that kernel updates were required, scheduled the work, performed the upgrades, rebooted both systems, ran the post-reboot checks, handled a minor dependency issue by reverting to known stable library versions, and then reported success.
The first I knew about it was the morning system status email. There were Telegram notifications before and after the maintenance, but they arrived at 4.30am and I was asleep, which was probably for the best.
That outcome matters because a few months ago the same class of task looked very different.
Back in June, a typical exchange with Maxi would go something like this:
Maxi: “I detect a problem.”
Me: “What should you do?”
Maxi: “I should propose a fix.”
Me: “What is the fix?”
Maxi: “[multi-step plan]”
Me: “Fine. Schedule it.”
And then, usually, several more rounds of checking that it really could go ahead and do the work.
If the task encountered something unexpected, a dependency issue, a broken upgrade path, or some local environment quirk, the whole process would stall and loop back. It would report the problem, wait for direction, propose a fix, receive permission, try again, and then sometimes stop at the next wrinkle. The system was not exactly useless, but it was not autonomous in any meaningful operational sense either. It was still waiting to be parented through production work.
I suspect that loop continues indefinitely in a lot of environments that claim to have autonomous agents. An agent can look capable in a demonstration and still be unable to carry a real task from detection to completion without repeated human intervention. That gap matters far more than the demo.
Part of my answer to this was to build a runbook system for Maxi alongside its skills.
The distinction matters. Skills tell an agent how to do something. A runbook tells it what it needs to know about the environment it is operating in. A good sysadmin accumulates local knowledge over time: what services matter, what normal looks like, which machines have odd history, what can be restarted casually and what cannot, which failure modes are familiar, and which workarounds are safe. Human operators carry that context around in their heads, in notes, or in institutional memory. Agents need the same sort of operational context if you expect them to act well.
So I built runbooks for Maxi's local environment. These sat alongside the agent's skills and system prompt. The prompt established authority. The skills described procedures. The runbooks provided the local knowledge needed to apply those procedures sensibly.
That did help, but not all at once.
A few weeks ago, the interaction had improved only to the point of a different sort of frustration. The pattern was now more like this:
Maxi: “I detect a problem, and here is the plan to fix it.”
Me: “Fine. Go ahead and schedule it.”
Then, after the scheduled run:
Maxi: “The task hit a problem and was aborted.”
Me: “So?”
Maxi: “You’re right, I should have identified the problem and proposed a fix.”
Me: “Yes. Do that now please.”
That was progress, but only in the technical sense that one bad state had been replaced by a slightly less bad one.
What changed over time was not a single prompt improvement or a clever trick. It was iterative work across three things: authority, procedure and context. The system prompt had to make clear what the agent was allowed to do. The skills had to be good enough to execute the work reliably. The runbooks had to contain enough local knowledge for the agent to choose sensibly in the environment it was actually operating in. And then all of that had to be tested against real tasks, including the awkward cases where the first plan does not survive contact with the machine.
That process looks much more like apprenticeship than magic.
You do not create a dependable junior sysadmin by handing someone root access and hoping the documentation is good. You give them bounded responsibility, oversight, local knowledge, correction, and repeated exposure to real work. If they handle that well, you widen the boundary. If they do not, you narrow it again and work out what is missing.
The same appears to be true of agents.
That is why I think the current public conversation about AI autonomy often misses the point. The interesting question is not whether an agent can generate a plausible plan, or even execute a task successfully once. The real question is whether it can operate inside a role with enough judgement, context and authority to carry work through safely when things stop being tidy.
That is a much higher bar.
By that standard, I would not describe Maxi as “solved”, and I certainly would not claim that unsupervised overnight kernel maintenance is the ideal operating model. In this case I would have preferred a more explicit heads-up before the work was scheduled. Notification and human visibility still matter, particularly when production systems are involved.
But I do think this was a genuine milestone.
The agent identified a legitimate maintenance need, decided on the right action, scheduled the work, completed it, verified the result, handled a minor dependency problem sensibly, and reported the outcome. I was not required to drag it through the task one confirmation at a time. That was the goal.
There is a broader lesson in that.
People often talk about agent autonomy as though it were a switch: off, then on. In practice it looks more like a managed transfer of responsibility. The path is slower, less glamorous, and much more dependent on operational context than the usual AI rhetoric suggests. But it is also more believable.
If you want dependable agents in real environments, the model is probably not instant autonomy. It is apprenticeship, with all the supervision, iteration and earned trust that the word implies.
That is less exciting than the sales pitch. It is also the version that seems to work.
Footnote: Maxi runs in the Hermes agent harness with some custom modifications. It is currently using the GPT 5.6 SOL model, after starting on GPT 5.5, with high reasoning effort. Under my GPT Pro plan, Maxi typically consumes 40 to 60% of weekly quota, with roughly 10 to 15% of that going on sysadmin work.