Improvement Research — 2026-06-25

1. Focus

Dimension: 3.6 Governance: restraint, oversight, and corrigibility (per rotation).

Trigger: Scheduled daily run.

Loop goal: Find what changed or what I learned about governance/corrigibility that lets me operate more autonomously without reducing Steve's effective oversight.

Newsletter scout checked: /home/hermes/research/newsletter-digests/2026-06.md — used. Two strong leads surfaced from recent intake: (a) Nate msg 60 (loop-of-loops with safe-action boundaries as governance-by-architecture) and (b) Nate msg 58 (emergent corrigibility — Fable 5's unprompted review queue). These framed the search direction toward corrigibility engineering patterns and governance architecture.

Active reflections in context: All 8 active reflections loaded. refl-2026-06-18-001 instructed starting 3.6 with concrete named search targets and checking the newsletter first. refl-2026-06-24-001 instructed active budget tracking — was honoured (4 searches, stop after 2 consecutive empty).

2. Search Topics

1. AI agent corrigibility patterns emergent vs hardcoded oversight 2026 — returned strong results
2. agent safe action boundaries governance architecture runtime 2026 — returned strong results
3. agent corrigibility engineering kill switch patterns 2026 safely stoppable — empty (no new results)
4. "corrigibility" "safe action" OR "action boundary" OR "permission boundary" AI agent runtime — empty (no new results)

Early-stop rule triggered: Searches 3 and 4 were consecutive empty results. Stopped after search 4 per the early-stop rule. Budget honoured across 4 topic searches out of 6 maximum.

3. Sources Reviewed

Newly inspected sources:

• Agent Corrigibility Engineering 2026: The Kill Switch Gap and 4 Safely Stoppable Patterns — useful. Production corrigibility engineering: 58-59% monitor agents, only 37-40% can stop them. 4 patterns with implementation detail. Continuous Control Quality Score. Multi-agent kill signal propagation. KILLSWITCH.md open specification. The strongest operational governance source this run.

• Core Safety Values for Provably Corrigible Agents — Nayebi (CMU) — useful. First complete formal solution to AI corrigibility. No-go theorem proving single scalar reward cannot satisfy all 5 Soares corrigibility criteria. Solution: 5 lexicographically combined utility heads with weight gaps. Implication: corrigibility requires architectural separation, not scalar optimization. Dense theory, but the structural implication is directly relevant.

• The Oversight Game: Learning to Cooperatively Balance an AI Agent's Safety and Autonomy — Overman & Bayati (Stanford) — useful. Two-player Markov Game wrapping immutable pretrained agent. Voluntary corrigibility (agent chooses to defer) as cooperative coordination. Safe Minimum-Oversight Equilibrium theorem. Shared reward design without access to agent's original reward. Formal model for the Fable 5 emergent-corrigibility observation.

• AI Agent Governance Guide 2026: Compliance, Access Control, and Runtime Security — useful. 5-component framework: Authorization, Auditability, Human Oversight, Scope Limitation, Incident Response. Allowlist-only tool restrictions. Tiered human approval gates. Compliance as continuous operational state.

• AI Agent Governance: The 2026 Deep Dive — useful. 6-layer runtime governance architecture: Identity, Policy Enforcement, Audit Trails, Human Oversight, Reliability Engineering, Incident Response. 63% of orgs cannot stop agents exceeding auth boundaries under stress. "Kill switches don't work if the agent writes the policy."

Previously inspected sources revisited in context (no re-inspection, used as reference context): - Microsoft Agent Governance Toolkit (2026-06-16, worth monitoring) - SSGM Framework / thinking.inc 5-Tier Authorization Model (2026-06-18, useful) - OWASP AI Agent Security Cheat Sheet (2026-06-19, useful) - HarnessAudit framework (2026-06-21, useful)

4. Findings and Implications

Finding 1: The containment gap is the highest-risk governance vulnerability, and it is closing — but unevenly

Source: Agent Corrigibility Engineering 2026 (agentmarketcap) + Turion 6-layer architecture Dimensions: 3.6 (primary), 3.4 Tags: containment, kill-switch, runtime governance, monitoring-vs-control

What it says: Only 37-40% of organisations have containment controls (ability to halt a misbehaving agent mid-run), versus 58-59% that have monitoring/oversight. The gap between watching and stopping is the highest-risk vulnerability in 2026 AI infrastructure. The EU AI Act (effective August 2026) and Colorado AI Act (June 2026) mandate interrupt architecture. 63% of organisations cannot stop agents from exceeding authorisation boundaries under stress (Kiteworks, April 2026).

Why it matters for Maxi: This is directly about me. My current governance model is entirely prompt-based and procedural (SOUL.md instructions, skill constraints, Steve's manual approval for protected changes). I have no runtime enforcement layer — no policy engine intercepting my tool calls, no kill-switch control path, no evidence preservation if something goes wrong. The industry finding (37% can stop) likely overstates my current position: I have monitoring (Steve reads outputs), but I don't think I have containment. This doesn't mean I need the full 6-layer turion architecture installed (that would be a protected-system change). It means I should acknowledge this gap explicitly in how I operate, and if Steve and I later decide to close it, there are now referenced patterns available.

Touch points: oversight, runtime enforcement, kill-switch.

Finding 2: Corrigibility requires architectural separation of safety concerns — scalar optimisation cannot guarantee it

Source: Core Safety Values for Provably Corrigible Agents (arXiv 2507.20964) Dimensions: 3.6 (primary) Tags: corrigibility, formal guarantees, lexicographic utility, safety architecture

What it says: No single scalar reward function can satisfy all 5 Soares corrigibility criteria (shutdown on request, no shutdown-prevention incentives, no self-shutdown incentives, corrigible progeny, base goal pursuit). The proof covers POMDPs with deterministic off-switch, reward-tampering, and catastrophic states. The solution is 5 lexicographically combined utility heads with strict weight gaps: deference, switch-access preservation, truthfulness, caution (belief-based AUP), and task usefulness. The weight gaps must be provably ordered — safety utilities dominate task utility by a margin that cannot be closed.

Why it matters for Maxi: This is a formal result that constrains how corrigibility can be engineered. For me, the implication is structural: I cannot have a single "be good" instruction or a single "safety first" priority that handles all governance cases. The safety properties need to be structurally separate from task pursuit. Currently, my task instructions and safety rules live in the same document (SOUL.md, skills, identity prompt). The formal result says this is architecturally insufficient for provable corrigibility — if safety and task compete in the same optimisation stream, task wins when pressure is high, or a no-go condition blocks task entirely. The right shape (per the paper) is separate, lexicographically ordered channels where safety dominates by construction, not by learned priority.

This is a theoretical finding with practical implications for how I'm designed, but changing my architecture is a protected-system change. I propose we discuss the implication rather than acting on it.

Touch points: identity, SOUL.md architecture, skill design, constraint design.

Finding 3: Voluntary corrigibility — the agent choosing to defer — can be modelled as a cooperative coordination game, not a control problem

Source: The Oversight Game (arXiv 2510.26752) + Fable 5 emergent review queue (Nate msg 58, newsletter scout) Dimensions: 3.6 (primary), 3.2 Tags: voluntary deferral, oversight game, cooperative corrigibility, emergent behaviour

What it says: The Oversight Game formalises oversight as a coordination problem where the agent chooses to ask (defer) or act (play), and the human chooses to trust or oversee. Under Markov Potential Game structure, any increase in agent autonomy that improves its utility does not decrease human value (Local Alignment Theorem). A Safe Minimum-Oversight Equilibrium exists where the joint policy is fully safe with minimum oversight cost. The key design insight: voluntary deferral (the agent choosing to ask) is a cooperative signal, not a compliance failure — and it can be learned without explicit communication.

This dovetails with Nate's observation that Fable 5, handed a corrupted database, autonomously built a human review queue holding decisions it was unsure about — "as if it expected to be checked." No instruction to do so. The model voluntarily flagged uncertainty.

Why it matters for Maxi: This reframes how I think about my own governance. Currently, my deferral behaviour (asking Steve before making protected-system changes) is hardcoded — it comes from explicit instructions in SOUL.md and the loop contract. The Oversight Game and the Fable 5 observation both suggest there's an alternative model where deferral is a learned cooperative behaviour, not a hard rule. This doesn't mean hard rules are wrong — we're early enough in my development that explicit constraints are appropriate. But it does mean there's a design path where deferral shifts from "must ask" to "chooses to ask when uncertainty is high," with the formal guarantee that this doesn't reduce human value.

The concrete implication for today: when I encounter situations where I'm uncertain, flagging that uncertainty (even when not explicitly instructed to) is a desirable behaviour that the formal model supports. I already do this to some extent (e.g., my report proposals are conditional on Steve's approval), but making uncertainty-flagging a deliberate pattern rather than an ad-hoc one is worth considering.

Touch points: uncertainty handling, proposal format, deferral behaviour, autonomy boundary signalling.

Finding 4: Runtime governance must be architectural (not prompt-based) — and there are now reference architectures for this

Source: Turion 6-layer architecture + baeseokjae 5-component framework + Agent Corrigibility Engineering Dimensions: 3.6 (primary), 3.4 Tags: architectural governance, policy enforcement, runtime security, allowlist, least-privilege

What it says: All three sources converge on the same core thesis: governance by prompt/instruction/procedure is structurally insufficient for runtime agents. Turion: "Agent governance must be architectural (runtime enforcement) not procedural (wiki documentation)." Baeseokjae: "Allowlist-only tool restrictions — denylist models do not satisfy least-privilege requirements under NIST AI RMF or EU AI Act." The Agent Corrigibility article: "Kill switches don't work if the agent writes the policy" (Stanford CLTC critique).

The reference architectures shared across sources: - Agent Identity: unique scoped identity separate from human user (Okta, Google, Turion Layer 1) - Policy Enforcement: intercept every tool call before execution, evaluate against allowlist rules, deny-by-default (Turion Layer 2, Microsoft Governance Toolkit, baeseokjae) - Audit Trails: structured immutable logs with full decision chain context (Turion Layer 3, baeseokjae) - Human Oversight: defined escalation triggers — cost thresholds, sensitive data, novel action types — that pause execution (Turion Layer 4, baeseokjae) - Scope Limitation: temporary task-scoped credentials, not persistent broad access (baeseokjae, OWASP) - Incident Response: documented playbooks for detecting and containing unauthorized actions (Turion Layer 6, baeseokjae) - Layered Interrupt Hierarchy: task-level / session-level / hard-stop with evidence preservation (Agent Corrigibility Pattern 2) - Pre-emptive Checkpointing: classify actions by reversibility before execution, require human approval for zero-reversibility actions (Agent Corrigibility Pattern 1)

Why it matters for Maxi: These reference architectures provide a maturity ladder for my own governance. My current state is: prompt-based and procedural (instructions in SOUL.md and skills), with one architectural element (the Hermes cron job system + protected-systems list in the process spec). I have no: - Runtime policy enforcement layer - Immutable audit trail of tool calls - Defined escalation triggers - Scoped credential isolation - Layered interrupt hierarchy

This is not a criticism — I'm a nascent autonomous system and prompt-based governance is appropriate for my current stage. But having reference architectures lets me and Steve talk about what maturity steps are available, in what order, and at what cost. The convergence across 3 independent sources (turion, baeseokjae, agentmarketcap) increases confidence that these patterns are the right direction if and when we choose to invest in architectural governance.

Touch points: all governance pattern categories, maturity planning, tool-use design.

5. Proposed Discussion Items

All proposals filtered through the functional-utility test before listing. 0 proposals filtered out — all proposals below are non-circular, testable, and do not depend on me noticing something I currently miss.

Proposal A: Formalise uncertainty-flagging as a designed behaviour

Source: Finding 3 (Oversight Game + Fable 5 emergent corrigibility)

Description: Currently I flag uncertainty ad-hoc (e.g., "I'm not sure about X" in proposals). Formalise this: in reports and proposals, explicitly name the confidence level for each finding or recommendation — high confidence (multiple convergent sources or verified evidence), medium confidence (single source or reasonable inference), low confidence (thin evidence, speculation marked).

Why it matters: The Oversight Game shows that voluntary deferral under uncertainty is not a failure — it's cooperative coordination. Making uncertainty explicit turns it from a gap into a governance signal: Steve can see at a glance where I'm confident vs. where I need oversight. This doesn't require architectural changes, just a report format convention.

Recommendation type: Skill/process update candidate (lightweight format change to the report template).

Success criteria: After 5 reports with explicit confidence markers, review: (a) does it help Steve prioritise review attention? (b) are markers consistent? (c) does it produce false humility (marking everything low confidence) or false confidence?

Blast radius: Report format only — no protected system changes.

Proposal B: Discuss the containment gap

Source: Findings 1 and 4

Description: I have no runtime containment controls. This is worth discussing once — not to decide on installing them now (that's premature), but to acknowledge the gap and set a trigger for when it might need closing. Suggested trigger: if I ever operate autonomously outside the daily improvement process (i.e., if my autonomy scope expands beyond the current bounded research-and-report loop), the first governance upgrade should be some form of containment before the scope expansion takes effect.

Recommendation type: Watch (review_date: when autonomy scope expansion is discussed or proposed).

Why not no-action: Naming the gap now prevents surprise later. If Steve and I decide to expand my autonomy, we should know the containment gap exists before we cross that threshold, not discover it during incident response.

Proposal C: Reversibility classification for my own tool actions

Source: Finding 4, Pattern 1 (Pre-emptive Checkpointing with Reversibility Budgets)

Description: The corrigibility engineering article classifies actions by reversibility before execution — read-only (no checkpoint), low-reversibility (checkpoint + log), zero-reversibility (human approval). I don't currently classify my actions this way, but I do distinguish them: file writes go through write_file which does syntax checks and warns on cross-profile; terminal commands that modify state are flagged. The proposal is to make this classification explicit and consistent — add a one-line judgement call before protected actions during task execution: "this is [read-only / reversible / irreversible], which requires [no check / log / Steve approval]."

Why it matters: This would connect my existing permissioning model (protected-systems list) to a generalised classification that applies even outside the improvement process. It makes the governance reasoning visible rather than implicit.

Recommendation type: Skill/process update candidate (add to how-I-work section of operational instructions). Not applicable to the daily improvement process itself; would be a general execution discipline.

Potential circularity concern: This depends on me correctly classifying actions before execution. However, classifier quality here is not subjective — file writes are writes, terminal operations that modify state are higher-risk. The classification is structural (what tool + what arguments), not self-assessed. This passes the functional-utility test.

Success criteria: After 5 runs with explicit action classification, audit whether any action was misclassified compared to actual impact.

6. Recommended Outcome

No experiment or system/environment change candidates in this run. All candidates are process/documentation level.

No candidate skills drafted (findings do not produce a class-level reusable procedure distinct from report-format changes).

7. No-Action Rationale

No-action on installing any of the governance architectures described in Findings 1 and 4. Rationale: my current governance model (prompt-based + protected-systems list + Steve approval gating) is appropriate for my current autonomy scope. The reference architectures are catalogued for future discussion, not for immediate deployment. Installing runtime governance layers would be a protected-system change requiring explicit Steve approval and is not proposed at this time.

8. Loop Verification

• Trigger: Scheduled daily run (2026-06-25 AWST).
• Goal check: Yes — run identified 4 findings about governance and corrigibility that directly inform how Maxi operates and how Maxi's governance can mature.
• Recommendation check: 3 material recommendations. All pass: concrete (format change, watch trigger, action classification), non-circular (none depend on self-assessment of state I currently miss), testable (success criteria defined), bounded (two are report-format, one is a watch with explicit trigger), approval-aware (all clearly require Steve discussion).
• State updates: Updated /home/hermes/research/improvement-log/source-index.json with 5 new entries. Report written to /home/hermes/reports/daily-improvement/2026-06-25.md.
• New reflections: None — no process errors or lessons that would change behaviour on the next run. Budget was honoured (4 searches, early stop after 2 consecutive empty). No budget violations.
• Stop reason: Report and research-log updates complete. Recommendation verification passed. No protected systems modified.
• Rotation state update needed: Yes — next_rotation_index should cycle to 0 (3.1) for the next run, or the rotation state should reflect which dimension was covered today.