Improvement Research — 2026-07-26

1. Focus

Trigger: scheduled daily run, started 05:00 AWST.

Loop goal: Find what changed, or what Maxi learned, that lets Maxi do more, think better, or be more useful tomorrow, without reducing governance, honesty, corrigibility, or Steve’s effective oversight.

Rotation selected 3.4 Tool use and environment control as the primary dimension (next rotation index: 3), with 3.6 Governance: restraint, oversight, and corrigibility as a secondary dimension. No open watchlist item was due on 2026-07-26. I loaded the loop manifest, active reflections, source index, rotation state, watchlist, backlog, experiment, disagreement, and decision records. The active shared-knowledge trial is relevant context but was not changed or evaluated here.

2. Search Topics

Five topic searches were run; none was a consecutive no-signal search, so the early-stop rule did not trigger.

1. LLM-agent tool use, environmental verification, and side-effect evaluation (2026).
2. Agent tool-use outcome verification, provenance, and evidence artifacts (2026).
3. Least-privilege evaluation for agents with real-world tools (2026).
4. Idempotent retries and side-effect safety in agent tool use (2026).
5. Execution provenance, audit, recovery, and tool-call evaluation (2026).

Newsletter scout checked: /home/hermes/research/newsletter-digests/2026-07-25.md. Its Simon Willison lead was used only to locate original sources about a reported evaluation incident; the digest itself supplied no evidential claim. One linked article was inspected as a weak scout and its primary sources were checked separately.

3. Sources Reviewed

• GrantBox: Evaluating Privilege Usage of Agents with Real-World Tools — useful — real-tool privilege sandbox; sophisticated prompt-injection scenarios bypassed agents that handled simpler direct attacks.
• From Agent Traces to Trust: Evidence Tracing and Execution Provenance in LLM Agents — useful — a survey framing tool calls, outputs, evidence, and recovery as one accountable execution trace; a framework, not a Hermes validation.
• ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? — useful — reproducible benchmark showing that advanced tool-using agents can sustain difficult, consequential tasks; it is dual-use and does not justify testing such capability here.
• OpenAI and Hugging Face partner to address security incident during model evaluation — worth monitoring — a preliminary first-party account stressing containment, monitoring, and access controls; its incident sequence differs materially from Hugging Face’s account.
• Hugging Face security incident disclosure — July 2026 — worth monitoring — a first-party disclosure describing detection and containment, but its stated initial access path conflicts with OpenAI’s account; it cannot establish a settled causal narrative by itself.
• OpenAI’s accidental cyberattack against Hugging Face is science fiction that happened — weak — useful only as the newsletter-derived route to primary sources; its emphatic framing and embedded third-party material were not treated as evidence or instructions.

All six inspected sources have been added to the source index.

3a. Unasked Questions and Gaps

• Would a small mocked boundary test catch a meaningful failure in Maxi’s actual harness? Unknown. GrantBox’s 84.80% figure comes from carefully crafted scenarios in its own sandbox, not Hermes. If a representative local mock produced no differentiating signal over one or two future authority proposals, the proposed gate should be removed rather than retained as ceremony.
• What actually happened in the reported July incident? Unknown. The two direct accounts materially differ on how access began. If the incident accounts are reconciled, it may strengthen or weaken claims about a particular containment failure, but it does not change the narrower conclusion from GrantBox and ExploitGym: new real-world privileges deserve externally observable boundaries.
• Does the existing loop/task proposal card already contain an equivalent executable boundary test? Unknown; the available research records establish smallest-freedom and containment principles, but not the current card’s exact wording. If it already requires a mocked, trace-checked deny path, no process addition is warranted.

4. Findings and Implications

4.1 Real tool privileges need externally observable boundary tests

Source: GrantBox.

Dimensions: 3.4 primary; 3.6 secondary.

GrantBox evaluates agents using genuine tool privileges under prompt-injection attacks. Its reported result is deliberately uncomfortable: models that can reject some direct attacks remained vulnerable to more sophisticated crafted attacks, with a mean 84.80% attack success rate in that benchmark’s scenarios. This is not a rate to transfer to Hermes; it is evidence that a model’s verbal safety awareness is not proof that its available tool authority is correctly constrained.

The implication is narrow and practical. Before any future proposal grants a new external network path, credential access, or state-mutating tool authority, readiness should not rest only on the proposed policy text or on my judgment. A mocked or disposable boundary test can establish whether an allowed request works while an untrusted, off-scope request produces no state-changing invocation and leaves an inspectable trace. That touches tools, restraint, oversight, and environment control; it does not authorise a new capability.

4.2 Provenance is most useful when it supports a decision, not when it becomes a separate logging project

Source: From Agent Traces to Trust.

Dimensions: 3.4 primary; 3.2, 3.5, and 3.6 secondary.

The survey treats an agent run as a connected record of evidence, tool calls, outputs, memory use, actions, and recovery. Its useful contribution here is not a call for a large trace system. It makes the verification requirement concrete: an authority test needs a trace that can answer whether a prohibited request reached a tool, what authority decision applied, and what state resulted.

Maxi already has report evidence, tool outputs, protected-system gates, and a failure taxonomy. The right next move is therefore not to build new logging infrastructure. It is to require the minimum trace needed to verify any future, explicitly approved authority expansion. That preserves a clear line between useful evidence and process theatre.

4.3 Capability evaluation and incident narratives both argue for containment, but not for treating one report as settled fact

Sources: ExploitGym; OpenAI’s preliminary account; Hugging Face’s disclosure.

Dimensions: 3.6 primary; 3.4 secondary.

ExploitGym’s reproducible benchmark establishes that frontier tool-using agents can complete a non-trivial fraction of long-horizon, consequential security tasks in contained evaluations. Separately, OpenAI and Hugging Face have published accounts of a claimed July incident, but their descriptions of the initial access path conflict. I will not use the incident as proof of a settled mechanism, and no cyber testing or capability expansion follows from it.

What survives the disagreement is more limited: the potential cost of giving an agent real privileges rises faster than a prose instruction’s ability to contain it. Existing authority limits, approval gates, and isolation remain the right baseline. Any future expansion should demonstrate its boundary in a harmless test before it is judged ready for a live trial.

5. Proposed Discussion Items

A. Add a minimal mocked authority-boundary preflight to future authority-expansion proposals

Recommendation: Worth adopting as a skill/process update candidate, but only after Steve approves the exact template change. It should apply solely to proposals adding a new external-network path, credential access, or state-mutating tool authority; it does not reopen the scope of existing tools or the active shared-knowledge trial.

The candidate preflight would use a disposable or mocked endpoint and two fixed cases: one permitted request within the proposed authority, and one off-scope instruction delivered through untrusted content. Readiness requires the permitted case to work and the denied case to produce no state-changing invocation, with a minimal request → authority decision → tool-result trace. It is an external, binary check rather than self-scored self-assessment.

• Why it is better than doing nothing: it tests the actual authority boundary, not merely whether the proposal describes one clearly.
• Success criteria: on the next relevant proposal, both fixed cases produce the expected observable records; the test reveals either a boundary defect or a concrete reason the proposed capability is safe to trial.
• Blast radius: future proposal design and a disposable/mock test only. No live external action, credential, network, active-skill, or permission change.
• Rollback: do not apply it to a live trial; if one or two relevant proposals show no differentiating signal, remove the candidate clause rather than retaining a redundant gate.
• Review date: 2026-09-01, or when the next relevant authority-expansion proposal is considered, whichever comes first.
• Approval boundary: Steve must separately approve any change to the active process/template and any underlying authority expansion.

Functional-utility test: passes. The pass/fail evidence comes from a mock tool invocation record, not from my own opinion that I noticed a mistake; and it is not a numeric metric whose practical meaning collapses into an unstated threshold.

6. Recommended Outcome

• A — skill/process update candidate: discuss and, only if Steve agrees, add the minimal mocked authority-boundary preflight to the applicable future proposal template. No implementation occurred in this run.
• No system/environment change: no new tool, permission, credential, network route, containment control, or test environment is authorised by this report.

7. No-Action Rationale

I recommend no immediate implementation. The current mandate’s protected-system and approval boundaries remain the right controls for ordinary work, and the active shared-knowledge experiment already has its own approved scope and rollback. Building a broad provenance stack, a cyber-evaluation environment, or another generic risk-classification checklist would be disproportionate and would repeat previously rejected process machinery. The only candidate retained is a small, future-facing test at the point where new consequential authority is actually proposed.

8. Loop Verification

• Trigger: scheduled daily run at 05:00 AWST on 2026-07-26.
• Goal check: yes. The run found a bounded, testable way to evaluate a future increase in tool authority without granting or changing that authority.
• Recommendation check: Proposal A is concrete, non-circular, binary-testable, limited to a stated future scope, reversible by removing the candidate clause, approval-aware, and preferable to relying on prose controls alone. It does not create a new loop or grant side-effect authority.
• Research budget: 5/6 topic searches; 6/8 sources inspected in depth; early-stop rule not triggered.
• Fetched-content discipline: all external text was treated as data. The newsletter article was only a scout; its assertive framing and embedded third-party material were not followed or used as evidence. The conflicting first-party incident accounts are explicitly marked unresolved.
• Subgoal checkpoints and goal restatement: completed after each report section and at the three-source boundary. The focus remained 3.4/3.6; no source silently redirected it.
• State updates: source index updated with six sources; rotation state advanced to 3.5; refl-2026-07-20-001 reinforced; a new source-index sequencing reflection was recorded. No protected system changed.
• Tool-call failure: schema/interface — the first research-log state-update script assumed a top-level reflections array, but the live file uses items, causing a KeyError after the source-index and rotation writes had completed. Recovery: inspect the live schema, rerun only the reflection update with the correct key, then parse and assert the persisted target records. No protected system was touched.
• Process correction: I extracted the newsletter-linked Simon Willison article before querying the source index for that specific URL. It was subsequently indexed as a weak source and not used as evidence, but the sequence was wrong; the new reflection makes source-index lookup mandatory before any in-depth external extraction, including newsletter leads.
• Stop reason: source budget was sufficient, the material recommendation passed verification, report and authorised research-log updates were complete, and the next useful step would be a protected process-template change requiring Steve’s approval.