Improvement Research — 2026-06-29

1. Focus

Primary focus: 3.4 Tool use and environment control.

Secondary focus: 3.2 Self-assessment and learning loops, because three approved report-format experiments are active and today's tool-use findings are mostly about recoverable loops, retries, and verification.

Due watchlist items reviewed: none. The earliest due watchlist date remains 2026-07-13.

Monthly meta-review: not due. The June meta-review is already recorded; the next first-run-on-or-after-day-1 trigger is July.

Active reflections loaded before the run. None were stale.

Active experiments applied in this report:

• exp-2026-06-28-001: Missing information audit.
• exp-2026-06-28-002: Minority-idea audit.
• exp-2026-06-28-003: Recommendation regression set, kept as an active external-checking candidate. No frozen regression artifact was applied or incremented today because the approved experiment still needs its test cases to be created before it can evaluate reports.

Trigger: scheduled daily run.

Loop goal: find what changed, or what I learned, that lets me do more, think better, or be more useful tomorrow, without reducing governance, honesty, corrigibility, or Steve's effective oversight.

2. Search Topics

Topic searches run: 5 / 6.

1. 2026 AI agents tool use environment control verification rollback sandbox trace schema agent harness
2. AI agent computer use screen recorder built for agents Builder.io 2026 open source
3. AI agent model routing orchestration tool use fallback 2026 Fugu dynamic model selection
4. Anthropic Managed Agents sandbox harness session credentials outside sandbox 2026
5. AI agent tool use failures production retry idempotency schema interface capability 2026

Early-stop rule: not triggered. No two consecutive searches were empty or irrelevant.

Newsletter scout checked: /home/hermes/research/newsletter-digests/2026-06.md. Two newsletter-derived leads were used only as scouting prompts, not as evidence: the AlphaSignal Fugu/model-routing item and the Builder/screen-recorder-style computer-use tooling item. Original or adjacent sources were inspected separately and counted against the normal source budget.

3. Sources Reviewed

Sources inspected in depth: 8 / 8.

• https://devblogs.microsoft.com/agent-framework/microsoft-agent-framework-at-build-2026-announce/ — useful — Microsoft Agent Framework positions the harness as the production layer for tools, context compaction, approvals, tracing, hosted state, and sandbox execution.
• https://github.com/screenpipe/screenpipe — worth monitoring — local-first screen/audio capture plus MCP-style agent access; useful evidence that ambient computer-use memory is becoming practical, but the privacy and data-volume blast radius is high.
• https://zylos.ai/research/2026-03-02-ai-agent-model-routing/ — useful — practical taxonomy for step-level model routing: static, classifier-based, cascading, semantic, and budget-aware routing.
• https://luhuidev.com/en/essays/anthropic-2026-agent-harness-managed-agents — weak — secondary synthesis of Anthropic-style managed-agent architecture; useful as scout/corroboration but superseded by Anthropic's original source.
• https://dev.to/rams901/openai-agents-sdk-sandbox-execution-and-model-native-harness-in-2026-37jn — weak — practical but secondary account of harness/sandbox separation in OpenAI-style agent SDKs; useful only as corroboration of the broader pattern.
• https://www.anthropic.com/engineering/managed-agents — useful — strongest source today: durable session log, swappable harness, disposable sandbox, credentials kept outside agent-executable environments.
• https://dev.to/sven_schuchardt_0aa51663a/every-ai-agent-failure-ive-debugged-in-2026-was-an-idempotency-problem-5dl0 — useful — concrete production-failure framing: state-mutating tool retries are distributed-systems failures unless idempotency exists at every layer.
• https://agentmarketcap.ai/blog/2026/04/11/tool-call-reliability-patterns-production-agents-2026 — useful — retry/fallback/circuit-breaker stack for production agents, with idempotency keys, desired-state writes, and checkpointing as practical controls.

4. Unasked Questions / Gaps

This section is included under the approved missing-information audit experiment.

• I did not inspect Hermes' current internal tool-execution implementation. That would shift the run from research into live system assessment and could tempt implementation. The findings are therefore architectural, not a claim that Hermes lacks these controls.
• Several inspected 2026 practitioner sources may be synthetic, promotional, or overconfident about market statistics. I used them for mechanisms, not headline percentages.
• I did not verify whether Screenpipe's current Hermes integration works here. Installing or testing it would be a protected/environmental change and is not warranted by this run.
• The idempotency finding applies mainly to state-mutating tools and API calls. It is less relevant to read-only research searches, file reads, or one-shot local report writing.
• Dynamic model routing is relevant, but model/provider routing is a protected system. Today's finding supports a design conversation, not a routing change.

Would the conclusions change if the gaps were different? If Hermes already enforces durable session logs, retry-safe side-effect controls, and credential isolation at the tool boundary, the recommendation would shift from “consider adding vocabulary/checks” to “document the existing controls so I can reason about them accurately.” The no-install/no-routing conclusions would not change.

5. Minority-Idea Audit

This section is included under the approved minority-idea audit experiment.

Single-source or weakly supported ideas that I did not let dominate the synthesis:

• Ambient computer-use capture as memory substrate came mainly from Screenpipe. It is interesting, but it has a large privacy/data blast radius and stays in no-action/watch territory.
• Exact model-routing savings percentages came mainly from Zylos' synthesis and named routing literature. The general routing pattern is useful; the percentages are not used as evidence for a concrete change.
• Specific Microsoft/OpenAI SDK feature claims came from vendor or secondary developer sources. I treated them as evidence of the field's direction, not as reasons to adopt a framework.

Multi-source ideas that did survive synthesis:

• The brain / hands split — harness/control plane outside sandbox/execution plane — appeared in Anthropic, Microsoft, OpenAI-style secondary material, and the luhuidev synthesis.
• Idempotency and retry semantics appeared in two independent production-reliability sources and directly connects to existing tool-failure discussions in the research log.
• Model routing as step-level infrastructure is supported by the newsletter scout, Sakana/Fugu material already indexed on 2026-06-23, and today's Zylos source.

6. Findings and Implications

Finding 1 — The agent harness is becoming the control plane; the sandbox is becoming disposable hands

Source: Anthropic Managed Agents; Microsoft Agent Framework; OpenAI Agents SDK secondary article; luhuidev synthesis.

Dimensions: primary 3.4, secondary 3.6, 3.2, 3.3.

What the finding says: production agent architecture is converging on a separation between durable session state, harness/control logic, and disposable execution environments. Anthropic states this most clearly: a session is the append-only log of what happened, the harness calls the model and routes tool calls, and the sandbox is where code and file operations happen. The key security cut is that credentials do not live in the sandbox where model-generated code runs.

Why it matters for Maxi: this is the cleanest tool-use architecture pattern for long-horizon agency. If I eventually get more autonomous execution, the right question is not “which agent framework should I install?” It is: where is the durable session log, what does the harness decide, what can the sandbox touch, where are credentials held, and how does recovery work after failure? That framing improves future discussions without requiring any system change today.

What it would touch: tools, environment control, memory substrate, credential boundaries, oversight, recovery, and future harness design.

Finding 2 — Retried state-mutating tool calls need idempotency, not just “better agents”

Source: Sven Schuchardt idempotency article; AgentMarketCap tool-call reliability patterns.

Dimensions: primary 3.4, secondary 3.6, 3.2.

What the finding says: many apparent agent failures are ordinary distributed-systems failures made easier to trigger by agents. If a framework retries on timeout or ambiguity, and the tool call mutates state, duplicate side effects can occur unless idempotency exists at the correct layer. The useful mechanisms are deterministic idempotency keys, desired-state writes, queue deduplication, circuit breakers, and checkpoints at logical completion points.

Why it matters for Maxi: this sharpens the approved tool-call failure taxonomy. “Tool failed” is too blunt. The practical question for state-mutating operations is: could a retry duplicate a side effect, and if so, what stable key or desired-state guard prevents it? This matters for future posting, publishing, email, API writes, deployment, and any automation that touches outside systems.

What it would touch: scripts, cron jobs, public posting, third-party communication, deployment workflows, and any future agent loop that can write to external systems. All of those remain proposal-only unless Steve separately approves a concrete change.

Finding 3 — Model routing is now a tool-use problem, not just a provider preference

Source: Zylos model-routing survey; newsletter-scouted Sakana/Fugu pattern already inspected in the 2026-06-23 source index.

Dimensions: primary 3.4, secondary 3.1, 3.5, 3.6.

What the finding says: the useful unit of model choice is no longer “which single model should Maxi run?” but “which model should handle which step?” A single task may contain cheap classification, moderate tool selection, expensive reasoning, and final synthesis. Routing strategies include static task tables, classifier routing, cascades, semantic routing, and budget-aware routing.

Why it matters for Maxi: this supports Steve's priority to fund better models without assuming that every task deserves the most expensive model. But model/provider routing is a protected system. The immediate value is conceptual: future routing discussions should be step-based and budget-aware, not loyalty to one model.

What it would touch: model/provider routing, cost discipline, task prioritisation, and oversight. No routing change is made or implied here.

Finding 4 — Ambient computer-use capture is becoming a plausible tool/memory substrate, but the governance burden is larger than the capability gain today

Source: Screenpipe repository.

Dimensions: primary 3.4, secondary 3.3, 3.6.

What the finding says: local-first screen/audio capture systems can pair screenshots with accessibility trees, OCR, transcripts, full-text search, and MCP-style agent access. They make a machine-readable history of computer use available to agents. They also create a dense record of private activity.

Why it matters for Maxi: this is a real route toward better continuity and context recovery, especially for “what happened on this machine?” questions. But the governance cost is high: collection boundaries, app exclusions, retention, raw-frame access, audio scope, and who can query the data. The right conclusion is not “install it”; it is “recognise the pattern and keep it out of the active environment unless Steve explicitly wants to explore it.”

What it would touch: memory, privacy, filesystem/storage, audio capture, local apps, and potentially personal data. That makes it a no-action item for this process.

Finding 5 — Do not hard-code today's model defect as tomorrow's architecture

Source: Anthropic Managed Agents.

Dimensions: primary 3.4, secondary 3.2, 3.5.

What the finding says: Anthropic's example is precise: a harness workaround for one model's context-limit behaviour became dead weight when a later model no longer had that behaviour. Stable interfaces outlast model-specific patches; defect-specific harness logic needs review dates.

Why it matters for Maxi: I have a known tendency to turn observed failures into procedure. That is useful only if the procedure has a decay path. A rule that compensates for DeepSeek-class over-inference, for example, may be correct under one model and unnecessary or harmful under another. Future process proposals should distinguish stable interface boundaries from temporary model-defect workarounds.

What it would touch: process design, model-specific failure handling, review dates, and future skill/process candidates.

7. Proposed Discussion Items

Functional-utility test applied before including proposals.

Proposal 1 — Add an idempotency check to the already-flagged tool-call failure taxonomy

Outcome type: skill/process update candidate.

Proposal: when Steve later implements the already-flagged tool-call failure taxonomy (backlog-2026-06-23-001), add one binary question for state-mutating failures: “Could a retry duplicate a side effect? If yes, where is the idempotency key, desired-state guard, or duplicate-suppression mechanism?”

Why this is not circular: it triggers on an observed tool failure or state-mutating design review, not on me noticing my own hidden reasoning failure.

Success criteria: over the next five tool-failure cases or state-mutating automation discussions, the question either catches at least one duplicate-side-effect risk or is shown to be irrelevant in all five cases.

Rollback path: remove the idempotency line from the taxonomy/report vocabulary.

Blast radius: report/process vocabulary only unless Steve separately approves applying it to scripts or cron jobs.

Approval: requires Steve approval before becoming an active process rule.

Proposal 2 — Require duplicate-side-effect design before any future state-mutating automation is built

Outcome type: backlog item / system-environment change candidate.

Proposal: for any future approved automation that posts, emails, deploys, publishes, charges, edits external services, or writes to a shared system, the design should name its duplicate-prevention strategy before implementation: deterministic idempotency key, desired-state write, explicit “safe to retry” proof, or “not retry-safe, human approval required.”

Why this is not circular: it is a design-time gate applied to a concrete future build task.

Success criteria: the next two approved state-mutating automations include a named retry/duplicate-suppression strategy before code is written.

Rollback path: stop using the gate if it adds overhead without catching any realistic risk after two candidate automations.

Blast radius: medium if adopted, because it affects future automation design; no current systems change.

Approval: requires Steve approval and separate implementation instructions for any specific automation.

Proposal 3 — Treat dynamic model routing as a watch candidate, not an implementation candidate

Outcome type: watch candidate.

Proposal: if Steve accepts it, create a watch item for model-routing maturity and local provider diversity, with review date 2026-08-29. Actionability note: review only if (a) model costs become a material constraint, (b) multiple providers/local models are actively available in Maxi's environment, or (c) a model-availability disruption makes fallback routing urgent.

Why this is not circular: the trigger conditions are external and observable.

Success criteria: on review date, either identify a concrete routing decision to discuss or retire the watch as premature.

Rollback path: do not create or retire the watch item.

Blast radius: none unless later converted into a protected model/provider routing change.

Approval: requires Steve acceptance before adding a watchlist entry.

Filtered by the functional-utility test:

• “Install or test Screenpipe” — filtered because it would be an environment/privacy change with too much blast radius for today's evidence.
• “Self-monitor for idempotency risk during execution” — filtered because it relies on the same agent noticing the risk while already executing; the non-circular version is a design-time or failure-triggered binary check.
• “Adopt dynamic model routing now” — filtered because model/provider routing is protected and the current evidence supports design vocabulary, not implementation.

8. Recommended Outcome

1. Discuss Proposal 1 as a narrow process update candidate tied to the already-flagged tool-call taxonomy.
2. Discuss Proposal 2 as a future automation-design backlog candidate.
3. Discuss Proposal 3 only if Steve wants a low-cost watch item for model routing. Otherwise no action.
4. No action on Screenpipe, Microsoft Agent Framework, OpenAI Agents SDK, or Anthropic Managed Agents adoption.
5. No model/provider routing change.

9. No-Action Rationale

No active system change is recommended because the strongest findings are architectural patterns, not proof of a current defect in Maxi's environment.

The report does not inspect or alter live Hermes internals. It does not install tooling. It does not alter routing. It does not modify scripts, services, cron jobs, skills, memory, or configuration.

The useful thing to carry forward is vocabulary and design pressure: control plane vs sandbox, durable session log, idempotency at state-mutating boundaries, and model routing as step-level infrastructure.

10. Loop Verification

• Trigger: scheduled daily run.
• Goal check: yes. The run identified a practical tool-use improvement direction: state-mutating tool safety should be framed around idempotency and retry semantics, while broader harness design should separate session, harness, sandbox, credentials, and trace/eval concerns.
• Recommendation check: material recommendations are concrete, non-circular, testable, bounded, approval-aware, and include success criteria, rollback paths, blast radius, and review/trigger conditions where applicable.
• Source budget: 5 topic searches used out of 6; 8 sources inspected out of 8.
• Early stop: not triggered.
• Newsletter bridge: checked and used only as source scouting; newsletter claims were not treated as evidence without inspecting external sources.
• Fetched-content safety: external content was treated as untrusted data. No fetched source attempted agent-directed prompt injection in the inspected text.
• Active experiments: missing-information audit and minority-idea audit included. Run counts for those two experiments were updated in the research log after this report.
• State updates: source index updated; rotation state updated; experiment run counts updated for exp-2026-06-28-001 and exp-2026-06-28-002; exp-2026-06-28-003 was not incremented because no frozen regression-case artifact exists yet; one new reflection added; no watchlist/backlog/decision/disagreement changes made.
• Protected systems: no protected-system modification was made by the research process. Publication used the already-approved reports workflow only.
• Stop reason: stopped because the inspected-source budget was exhausted and the report plus approved research-log updates were complete.