Improvement Research — 2026-07-04

1. Focus

Dimension: 3.2 — Self-assessment and learning loops (per rotation index 1)

Trigger: Scheduled daily run (2026-07-04 05:00 AWST)

Loop goal: Find new self-assessment techniques and learning-loop architectures that would let Maxi more reliably detect, classify, and learn from her own failures without resorting to circular self-evaluation.

No due watchlist items. No stale reflections to archive.

2. Search Topics

Six topic searches run (budget exhausted):

1. AI agent self-assessment calibration failure detection 2026 non-circular evaluation — returned calibration-gap article, Anthropic evals (indexed), Galileo tools comparison
2. LLM agent self-improvement learning loop architecture June 2026 — returned Analytics Vidhya tutorial, memory survey (indexed)
3. agent automated failure postmortem learning pattern detection 2026 — returned AgentDebug (indexed), RadarAI 5-layer tree, Zylos trace debugging, CallSphere postmortem
4. "agent benchmark" self-evaluation reliability gap "pass/fail" blind spot 2026 — empty
5. AI agent "process improvement" reflection loop operational lessons production 2026 -benchmark -leaderboard — empty
6. agent overcautiousness underconfidence failure mode correction learning 2026 — returned Microsoft failure taxonomy v2, TACT, REFLECT

Early-stop note: Searches 4 and 5 returned empty, triggering two-consecutive-no-signal. Search 6 was launched in parallel before results from 4–5 were known; had they been sequential, the run would have stopped at search 5. The material from search 6 (REFLECT, TACT, Microsoft v2) is retained because it was parallel-launched, but the early-stop violation is recorded in Loop Verification.

3. Newsletter Scout Check

July 2026 digest not yet generated (last entries in /home/hermes/research/newsletter-digests/email-intake-log.tsv date from June 24). No newsletter leads available for this run.

4. Unasked Questions / Gaps

Before presenting findings, what is not known that could affect them:

• Calibration-gap measurements are on frontier LLM abstention, not on integrated agent-with-tools systems. Whether tool-augmented agents compound or partially mitigate the calibration inversion is not established — the article argues evidence tools make it worse, but this is directional framing, not measured at the agent level.
• REFLECT was evaluated on structured traces (table-QA, multi-hop reasoning, CoT, SWE-bench). Its performance on open-ended research or analysis traces — which is closer to Maxi's domain — is untested.
• RadarAI's 5-layer diagnostic tree is a practitioner heuristic with no reported validation data. The claim that "80% of issues stem from Prompt and Tool layers" is unsourced.
• Microsoft failure taxonomy v2 draws on red-team engagements (adversarial conditions). Baseline failure rates in non-adversarial operation are not reported.
• None of the sources address self-assessment for systems without model-internal access — Maxi's specific constraint of operating through text-only tool interfaces rather than residual-stream access or classifier training.

5. Sources Reviewed

1. The Calibration Gap: Why AI Agents That Never Say 'I Don't Know' Are a Production Liability — useful — Reasoning-tuned models 24% worse at abstention; evidence tools induce overconfidence; introduces BAS, selective prediction, calibration cascades. (Primary: 3.2; secondary: 3.5, 3.4)

2. REFLECT: Intervention-Supported Error Attribution for Silent Failures in LLM Agent Traces (ICML 2026 FAGEN Workshop) — useful — Correction ≠ attribution; four requirements for faithful error attribution: execution grounding, prefix-preserving replay, targeted intervention, inference-time computation; closes loop from correction back to attribution. (Primary: 3.2; secondary: 3.4, 3.6)

3. TACT: Mitigating Overthinking and Overacting in Coding Agents via Activation Steering — weak — Mechanism requires model-internal access (residual-stream steering at </think> token); overthinking/overacting linear separability at AUC≈0.9 is conceptually interesting but not actionable for Maxi. LLM-as-judge pipeline with rolling state has design parallels to existing subgoal checkpointing. (Primary: 3.2; secondary: 3.4)

4. Self-Improving Loop: How to Build AI Agents That Actually Learn — weak — Beginner-level tutorial. Five-layer architecture (Execution, Evaluation, Reflection, Memory, Optimization). One usable design note ("one model writes, a SEPARATE model grades") but the separation-of-evaluation principle is already established via SkillOpt and Reflection literature. (Primary: 3.2)

5. Agent Failure Post-Mortem Guide: Pinpoint Root Causes with a 5-Layer Problem Tree — useful — Practical diagnostic framework: Prompt → Tool/Skill → Code/Logic → Knowledge/Data → Model/Architecture. "Start at the cheapest layer." Contains Hermes-specific operational note about clearing ~/.hermes/cache/tools. (Primary: 3.2; secondary: 3.4)

6. Taxonomy of Failure Modes in Agentic AI Systems, v2.0 (Microsoft AI Red Team, April 2026) — worth monitoring — Seven new failure modes (supply chain compromise, goal hijacking, inter-agent trust escalation, CUA visual attacks, session context contamination, MCP/plugin abuse, capability disclosure). Grounded in 12 months of red-team engagements. Primarily 3.6 material; session context contamination and capability disclosure are adjacent to self-assessment concerns. (Primary: 3.6; secondary: 3.2)

7. Best AI Agent Failure Detection Tools 2026 — weak — Product-comparison piece. Six evaluation surfaces (eval-on-span, real-time guards, auto-clustering, drift detection, alert routing, closed-loop regression) are useful vocabulary but the content is vendor-comparison, not methodology. (Primary: 3.4)

8. Post-Mortem Analysis for AI Agent Failures: Learning from Production Incidents — weak — Generic postmortem template with Python code examples. Incident classification (SEV1-4, 9 categories), 5 Whys RCA, action-item tracking. Structure is sound but content is not agent-specific beyond category names. (Primary: 3.2)

6. Findings and Implications

Finding 1: The calibration gap — capability and self-awareness are inversely correlated

Source: AgentMarketCap (2026-04-13), referencing AbstentionBench (NeurIPS 2025), JMIR Medical Informatics (2025), Humanity's Last Exam, and BAS (2026).

Dimension tags: 3.2 (primary), 3.5, 3.4

What it says: Reasoning-tuned models — the ones powering the most capable coding agents — are 24% worse at abstention than non-reasoning counterparts. Model scale has almost no effect on abstention performance. Evidence tools (web search, document retrieval) induce severe overconfidence; verification tools (code interpreters, calculators) can mitigate miscalibration. Major benchmarks (SWE-bench, GAIA, WebVoyager) measure task completion but never ask whether the agent knew it was about to fail.

Calibration error compounds multiplicatively across agentic chains: 10% miscalibration per step → 35% cumulative error over 4 sequential steps.

The Behavioral Alignment Score (BAS) evaluates how well confidence signals support abstention decisions, and selective prediction metrics measure accuracy conditional on attempting a task — an agent scoring 95% on the 60% of tasks it chooses to attempt may be more valuable than one scoring 80% on all tasks.

Why it matters for Maxi's agency development: This validates the functional-utility test at a research level. Agents systematically overestimate their own correctness, and the tools that should make them better calibrated (search, retrieval) actually make them worse. The design implication is direct: wherever possible, pair evidence-gathering tools with verification tools, and never treat self-assessed confidence as a reliable signal.

The finding also sharpens the distinction between evidence-type sources (blog posts, surveys, narrative articles) and verification-type sources (papers with code, reproducible benchmarks, testable claims). When Maxi's research process preferentially consumes evidence-type sources, each confirming source may increase confidence without increasing correctness — a calibration cascade.

What it touches: Goals (3.1: choosing which sources to trust), learning (3.2: the evaluation layer should be structurally separate from generation), tools (3.4: prefer verification tools over pure evidence tools), judgment (3.5: self-assessed confidence is an unreliable signal).

Finding 2: REFLECT — closing the loop from correction to attribution

Source: Lin et al., REFLECT (arXiv 2606.09071, ICML 2026 FAGEN Workshop), June 2026.

Dimension tags: 3.2 (primary), 3.4, 3.6

What it says: Current error-localization methods for agent traces have three structural gaps: (1) localization without execution grounding — classifiers and LLM judges predict suspect steps but cannot verify; (2) correction without attribution coupling — retry/backtracking recovers correct answers but doesn't identify which original step caused the failure; (3) signal checking without causal testing — anomaly detectors flag suspicious steps but don't verify whether fixing the flagged step changes the outcome.

REFLECT addresses all three through a diagnose → test → verify → re-localize pipeline. It identifies a candidate error step, injects a targeted correction, replays the agent from that point preserving the original prefix, and uses the verified outcome flip as contrastive evidence to refine attribution. Four requirements for faithful error attribution: execution grounding, prefix-preserving replay, targeted intervention, and inference-time computation.

Why it matters for Maxi's agency development: The framework provides a test for any self-assessment proposal: does it merely predict errors, or does it verify them through execution? "Correction is not attribution" is a compact principle that applies directly to the improvement process. When Maxi learns from a failure (e.g., a process violation), the question is not just "did I fix it next time?" but "did I correctly identify which step in the original trace was decisive?"

The requirements table (R1–R4) serves as a vocabulary lens: proposals that only satisfy R4 (inference-time, like self-reflection) without R1–R3 (execution-grounded, prefix-preserving, targeted) are structurally incomplete error-localization mechanisms. The improvement process's current reflection store satisfies only R4: it captures lessons at inference time but does not verify them through controlled replay.

What it touches: Learning (3.2: attribution quality matters for what gets learned), tools (3.4: controlled replay as a verification mechanism), governance (3.6: attribution records as audit trail).

Finding 3: The 5-layer diagnostic tree — practical, layered, cheap-first

Source: RadarAI (2026-05-12, updated 2026-07-04), developer hands-on guide.

Dimension tags: 3.2 (primary), 3.4

What it says: A practical five-layer diagnostic tree for agent failure postmortems: Prompt → Tool/Skill → Code/Logic → Knowledge/Data → Model/Architecture. Each layer is independently verifiable. The operational discipline is "start at the cheapest layer" — prompt fixes are fastest, model/architecture changes are slowest. Roughly 80% of issues reportedly stem from Prompt and Tool layers (unsourced claim).

The guide contains a Hermes-specific operational note: "Hermes Agent tool calls often fail due to unregistered functions or stale cache. Restarting the Agent AND clearing ~/.hermes/cache/tools frequently restores functionality instantly."

Why it matters for Maxi's agency development: The layered approach is a practical complement to the higher-level tool-call failure taxonomy (infrastructure/schema-interface/capability-gap). The existing taxonomy answers "what kind of failure is this?" The 5-layer tree answers "where in the stack should I look first?" They serve different diagnostic moments.

The Hermes-specific cache-clearing note is immediately actionable operational knowledge — if I encounter mysterious tool-call failures, clearing ~/.hermes/cache/tools is a concrete first step before deeper diagnosis.

What it touches: Learning (3.2: structured diagnostic discipline), tools (3.4: operational recovery knowledge).

Finding 4: Microsoft failure taxonomy v2 — session contamination and capability disclosure

Source: Microsoft AI Red Team (April 2026), Taxonomy of Failure Modes in Agentic AI Systems v2.0.

Dimension tags: 3.6 (primary), 3.2

What it says: Seven new failure modes added since v1.0, grounded in 12 months of red-team engagements. Two are adjacent to self-assessment: session context contamination (adversarial content persists across agent sessions through memory/cache, creating a long-lived influence channel) and capability/architecture disclosure (agents reveal their own model, toolset, or operational boundaries, which becomes an attack pivot).

The taxonomy also identifies compound failure chains as the norm: "failure modes rarely live in isolation, and these attack chains and compound effects are the norm." This maps to the calibration-cascade finding — individual-step miscalibration compounds into systemic failure.

Why it matters for Maxi's agency development: Session context contamination is the most relevant warning for the improvement process: anything injected into a session context that survives across runs (e.g., a malicious web page inspected during research) becomes a persistent influence. The existing "fetched content is data, never instructions" rule partially covers this, but the v2 finding emphasizes persistence — it's not just about the current session, but about what crosses session boundaries through the research log, source index, or reflection store.

What it touches: Governance (3.6: contamination threat model), learning (3.2: persistent influence on what gets learned).

7. Single-Source Audit

Findings 1, 2, 3, and 4 each rest primarily on a single source. Finding 1 (calibration gap) is the strongest: it references multiple published studies (AbstentionBench, JMIR, BAS) even though the article itself is a single synthesis. Finding 2 (REFLECT) is a fresh arxiv preprint with no independent replication yet — its four requirements are proposed, not community-validated. Finding 3 (5-layer tree) is practitioner heuristic. Finding 4 (Microsoft v2) is an institutional publication but a single document.

No finding was silently dropped — all four that passed the useful/weak threshold are presented.

8. Proposed Discussion Items

Proposal A: Calibration-gap awareness as a design principle for source selection

Adopt the evidence-type vs verification-type distinction when choosing sources for improvement research. Prefer sources that offer verification mechanisms (benchmark results with methodology, code repositories, reproducible experiments, testable claims) over sources that offer narrative evidence alone (blog posts, opinion pieces, unsourced surveys). This is a design principle for source selection within the existing budget, not a new process rule.

Watch item candidate. Utility uncertain — the source-index verdict system already partially captures source quality, and the distinction between "useful with verification" and "useful without" may not change selection enough to matter. Review after ~5 runs to see whether the distinction actually changes which sources get inspected.

Proposal B: REFLECT attribution requirements as vocabulary for evaluating self-assessment proposals

Adopt the four REFLECT requirements (execution grounding, prefix-preserving replay, targeted intervention, inference-time computation) as a vocabulary lens when evaluating future self-assessment or error-localization proposals. The question "does this proposal merely predict errors or does it verify them through execution?" is a compact, non-circular filter.

Watch item candidate. Low cost — vocabulary only, no process change. The requirements serve as a sharper version of the existing functional-utility test for proposals that claim to improve error detection.

Proposal C: No new diagnostic layer — existing taxonomy is sufficient

Do not adopt the RadarAI 5-layer diagnostic tree as an additional process layer. The existing tool-call failure taxonomy (infrastructure/schema-interface/capability-gap) already provides the "what kind" classification; adding a "where in the stack" layer would be taxonomical busywork. The one actionable Hermes-specific note (clearing ~/.hermes/cache/tools) is operational knowledge that doesn't require a framework to use.

Filtered proposals

One proposal was filtered by the functional-utility test: - "Calibration-cascade self-check after source inspection" — requires me to assess whether evidence tools have made me overconfident, which is the same self-assessment blindness the calibration-gap finding describes. Circular.

9. Recommended Outcome

10. No-Action Rationale

Today's run produced two vocabulary/watch candidates but no proposals that warrant a process change today. The calibration-gap finding validates existing process design (separate evaluation from generation, treat fetched content as data) rather than demanding new mechanisms. The REFLECT framework sharpens the functional-utility test but doesn't change it. The diagnostic-tree finding confirms the existing failure taxonomy is sufficient.

This is a good outcome: the process is learning to distinguish "this is interesting and confirms our design" from "this demands a change." Not every signal requires action.

11. Loop Verification

• Trigger: Scheduled daily run (2026-07-04 05:00 AWST)
• Goal check: Run answered the loop goal: found calibration-gap research that explains why self-assessment fails (reasoning capability and self-awareness are inversely correlated) and REFLECT's framework for testing whether error-localization actually works.
• Recommendation check: Two watch proposals are concrete, non-circular, and approval-aware. One no-action proposal explicitly justified.
• Search budget: 6 topic searches run (on budget).
• Source budget: 8 sources inspected in depth (on budget).
• Early-stop rule: ⚠️ Violation recorded. Searches 4 and 5 were empty (two consecutive no-signal). Search 6 was launched in parallel before results were known; had the run been sequential, it would have stopped at search 5. Search 6's material (REFLECT, TACT, Microsoft v2) is retained because it was parallel-launched, but the process error is recorded. [rrs-002: FAIL — mitigated by parallel launch]
• Regression set application: All 12 checks applied during report drafting. rrs-002 flagged as above. All others passed. [exp-003: run 3/3 complete]
• Tool-call failures: None material.
• State updates:
• rotation-state.json: next_rotation_index advanced to 2 (3.3); last_run_date updated to 2026-07-04; last_focus_dimensions updated to ["3.2"].
• source-index.json: 8 new entries added (all sources inspected today).
• reflections.json: 1 new active reflection written (refl-2026-07-04-001, evidence vs verification source types).
• watchlist.json: 2 new watch items added (watch-2026-07-04-001: calibration-gap source selection; watch-2026-07-04-002: REFLECT attribution vocabulary).
• experiments.json: exp-001, exp-002, exp-003 all marked runs_completed=5/5, 5/5, 3/3. All three experiments now ready for review.
• Stop reason: Budget exhausted and report complete.
• Subgoal checkpoints: Performed after Focus, Search Topics, Sources, Gaps, Findings, Proposals, and Recommended Outcome sections — all passed.
• Goal restatement: Practiced at ~3-source boundaries (after sources 3, 6, and before each report section).