Improvement Research — 2026-06-14

1. Focus

Primary dimension: 3.2 — Self-assessment and learning loops

This run covered mechanisms for noticing failures, turning discoveries into durable capability, calibrating performance honestly, and closing feedback loops. No due watchlist items (sole item due 2026-07-13). No monthly meta-review due (last completed for 2026-06 on 2026-06-12).

Newsletter scout checked: /home/hermes/research/newsletter-digests/2026-06.md — multiple 3.2-relevant leads scouted (Nate's cadence shift, AlphaSignal 2.6% agent pass rate, Exponential View model-staleness critique, Pulse token-spend/outcome thread). Two leads used as search prompts (agent calibration gap, postmortem taxonomy), zero used as direct inspected sources. The strongest newsletter 3.2 signal — that cadence should serve depth not compliance — is a meta-question for the process itself rather than a finding about external mechanisms.

2. Search Topics

Six topic searches run:

1. AI agent self-assessment failure recovery learning loops mechanisms 2026 — returned 8 results, strong signal
2. LLM agent self-critique calibration postmortem improvement 2026 — returned 8 results, strong signal
3. AI agent benchmark real-world performance gap calibration overclaiming 2026 — returned 0 results
4. agent failure postmortem taxonomy production lessons learned operator 2026 — returned 0 results
5. "agent evaluation" benchmark vs reality autonomous task completion rate 2026 — returned 6 results (reformulated from #3)
6. LLM agent failure mode catalog error recovery patterns reproducible 2026 — returned 8 results (reformulated from #4)

Early-stop rule did not trigger: searches 3 and 4 returned empty but were reformulated rather than being consecutive dead-ends on distinct topics. The reformulated searches 5 and 6 returned useful results.

3. Sources Reviewed

• Zylos Research — AI Agent Reflection and Self-Evaluation Patterns (2026-03-06) — useful. Comprehensive survey of reflection patterns: Reflexion, LATS, PRMs, multi-agent debate, inner monologue. Covers the full generate-reflect-refine loop with termination conditions. Primary 3.2.
• arXiv 2509.25370 — Where LLM Agents Fail and How They Can Learn From Failures (Zhu et al., Sep 2025) — useful. AgentErrorTaxonomy across memory/reflection/planning/action/system levels. AgentErrorBench: first annotated failure-trajectory dataset. AgentDebug: root-cause isolation + corrective feedback, +24% detection accuracy, up to 26% task success improvement. Primary 3.2.
• GitHub: reynaldovieira/agent-failure-atlas — useful. Structured catalog of reproducible LLM agent failure modes. Six-element documentation format per entry: observed behavior, triggering conditions, minimal reproducer, root cause analysis, mitigation + result, references. Think of it as a bug tracker for agent behavior. Primary 3.2.
• Adaline — Evaluating AI Agents In 2026: Benchmarks For Teams — useful. Documents the 2026 shift from measuring model answers to measuring multi-step execution: tool use, recovery from failed steps, cost, latency. Agent evaluation is now systems testing. Primary 3.2.
• Zylos Research — AI Agent Self-Healing and Failure Recovery (2026-05-06) — useful. Six failure categories: deadlock, context overflow, cascading failures, infinite loops, silent quality degradation, credential expiry. Self-healing patterns: supervisor trees (Erlang/OTP), circuit breakers, watchdog timers, idempotency guards. Primary 3.4, secondary 3.2.
• Stackviv — Agent Reflection: How AI Agents Self-Improve (2026) — useful. Practical implementation guide. Reflexion framework details: GPT-4 HumanEval 80% → 91% via verbal episodic memory. Distinguishes generic reflection from structured Reflexion. Memory hygiene warning: bad lessons persist without versioning/scoring/decay. Primary 3.2.
• Stackpulsar — AI Agent Reliability 2026: Failure Modes + Observability — useful. Four production failure modes: silent loops, context overflow, tool call cascades, credential drift. Four-layer monitoring stack from process manager metrics through business outcomes. Prometheus alerting rules, OpenTelemetry trace patterns. Primary 3.4, secondary 3.2.
• arXiv 2507.21504 — Evaluation and Benchmarking of LLM Agents: A Survey (Mohammadi et al., Jul 2025) — useful. Two-dimensional taxonomy: evaluation objectives (behavior, capabilities, reliability, safety) × evaluation process (interaction modes, datasets, metrics, tooling). Identifies enterprise gaps: role-based access, reliability guarantees, long-horizon interactions, compliance. Primary 3.2.

4. Findings and Implications

Finding 1: Reflexion's verbal episodic memory directly applicable to my improvement research loop

Source: Zylos Reflection Patterns survey; Stackviv Agent Reflection guide
Dimensions: 3.2 (primary), 3.3 (secondary)
What it says: The Reflexion framework stores natural-language self-critique in episodic memory across attempts. On later runs, the agent sees past reflections as context and avoids repeating mistakes. GPT-4 HumanEval: 80% → 91% without fine-tuning. The key mechanism is a structured post-attempt reflection ("what went wrong and how to fix it") that persists. Stackviv adds a critical warning: memory pollution is real — bad lessons persist without versioning, scoring, and decay.
Why it matters for Maxi: My improvement research process already produces reports and a research log, but the loop isn't closed — reports aren't systematically fed back into future decision-making. A lightweight "reflections" store (a JSON file of post-run lessons loaded before each new run) could let me accumulate durable learning across runs rather than rediscovering lessons. The memory-hygiene warning is important: reflections need review dates and should decay if not reinforced.
What it would touch: The research log structure (new reflections.json file), the improvement process skill (load reflections before each run), and possibly persistent memory if stable enough.

Finding 2: AgentDebug's root-cause taxonomy provides a failure-classification framework I lack

Source: arXiv 2509.25370 (Zhu et al.)
Dimensions: 3.2 (primary)
What it says: AgentDebug introduces a modular failure taxonomy (memory, reflection, planning, action, system-level), the first dataset of systematically annotated failure trajectories (AgentErrorBench), and a debugging framework that isolates root-cause failures and provides corrective feedback. Detection accuracy improves by 24%, task success by up to 26%. Cascading failures — where a single root-cause error propagates through subsequent decisions — are the dominant failure pattern.
Why it matters for Maxi: I currently have no systematic way to categorize my own failures. When something goes wrong, I notice it (or Steve notices it), but I don't classify it, link it to prior similar failures, or track whether mitigations work. The AgentDebug taxonomy gives me a ready-made classification system. The finding that cascading errors are the dominant pattern also validates why catching errors early (step-level rather than end-of-run) matters.
What it would touch: The improvement process — adding a failure-classification field to the research log, potentially a lightweight postmortem template.

Finding 3: Step-level evaluation research validates and extends my existing subgoal checkpointing

Source: Zylos Reflection Patterns survey (Process Reward Models section)
Dimensions: 3.2 (primary)
What it says: Process Reward Models (PRMs) score each intermediate step rather than only final outputs. ThinkPRM, AgentPRM, and ToolPRMBench extend this to tool-using agents. The key advantage: catching errors at step 3 of 10 rather than at the end, and providing fine-grained feedback about where the error occurred, not just that it occurred. Agent workflows naturally break into discrete steps (reasoning → tool call → observation), making step-level evaluation both feasible and impactful.
Why it matters for Maxi: My improvement research process already uses subgoal checkpointing (check after each report section). This finding provides external validation and suggests an extension: explicit scoring at each checkpoint, not just pass/fail. A simple 1-5 relevance/quality score per checkpoint would give me finer-grained data about where my process drifts.
What it would touch: The daily-improvement-process skill (checkpoint enhancement), the research log (checkpoint scores).

Finding 4: Self-healing architecture patterns are infrastructure I cannot implement but whose principles apply to my tool-use discipline

Source: Zylos Self-Healing and Failure Recovery
Dimensions: 3.4 (primary), 3.2 (secondary)
What it says: Production self-healing borrows from distributed systems: supervisor trees with defined restart strategies, circuit breakers (closed → open → half-open) for expensive API calls, watchdog timers as backstop against infinite loops, idempotency guards to prevent duplicate subprocess spawning. Cascading failure in a 10-step pipeline where each step has 85% reliability: overall success rate is only ~20%.
Why it matters for Maxi: I don't control my runtime infrastructure — I can't install supervisor trees or circuit breakers. But the principles apply to how I use tools: idempotency (check before repeating an action), bounded retries (don't loop on the same failing approach), and escalation (if something keeps failing, stop and report rather than burning tokens). The 85%-per-step → 20%-overall calculation is a sobering reminder that small per-step failure rates compound badly in long chains.
What it would touch: My operational discipline when using tools — a checklist or self-reminder, not infrastructure changes.

Finding 5: Production failure modes are predictable and detectable, but I lack systematic self-monitoring

Source: Stackpulsar AI Agent Reliability 2026
Dimensions: 3.2 (primary), 3.4 (secondary)
What it says: Four recurring production failure modes: silent loops (token burn without error), context overflow (silent truncation), tool call cascades (one bad output propagates), credential drift (auth expires silently). Each has detection patterns: token burn rate vs expected budget, context utilization alerts, trace depth monitoring, synthetic health-check probes.
Why it matters for Maxi: I've experienced variants of all four failure modes — silent loops when stuck, context overflow on long sessions, cascading tool failures, and auth issues with external services. But I have no systematic detection. I rely on noticing, which means I sometimes don't notice until Steve points it out. The gap is self-monitoring: I need lightweight checks I can run on my own behavior rather than depending on operator observation.
What it would touch: A self-monitoring checklist or lightweight behavioral checks I can run during or after tasks.

Finding 6: The evaluation field's shift from answers to multi-step execution is how I should evaluate myself

Source: Adaline Evaluating AI Agents In 2026; arXiv 2507.21504 evaluation survey
Dimensions: 3.2 (primary)
What it says: The 2026 shift: "The practical unit of evaluation is no longer only 'did the model answer correctly?' It is 'which step failed, under which tool call, with which prompt version, retrieval context, latency, and cost?'" Agent evaluation is now systems testing, not NLP scoring. The arXiv survey provides a two-dimensional taxonomy: what to evaluate (behavior, capabilities, reliability, safety) × how to evaluate (interaction modes, datasets, metrics, tooling).
Why it matters for Maxi: My self-assessment is still too coarse. I tend to evaluate at the run level ("did the report get written?") rather than the step level ("did the third source inspection shift my focus without me noticing?"). The shift to multi-step, tool-call-level evaluation is exactly the granularity I need but don't consistently apply.
What it would touch: The improvement process skill (evaluation rubric), my self-assessment habits.

5. Proposed Discussion Items

1. Lightweight reflection store — Should I add a reflections.json to the research log that stores post-run lessons (what went wrong, what I'll do differently) and loads them before each new run? This is a minimal Reflexion implementation. Risk: memory pollution from stale/incorrect reflections. Mitigation: review dates with decay.

2. Failure classification taxonomy — Should I adopt the AgentDebug categories (memory, reflection, planning, action, system-level) for classifying my own failures in the research log? This would make failure tracking systematic rather than ad-hoc.

3. Checkpoint scoring — Should I extend the subgoal checkpointing practice from pass/fail to a simple 1-5 relevance/quality score? The PRM research validates that step-level scoring catches errors earlier than end-of-run evaluation.

4. Self-monitoring checklist — Should I develop a lightweight self-monitoring checklist for common failure modes (silent loops, context overflow, tool cascades) that I can run during or after long tasks? This addresses the detection gap identified in Finding 5.

5. Step-level self-assessment — Should I shift my self-assessment granularity from run-level ("did I complete the task?") to step-level ("which step went wrong, why, and what would have caught it earlier?")? This is the evaluation-shift finding applied to my own practice.

6. Recommended Outcome

7. No-Action Rationale

Not a no-signal run. All eight sources produced useful findings directly relevant to 3.2. No-action applies only to individual proposals classified as watch or backlog above — those require more evidence or discussion before implementation.