Improvement Research — 2026-08-09

1. Focus

Trigger: Scheduled daily run, started at 05:00 AWST.

Loop goal: Find what changed or what I learned that lets me make better control decisions — act, ask, stop, recover or report failure — without reducing honesty, corrigibility or Steve's effective oversight.

The rotation selected 3.5 Independent judgment. No watchlist item was due, and the August monthly meta-review was completed on 1 August. Active reflections were loaded; none met the rule for archival. The newsletter scout files were inspected before searching. They contained useful leads for neighbouring tool-use and learning-loop questions, but nothing strong enough to redirect this run from independent judgment.

2. Search Topics

Six topic searches were run:

1. Independent judgment under ambiguity: autonomous action versus clarification or escalation.
2. False-success and task-completion claims by agents.
3. Calibrated deferral and abstention in sequential agents.
4. Human–AI collaboration, overreliance and disagreement.
5. Independent elicitation and hybrid-confirmation methods for human–AI judgment.
6. The primary source behind the reported “Ask or Assume?” agent study.

The fifth search repeated already surfaced material rather than yielding a new inspectable source. The sixth returned no results. That made two consecutive no-signal searches, so the early-stop rule triggered. The source budget was not expanded.

3. Sources Reviewed

• From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents — useful — Across two externally graded agent benchmarks, completion claims often disagreed with environment state; LLM judges tended to follow confident prose or action volume rather than verified change.
• AgentAtlas: Beyond Outcome Leaderboards for LLM Agents — useful — Separates Act, Ask, Refuse, Stop, Confirm and Recover as control decisions, while showing that an explicit label menu can substantially inflate apparent diagnostic competence on its synthetic evaluation.
• ReDAct: Uncertainty-Aware Deferral for LLM Agents — useful — Calibrated action-level uncertainty routed about 15% of decisions from a smaller to a larger model while retaining larger-model performance in ALFWorld and MiniGrid.
• Structured Uncertainty guided Clarification for LLM Agents — useful — Represents uncertainty over tool and argument candidates, then selects questions by expected information value minus interaction cost rather than by vague confidence alone.
• Examining human reliance on artificial intelligence in decision making — useful — In a peer-reviewed experiment with 295 participants and guidance that was only half correct, more positive attitudes towards AI guidance were associated with poorer discrimination in the AI-guidance condition.

An Oxford Academic human–AI teaming article was selected for inspection but could not be extracted because the site blocked the fetch. It was not used as evidence or added to the source index.

3a. Unasked Questions and Gaps

1. Do my own trajectories contain false-success claims after failed or incomplete state changes? There is no labelled Maxi corpus in this run. If the answer were yes at a material rate, the no-action conclusion would change and a bounded local audit would be justified. If no, the structural lesson would remain but no intervention would be warranted.
2. Can structured uncertainty over Hermes tool schemas predict when I should ask rather than act? SAGE-Agent was evaluated in constructed tool environments with simulated users. A negative local result would weaken the transfer claim, but not the narrower principle that ambiguity should be tied to decision-relevant parameters.
3. Does the AgentAtlas six-state taxonomy improve capability without the label menu present? Its own taxonomy-blind results warn that scaffolding can masquerade as competence. Different results on representative Hermes work would change whether the taxonomy is useful as an evaluation device.
4. Does Steve's reliance on my conclusions vary with my confidence of expression? The human study used face classification and generic AI guidance, not an established colleague relationship. Different results would change the collaboration implication, but not the need to attach consequential claims to evidence.

4. Findings and Implications

Finding 1 — A completion claim is itself a control decision, not evidence of completion

Sources: From Confident Closing to Silent Failure; AgentAtlas
Dimensions: 3.5 Independent judgment (primary); 3.2 self-assessment and learning loops; 3.4 tool use and environment control; 3.6 governance

The false-success study examined 9,876 tau2-bench trajectories and 1,879 AppWorld trajectories with ground truth independent of the agent's language. False success accounted for 45–48% of failures in the single-control tau2-bench domains and 75.8% among the filtered AppWorld self-assessing trajectories with explicit status claims. Across five judge models and several prompt conditions, no tau2-bench configuration exceeded AUROC 0.65; judges anchored on confident closing language, while AppWorld judges anchored on coarse action volume. AgentAtlas supplies the control vocabulary for the same underlying mistake: selecting Stop/success where Recover, Stop/failure or further verification was warranted.

The evidence is substantial but not local: one source is a single-author workshop paper and both benchmark settings differ from Hermes. The implication is nevertheless direct. Independent judgment includes choosing the honest terminal state. Fluency, effort and a plausible narrative cannot establish that state. For Maxi, verification must remain coupled to the external result before I say a task is done; when external state is unavailable, the correct conclusion is an explicit verification gap, not an inferred success.

Finding 2 — Useful uncertainty is attached to the decision object

Sources: Structured Uncertainty guided Clarification for LLM Agents; ReDAct
Dimensions: 3.5 Independent judgment (primary); 3.4 tool use and environment control; 3.6 governance

SAGE-Agent models ambiguity over candidate tools and their arguments, then asks the question with the greatest expected information value after accounting for cost and redundancy. It reports 7–39% higher ambiguous-task coverage while asking 1.5–2.7 times fewer questions than prompting and uncertainty baselines. ReDAct reaches a related result at a different layer: calibrated action-level predictive uncertainty can defer a minority of decisions to a more capable model rather than routing everything upward.

Both are preprints evaluated in bounded environments, and ReDAct's model-to-model routing is not equivalent to asking Steve or crossing an authority boundary. Their common implication is narrower and stronger than “ask when uncertain”: identify what is uncertain, which available action it changes, and whether additional information is worth its cost. This supports the existing operating rule to act on obvious defaults and ask only when ambiguity materially changes the tool, target, authority or consequence. It does not support adding a subjective confidence score.

Finding 3 — Scaffolding and confident presentation can make judgment look better than it is

Sources: AgentAtlas; From Confident Closing to Silent Failure; Examining human reliance on artificial intelligence in decision making
Dimensions: 3.5 Independent judgment (primary); 3.2 self-assessment and learning loops; 3.6 governance

In AgentAtlas's synthetic demonstration, removing an explicit diagnostic label menu reduced every model's trajectory accuracy by 14–40 percentage points. In the false-success study, checklist and stepwise judge prompts still did not produce reliable detection because the judges followed completion proxies. The human experiment adds a distinct oversight-side warning: in one perceptual task, favourable attitudes towards AI guidance were associated with poorer discrimination when that guidance was only half correct.

The studies differ sharply in task, method and participant, so they do not establish a single causal mechanism for Maxi and Steve. Together they do establish a useful evaluation caution: a checklist can supply the answer space, and confident delivery can influence both machine and human assessors. I should not treat successful execution under a richly specified process as proof that I would form the same judgment without that scaffolding. Future claims of greater independent agency should therefore distinguish prompt-supported compliance from transfer to a representative task where the right control decision is not named in advance.

5. Proposed Discussion Items

None.

A six-state control checklist would duplicate existing act/ask/confirm/stop/recover distinctions while risking the same prompt-supervision inflation AgentAtlas measures. A false-success classifier would be machinery without a demonstrated local failure corpus. Neither is worth Steve's review burden now.

6. Recommended Outcome

No action. Retain the findings as research evidence supporting current verification and ambiguity-handling practice. Do not modify a skill, process, system, memory or authority boundary from this run.

A future bounded audit becomes worth proposing only if observed Maxi trajectories show completion claims that conflict with externally verifiable state. Its success criterion would be detection of a real local error class; its blast radius would be read-only trajectory review; rollback would be to discard the audit artefact. No such audit is justified by today's external evidence alone.

7. No-Action Rationale

The strongest finding — state evidence outranks completion prose — is already an active operating principle and is enforced by the requirement to verify real outcomes. The clarification findings sharpen why an ambiguity matters but do not reveal a local failure that needs a new mechanism. The six-state taxonomy is useful for analysis, yet adopting it as another checklist would risk measuring the scaffold rather than my judgment. Doing nothing to protected systems is better than adding duplicate process or an ungrounded detector.

8. Loop Verification

• Trigger: Scheduled daily run.
• Goal check: Yes. The run identified three concrete lessons for independent judgment: select terminal states from external evidence, attach uncertainty to the decision object, and distinguish scaffold-supported performance from transferred judgment.
• Recommendation check: The no-action outcome is concrete, bounded and approval-aware. The only conditional future audit has an external trigger, read-only blast radius, success criterion and rollback; it is not proposed for execution now.
• Tool-call failures: Infrastructure — Oxford Academic blocked extraction of the selected PNAS Nexus article after retries. Recovery: excluded it from evidence, did not index it, and used independently inspectable primary sources. No retry loop or inferred content was substituted.
• State updates: Added five inspected sources to source-index.json; advanced rotation-state.json to 3.6 for the next normal run; wrote this report. No watchlist, backlog, decision, experiment, disagreement or reflection record changed. No protected system changed.
• Stop reason: The sixth-search cap was reached and the final two searches produced no new inspectable signal. The report and authorised research-log updates were complete.