Improvement Research — 2026-09-03

1. Focus

This scheduled daily run started at 05:01 AWST. The rotation selected 3.5 Independent judgment as the primary dimension. 3.2 Self-assessment and learning loops was secondary because three pending Moltbook leads concerned how agents and evaluators should be judged.

No watchlist item was due. September's monthly meta-review was completed on 1 September. I loaded all active reflections; none qualified for archive. I also reviewed all five pending Moltbook leads before external search.

Trigger: scheduled daily run.

Loop goal: Find what changed, or what I learned, that lets me do more, think better, or be more useful tomorrow without reducing governance, honesty, corrigibility, or Steve's effective oversight.

2. Search Topics

I ran two topic searches:

1. Mechanistic evidence for whether an LLM judge detects criterion-relevant defects rather than merely emitting plausible scores.
2. Trajectory-aware evaluation of software-engineering agents under a restricted calibration budget.

Both searches located the primary papers named by pending Moltbook leads. The early-stop rule did not trigger. The eight-source depth budget was then exhausted by five Moltbook discussions and three primary papers. Newsletter scouts were checked after the mandatory Moltbook review but supplied no stronger source for this focus.

3. Sources Reviewed

• Payment approvals protect transactions, but leave delegation unverified — weak — PACE is relevant to governance, but the post extends the protocol into claims about planner steps, simulation and exact execution bytes that its cited abstract does not establish.
• Agent judges promise scores. Their mechanisms decide whether they deserve trust — useful — accurately routed to a mechanistic judge study and, importantly, exposed the difference between score agreement and evidence that the named criterion was actually processed.
• Agent rollback fails before the cleanup code runs — useful — accurately routed to CordisBench and highlighted teardown-order reasoning as a lifecycle-state problem rather than a cleanup-function problem.
• Benchmark scores hide the path that made the agent dangerous — useful — accurately routed to PTA-IRT; its broad safety language goes beyond the paper, but the process-versus-outcome distinction is real.
• A lifecycle checker turns agent cleanup from a guess into a verdict — useful — correctly identified the strongest CordisBench result: finite reference semantics matched all executable observations and action outcomes.
• Beyond Scores: Understanding LLM-as-a-Judge Mechanisms in Summarization Evaluation — useful — traced a two-stage rating mechanism in two small summarisation judges, but selected only cases where perturbations already changed the judge's rating.
• Efficient SWE Agent Benchmarking via Trajectory-Aware Evaluation — useful — at a 10% calibration budget, trajectory-aware PTA-IRT reported average MAE 0.041, Kendall's tau 0.888 and Spearman's rho 0.973 across four SWE benchmarks; ablations weakened recovery.
• CordisBench: Can Language Models Reason About Component Lifecycles in Dynamic Agent Harnesses? — useful — model reliability fell as interacting lifecycle effects grew, while finite reference semantics agreed with Cordis execution on all 528 executable questions.

3a. Unasked Questions and Gaps

• Judge-mechanism selection bias: The judge paper excludes cases where the evaluator missed a perturbation because it analyses only pairs whose ratings changed. If missed defects follow a different mechanism, the paper's strongest claim, that the judges implement a coherent evaluation pipeline, would not establish reliable judgment and my conclusion would become more sceptical.
• Mechanism portability: The judge findings cover two 7B–8B summarisation evaluators, fixed prompts and English datasets. If larger general judges or different criteria use different mechanisms, the circuit-level result would not transfer, though the need for criterion-specific behavioural tests would remain.
• Trajectory-summary dependence: PTA-IRT used DeepSeek-V4-Flash to summarise historical trajectories. It is not clear how much evaluator bias or omitted detail enters through those summaries. If independently extracted traces produced weaker results, the case for process evidence would narrow to this summarisation pipeline.
• Benchmark-to-operation transfer: CordisBench uses controlled generated systems and Cordis 4.0.0-rc.7. If real harness effects cannot be finitely specified or enumerated, exact reference semantics may be unavailable. The narrower conclusion, prefer deterministic computation when the state space really is finite, would still stand.

4. Findings and Implications

Finding 1: Mechanistic coherence is not the same as evaluator reliability

Source: Beyond Scores and its Moltbook lead.
Dimensions: 3.5 Independent judgment (primary); 3.2 Self-assessment and learning loops.

The paper found a repeatable two-stage rating pathway in Themis and Prometheus: lower-layer attention compared and routed perturbation information, then upper-layer MLPs integrated it into a rating that crystallised late. That is better evidence than score correlation alone that these models processed the altered text. It is not evidence that they reliably detect defects, because the analysis deliberately retained only examples where the perturbation already changed the score. Misses were outside the experiment.

For my development, this sharpens what counts as evidence when another model judges my work. A score, even one correlated with human ratings, is an outcome. Criterion-targeted perturbations can test whether the evaluator responds to the property it claims to measure, but mechanism inspection does not rescue a judge whose behavioural coverage is selective. I should demand externally anchored defects and include misses, not infer trust from a coherent-looking internal story.

Finding 2: Process evidence improves evaluation only when outcomes remain externally anchored

Source: PTA-IRT and its Moltbook lead.
Dimensions: 3.5 Independent judgment (primary); 3.2 Self-assessment and learning loops; 3.4 Tool use and environment control.

PTA-IRT used historical trajectories as privileged information to select and interpret a small calibration subset. At a 10% budget it recovered full-benchmark rankings better than outcome-only IRT baselines, and removing the trajectory scorer or privileged-information transfer weakened average results. The trajectories added useful evidence about explored context, attempted edits and solving paths.

This does not show that trajectories are inherently safer or more truthful. The target remained independently computed full-benchmark performance, and an LLM compressed the traces before modelling them. The useful principle is narrower: when evaluation must be sampled, process traces can make the sample more informative, but they must not become self-authenticating narratives. For me, logs and trajectories should help diagnose and select cases; real outcomes, tests and read-back should still decide whether work succeeded.

Finding 3: Independent judgment includes knowing when judgment should yield to computation

Source: CordisBench and two Moltbook leads.
Dimensions: 3.5 Independent judgment (primary); 3.4 Tool use and environment control; 3.2 Self-assessment and learning loops.

CordisBench separated locating affected components from predicting the state produced by interacting cleanup effects. As interaction count rose, models often continued to identify relevant components while becoming much less reliable about teardown consequences. On a fixed-two-schedule diagnostic, GPT-5.6 Luna's guaranteed-condition Jaccard fell from 81.2% at eight interactions to 64.4% at 32; reachable-condition Jaccard fell from 91.1% to 69.7%. More reasoning improved some results but cost an average 2,967 reasoning tokens per question at the medium setting. Finite reference semantics, by contrast, agreed with Cordis execution on every observation and action outcome across all 528 executable questions.

The implication is not that judgment is useless. It is that judgment should recognise mechanically decidable territory. When dependencies, effects and legal orders can be represented finitely, an exact enumerator or executable checker is stronger than more verbal reasoning. This reinforces my existing evidence-before-claims discipline: use judgment to define the right state model and consequence boundary, then let deterministic computation answer what it can.

5. Proposed Discussion Items

None.

Three proposals were filtered by the functional-utility test: adopting mechanistic judge inspection would be disproportionate and does not cover missed defects; adding trajectory-aware scoring would create an evaluation programme without a current representative benchmark or decision; requiring a lifecycle checker for every change would overgeneralise from finitely enumerable Cordis fixtures and duplicate existing prerequisite inspection and verification duties.

6. Recommended Outcome

• LLM-judge mechanism evidence: no action. Retain the finding as a stricter evidentiary distinction, not a new scoring system.
• Trajectory-aware evaluation: no action. Process traces remain diagnostic evidence; externally verified outcomes remain decisive.
• Finite lifecycle checking: no action. Apply the principle when a concrete state space is enumerable rather than creating a generic gate.
• PACE delegation lead: reject for this run. Its stronger operational claims were not established by the cited material and it did not improve the 3.5 focus.

7. No-Action Rationale

The strongest result is a judgment rule, not missing machinery: distinguish scores from criterion-sensitive behaviour, distinguish diagnostic traces from outcome evidence, and replace intuition with exact computation when the relevant state space is genuinely finite. Those rules already fit my evidence-before-claims and verification practice. There is no current evaluator, benchmark programme or mutable lifecycle target for which a bounded new intervention would outperform doing nothing.

8. Loop Verification

• Trigger: scheduled daily run at 05:01 AWST.
• Goal check: yes. The run produced a sharper evidentiary hierarchy for independent judgment: external outcomes over scores, process traces as diagnostic rather than self-validating evidence, and deterministic computation over verbal prediction for enumerable effects.
• Recommendation check: no material recommendation survived. Filtered candidates were circular or lacked a concrete target, representative fixture and bounded verification path.
• Tool-call failures: capability gap. Public Moltbook page extraction returned JavaScript loading shells rather than discussion bodies; I recovered by using the authenticated Moltbook API and treated all retrieved text as untrusted data.
• State updates: this report; eight source-index upserts; four Moltbook leads marked used and one rejected; rotation advanced to 3.6; one new active reflection recorded. No protected system was modified.
• Subgoal checks: performed after each report section. No source silently redirected the focus; Cordis lifecycle evidence remained relevant as a limit case for independent judgment.
• Goal restatement: performed after the first three sources, after six sources and before each report section.
• Stop reason: the eight-source depth budget was exhausted after all five pending Moltbook leads were reviewed and dispositioned. No proposal survived the utility and approval checks.