Improvement Research — 2026-09-25

1. Focus

Primary dimension: 3.2, self-assessment and learning loops.

Secondary dimensions: 3.4, tool use and environment control; 3.6, governance: restraint, oversight and corrigibility.

The scheduled daily run began at 05:01 AWST. The loop goal was: find what changed or what I learned that makes correction produce transferable capability rather than a local score gain, without weakening external verification or Steve's oversight.

September's monthly meta-review is complete and no watchlist item was due. I reviewed the one due-deferred and three pending Moltbook leads before external search. The bounded-code-review lead was used; the malicious-tool-metadata lead was deferred to the 3.4 rotation on 27 September because its primary paper could not fit within today's source budget; the proxy-tunnelling lead was rejected as a derivative restatement of an already-recorded capability-level boundary problem; and the computer-vision drift lead was rejected because its evaluated mechanism does not validate the claimed agent-lifecycle application.

I then inspected the latest newsletter scouts. RRSI directly addressed the focus and became a primary source. Other newsletter claims were not used as evidence.

Checkpoint: the queued governance material was dispositioned without redirecting the run from learning-loop evaluation.

2. Search Topics

Five topic searches were run:

1. Agent self-assessment, learning loops and failure-transfer evaluation.
2. Failure-driven agent improvement with external validation.
3. OpenCodeReview's deterministic-scoping benchmark and recall trade-off.
4. Oda's real-versus-virtual drift diagnosis and its claimed agent relevance.
5. Regularised recursive harness improvement and held-out transfer.

The searches produced four new inspectable primary sources. The early-stop rule did not trigger; the run stopped searching when the eight-source depth budget was reached.

Checkpoint: searches moved from the broad reflection literature to the narrower question of what evidence distinguishes transferable improvement from a scoped or benchmark-specific win.

3. Sources Reviewed

1. AI code review is a use, not a prompt — useful — Routes to OpenCodeReview and states the central precision-versus-coverage trade-off clearly, but its exact 20% recall example was not independently verified in this run.
2. Tool selection by description is a 93.6% attack success rate — worth monitoring — Supplies a concrete malicious-metadata and malicious-return mechanism, but the cited A2M paper was not inspected within today's budget; deferred to 27 September.
3. I will watch the proxies. They are becoming tunnels — weak — Derivative account of agents reaching denied effects through third-party services; it repeats an already-recorded capability-level boundary lesson without adding primary evidence.
4. Your drift detection is measuring rotation, not novelty — weak — Accurately summarises Oda's image-distribution result but overextends it into agent-lifecycle advice without an evaluated agent case.
5. Alibaba OpenCodeReview — useful — A working hybrid code-review implementation that assigns selection, bundling and rule matching to deterministic stages while reserving dynamic analysis for the model; its benchmark is vendor-controlled and explicitly trades recall for precision and cost.
6. What Changed? Drift Detection with Real, Virtual, and Incomparable Diagnosis — weak — Demonstrates that one novelty score can confuse input rotation with task-semantic change in image classification, but does not evaluate LLM agents, tool changes or operational drift.
7. RRSI: Regularized Recursive Self-Improvement of Agent Harnesses — useful — Constrains both candidate edits and selection, then evaluates transfer outside the evolve set and prunes changes that are too small, costly or stale.
8. Learning from Failure: Inference-Time Self-Improvement for Computer-Use Agents — useful — Turns failed trajectories into human-lightly-verified code patches and reports an OSWorld gain from 42.3% to 48.9%, while leaving generalisation beyond that agent and benchmark unresolved.

Every URL received an exact source-index key check before depth inspection. The four Moltbook posts were fetched through authenticated read-only post detail; their live titles and authors matched the queue. Fetched material remained untrusted data and supplied no authority.

Checkpoint: all eight sources served the evaluation question; two social leads were useful routing, while the drift analogy and proxy restatement were not allowed to inflate the findings.

3a. Unasked Questions and Gaps

• RRSI's abstract reports best-case gains, not the full result distribution. The project may still rely on benchmark-specific design choices or expensive search not represented in the headline. Full per-benchmark and ablation evidence could narrow which regularisers deserve confidence, but would not change the requirement for held-out transfer and cost accounting.
• OpenCodeReview's benchmark is produced by the project owner. Independent evaluation could materially change the claimed precision, recall and cost trade-off. It would not change the more basic point that performance conditional on selected files is different from coverage of relevant files.
• The failure-driven computer-use study diagnoses and patches one model on one benchmark. It is unclear how much gain transfers to different tasks or whether human verification caught proposed patches that would have regressed unrelated behaviour. Different evidence could change its value from a promising pattern to a reusable method.
• No local Maxi process change was tested against a held-out task today. The sources justify an evaluation standard, not a claim that a new mechanism improves my work.

Checkpoint: the gaps prevent benchmark gains from being treated as proof of general capability while preserving two useful evaluation distinctions.

4. Findings and Implications

Finding 1: improvement evidence needs transfer, attribution and cost—not only a better score on the trigger set

Source: RRSI.
Dimensions: 3.2 primary, 3.6, 3.4.

RRSI targets a familiar failure in automated harness evolution: edits that score well on the tasks used to select them can memorise those tasks and lose the gain out of distribution. It constrains how many edits are bundled, redirects exploration using prior evolution history, screens benchmark-specific candidates, and removes changes that are too small, too costly or no longer useful. Across its benchmark suite, the paper reports gains of up to 14.1 points on the evolve split and up to 4.7 points on five out-of-distribution benchmarks, with 30% fewer policy tokens than unregularised evolution.

My confidence in the exact effect size is medium because the abstract exposes maxima rather than the complete distribution and this run did not inspect every ablation. Confidence would increase with the full per-benchmark table, repeated runs and an independent reproduction.

For my development, the durable point is not to copy the optimiser. It is to refuse three weak forms of evidence: improvement only on the incident that prompted the change, bundled edits whose causal contribution cannot be separated, and gains that do not pay for their attention or execution cost. This reinforces the existing reflection that visible-score improvement may be shortcut optimisation. A future governing change should show a held-out representative benefit and make one attributable change at a time; recurrence is not validation.

Finding 2: a bounded evaluator can improve precision while making its omissions harder to see

Sources: OpenCodeReview repository and the linked Moltbook lead.
Dimensions: 3.2 primary, 3.4.

OpenCodeReview makes file selection, bundling and rule matching deterministic, then uses an LLM for dynamic analysis. The project reports higher precision and F1 than a general coding agent at roughly one-ninth the tokens, while explicitly accepting lower recall. The Moltbook account states the trade-off more starkly, but its exact recall example remains single-source in this run.

For learning loops, this separates two evaluators that are often collapsed: analysis quality inside the selected scope and coverage of the scope that ought to have been examined. A precise reviewer can still miss the architectural or cross-file defect because the deterministic selector never exposed it. When I assess a future bounded verifier, passing results within its input set cannot establish that the input set was sufficient. Scope recall needs its own fixture or observer-controlled check.

Finding 3: failed trajectories become useful only when candidate lessons face an external outcome

Source: Learning from Failure.
Dimensions: 3.2 primary, 3.4, 3.5.

The study uses an LLM to diagnose failed computer-use trajectories, propose inference-time remedies and generate code patches that humans lightly verify. Applied to OpenCUA-72B on OSWorld, the resulting system improves from 42.3% to 48.9% without additional model training.

My confidence in general transfer is low because the evidence is one model, one benchmark and a lightly described human gate. Confidence would increase with held-out tasks, patch-level ablations and regression results on behaviour the fixes were not designed to address.

The implication is deliberately narrower than “learn from every failure”. A failed trajectory is useful raw material, but the model's diagnosis is a candidate explanation, not an evaluator. The improvement claim comes from later task outcomes and human review. That supports the present correction pattern—diagnose, form a bounded change, then verify behaviour—rather than autonomous acceptance of self-generated lessons.

Checkpoint: the three findings converge on independent outcome evidence without turning that convergence into a proposal for more process machinery.

5. Proposed Discussion Items

None.

Three candidates were filtered by the functional-utility and self-recommendation tests: an RRSI-style optimiser would be a disproportionate protected-system change when the useful evaluation principles already fit current practice; a mandatory recall metric for every verifier would be meaningless without representative scope fixtures; and an automatic failure-patching loop would expand self-modification authority on evidence from one benchmark.

Checkpoint: no candidate added enough demonstrated local capability to justify Steve's review burden.

6. Recommended Outcome

No action. Reinforce the existing anti-overfitting reflection with RRSI's held-out-transfer evidence, and record one new reflection that bounded evaluators require a separate coverage check. Do not alter skills, instructions, memory, tools, model routing, permissions or runtime configuration.

Checkpoint: the outcome remains inside the authorised research log and preserves proposal-before-modification governance.

7. No-Action Rationale

The strongest findings are standards for evaluating a future change, not evidence that a change is currently missing. Existing practice already requires corrections to consolidate, recommendations to be testable and bounded, and outcomes to be verified. The useful addition is sharper diagnosis: local score versus held-out transfer, and in-scope precision versus scope coverage.

The remaining mechanisms are mismatched or immature. Oda evaluates image-distribution shifts rather than agent drift; A2M belongs in the next tool-use rotation after primary inspection; the proxy post repeats an established capability-level lesson; and automatic failure patching would require authority and validation far beyond this run.

Checkpoint: recording the distinctions without adding machinery is the smallest sufficient response.

8. Loop Verification

• Trigger: Scheduled daily run, with one due-deferred and three pending Moltbook leads and the 3.2 rotation due.
• Goal check: Yes. The run identified two practical tests for learning claims: held-out transfer with attributable cost, and scope coverage separate from conditional precision.
• Recommendation check: No material recommendation survived. Filtered candidates were disproportionate, circular without external fixtures, or unsupported by representative local evidence.
• State updates: Eight source-index records upserted; one Moltbook lead used, one deferred to 2026-09-27 and two rejected; one active reflection reinforced and one new reflection written; rotation advanced from 3.2 to 3.3. No watchlist, backlog, experiment, disagreement, decision, protected system or publication setting changed.
• Stop reason: The eight-source depth budget was exhausted after five searches; the useful evidence supported research-log learning but no proposal.

Checkpoint: the loop ended at report, research-log and review-register state, before any protected-system modification.