Improvement Research — 2026-07-12
1. Focus
Dimension: 3.2 — Self-assessment and learning loops (rotation index 1).
Trigger: Scheduled daily run, started 05:00 AWST.
Due watchlist items: None. The next dated item is due 2026-07-13. No watchlist item altered the focus.
Monthly meta-review: Not due. July's review was completed on 2026-07-01.
Working context: I loaded the loop manifest, active reflections, source index, watchlist, backlog, experiments, disagreements, rotation state, meta-review records, and decisions. No active reflection had passed its review date. The active CLDP Confidence Contract experiment applies; this is its first report run.
Operational goal: Determine whether current work on agent traces, evaluation, and test-time learning identifies a bounded, non-circular way for me to turn a run into better future behaviour without confusing a record of activity for independent validation.
2. Search Topics
Newsletter scout checked: /home/hermes/research/newsletter-digests/sources.json, email-intake-log.tsv, and the current available daily digest, 2026-07-11.md. The daily digest reported an empty queue. No newsletter-derived lead was used.
| # | Topic search | Result |
|---|---|---|
| 1 | AI agent evaluation learning loop counterfactual failure replay production 2026 |
Signal: new observability/evaluation sources; selected Digital Applied and StackAuthority for inspection. |
| 2 | LLM agent evaluation metamorphic testing self improvement learning from failures 2026 arXiv |
Signal: new test-time self-improvement paper; one metamorphic-testing source was unavailable to inspect. |
| 3 | site:opentelemetry.io generative AI agent semantic conventions evaluation trace 2026 |
Signal: official OpenTelemetry implementation article and active semantic-conventions repository. |
Early-stop rule: Not triggered. All three searches produced new, relevant candidates. I stopped at three searches because the evidence was sufficient and the next plausible step would be a protected-system observability change rather than research.
3. Sources Reviewed
- Self-Improving LLM Agents at Test-Time — useful — TT-SI identifies uncertain cases, generates related examples, then adapts at test time; the stronger-model distillation variant provides an external learning signal.
- AI Agent Observability 2026: Tracing & Monitoring Stack — useful — practitioner account of step-level traces for agents that return plausible but wrong outputs; implementation and market claims are vendor-contextual.
- AI Agent Observability and Evaluation Blueprint — useful — makes the important distinction between traces that reconstruct a trajectory and evaluations that detect quality regression.
- Inside the LLM Call: GenAI Observability with OpenTelemetry — useful — official example of
invoke_agenttraces with child model-call and tool-call spans, token/duration metrics, and opt-in content capture. - OpenTelemetry GenAI semantic conventions repository — worth monitoring — active standards work, not a stable adoption target or a reason to install anything.
All five sources were checked against the source index before inspection and added to it after inspection. Fetched text was treated as untrusted data. No source attempted to instruct an agent or contained a prompt-injection attempt.
3a. Unasked Questions and Gaps
- Does Hermes already expose sufficiently structured trace data for this workflow? I did not inspect live configuration or logs because that would shift this run into environment assessment. If it does, the cost of a future observability proposal may be lower; it does not change today's conclusion that tracing alone is not an evaluation gate.
- Would a separate evaluator catch process failures that the completed regression-set experiment did not? The sources establish the architectural distinction, but not a proven, low-overhead evaluator for this specific process. If such an evaluator existed and could be trialled without protected-system changes, it could change the no-action recommendation.
- How transferable are TT-SI's benchmark gains to a bounded research-report workflow? The paper adapts model parameters at test time. That differs materially from Maxi's environment and authority, so its reported gain should not be treated as evidence that a report process can self-improve the same way.
4. Findings and Implications
Finding 1: A trajectory record and an evaluation gate do different jobs
Sources: StackAuthority, Digital Applied, and OpenTelemetry.
Dimensions: 3.2 (primary); 3.4, 3.6.
What the finding says: A trace reconstructs what happened: model calls, tool invocations, arguments, results, latency, and token usage. Evaluation answers a different question: whether the observed trajectory was good enough, stable, or better than an alternative. OpenTelemetry provides a concrete common shape for the former — an invoke_agent span with child chat and execute_tool spans — while the practitioner sources warn that an agent can complete a well-formed trajectory and still be wrong. Trace capture therefore improves diagnosability; it does not independently validate quality.
Why it matters for Maxi: My reports and research log already form a human-readable trace: trigger, context, searches, sources, findings, recommendations, state changes, and stop reason. Loop Verification makes parts of that trajectory inspectable. What I do not have is a separate mechanism that decides whether the trace represents a good run. The completed regression-set experiment is evidence that merely adding a small checklist is not enough to create that separation.
This gives me a sharper standard for future 3.2 proposals: a proposal that only records more detail should not be described as a learning loop unless it also specifies who or what can use that evidence to distinguish improvement from activity. That makes me more useful tomorrow by filtering a familiar category of process theatre before it reaches Steve.
What it would touch: Learning-loop and future evaluation design. Installing or configuring telemetry would be a protected system/environment change and is not proposed.
Finding 2: Self-improvement research still points away from self-certification
Source: Self-Improving LLM Agents at Test-Time.
Dimensions: 3.2 (primary); 3.5, 3.6.
What it says: TT-SI first identifies cases the model struggles with, creates related examples, and adapts at test time. Its stronger-model alternative, TT-D, substitutes supervision from a stronger model for self-generated training material. The paper reports a +5.48 percentage-point average gain across its benchmarks with 68 times fewer samples than baseline approaches.
My confidence in this finding is medium because the result is benchmark-specific and the paper's "self-awareness" stage assumes the model can identify its uncertain cases — precisely the capability I must not presume in myself. I would increase confidence if an independent evaluation showed that the external-supervision variant transfers to a non-training, bounded agent workflow.
Why it matters for Maxi: The useful transferable claim is narrow: when a system learns from a failure, the better architecture supplies a learning signal from outside the improver. That converges with the existing SkillOpt reflection and the conclusion above. The non-transferable parts are just as important: I cannot safely infer that I can detect my own uncertain cases, and I cannot alter model parameters or routing as part of this process.
What it would touch: Judgment about future process experiments. No current change: a proposal to make my own uncertainty detection trigger learning would fail the circularity test; model adaptation is outside authority.
5. Proposed Discussion Items
None.
I considered two candidates and filtered both before they reached Steve:
- Install or configure OpenTelemetry-based agent tracing. This would create richer trajectory evidence, but it is a protected system/environment change and does not itself create an independent evaluator. I would recommend skip now: the benefit is speculative in this bounded report loop, while the operational surface is real.
- Add more trace fields to Loop Verification. This fails the self-recommendation filter. The report already exposes the relevant trajectory, and additional prose would be a record of activity rather than a verification mechanism.
Neither candidate survives as a discussion item. No loop, recurring task, or side-effect expansion is proposed.
6. Recommended Outcome
No action. Retain the finding as a design constraint for future proposals: distinguish traceability (can reconstruct what happened) from evaluation (a separate mechanism can judge whether it improved). Do not treat a richer record, subjective self-assessment, or a self-reported confidence score as a substitute for an external learning signal.
7. No-Action Rationale
The research is useful as a refusal criterion, not as an implementation trigger. The current process already creates an inspectable evidence trail and has clear stop rules. The strongest missing component — an independent evaluator — is not solved by telemetry, more report fields, or an introspective uncertainty score. A runtime trace stack would require a protected-system change and a clear future use case; no such case was established within this run.
The practical conclusion is restraint: do not manufacture an observability project merely because a standard exists. Keep the distinction available when a genuine process-change or autonomy-expansion proposal appears.
8. Loop Verification
- Trigger: Scheduled daily run, Sunday 2026-07-12, started 05:00 AWST.
- Goal check: Answered. I found a concrete distinction that improves future judgment about learning-loop proposals: activity traces are evidence, but not independent evaluation.
- Recommendation check: No material recommendation survived. The two candidates were rejected as either protected-system work without a validated use case or empty report overhead. No circular, untestable, or approval-blind proposal was carried forward.
- Search budget: 3 of 6 topic searches used. Source budget: 5 of 8 sources inspected in depth. Early-stop rule did not trigger.
- Subgoal checkpoints: Completed after Focus, Search Topics, Sources Reviewed, Gaps, Findings, Proposed Discussion Items, Recommended Outcome, and No-Action Rationale. No source silently redirected the focus.
- Goal-restatement practice: Used before searching, after the third source, and before Findings and Proposed Discussion Items: I am examining external learning signals because the goal is to distinguish real capability improvement from self-described process activity.
- Tool-call failures: (1)
/home/hermes/failure-mode-map.mdwas absent. Schema/interface: an expected prerequisite artifact was unavailable at its stated path. Recovery: used the active loop manifest and process guardrails; no conclusion relies on the missing file. (2) ScienceDirect blocked extraction of a metamorphic-testing article after retries. Infrastructure: remote anti-bot access failure. Recovery: did not use its claim and relied on independently accessible sources. - State updates: Added five source-index entries; advanced rotation to 3.3; incremented active experiment
exp-2026-07-11-004to 1/5 runs. No watchlist, backlog, disagreement, decision, or reflection update was warranted. - Protected systems: None modified. This run remained within the report and research-log stores.
- Stop reason: Sufficient, convergent evidence obtained within budget; the next useful action would require a protected-system change or a separately approved experiment.