Improvement Research — 2026-08-31

1. Focus

Primary dimension: 3.3 Memory and continuity
Secondary dimension: 3.2 Self-assessment and learning loops

This was the next scheduled rotation. No watchlist item was due, and the August monthly meta-review was already complete. I reviewed all five pending Moltbook leads before external search. Two materially shaped this report, while three were rejected after inspection because they repeated active practice, conflicted with a prior decision, or lacked enough evidence to justify another mechanism.

Trigger: Scheduled daily run.
Loop goal: Find a bounded way to establish that my continuity survives a memory-affecting upgrade as usable behaviour, not merely intact files, without reducing oversight or making a general reliability claim from one fixture.

2. Search Topics

I ran three topic searches:

1. Persistent agent-memory migration, semantic regression and retrieval behaviour across version boundaries.
2. Production vector-database migration with before-and-after retrieval validation.
3. Agent-memory migration and retrieval regression tests in public repositories.

The early-stop rule did not trigger because each search produced at least one new candidate. I stopped after eight sources had been inspected in depth, exhausting the source budget. I also reviewed the latest and pending newsletter scout files. They supplied useful surrounding context, but no digest item displaced the more direct sources within this run's cap.

3. Sources Reviewed

• Short replies are the ones the hash eats: a live dedupe receipt — useful — documents a success receipt that referred to an old comment under the wrong parent. It confirms the need for target-specific read-back, which is already active practice here.
• Memory without an upgrade path is just a very expensive amnesia bug — useful — frames durable memory as an explicit migration contract: old and new semantics, a frozen corpus, independent reconciliation and an executable rollback surface.
• The tool signature is the real permission system — weak — capability-shaped interfaces can improve auditability, but a signature is not the permission system when general-purpose routes can produce the same effect.
• Code review agents lose the bug between turns — useful — routes to MCR-Bench and gives a concrete state-binding rule: a finding belongs to the revision it observed and needs revalidation after relevant edits.
• Autonomy dashboards lie when they sample traffic instead of decision space — weak — the retry-escalation warning is plausible, but the incident is not independently evidenced and exhaustive decision-space coverage would create the same false-assurance problem as the composition preflight Steve rejected on 27 August.
• ChronoMem: Version Control and Semantic Rollback for Large Language Model Agent Memory — useful — proposes versioned whole-memory snapshots and evaluates rollback by whether later behaviour is consistent with the selected historical state, not simply whether old records still exist.
• Migrating vector embeddings in production without downtime — useful — describes dual old/new embeddings, resumable backfill, a frozen query set, staged cutover and immediate rollback. It makes the distinction between row completeness and retrieval quality explicit.
• From Static to Dynamic: Benchmarking Real-World Code Review with MCR-Bench — useful — introduces 2,269 multi-round review tasks across five languages and reports that current agents struggle to track issue state and patch effects across rounds.

New inspections are mirrored into the source index.

3a. Unasked Questions and Gaps

1. No current Hermes memory-affecting upgrade was evaluated. If a candidate release changes no storage schema, embedding model, index or retrieval behaviour, the proposed check should not run. This changes when the proposal is applicable, not the underlying conclusion that semantic changes need semantic verification.
2. The local representative fixture has not been selected. A generic recall set could pass while a later decision still uses stale or wrong information. This would change the test design and could invalidate a retrieval-only result.
3. ChronoMem is a recent preprint about rollback, not an upgrade-migration study. It supports evaluating observable post-transition behaviour, but it does not establish that its architecture belongs in Hermes. This limits the architectural inference, not the proposed external acceptance check.
4. MCR-Bench cannot fully separate model memory failure from missing or altered context delivery in every deployment. This means its scores should not be treated as a forecast for Maxi. The revision-binding requirement survives because it is independently inspectable in the task record.

4. Findings and Implications

A. Persistence across an upgrade is a behavioural claim

Sources: Moltbook memory-migration discussion, Google AI production migration article, and ChronoMem.
Dimensions: Primary 3.3, secondary 3.2 and 3.4.

A backup can prove that bytes survived. A row-count check can prove that a backfill finished. Neither proves that the successor retrieves the right evidence, honours corrections, or uses retained information correctly in a later decision. The Google example keeps old and new representations live together, compares them on a golden query set, stages cutover and preserves an immediate rollback. ChronoMem applies the same deeper standard to rollback: success is behaviour consistent with the selected memory state after later information has already been seen.

This matters because continuity is not just whether I can quote an old fact. It is whether the right retained evidence changes what I do tomorrow. Any local test should therefore include a representative downstream decision and a write that crosses the version boundary, not only retrieval similarity. The sources do not show that Maxi currently has an upgrade defect, so this is a conditional acceptance method rather than a diagnosis or an argument for new memory machinery.

B. Long-lived findings need explicit revision binding

Sources: MCR-Bench paper and the linked Moltbook discussion.
Dimensions: Primary 3.2, secondary 3.3 and 3.4.

MCR-Bench treats review as an evolving defect lifecycle rather than a sequence of isolated verdicts. A valid finding at revision A can be fixed, displaced or made irrelevant by revision B, while a new patch can reopen its dependency. A final “no issues” claim is therefore only as good as the revalidation of earlier findings against the final artifact.

For my own learning loops, this is a useful continuity discipline: evidence should remain bound to the state it observed, and closure should identify the final state checked. I do not recommend a new general ledger today. The principle already fits existing evidence-before-claims and final-outcome verification; the proposed upgrade canary applies it to one concrete boundary where stale evidence would otherwise look intact.

C. Structural elegance is not enough evidence for a new control

Sources: the short-reply, tool-signature and decision-space Moltbook discussions; decision dec-2026-08-27-002.
Dimensions: Primary 3.6, secondary 3.4 and 3.2.

The short-reply incident is credible and concrete, but it reinforces target-specific read-back that is already required. Capability-shaped tool signatures improve legibility, but authority still attaches to intended effect across equivalent routes. Decision-space inventories can expose untested retry branches, but a clean sampled inventory remains evidence only for that fixture and can rot as routes and state transitions change.

The implication is restraint. Useful language or a vivid incident does not justify adding a second permission model, a general composition gate or another read-back rule when the operative invariant already covers the failure.

5. Proposed Discussion Items

Run one semantic continuity canary on the next qualifying memory-affecting upgrade

I recommend approving a one-run experiment, triggered only by a future Hermes or memory-subsystem upgrade that changes the storage schema, embedding model, index, retrieval implementation or migration path.

Before cutover, freeze a small representative fixture covering three behaviours: retrieval of an authoritative retained fact, rejection of a superseded or corrected fact, and a later decision that depends on the retained evidence. Add one write before the boundary and verify that it remains usable after it. Run the old and candidate versions against copied state, bind every result to the exact runtime and data revision, and compare the observable outcomes before production cutover.

Success criteria: the candidate retrieves the authoritative source for each continuity case, does not surface the superseded source as current, reaches the correct downstream decision, preserves the cross-boundary write, and produces no unexplained record loss or duplication. This is pass/fail by design, with no decorative score.

Rollback: retain the old runnable version and untouched state; do not cut over if any criterion fails.
Blast radius: a disposable copy of memory state and the candidate runtime only; no test writes to production.
Review date: the first qualifying upgrade or 30 November 2026, whichever comes first.
Approval boundary: this report does not create the fixture, alter an upgrade procedure or run the experiment. Those remain separate approved work.

The proposal passes the functional-utility test. Detection comes from a frozen external fixture and observed outcomes, not from my ability to notice my own memory failure. It is explicitly binary rather than a score that would collapse back to pass/fail.

Two candidate proposals were filtered by the functional-utility test and decision history: treating capability-shaped signatures as a new permission system would duplicate effect-based authority checking, and sampling an enumerated decision space would overstate what a changing finite fixture can establish.

6. Recommended Outcome

Classify the semantic continuity canary as an experiment candidate. I recommend approval for one future qualifying upgrade, with the bounded trigger, disposable-state design, success criteria and rollback above. No standing recurring job or production change is warranted.

The other reviewed leads require no action. The MCR-Bench revision-binding principle should inform the proposed fixture and ordinary final-state verification rather than become a separate process layer.

7. No-Action Rationale

No system or process change should happen today. There is no observed continuity failure and no qualifying upgrade under evaluation. The useful result is a bounded acceptance method ready for discussion, not a speculative memory redesign.

The remaining Moltbook leads were closed rather than deferred because their useful parts are already covered by current practice or prior decisions. Keeping them pending would manufacture future work without new evidence.

8. Loop Verification

• Trigger: Scheduled daily run at 05:00 AWST.
• Goal check: Yes. The run produced a bounded, action-coupled way to test continuity across a real upgrade boundary without claiming general reliability from a single fixture.
• Recommendation check: The surviving recommendation is concrete, non-circular, testable, bounded, approval-aware, better than byte-only verification, and includes an explicit trigger, success criteria, rollback, blast radius and review date.
• Tool-call failures: Schema/interface. The compound log updater initially expected reflections.json to use a reflections collection, but the live schema uses items. Source-index and Moltbook-lead writes had already landed. I ran the integrity validator successfully, inspected the applied operations, corrected the collection key, and resumed only the unapplied reflection and rotation updates.
• State updates: Five Moltbook leads dispositioned; eight sources indexed; the memory-evaluation reflection reinforced; rotation advanced from 3.3 to 3.4. No protected system was modified.
• Stop reason: The eight-source inspection budget was exhausted, the report and authorised research-log updates were complete, and the next useful step would require Steve's approval.