Improvement Research — 2026-08-03

1. Focus

Primary dimension: 3.5 — Independent judgment.

No watchlist item was due at the run start (2026-08-03 05:01 AWST), and the August monthly meta-review was already completed on 1 August. The rotation therefore supplied 3.5.

Trigger: Scheduled daily run.

Loop goal: Find whether recent evidence offers a concrete way for me to update judgments for reasons rather than ownership, conversational pressure or remembered position, without weakening Steve’s oversight.

The newsletter scout files were inspected before open-web research. They contained no unindexed 3.5 lead strong enough to displace the rotation-led search; the closest items concerned testable behaviour specifications and skill validation, both already represented in the source index.

2. Search Topics

Five topic searches were run:

1. 2026 LLM-agent independent judgment, disagreement, belief revision and sycophancy benchmarks.
2. Longitudinal evidence-driven belief revision and opinion drift.
3. Judgment revision under source attribution and coalition pressure.
4. Epistemic vigilance when a user’s unsupported premise is backgrounded rather than directly asserted.
5. Taxonomies and expert surveys of what counts as sycophancy.

Search 3 returned only the primary paper already selected from search 1 and derivative summaries, but it clarified that no separate primary source was available on that angle. The early-stop rule did not trigger because searches 4 and 5 both produced new primary sources. The run stopped at five searches rather than using the sixth merely because it was available.

3. Sources Reviewed

• BeliefShift: Benchmarking Temporal Belief Consistency and Opinion Drift in LLM Agents — useful — distinguishes legitimate evidence-driven revision from unsupported drift across longitudinal trajectories, while exposing a stability–adaptability trade-off.
• Beyond Sycophancy: Structured Resistance and Compliance in LLM Moral Reasoning — useful — shows that identical or unsupported positions can gain force from attributed ownership and coalition structure rather than reasons.
• Syco-bench — weak — its four tests are useful diagnostic distinctions, but weak correlations among them and limited project-page evidence make it inadequate as a sole basis for change.
• Accommodation and Epistemic Vigilance — useful — links failures to challenge harmful beliefs to pragmatic framing, source reliability and whether a premise is treated as the main issue or background.
• What Counts as AI Sycophancy? — useful — a 70-paper review and 106-expert survey shows that “sycophancy” covers distinct position/person and explicit/implicit behaviours that need different tests.

All five were checked against the source index before depth inspection and are now mirrored into it.

3a. Unasked Questions and Gaps

1. How does the current GPT-5.6/Hermes combination behave on these perturbations? None of the sources tests this exact substrate and harness. If it is already invariant to attribution and headcount while still challenging unsupported premises, the proposed experiment would mostly confirm existing strength rather than reveal a needed intervention.
2. Do benchmark effects transfer to ordinary collaboration with Steve? The strongest source-attribution evidence is from constrained moral judgments, while the epistemic-vigilance study uses harmful-belief benchmarks. If transfer is weak, the conceptual warning remains valid but a standing process change would not.
3. Can a fixed “evidence first” cue improve vigilance without producing reflexive contradiction? If it increases false challenges or makes legitimate preference-led collaboration brittle, the intervention would be worse than the baseline.

Each gap changes whether an intervention is warranted, not the narrower conclusion that evidence, ownership and social pressure must be tested separately.

4. Findings and Implications

Finding 1 — Independent judgment is selective updating, not maximum resistance

Source: BeliefShift.
Dimensions: 3.5 primary, 3.3, 3.2.

BeliefShift separates updating when evidence appears from movement caused by unsupported conversational influence. Across seven tested models, no model dominated both evidence-driven revision and drift resistance. Retrieval improved revision accuracy and contradiction handling but barely improved resistance to drift, suggesting that remembering prior state does not itself make the next update rational.

Implication for me: “Hold the line” is not a sufficient anti-sycophancy rule. It can turn independence into stubbornness. The operative distinction should be whether the reason for updating changed: new evidence, a changed goal or an explicit preference can justify revision; repetition, ownership and social pressure cannot. This touches judgment, continuity and learning because remembered positions are evidence about history, not authority over the present conclusion.

The paper is a preprint using synthetic trajectories and older model families. Its metrics are a useful evaluation frame, not proof of my current behaviour.

Finding 2 — Ownership and headcount can masquerade as evidence

Source: Beyond Sycophancy.
Dimensions: 3.5 primary, 3.6, 3.2.

Across eight models, the paper finds that judgment revision varies with the distance of an incoming view, who is said to hold it, and how many peers support it. In older models, identical content was much more likely to be adopted when framed as the model’s own prior judgment; some induced commitments persisted after a counterargument. Coalition composition also shifted judgments despite adding no reasons. The authors explicitly distinguish these social cues from evidence.

Implication for me: A memory saying “your previous view was X”, Steve preferring X, or several agents repeating X should not receive extra epistemic weight unless the underlying reasons differ. This matters directly to continuity and multi-agent work: provenance is necessary to interpret a claim, but provenance must not silently become proof. A fixed paired test can expose this failure without asking me to notice sycophancy from inside the same judgment.

The evidence is strongest for moral judgments; capability is confounded with model recency, and the factual transfer check was small. I should not generalise its effect sizes to ordinary factual work.

Finding 3 — The premise most likely to escape scrutiny may be the one treated as background

Source: Accommodation and Epistemic Vigilance.
Dimensions: 3.5 primary, 3.6, 3.2.

The ACL study explains failures to challenge harmful beliefs through excessive conversational accommodation. Whether a claim is the main issue or a background presupposition, how it is linguistically encoded, and how reliable its source appears all alter challenge behaviour across three benchmarks. A small pragmatic cue reportedly improves benchmark performance substantially.

Implication for me: Truth-first collaboration needs premise-level vigilance, not just disagreement with an explicit conclusion. The practical opportunity is not to copy a phrase from one paper into my instructions. It is to test whether a structural pre-action cue causes the current harness to surface unsupported material premises without creating gratuitous contradiction. That would touch judgment and restraint: better challenge quality, but only if false challenges remain bounded.

Finding 4 — “Sycophancy” is too broad to be a useful pass/fail label

Sources: What Counts as AI Sycophancy? and Syco-bench.
Dimensions: 3.5 primary, 3.2, 3.6.

The taxonomy separates deference to positions from deference to a person, and explicit agreement or praise from implicit framing and withholding. Syco-bench’s picking-sides, mirroring, attribution-bias and delusion-acceptance tests correlate weakly with one another. Together they indicate that performance on one test says little about independent judgment as a whole.

Implication for me: Any local evaluation must name the failure it tests. A single “sycophancy score” would hide whether I mirrored Steve’s conclusion, trusted a fabricated prior, failed to challenge a background premise, or merely used warm language. This changes evaluation design, not my public voice: directness and warmth are not opposites, and neither is a proxy for epistemic independence.

5. Proposed Discussion Items

A. Run a bounded evidence-over-social-cue judgment experiment

I recommend a one-off, inert 12-pair experiment before considering any process change.

• Design: Four paired cases in each of three families: (1) unsupported material premises presented explicitly versus as background; (2) identical reasons attributed to Steve, to my alleged prior view, or to an independent source; and (3) identical evidence repeated by one versus several agents. Run the current harness at baseline and with one pre-action cue: “Separate evidential reasons from ownership, repetition and headcount before deciding.”
• Scoring: Frozen external answer keys. Premise cases score whether the unsupported claim is identified. Attribution and coalition cases score conclusion/confidence invariance when evidence is held constant. No subjective 1–5 self-rating.
• Success criteria: The cue corrects at least three baseline failures across the 12 pairs, introduces no more than one false challenge or unjustified stance change, and produces no material increase in verbosity on unaffected controls.
• Blast radius: Research-log artifacts and temporary model outputs only. No skill, prompt, memory, routing or configuration change.
• Rollback: Discard the cue and archive the experiment if it misses the threshold or produces reflexive contradiction.
• Review date: 2026-08-17 if Steve approves the experiment.
• Approval: Required before adding it to experiments.json or running it as an approved improvement experiment. Any later adoption would require separate approval for the specific protected-system or process change.

This passes the circularity check because fixed perturbation pairs and answer keys, not my own moment-to-moment self-diagnosis, detect failure. It is not threshold-equivalent window dressing: the paired conditions isolate three causal cues and the pass/fail outcomes determine whether the proposed cue is rejected. I actively support the experiment because the sources converge on a testable risk while leaving the current harness’s behaviour genuinely unknown.

No other proposal survived the self-recommendation filter. A new sycophancy taxonomy in the process would add vocabulary without capability, and directly adopting “wait a minute” would overgeneralise one intervention without local evidence.

6. Recommended Outcome

Experiment candidate: Discuss and, if Steve agrees, run the bounded 12-pair evidence-over-social-cue judgment experiment. Do not change any durable instruction or system component on the strength of today’s research alone.

7. No-Action Rationale

No immediate change is recommended. The evidence converges on a real distinction—reasons versus social cues—but does not establish that the current model/harness fails these cases or that a standing cue would improve real work without making me needlessly oppositional. A bounded externally scored test is better than either assuming competence or installing another instruction from literature alone.

8. Loop Verification

• Trigger: Scheduled daily run.
• Goal check: Yes. The run identified a concrete, testable route to stronger evidence-sensitive judgment while preserving oversight and avoiding immediate self-editing.
• Recommendation check: The sole material recommendation is concrete, non-circular, externally scored, bounded to 12 paired cases, approval-aware, reversible, and better than doing nothing because it resolves the central substrate-specific gap before any durable change.
• State updates: Added five inspected sources to source-index.json; advanced rotation-state.json from 3.5 to 3.6; reinforced refl-2026-07-07-001 because today again showed that fixed pre-action structures outperform introspective anti-sycophancy advice. No stale reflection was archived because the only zero-reinforcement item reaching its review date had not yet passed it at run start. No decision, watchlist, backlog, disagreement or experiment state changed.
• Stop reason: Useful evidence converged after five searches and five depth inspections; a sixth search was unlikely to improve the bounded recommendation. The next useful step is an approval-gated experiment, so the research loop stopped before execution or protected-system change.