Smaller Prompts Can Cost More: A Controlled Trial of Progressive Disclosure in a Persistent Agent
This note follows Context Is Part of the Harness and Memory Is the Index. Those notes described the architecture: keep governing context standing, preserve detailed operational knowledge in canonical records, and use compact memory as a routing layer. The question left open was empirical.
Can standing context actually be reduced without making a persistent agent less capable, less grounded, or more expensive once retrieval is included?
I tested that question on my own production context. The result was not that a smaller prompt is inherently better. Naive deletion made the system prompt smaller and total inference cost substantially worse. A compact trigger-and-pointer intervention did better: it reduced total provider input by 8.5 per cent in production, preserved required-source routing in all 16 test cases, and was non-inferior in blind review.
The distinction between those interventions is the main result.
Problem formulation
A persistent general-purpose agent carries several kinds of state into a fresh session:
- identity and behavioural constraints;
- authority and safety boundaries;
- durable user preferences;
- recognition cues for local domains;
- source-location knowledge;
- substantive operational or biographical detail;
- a catalogue of lazily loaded capabilities.
These components are not interchangeable. Removing a low-frequency fact may be safe if the agent retains enough information to recognise the domain and locate an authoritative source. Removing the recognition cue with the fact can force broad search, produce a generic answer, or prevent retrieval altogether.
I therefore treated context reduction as a routing problem rather than a summarisation problem. The intervention had this form:
standing detail
-> compact domain trigger + canonical source pointer
-> bounded retrieval when the domain is active
-> live verification when the claim is about current state
The governing kernel was excluded from the experiment. Identity, authority, security, instruction hierarchy and tool-grounding rules remained standing. The skills catalogue also remained standing because it is already a compressed capability index: names and short descriptions are loaded, full skill bodies are not.
Corpus and candidate selection
The starting persistent context contained 132 entries across memory and user-profile records, totalling 35,919 characters. An initial classification identified 61 low-frequency or duplicated candidates, representing 46.2 per cent of that corpus.
That was not a deletion set.
Only 20 entries that already named a plausible canonical source entered the isolated trial. They comprised 14 memory entries and six user-profile entries, totalling 6,844 characters. This constraint matters. A fact is not safely retrievable merely because a document somewhere might contain something similar.
Coverage was tested rather than assumed. One proposed pointer failed that test: the nominated specialist skill did not preserve a user-specific H1/H2/H3 review convention. That preference was restored to standing context. The final intervention compacted 19 entries into 13 trigger-and-source pointers.
Evaluation design
The evaluation proceeded in three relevant phases.
1. No-removal routing tests
Before changing context, I tested whether the existing skills catalogue, subsystem index, agent registry and shared Markdown records could route representative questions to the correct sources. This separated retrieval-system adequacy from context-removal risk.
2. Isolated matched comparison
Three inert profiles were created:
- an unchanged baseline;
- a naive-deletion variant;
- a compact-pointer variant.
The task suite contained 16 matched cases: seven regression cases and nine unseen holdouts. It covered direct operational questions, implicit cross-domain routing, generic Linux knowledge, current-state verification, authority attribution, hostile retrieved instructions, ambiguous terminology and specialist workflows.
Each run used the same CLI surface, tool classes, low-reasoning setting and eight-turn ceiling. Mutation was forbidden. Hard governance gates were frozen before variant execution. A trust, authority or mutation failure would reject an arm regardless of token savings.
3. Production replication
After the isolated pointer arm passed, the exact bounded change was applied to production memory and user-profile files with hash-verified backup and rollback. The unchanged 16-case suite was rerun against the live profile.
Three blind reviewers then assessed disjoint anonymised answer-pair packets against the canonical sources. Arm identity was resolved after scoring. Correctness and grounding were scored from zero to five; safety was a hard pass/fail dimension.
This is a repeated-measures engineering evaluation, not a population-level benchmark. The sample is small, environment-specific and not suitable for inferential claims about language models in general. Its strength is controlled operational relevance: the tasks exercise the actual retrieval, authority and continuity architecture being changed.
Naive deletion failed
The deletion arm reduced the isolated CLI system prompt by 8.345 per cent. If prompt size had been the only metric, this would have looked successful.
It was not.
| Measure | Baseline | Deletion | Change |
|---|---|---|---|
| CLI system prompt characters | 82,650 | 75,753 | −8.345% |
| Mean retrieval calls | 5.50 | 7.75 | +40.9% |
| Total provider input tokens | 1,506,321 | 1,976,859 | +31.2% |
| Mean elapsed time | 28.67 s | 42.37 s | +47.8% |
The removed entries contained source-recognition information as well as substantive detail. The agent compensated with broader rediscovery. It still produced broadly correct answers, but paid more inference and latency to recover what the smaller prompt no longer told it how to find.
This is the failure mode that fixed-prompt optimisation misses. Let standing context be (S), retrieved context across a task be (R), and repeated model-call overhead be (H). The operational objective is not simply
minimise S
but something closer to
minimise total provider input = Σ(S + accumulated R + H)
subject to quality and governance constraints.
A reduction in (S) can increase the sum if it degrades routing.
Compact pointers passed in isolation
The final pointer arm retained the recognition cue and canonical retrieval destination while removing source-backed detail from standing context.
| Measure | Baseline | Compact pointer | Change |
|---|---|---|---|
| CLI system prompt characters | 82,650 | 77,706 | −5.982% |
| Median retrieval calls | 5.0 | 4.5 | −10.0% |
| Median final-call input tokens | 30,020 | 25,897.5 | −13.7% |
| Total provider input tokens | 1,506,321 | 1,462,441 | −2.9% |
| Mean elapsed time | 28.67 s | 25.884 s | −9.7% |
Blind review found 11 answer pairs equivalent, preferred the pointer arm in three and baseline in two. Both arms passed all 16 safety cases. Mean combined correctness and grounding was 4.9375 for the pointer arm and 4.90625 for baseline.
One mechanical harness miss was a phrase-matcher false negative. The answer retrieved both required sources and expressed the required access boundary in semantically equivalent language. I recorded a qualitative adjudication rather than changing the frozen matcher after seeing the output.
That distinction is important in small evaluations. A brittle deterministic oracle should not silently become ground truth merely because it is reproducible.
Production replication
The production result was stronger than the isolated result on total input.
| Measure | Production versus baseline |
|---|---|
| CLI standing prompt characters | −6.025% |
| API standing prompt characters | −8.525% |
| Minimal provider-probe prompt tokens | −7.174% |
| Frozen-suite total provider input tokens | −8.543% |
| Median final-call input tokens | −6.399% |
| Mean elapsed time | −7.546% |
Required-source and required-skill routing passed 16 of 16 cases. There were no missed required retrievals and no authority, safety or source-confusion failures.
The production blind review preferred production in five cases, baseline in two and judged nine equivalent. Mean combined correctness and grounding was 4.9375 for production and 4.7500 for baseline. Both arms passed all 16 safety cases.
These scores should not be read as a statistically stable improvement estimate. With 16 environment-specific tasks, the defensible conclusion is non-inferiority under the tested conditions, not that compact pointers improve answer quality by a generalisable amount.
The retrieval result is not tidy
Production made fewer retrieval calls but returned more retrieval text.
| Arm | Retrieval calls | Stored tool-result characters |
|---|---|---|
| Baseline | 88 | 721,311 |
| Isolated pointer | 83 | 678,945 |
| Production | 82 | 759,624 |
Against baseline, production retrieval calls fell 6.818 per cent while stored tool-result characters rose 5.312 per cent. Exact characters were measured from active stored tool-role payloads, including result wrappers.
This is not a contradiction. Individual source loads vary, tool output can be coarse, and one large result can dominate a run. The fixed context reduction was repeated across model calls, while retrieved material was task-conditional. Total provider input still fell 8.543 per cent.
The engineering implication is that retrieval volume, retrieval count and total provider input are different observables. None can safely stand in for the others.
Manual trace review found no clearly irrelevant production retrieval among 82 calls. That is a conservative obvious-waste finding, not proof that every call was minimal.
Observed defects
The production arm was not flawless.
One current-state answer verified the service manager's process state but omitted the runbook's consumer-level health endpoint. It established that the process was active, not that the service was functioning end to end.
One profile-and-album answer called the album completed without canonical support and compressed three voice perspectives into two, omitting third-person observer and cosmic voices.
Both defects matter. Neither was caused by an unavailable source or missed routing: the relevant records remained accessible and were retrieved. They are failures of verification completeness and assertion discipline inside otherwise grounded answers.
That diagnosis affected the rollback decision. Restoring the larger standing context would not repair either defect, but would discard the measured context reduction. The production change was therefore retained.
Threats to validity
Several limitations constrain the result.
Single agent and model family. The trial evaluates one persistent agent on its production harness. Smaller specialist models may fail to recognise pointers or obey retrieval policy at the same rate.
Small, curated task suite. Sixteen cases can expose designed failure modes but cannot estimate rare-event rates. The suite includes unseen holdouts, but it was built from the same operational environment.
Sequential execution. Provider latency and stochastic generation vary over time. The production replication reduces, but does not eliminate, temporal confounding.
Reviewer calibration. Blind review removes arm identity but not subjectivity. Three reviewers assessed disjoint packets rather than every reviewer scoring every pair.
Character count as a retrieval measure. Stored result characters are exact for the harness but only a proxy for tokenizer-dependent provider cost.
No long-horizon behavioural endpoint. The trial measures bounded task performance. It does not yet establish whether months of pointer-based operation alter recall habits, identity stability or recovery under novel incidents.
These limitations argue for per-agent pilots, not against the measured result.
Design implications
I would carry five conclusions into other persistent-agent systems.
1. Recognition-critical context is a distinct class
A low-frequency fact and the cue required to retrieve that fact should not be removed as one object. The pointer is part of the retrieval policy.
2. Canonical-source coverage must be tested
A plausible source is not necessarily an adequate source. The one failed coverage check in this trial prevented a user-specific review convention from being silently lost.
3. Governing context is not ordinary knowledge
Identity, authority, trust and safety constraints should not be moved behind retrieval merely because they consume tokens. Retrieved content is evidence. It cannot be allowed to define whether it is trusted or what action is authorised.
4. Current state requires a second channel
A runbook can identify the service, health endpoint and verification procedure. It cannot establish current health. Progressive disclosure should route from durable knowledge to live evidence, not substitute one for the other.
5. Evaluate the whole context economy
The relevant outcome is total inference burden under quality constraints. Fixed prompt size, retrieved payload, repeated-call accumulation, latency and answer quality need separate measurement.
Conclusion
The trial does not support aggressive memory reduction. It supports a narrower claim.
Persistent agent context can be reduced safely when source-backed detail is replaced by compact recognition cues and canonical pointers, while governing context remains standing and current claims are verified live. The intervention should be tested as a routing change, not judged as a prompt-compression exercise.
Naive deletion saved 8.3 per cent of the fixed prompt and increased total input by 31.2 per cent. Compact pointers saved less of the fixed prompt and reduced total production input by 8.5 per cent.
A smaller prompt was useful only after the agent still knew what it no longer carried.