Self-tuning
Letting ROSE propose and validate its own retrieval improvements.
Every stage of ROSE is corrected by an outcome. A compression that fails replay is rejected; a dropped detail that keeps rescuing failures is folded back; consolidation follows what actually got used together. One stage was exempt: the criteria that decide what gets recalled — the relevance prompt and a handful of constants — could only change when a person had an idea.
People are not good at this. Of six hand-written attempts to improve the relevance prompt, five made retrieval worse. Each was plausible, each had an argument, and the common shape of the mistake was the same: buying a little less noise by dropping lessons that mattered.
rose tune # one proposal, measured rose tune --rounds 3 rose tune --history # every attempt, including the rejected rose tune --dry-run # propose without applying
The loop#
- Measure. Run
eval-recallto get a baseline. - Show the failures. Not the score — the actual cases, in both directions, misses first. A number says the stage is wrong; only the cases say how.
- Take one proposal. Either a replacement prompt or one constant.
- Apply it in a sandbox and measure again.
- Keep it only if it wins on precision and recall.
- Write the attempt down either way.
Why it is safe to leave running#
| Property | Reason |
|---|---|
| It cannot reach a correctness gate | Only recall-shaped constants are tunable, and only within a range. A tuner that can move the thresholds it is scored against can pass its own exam. |
| Rejected changes are reverted unconditionally | Damage that arrives labelled as an improvement is worse than no tuning at all. |
| Only strict improvements are kept | Precision is bought by serving less and recall by serving more, so a change must be at least as good on both. A trade is a preference, and preferences are yours to make. |
| Failures are remembered | The ledger is fed back into the next proposal. A loop that forgets re-proposes forever, and the failures are the more informative half. |
Overriding a prompt by hand#
Tuning writes to .rose/prompts/. You can write there yourself —
a file replaces the shipped prompt of the same name, and anything malformed
falls back to the original rather than failing the call, because recall runs
in a hook and an experiment must never take memory offline.
.rose/prompts/relevance.md # which lessons apply to this work .rose/prompts/warm_relevance.md # same, when the candidates are already in context .rose/prompts/related.md # is this new lesson about the same thing as that one
{placeholder} the original had, or the
call throws and recall reports itself as degraded. rose tune
checks this before applying; if you edit by hand, check it yourself with
rose recall -p "…".
Changing a prompt changes the fingerprint that keys the judgement cache, so old verdicts become unreachable rather than silently authoritative. This matters more than it sounds: before it was fixed, a complete rewrite of the relevance prompt produced a byte-identical evaluation report, because every question had been answered from the cache of the text it replaced.
What it found#
The first round proposed something six manual attempts had not: classify the work as BUILDING (code is about to change) or CONVERSING (the user is thinking aloud and wants an answer) before judging any lesson, on the grounds that the same lesson is load-bearing in one mode and pure noise in the other.
| Before | After | |
|---|---|---|
| Precision | 47% | 51% |
| Recall | 77% | 88% |
Kept, because both improved. The honest caveat: one round, one store, six episodes. What that supports is that the loop can find and validate a non-obvious change — not that it will keep doing so.