Evaluation
Scoring retrieval and compression against recorded outcomes.
ROSE's claim is falsifiable, so rose eval tries to falsify it:
replay held-out episodes against every level of a lesson and see whether
the shorter forms still transfer.
- A control arm. Every case also runs with no lesson at all. Without it you measure the model's prior, not the lesson.
- A ceiling arm. The original L0 sets what full detail achieves, so a low score at L2 can be told apart from a lesson that never helped.
- Held-out episodes. Compression is accepted by replaying episodes; scoring it on those same episodes measures memorisation. The split is deterministic and the compressor never sees the test half.
- Blind grading. The grader sees the task, the known-good outcome and a candidate — never the lesson, and never which arm produced it.
- Repeats. A single sample per arm is a coin toss.
The headline result is not a number but a curve: transfer against tokens, across levels. The thesis holds if transfer stays flat while tokens fall, and is refuted if transfer degrades with depth. Nothing in the eval path writes to the store — an eval that mutates what it measures is measuring itself.
ROSE-Bench is a hand-written set for procedural memory under compression, organised around the four ways it can fail: transfer, retention, retrieval and cost. Every case names the trap — the specific wrong thing a good model does by default — because a case whose control passes should be deleted, not celebrated.
Scoring retrieval#
Compression is checked by replay against the episodes a lesson came from, episodes, a dropped detail earns its way back by rescuing a failure. Recall — the stage that decides what enters your context on every prompt — was checked against nothing, so every number anyone quoted about it was an assertion.
rose eval-recall # score retrieval as it stands rose eval-recall --save baseline # keep this run to compare against rose eval-recall --against baseline # A/B a change
How it scores#
Each recorded episode is replayed against exactly the lessons it was served, and the judge's picks are compared with which lessons a later judgement said had actually borne on the work.
Precision and recall are always printed together, and never one alone:
| Number | Meaning | How to cheat it |
|---|---|---|
| Precision | Of the lessons recall would serve, how many bore on the work. | Serve nothing. |
| Recall | Of the lessons that would have helped, how many are still delivered. | Serve everything. |
| Noise tokens | Context spent on lessons that were never used. | — |
Either number on its own makes any change look like progress, which is not hypothetical: four separate attempts to improve the relevance prompt each reduced noise and each did it by dropping lessons that mattered.
What it is not#
episode.used is a judgement, made once, about a session that has
since ended. It is a real observation of an outcome rather than an assertion,
and it is the same signal attribution already runs on — but it is not ground
truth, and a store with six episodes supports a direction, not a decimal
place.
It also cannot see coverage. The score is computed over candidates the system
chose to consider, so it is blind to lessons never considered at all — which
is exactly how a truncation bug hid fourteen unreachable lessons behind
healthy-looking precision. Check reachability separately with
rose tree and rose recall.
Scoring compression#
rose eval answers the other question: does a lesson still transfer
what it knew after being shortened? It holds out episodes, replays them
against each level of a lesson, and reports the pass rate per level — so
"compressed to 56% and still solves everything it used to" is measured rather
than claimed.
rose eval --holdout 0.3 --samples 3
recall.strategy exists so the descent policy can be compared the
same way rather than argued about. See
Descent for what the strategies do.