Judge Reliable Lift Alongside Mean Lift
Report a lower-percentile held-out "reliable lift" and repeatability alongside mean gain when judging harness changes, because search procedures can select brittle edits with large average improvements.
Evidence: weak
A harness change that raises the average score can still be a bad change. If the gain comes from a few lucky runs, or collapses on tasks outside the tuning set, the mean hides it. Automated search over harness edits makes this worse, because it will find whatever edit scores highest, brittle or not.
Two lines of work address this. τ-bench, a 2024 benchmark from Sierra, introduced pass^k, the rate of succeeding on the same task in all k independent trials, and showed how far it falls below single-run success: GPT-4o succeeded on under half of tasks, and pass^8 in the retail domain was below 25%.
A 2026 preprint from Airbnb, "Beyond Prompts", adds metrics for effectiveness, stability, repeatability and efficiency, and a reliable-lift estimator called RelLift95, a lower-percentile held-out gain that the paper reports next to the mean as a conservative companion to it, not a replacement. Its middleware optimiser shows the gap between the two:
| Benchmark | Mean held-out lift | Reliable lift |
|---|---|---|
| Multi-round | 14.2 points | 12.0 points |
| Retail | 14.9 points | 10.1 points |
The abstract warns that some search procedures find large gains yet still select brittle updates.
τ-bench is established. RelLift95 is a single preprint, so the specific estimator is weakly supported even though the principle is well grounded.
The practical rule: judge any change to an agent's prompts, tools or playbooks, or to its harness settings, on repeated runs over held-out tasks, and report both the mean and a lower-percentile result. The mean is the expected benefit; the lower percentile is a robustness constraint. Don't discard the mean: a change with a much larger mean gain and an acceptable lower tail can beat one that is consistently but only modestly better. What the lower percentile rules out is a large mean that rests on a brittle tail. This is the acceptance criterion inside in-loop regression control and in guarded self-modification. Disclose what changed through harness cards, and read the transcripts of the runs that fail to see why a change is brittle.
Sources
- Zhao, Ruan, Chen, Tu, Abbasi, Hesch (Airbnb), arXiv 2609.05736, 4 Sep 2026, §3 and Table 2 (preprint) — https://arxiv.org/html/2609.05736v1
- Yao, Shinn, Razavi, Narasimhan (Sierra), arXiv 2406.12045, 17 Jun 2024 — https://huggingface.co/papers/2406.12045