Rejection Sampling Best-of-N

Sample n candidate completions from a base policy and pick the highest-scoring one under a reward or judge model — more inference compute, no extra training.

Best-of-n (rejection sampling) draws a fixed n answers from a behavior- cloned or RL policy and keeps the one ranked highest by a scorer (WebGPT: a preference reward model; harnesses may use an LLM judge or task metric). It optimizes against the scorer without further training, trading inference compute for quality — a sibling of self-consistency decoding that ranks with an external score rather than majority vote.

Use when you already have a reliable offline scorer and can afford n full episodes or answers. Watch for reward overoptimization as n grows; keep offline prompt evaluation on held-out labels. This is harness inference design, not the RLHF manufacturing pipeline that trained the reward model.

---

Sources

• WebGPT (Nakano et al.), §3.2