Speculative decoding

Speculative decoding speeds generation by drafting tokens with a cheaper model and verifying them with the target model in parallel.

A small draft model proposes several future tokens; the large model checks them in one forward pass and accepts a prefix that matches what it would have sampled. Rejected tokens fall back to the large model's own sample. When draft quality is high, tokens per second rise without changing the target distribution in the ideal case. It needs extra memory for the drafter and helps most on long output...