Hyperparameter tuning

Hyperparameter tuning searches training or decoding settings that are not learned as weights, such as learning rate or temperature.

Hyperparameters are chosen before or around training: learning rate, batch size, LoRA rank, temperature, top-p. Search methods include grid search, random search, and Bayesian optimization. For LLMs, decoding hyperparameters should be pinned in eval logs because they change task scores. Tune on a validation split, then freeze the setting for test and production.