LLM Temperature: Equation and Effect

Problem What is temperature in LLMs? Give the equation and explain its effect on outputs.

Be ready to discuss

• The equation: temperature scales the logits before the softmax, p_i = exp(z_i / T) / sum_j exp(z_j / T).
• Low T (< 1) sharpens the distribution toward the highest-probability tokens — more deterministic and more repetitive output; in the limit T -> 0 this degenerates to greedy argmax decoding.
• High T (> 1) flattens the distribution — more diverse and creative, with rising risk of incoherence and factual drift. T = 1 leaves the raw softmax distribution unchanged.
• Why it scales logits rather than probabilities, and why that makes the effect non-linear across the vocabulary.
• How temperature interacts with the other decoding knobs: top-k and top-p/nucleus truncate the candidate set first, so temperature only reshapes what survives truncation.
• Picking a value by task: near-zero for extraction, classification, and tool-argument generation where reproducibility matters; higher for brainstorming and copywriting.