mathematics//information theory//entropy
Entropy is a measure of the average uncertainty of a probability distribution, the number of bits needed on average to identify an outcome, and it is used to size what a signal, a log or a telemetry stream really has to carry. An outcome of probability \(p\) carries a **surprisal** of \(-\log_2 p\) bits (a certain one none, a one-in-a-thousand one almost ten), and entropy is that surprise averaged over the outcomes, the uncertainty of a variable before it is read.
Entropy is a measure of the average uncertainty of a probability distribution, the number of bits needed on average to identify an outcome, and it is used to size what a signal, a log or a telemetry stream really has to carry. An outcome of probability ppp carries a surprisal of −log2p-\log_2 p−log2p bits (a certain one none, a one-in-a-thousand one almost ten), and entropy is that surprise averaged over the outcomes, the uncertainty of a variable before it is read.
H(X)=−∑ip(xi)log2p(xi)H(X) = -\sum_i p(x_i) \log_2 p(x_i)H(X)=−i∑p(xi)log2p(xi)
It is maximum when all outcomes are equally likely and zero when the outcome is certain; a fair coin has one bit. It is the foundational quantity of information theory: mutual information, channel capacity and the coding theorems all build on it.
H, bits2.15 maximum, log2 62.58 effective outcomes, 2^H4.5 Outcomes A 45 %, B 20 %, C 15 %, D 10 %, E 6 %, F 4 %. The entropy is the average surprise, each outcome's −log2 p weighted by how often it happens: the most when every outcome is equally likely, zero when one is certain.
Drag the bars toward equal heights and then pile everything on one, and move the temperature to see what sampling does to the same distribution.
A reading that is almost always the same says almost nothing.
A detector that alarms on 0.1 % of its readings carries 0.011 bits per reading, so nearly every sample it sends repeats what the receiver already knew. That is why report-on-change telemetry (deadband) cuts bandwidth so far with little loss.
Among all distributions with a given mean and variance, the Gaussian has the largest entropy: it is the maximum entropy distribution for those two numbers, the one that assumes least beyond them. Knowing only μ\muμ and σ\sigmaσ, assuming a Gaussian is the humblest choice. Its entropy grows as log2σ\log_2\sigmalog2σ, so halving the standard deviation of an estimate is worth exactly one bit.
In ML the output distribution over tokens has an entropy: high when many next tokens are plausible, low when one dominates. Temperature in sampling controls it, flattening the distribution when raised and sharpening it when lowered, and extended thinking can be read as lowering it, each reasoning token narrowing the distribution until the continuation falls into a basin of attraction where correct answers cluster (a distributional shift of the model's own output).
Cross-entropy loss decomposes as H(p,q)=H(p)+DKL(p∥q)H(p, q) = H(p) + D_{\mathrm{KL}}(p | q)H(p,q)=H(p)+DKL(p∥q), so minimizing it means aligning the model's distribution with the true one (KL divergence).
In physics, the Boltzmann entropy S=kBlnWS = k_B \ln WS=kBlnW counts microstates, while the Shannon entropy counts distinguishable messages: the same mathematics in different domains. Entropy measures uncertainty, never meaning or usefulness: a random stream has the most of it and tells nothing about the pump.