mathematics//information theory
Information theory is the branch of mathematics that measures uncertainty and information in bits and asks how much of it can be transmitted reliably through a noisy channel, and its measures are used from radio links and telemetry budgets to the training loss of ML models and the choice of which sensor to read next. Claude Shannon founded it in *A Mathematical Theory of Communication* (1948), and the step that made it possible was to count information apart from meaning: a bit is a bit whether it encodes a pump alarm, a poem or noise.
Information theory is the branch of mathematics that measures uncertainty and information in bits and asks how much of it can be transmitted reliably through a noisy channel, and its measures are used from radio links and telemetry budgets to the training loss of ML models and the choice of which sensor to read next. Claude Shannon founded it in A Mathematical Theory of Communication (1948), and the step that made it possible was to count information apart from meaning: a bit is a bit whether it encodes a pump alarm, a poem or noise.
The unit is the surprise of an event, −log2p-\log_2 p−log2p bits for an event of probability ppp: a certain event carries none, a fair coin one bit, a one-in-a-thousand event about ten. Every quantity of the theory is built from that. Entropy is the mean surprise of a variable before it is read; mutual information is how much reading one variable lowers the entropy of another; the KL divergence is the price of believing one distribution while the world follows another; and channel capacity is the most mutual information a link can carry per use.
Information is counted relative to a question.
A random message has maximum entropy and is useless to its reader; what an engineer wants is information about the thing that matters, which is mutual information. Whether that information is worth paying for is a further question, answered by the value of information once a decision and its costs are on the table.
Entropy is the tool for budgets: how many bits a telemetry stream, a log or a compressed signal really needs. A detector that rarely fires carries almost nothing per reading, which is why sending only the changes saves so much bandwidth.
Mutual information is the tool for choosing measurements: which sensor to read, where to point a camera, which test to run next. It counts every kind of dependence, linear or not, and the data processing inequality says no filter or model downstream can add to it.
The KL divergence is the tool for comparing distributions: production data against training data in a drift monitor, or a model's predictions against the data in the cross-entropy loss that trains classifiers. Pretraining a language model is, at its core, the minimization of that surprise, measured in bits per token.
Capacity is the ceiling of a link whatever the code: below it errors can be made to vanish, above it they are inevitable. It is the number a radio telemetry link is checked against before anyone argues about protocols, and it lends an analogy to model capacity (a model too small for a task keeps making errors however long it is trained), which is intuition with no theorem behind it.
Boltzmann's entropy of a physical system and Shannon's entropy of a message have the same mathematical form: one counts microstates, the other distinguishable messages.