Information Theory

Founded by Claude Shannon in "A Mathematical Theory of Communication" (1948) — the paper that launched the digital age.

Founded by Claude Shannon in "A Mathematical Theory of Communication" (1948) — the paper that launched the digital age.

Core question: how much information can be transmitted reliably through a noisy channel?

Key quantities: entropy (uncertainty of a source), mutual information (shared information between variables), channel capacity (maximum reliable transmission rate)

Shannon showed that information is quantifiable, independent of meaning — a bit is a bit whether it encodes poetry or noise.

Deep connection to thermodynamics: Boltzmann entropy measures disorder in physical systems, Shannon entropy measures uncertainty in messages. The formulas are structurally identical.

Connection to ML

In ML: the cross-entropy loss function is a direct application — it measures how surprised the model is by the data, in bits. Pretraining is, at its core, entropy minimization.

Shannon's noisy-channel coding theorem: if you transmit below channel capacity, error-free communication is possible. Above it, errors are inevitable. This has a deep analogy to model capacity — a model too small for the task will always make errors, no matter how long you train it.