mathematics//information theory//channel capacity
The channel capacity is the fastest rate at which information can cross a noisy channel and still arrive intact, measured in bits per use of the channel. It belongs to the channel, not to the message: the noise, the bandwidth and the power allowed at the input fix it, and no choice of code can move it. What a code can do is get close to it.
The channel capacity is the fastest rate at which information can cross a noisy channel and still arrive intact, measured in bits per use of the channel. It belongs to the channel, not to the message: the noise, the bandwidth and the power allowed at the input fix it, and no choice of code can move it. What a code can do is get close to it.
The result that gives the number its meaning has two halves. Below capacity there are codes whose errors vanish as the blocks get longer, so noise costs delay and cleverness but not correctness. Above capacity no code works: the errors stay, whatever the scheme. The surprise is that this is a threshold and not a slope. Noise does not trade a little rate for a little reliability all the way down; it draws a line, and on the near side of the line reliable communication is free.
For the textbook channel, a band-limited signal of power SSS in Gaussian noise of power NNN over a bandwidth BBB, the capacity has a closed form, the Shannon-Hartley theorem.
C=Blog2 (1+SN)C = B \log_2\!\left(1 + \frac{S}{N}\right)C=Blog2(1+NS)
The shape of the formula is the lesson. Capacity grows linearly with bandwidth and only logarithmically with the signal-to-noise ratio: doubling the bandwidth doubles the rate, while doubling the power adds about one bit per second per hertz when the signal is already strong, and much less when it is weak. A wider channel beats a louder transmitter, and the last decibels of power are the most expensive way to buy throughput.
What the code has to overcome is the entropy the noise adds to every symbol. Capacity is the mutual information between input and output, taken over the best input distribution, so rate above capacity is not harder to decode; it is information that never arrived.
Model capacity is an analogy, not an instance. A model too small for a task keeps making errors however it is prompted, the way a channel driven above capacity keeps making errors however it is coded, and scaling laws read like empirical measurements of that ceiling. Extended thinking then looks like widening the channel: more compute per problem, more bits to reduce the uncertainty of the answer, up to a ceiling fixed by pre-training. The analogy is useful for intuition and has no theorem behind it.