ML//parameters and weights

Parameters are the values a model learns during training. Weights are the parameter tensors that scale and combine signals, although ordinary ML conversation often uses both words almost interchangeably; the biases, one per output unit, are parameters too, and a model's **parameter count** is weights plus biases.

Parameters are the values a model learns during training. Weights are the parameter tensors that scale and combine signals, although ordinary ML conversation often uses both words almost interchangeably; the biases, one per output unit, are parameters too, and a model's parameter count is weights plus biases.

Counting them is simple arithmetic and worth doing before choosing hardware. A fully connected layer from nnn inputs to mmm outputs has nm+mnm+mnm+m: 4,160 for 64 to 64, so a small fault classifier with 40 inputs and two hidden layers of 64 has 6,849 and fits in a few kilobytes at int8 on a microcontroller. A convolutional layer depends only on its kernel and channels, never on the image size (weight sharing). At the other end, a language model of 7 billion parameters needs about 14 GB in 16-bit floats and about 4 GB at 4 bits (quantization), enough for a laptop GPU.

The count drives three costs at once: memory to store the model, labelled data to fit it without overfitting, and compute per prediction. Training costs more than the count suggests, since every activation is kept for backpropagation and an optimizer such as Adam stores two more numbers per parameter.

Parameter count estimates model storage and some compute, but it does not equal total inference memory (model memory footprint). Activations, runtime workspaces and the KV cache may dominate under large batches or long contexts.

Nor does a larger parameter count guarantee a better model; architecture, data and training determine what those numbers encode.

Weights are frozen experience. Context is today's paperwork.