mathematics//optimization//argmax

The argmax is the operator that returns the element of a set at which a function takes its largest value, not the value itself, and it is the standard way to say *keep the best one* in code, in optimization and in machine learning. Over the scores 0.1, 0.7 and 0.2 the maximum is 0.7, while the argmax is the position of that score (index 1, counting from 0) or the candidate it belongs to.

The argmax is the operator that returns the element of a set at which a function takes its largest value, not the value itself, and it is the standard way to say keep the best one in code, in optimization and in machine learning. Over the scores 0.1, 0.7 and 0.2 the maximum is 0.7, while the argmax is the position of that score (index 1, counting from 0) or the candidate it belongs to.

It is a hard selection: one winner takes everything and every other element counts zero, however close it came. That is what makes it useful as a decision (pick the class, pick the move, pick the token) and useless as a training signal, because a small change in the scores either changes nothing or flips the winner, so its gradient is zero almost everywhere.

Argmax decides, softmax weighs.

The softmax is its soft counterpart: instead of all the weight on one element it gives each a share that grows with its score, and it approaches the argmax (written as a one-hot vector) as the temperature goes to zero.

In a classifier, the prediction is the argmax of the output probabilities; the probabilities themselves are kept only when the confidence matters.

In text generation, greedy decoding is the argmax of the next-token distribution at every step, which is why a temperature of 0 makes the output deterministic.

In attention the lookup is deliberately soft: an argmax over the keys would return a single value and could not be trained by gradient, so the softmax mixes all of them.

Ties need a rule. Most libraries return the first index that reaches the maximum, while the mathematical definition returns the whole set of maximizers.