ML//transformer//encoder

An encoder is the BERT-style architecture, in which self-attention is fully bidirectional, with no masking. Its counterpart is the decoder, which sees only the past and generates; the encoder sees everything and understands, the fundamental split of transformer architectures.

An encoder is the BERT-style architecture, in which self-attention is fully bidirectional, with no masking. Its counterpart is the decoder, which sees only the past and generates; the encoder sees everything and understands, the fundamental split of transformer architectures.

Every token attends to every other token freely, so early words get fully disambiguated by later context.

Its output is one vector per token, and the [CLS] token aggregates the meaning of the whole sentence for classification.

Without masking every position is enriched better, but the encoder cannot generate text, because seeing the future defeats autoregressive generation.

It is used for understanding tasks: classification, NER, similarity. Downstream layers sit on top to convert vectors into task outputs.