ML//transformer//encoder-decoder//T5

T5 (Text-to-Text Transfer Transformer) is an encoder-decoder language model released by Google in 2019 that treats every NLP task as a text-to-text problem, so that one model serves translation, summarization and classification alike. An input starting with `translate English to French:` produces French text, one starting with `summarize:` a summary, one starting with `classify:` a label.

T5 (Text-to-Text Transfer Transformer) is an encoder-decoder language model released by Google in 2019 that treats every NLP task as a text-to-text problem, so that one model serves translation, summarization and classification alike. An input starting with translate English to French: produces French text, one starting with summarize: a summary, one starting with classify: a label.

A unified text-to-text format means one architecture, one loss function and one training procedure for all tasks, with no task-specific heads or output layers. The model learns task selection from the prefix.

It is the canonical encoder-decoder model: the encoder processes the full input with bidirectional attention (like BERT), the decoder generates the output autoregressively (like GPT). This combination is more natural for tasks with distinct input and output.

It was trained on C4 (Colossal Clean Crawled Corpus), a cleaned version of Common Crawl. The T5 paper was also a massive empirical study comparing pre-training objectives, architectures, and transfer strategies, establishing many best practices.

Against decoder-only models (GPT-style), encoder-decoder is better when the input is a fixed, known text and the output is a transformation of it (translation, summarization). Decoder-only is better for open-ended generation where input and output blur together (conversation, creative writing).

It spawned the Flan-T5 family (SFT on 1800+ tasks), which showed that instruction tuning on diverse tasks dramatically improves zero-shot and few-shot performance. This was an early signal that SFT could unlock capabilities that pre-training alone couldn't.