ML//multimodal//TTS

TTS (text-to-speech, also speech synthesis) is the generation by machine of audio of a voice reading a written text, and it is used wherever a program has to read text aloud, language-learning audio among them. Pronouncing the letters is the easy part. The hard part is what a human reader supplies without noticing: how to say a number or an abbreviation, which syllable carries the stress, where to pause and how the pitch moves across a sentence (the **prosody**). A system with the right sounds and the wrong prosody sounds exactly like a machine.

TTS (text-to-speech, also speech synthesis) is the generation by machine of audio of a voice reading a written text, and it is used wherever a program has to read text aloud, language-learning audio among them. Pronouncing the letters is the easy part. The hard part is what a human reader supplies without noticing: how to say a number or an abbreviation, which syllable carries the stress, where to pause and how the pitch moves across a sentence (the prosody). A system with the right sounds and the wrong prosody sounds exactly like a machine.

Older systems took two routes. Concatenative synthesis cuts a long recording of one speaker into small units and stitches the right ones together, natural inside a unit and audible at the joins. Statistical parametric synthesis predicts a smooth trajectory of acoustic features and hands it to a vocoder, which removes the joins and sounds muffled. Neural TTS keeps the two stages with learned parts: an acoustic model turns text into an intermediate representation (usually a mel spectrogram) and a neural vocoder turns that into the waveform, predicting prosody along the way.

SSML (Speech Synthesis Markup Language, an XML format standardised by the W3C) is how a caller controls the reading. Tags choose the voice and language, change rate, pitch and volume (prosody), insert pauses (break) and fix pronunciations. Each provider implements a subset and adds extensions of its own: Azure's Speech is one hosted implementation, and its SSML dialect, voice catalogue and per-character bill are that provider's choices.

A voice is a trained speaker tied to a locale, the language plus its regional variant (zh-CN for Mandarin as spoken in mainland China). The locale sets the pronunciation rules; the voice within it sets the timbre.

Language-learning audio is where that control earns its keep. A learner wants a slower rate than a native speaker would use, one voice across the whole deck, and a short silence before the first syllable, because some players clip the start of a file. All three are SSML settings.

Cloud TTS is billed per character, so a batch job should never pay twice for the same string. Each file is named after a hash of the text plus every setting that changes the sound (voice, rate, format): an unchanged string finds its file and is skipped, and a new voice changes every hash, so the whole set regenerates with no stale files mixed in.

Near-human quality arrived around 2024 (NaturalSpeech from Microsoft, now at v3, ElevenLabs, Cartesia), and the frontier then merged with omnimodels: GPT-4o and Gemini 2.0 speak natively. Full-duplex systems (Moshi from Kyutai, the OpenAI Realtime API) listen and speak at the same time. Whisper runs the opposite direction, audio to text, and a voice assistant chains the two around a language model.