6.2.1. Speech as an Agent Modality
💡 First Principle: Voice interaction is a text core wrapped in speech services. Speech-to-text (STT) converts spoken input to text the model reasons over; text-to-speech (TTS) converts the model's text reply to audio. Keeping the middle stage in text — rather than an opaque audio-to-audio model — is what lets you ground, evaluate, and trace the agent's behavior.
STT transcribes audio to text on input and TTS synthesizes audio from text on output, bracketing the agent's text reasoning so it can participate in voice interactions without changing its core. Custom speech models adapt STT to domain vocabulary, accents, or noisy environments where the base model underperforms — the speech analog of fine-tuning for a specialized recognition task. Multimodal reasoning from audio inputs lets a model reason directly over audio content (not just a transcript) where the task needs it. Speech translation converts spoken input in one language to text or speech in another, combining recognition and translation. For agentic design, the advantage of the STT-text-TTS pattern over a single audio-to-audio model is observability: a text middle stage can be grounded against sources, scored by text evaluators, and traced.
⚠️ Exam Trap: Don't feed raw microphone audio straight into a text chat-completion and expect a voice assistant — a text model reasons over text, so the design needs STT on input and TTS on output. And reach for a custom speech model only when the base recognition underperforms on domain vocabulary or conditions, not by default.
Reflection Question: A voice assistant gives answers that are hard to evaluate and often ungrounded. Why does keeping the reasoning stage in text — STT in, TTS out — fix both problems better than swapping to a single audio-to-audio model?
