AssemblyAI details voice agent build with STT, LLM Gateway and TTS
AssemblyAI has published a hands-on guide for building a voice agent on its own stack, combining streaming speech recognition, LLM access and text-to-speech on one API key. The company says the approach is meant to replace the older pattern of wiring together separate vendors for listening, reasoning and speaking.
What the guide covers
In the Oct. 29, 2026 guide, AssemblyAI says a voice agent can be assembled from Universal-3.6 Pro Realtime for streaming speech-to-text, LLM Gateway for tool-calling language model requests, and AssemblyAI Text-to-Speech for the spoken response. The example uses standard JSON over WebSocket or HTTP, and the company says the loop can be written directly or handled through orchestrators such as LiveKit or Pipecat.
The guide also lays out the protocol details it says developers need to manage, including authentication with a raw API key in the Authorization header, 16 kHz 16-bit mono PCM input for speech recognition, and 24 kHz output from the text-to-speech service. AssemblyAI says Universal-3.6 Pro Realtime sends Turn messages with built-in end-of-turn detection, while the TTS side supports Cancel for interruption handling.
Accuracy claims and pricing
AssemblyAI cites its English voice-agent benchmark, which it says includes 12,460 scripted scenarios. In those results, Universal-3.6 Pro Realtime is listed at a 5.19% word error rate and a 14.4% entity error rate, compared with 7.78% and 18.5% for ElevenLabs Scribe v2, 8.64% and 26.1% for Deepgram Nova-3, and 13.50% and 30.1% for Deepgram Flux EN. The company says the entity-error metric matters most when agents must capture names, phone numbers or email addresses correctly.
For usage, AssemblyAI says the services are billed separately under one key, with streaming speech-to-text priced at $0.45 per hour and text-to-speech at $15 per 1 million characters, while LLM Gateway is billed per token based on the selected model. In background material supplied with the guide, AssemblyAI also says its Voice Agent API is generally available and that the broader all-in voice-agent pricing is $4.50 per hour, covering speech recognition, reasoning and speech synthesis.
Why the update matters
The guide positions AssemblyAI as infrastructure for developers building conversational agents, rather than as a single-purpose transcription vendor. It emphasizes turn detection, interruption handling and tool calling as part of the same stack, and says the setup can be used both with direct WebSocket code and with existing agent frameworks. The practical significance is that the company is trying to collapse several integration layers into one workflow, while keeping the underlying protocols standard.
AssemblyAI also says the stack is multilingual. Universal-3.6 Pro Realtime supports 32 languages, while AssemblyAI Text-to-Speech supports English, Spanish, French, German, Italian and Portuguese. The company says the TTS model can speak all supported voices across those languages, with language selected per session rather than inferred from the text.
Timing for the next product change is not stated in the material provided.