AssemblyAI details a compliant AI cold-calling agent build
AssemblyAI published a technical guide on Oct. 29, 2026, describing how to build an outbound AI cold-calling agent that dials prospects, qualifies them in a structured conversation and writes outcomes back to a CRM. The guide centers on a fail-closed compliance gate, Twilio for telephony, Universal-3.6 Pro Realtime for speech recognition, an LLM through LLM Gateway for the script and tools, and AssemblyAI Text-to-Speech for replies.
Evidence and context
The article says the call flow should be built around four pieces: a compliance gate that blocks the call before it is placed, a telephony path, an agent that runs the script and a tool layer that records the outcome. It specifies that one AssemblyAI API key can cover Universal-3.6 Pro Realtime, LLM Gateway and Text-to-Speech, while Twilio places the call and carries the audio. The implementation example uses a single Python file, server.py, and a Media Streams WebSocket to bridge audio between Twilio and the server.
AssemblyAI framed the system as suitable for structured qualification rather than open-ended persuasion. The guide argues for a closed set of outcomes, with two or three known questions, explicit branching and one of six dispositions written to the CRM. It also says the agent should disclose that it is AI in the first sentence and, where recording is enabled, disclose that as well.
The compliance section places the gate before the dial and recommends fail-closed behavior for unknown timezones or unavailable do-not-call checks. It lists internal suppression lists, registry scrubbing, local calling windows and consent or disclosure rules as pre-dial checks. The article says U.S. telemarketing rules, including the TCPA, can impose statutory damages per violating call, and that some states and other countries add their own rules.
On the telephony side, the guide uses Twilio's Calls.create() API with answering machine detection, status callbacks and a call time limit. If Twilio detects a machine, the example hangs up and logs a no_answer disposition. If a human answers, Twilio opens a Media Stream and passes the lead ID to the server.
The bridge code described in the article uses AssemblyAI's speech-to-text streaming with encoding=pcm_mulaw and sample_rate=8000, matching Twilio's 8 kHz μ-law audio, while Text-to-Speech returns audio in the same format after server-side resampling from its native 24 kHz output. AssemblyAI also says the TTS session can be kept alive across the call and that turn detection should generally stay at default settings unless measured evidence suggests otherwise.
The script is built as a system prompt with explicit branches for discovery, objections and call closure. The article says the prompt should avoid improvisation, never state prices, customer names, statistics or commitments unless those values came from a tool result, and limit replies to one or two short sentences.
AssemblyAI also described four tools for the call: book_meeting, log_disposition, honor_dnc and mark_callback. The guide says tool arguments should be validated before any CRM write occurs, with the meeting time limited to the slots offered during the call and the email matching what the prospect actually said. It also says opt-outs should be added to the suppression list immediately.
For status reporting, the article says Twilio callbacks should be verified with X-Twilio-Signature and handled idempotently because Twilio retries failed deliveries. Those callbacks provide the call status, duration and answering-machine result, which the guide positions as the source of connection-rate, conversation-rate and book-rate reporting.
AssemblyAI backed the technical pitch with performance claims from its English voice-agent benchmark of 12,460 scripted scenarios. It said Universal-3.6 Pro Realtime recorded a 5.19% word error rate and a 2.4% phone-number error rate, compared with 13.50% and 11.5% respectively for Deepgram Flux EN in the same table. The company also cited a 307 ms median time to final transcript on Pipecat's open benchmark, though those comparisons were presented as vendor claims rather than independently verified measurements.
The guide also included pricing context from AssemblyAI's own rates: Universal-3.6 Pro Realtime streaming at $0.45 per hour and Text-to-Speech at $15 per 1 million input characters. It said a 90-second qualification call would use about $0.011 of streaming speech-to-text, before LLM, TTS and carrier costs are added.
AssemblyAI linked the article to earlier product posts dated Sept. 29, 2026 on Universal-3.6 Pro Realtime and Sept. 30, 2026 on real-time code-switching transcription, along with an Oct. 20, 2026 comparison of voice agent APIs. The new guide extends that product narrative into a full implementation pattern for outbound qualification calls, but the company's performance claims remain vendor-supplied and are not independently verified in the source material.