AI-Driven Voice Interaction Models

Voice is the next frontier for AI Agents, but most builders struggle to navigate this rapidly evolving ecosystem. After seeing the challenges firsthand, I've created a comprehensive guide to building voice agents in 2024. Three key developments are accelerating this revolution: -> Speech-native models - OpenAI's 60% price cut on their Realtime API last week and Google's Gemini 2.0 Realtime release mark a shift from clunky cascading architectures to fluid, natural interactions -> Reduced complexity - small teams are now building specialized voice agents reaching substantial ARR - from restaurant order-taking to sales qualification -> Mature infrastructure - new developer platforms handle the hard parts (latency, error handling, conversation management), letting builders focus on unique experiences For the first time, we have god-like AI systems that truly converse like humans. For builders, this moment is huge. Unlike web or mobile development, voice AI is still being defined—offering fertile ground for those who understand both the technical stack and real-world use cases. With voice agents that can be interrupted and can handle emotional context, we’re leaving behind the era of rule-based, rigid experiences and ushering in a future where AI feels truly conversational. This toolkit breaks down: -> Foundation layers (speech-to-text, text-to-speech) -> Voice AI middleware (speech-to-speech models, agent frameworks) -> End-to-end platforms -> Evaluation tools and best practices Plus, a detailed framework for choosing between full-stack platforms vs. custom builds based on your latency, cost, and control requirements. Post with the full list of packages and tools as well as my framework for choosing your voice agent architecture https://lnkd.in/g9ebbfX3 Also available as a NotebookLM-powered podcast episode. Go build. P.S. I plan to publish concrete guides so follow here and subscribe to my newsletter.

VoiceTextBlender introduces a novel approach to augmenting LLMs with speech capabilities through single-stage joint speech-text supervised fine-tuning. The researchers from Carnegie Mellon and NVIDIA have developed a more efficient way to create models that can handle both speech and text without compromising performance in either modality. The team's 3B parameter model demonstrates superior performance compared to previous 7B and 13B SpeechLMs across various speech benchmarks whilst preserving the original text-only capabilities—addressing the critical challenge of catastrophic forgetting that has plagued earlier attempts. Their technical approach employs LoRA adaptation of the LLM backbone, combining text-only SFT data with three distinct types of speech-related data: multilingual ASR/AST, speech-based question answering, and an innovative mixed-modal interleaving dataset created by applying TTS to randomly selected sentences from text SFT data. What's particularly impressive is the model's emergent ability to handle multi-turn, mixed-modal conversations despite being trained only on single-turn speech interactions. The system can process user input in pure speech, pure text, or any combination, showing impressive generalisation to unseen prompts and tasks. The researchers have committed to publicly releasing their data generation scripts, training code, and pre-trained model weights, which should significantly advance research in this rapidly evolving field of speech language models. Paper: https://lnkd.in/dutRcaAA Authors: Yifan Peng, Krishna C. Puvvada, Zhehuai Chen, Piotr Zelasko, He Huang, Kunal Dhawan, Ke Hu, Shinji Watanabe, Jagadeesh Balam, Boris Ginsburg #SpeechLM #MultimodalAI #SpeechAI

The fastest LLM is NOT the best LLM for Voice AI. I tested 23 models to prove it. The winner? Not who you'd expect. At Pype AI, we were losing 15% of patient calls to latency. I thought it was our infrastructure. Our prompts. Our network. Nope. It was the LLM. So I benchmarked everything: 23 models, 6 providers, 2500+ API calls over 36 hours. Here's what surprised me: Groq was the fastest. 173ms average response time. But during peak hours? It spiked to 2,458ms. That's a 14x swing. One call feels instant. The next? 2.4 seconds of silence. The winner? Google Gemini 2.5 Flash Lite. Not the fastest (489ms average). But rock solid consistent. 418ms to 537ms. Every single call. Peak or off-peak. Quick breakdown: - Groq: 384ms avg, 318% variation (⚡ fast, 🎲 unreliable) - Google: 518ms avg, 38% variation (✅ winner) - OpenAI: 739ms avg, 62% variation (⚠️ decent) - Anthropic: 1,757ms avg (❌ too slow) - Azure India: 1,209ms avg (❌ cross-region lag) *Note: Azure was India → US. Testing US deployment next.* Here's why consistency matters MORE than speed: Once you have a consistent LLM, you can layer in UX magic - filler words, natural pauses, "hmm" sounds. These tricks make voice AI feel human. But if your LLM randomly freezes? No amount of "umm" saves you. You can't polish unpredictable latency. Consistency first. UX polish second. So if you're building Voice AI use: ✅ In Production: Google Gemini 2.5 Flash Lite ⚠️ Backup: OpenAI GPT-4.1-mini 🧪 Dev: Groq (low traffic only) ❌ Skip: Anthropic (not for real-time) I'm running this test continuously for the next 7 days, then for 30 days straight. Full analysis + raw data will be public via Whispey (open source). But if you want early access comment "DATA" below and I'll send it over. #VoiceAI #LLM #BuildInPublic #AIEngineering

Cartesia Sonic-3 is the first AI voice model I’ve seen that nails Hindi perfectly. For years, even the best text-to-speech (TTS) models struggled with Hindi. The rhythm, tonality, and emotional micro-expressions just didn’t sound human and the accent was inaccurate. This model doesn’t just translate Hindi. It is specially trained for it, with precise control over pacing, expressions and  tonality, all rendered in real time. Under the hood, Sonic-3 is engineered for low-latency voice generation optimized for conversational AI agents, clocking in 3–5x faster than OpenAI’s TTS while maintaining superior transcript fidelity. What makes it stand out technically: → 𝗚𝗿𝗮𝗻𝘂𝗹𝗮𝗿 𝗰𝗼𝗻𝘁𝗿𝗼𝗹 𝘁𝗮𝗴𝘀 let developers dynamically modulate speed, volume, and emotion inside the transcript itself. ("Can you repeat that slower?" now works in production.) → 𝟰𝟮-𝗹𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗺𝘂𝗹𝘁𝗶𝗹𝗶𝗻𝗴𝘂𝗮𝗹 𝗺𝗼𝗱𝗲𝗹 built on a single unified speaker embedding, so one voice can switch between languages like Hindi, Tamil, and English natively while maintaining accent continuity. → 𝟯-𝘀𝗲𝗰𝗼𝗻𝗱 𝘃𝗼𝗶𝗰𝗲 𝗰𝗹𝗼𝗻𝗶𝗻𝗴 powered by a low-sample adaptive cloning pipeline that enables instant personalization at scale. → 𝗥𝗲𝗮𝗹-𝘁𝗶𝗺𝗲 𝗶𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝘀𝘁𝗮𝗰𝗸 achieving sub-300 ms end-to-end latency at p90, tuned for live interactions like support agents, NPCs, and healthcare assistants. → 𝗙𝗶𝗻𝗲-𝗴𝗿𝗮𝗶𝗻𝗲𝗱 𝘁𝗿𝗮𝗻𝘀𝗰𝗿𝗶𝗽𝘁 𝗮𝗹𝗶𝗴𝗻𝗺𝗲𝗻𝘁 that handles heteronyms, acronyms, and structured text (emails, IDs, phone numbers) which usually break realism in production systems. 🎧 Here is example of me trying Sonic-3’s Hindi. You have to hear it to believe it. If you’re building voice agents, conversational AI, or multimodal assistants, keep an eye on Cartesia. They’ve raised $100M to build the most human-sounding voice models in the world, and Sonic-3 just set a new benchmark for multilingual voice AI. #CartesiaPartner

Most voice AI is just a chatbot with a microphone. One company was purpose-built for real phone calls from day one. The architecture lesson most AI builders learn too late: Everyone builds text agents first. Winners build voice first. Text agents get retries. Formatting. Autocorrect. Voice agents get one shot. Real-time. No edits. Then production hits ↓ → Latency: Model takes 3 seconds. Customer hangs up. → Context: "Uh, yeah, so I need to, wait — can you also check my..." → Interruptions: Humans talk over each other. Chat agents break. → Compliance: Every voice interaction is regulated differently. Two traps I see teams fall into: Trap 1: Bolt STT onto a chat agent. Add TTS on output. Call it "voice AI." That's a wrapper. Wrappers break in production. Trap 2: Build your own with Pipecat, LiveKit, Vapi. 6 months later you're managing STT providers, TTS rate limits, LLM deprecations, infrastructure scaling, compliance audits. You wanted a voice assistant. Now you're a voice infrastructure company. PolyAI solved this differently. Full stack built for voice since 2017: → Proprietary ASR + LLM trained on real customer service transactions → 45+ languages. 24/7. Unlimited scale. → Handles surges instantly — storms, outages, promos — zero staffing panic Not just handling calls — generating revenue: → Turning bookings into room upgrades → Enrolling callers into rewards mid-conversation → QA Agents scoring every call automatically → Analyst Agents surfacing patterns no human team catches One healthcare company found fewer complaints from the AI than human reps — on the hardest, most emotional calls. Marriott. FedEx. Caesars. PG&E. 25+ countries. 391% ROI. $10.3M average savings. Payback under 6 months. The companies still running "press 1 for sales, press 2 for support"? Not behind on technology. Behind on architecture. That gap compounds every quarter. My stress test for any voice AI: → Noisy environment → Regional accent → Language switch mid-sentence → Multi-step transaction → Worth 15 minutes if you're evaluating: https://poly.ai/gordon Build or buy — what's your current approach to voice?

Voice AI can now listen and speak at the same time. NVIDIA released a model that handles continuous audio input and output, allowing conversations to flow the way they do between people, with interruptions, back-channel cues, and natural pacing instead of rigid turns. Until now, most voice systems followed a simple pattern. Listen. Pause. Respond. It works, but it always feels artificial. The moment a conversation gets messy or human, the illusion breaks and you are reminded that you are talking to software. That barrier is starting to disappear. When voice feels natural, it unlocks use cases where scripted exchanges fall apart, from sales calls and customer support to internal workflows, gaming, and interactive NPCs that can react in real time instead of repeating prewritten lines. Anywhere timing, nuance, and responsiveness matter, this changes the experience. At the same time, as voice becomes indistinguishable from real conversation, the bar for reliability, guardrails, and verification quietly rises. These systems need to hold up under real users, real pressure, and real incentives. We’re getting closer to voice AI that feels like conversation!

3 open voice models in one week 🔥 Momentum is building, the stack is opening up, and agentic AI is quickly moving to production. Here’s what stood out 👇 - Mistral AI launches Voxtral TTS: Open-weight 4B TTS model. 9 languages, 90ms TTFA, 6x RTF. Mistral claims it beats ElevenLabs on quality benchmarks via Ivan Mehta for TechCrunch - Cohere releases Transcribe: Open-source 2B ASR model built for edge. 14 languages, 5.42 avg WER on HF Open ASR leaderboard, beating Zoom Scribe v1, IBM Granite 4.0, ElevenLabs Scribe v2, and Qwen3-ASR. - Google ships Gemini 3.1 Flash Live + Search Live goes global: Real-time voice/video model with native function calling. 90.8% on ComplexFuncBench Audio (~20% jump over prev gen). Now powers Search Live in 200+ countries with voice and camera input via Aisha Malik for TechCrunch - smallest.ai launches Lightning V3: 3.89 MOS in conversational evals, claims to beat OpenAI, Cartesia, and ElevenLabs. 15 languages with auto-detection and mid-sentence switching. Voice cloning from 5-15s of audio. - Amazon Polly adds Bidirectional Streaming: Stream text to Polly token-by-token as your LLM generates it, get audio back in real time over HTTP/2. 39% faster than batch approach, collapses 27 API calls to 1 on a 970-word passage. - Amazon Web Services (AWS) adds WebRTC to Bedrock AgentCore: Pipecat voice agents now run on AgentCore Runtime with bidirectional WebSocket and WebRTC. Ready-to-deploy examples with Pipecat, Nova Sonic, LiveKit, and Strands SDK. - Genesys reports record Q4: Genesys Cloud at ~$2.6B ARR, 35%+ YoY growth. 70%+ of customers now on AI. AI-powered conversations up 120% YoY. - Artificial Analysis updates voice benchmarks: ElevenLabs Scribe v2 leads at 2.3% WER. Best value: Mistral Voxtral Small at 3.0% WER / $4 per 1K min. TTS Arena: Inworld TTS-1.5-Max at #1, ELO 1,160. - AI chatbots handle 60%+ of banking support: Bank of America's Erica: 1.5B+ interactions, 98% resolved without human. Klarna: 66% of inquiries, saving $40M/yr. Gartner projects $80B in contact center labor cost cuts in 2026. - The economics of AI vs human agents: Voice AI now costs ~$0.40/call vs $7-12 for a human agent: 90-95% cost reduction per interaction. Analysis of how this is reshaping contact center staffing via Medium - Agentic Voice AI goes mainstream: 1 in 10 customer service interactions projected to be fully automated by agentic voice AI in 2026. 80% of businesses plan to deploy. RingCentral shipped AIR Pro, an agentic voice platform embedded in its comms stack. - Salesforce Agentforce Contact Center: Native CCaaS unifying voice, digital channels, CRM, and AI agents in one stack. - Otter.ai hits 35M users, $100M ARR - Sam Liang interview: $100M ARR with <200 employees ($500K+ rev/employee). Liang says 2026 is “the year of the voice.” The stack is getting faster, cheaper, and more open at the same time. The shift to voice-first systems is already underway.