Optimizing for Voice-Enabled Devices

🎙 The hidden speech-to-text bottlenecks most teams miss 🎙 Most teams obsess over Word Error Rate when optimizing STT, but our analysis of top-performing voice agents shows that’s only part of the equation. Here are three counterintuitive insights that drive real performance gains: ⚡ Perceived speed > raw accuracy A lower time-to-first-token (TTFT) makes voice AI feel more responsive—even if total processing time stays the same. Shaving 100-200ms off TTFT can dramatically improve user experience. 🎯 The fine-tuning paradox Domain-specific tuning can 3-5x accuracy for specialized vocabulary (legal, medical, automotive), but it plateaus quickly. Instead of overfitting, focus on Keyword Recall Rates to ensure mission-critical terms are always captured. 🌎 Accent gaps are killing your accuracy Most voice agents show a 30% accuracy gap between native and non-native speakers. Stop training on "Californian accents reading newspapers" and start collecting conversational speech reflecting your actual users. For global applications, consider accent-specific models that treat speech variations as unique linguistic systems. 💡 Pro tip: Simulate real user speech in pre-production evals to catch failures before they hit production - with Coval. What STT levers have you pulled to optimize your voice agents? Share below 👇 In the next few days, I’ll be sharing more on building the ultimate Voice AI stack—follow along for more insights!

Google's voice search algorithm is the blueprint for AI search. And most people missed it. Here's why this matters: When voice search launched, Google had to solve a problem: People don't speak like they type. Typed search: "plumber near me" Voice search: "Hey Google, who's a good plumber in my area that can come today?" Google had to understand: Natural language Context Intent Conversational queries Sound familiar? That's exactly what AI search does now. ChatGPT, Perplexity, Google AI Overviews, Claude... They all process queries the same way voice search does. Here's the pattern: Voice search taught Google: "What's the weather" = user wants current local weather "How do I fix a leaky faucet" = user wants step-by-step instructions "Best pizza place open now" = user wants immediate recommendation with hours AI search uses the same logic: Question-based queries Context awareness Conversational understanding Intent matching This is why voice search optimization = AI search optimization. How to optimize for both: 1. Write like people speak Don't write: "Our plumbing services provide residential and commercial solutions." Write: "We fix leaky faucets, clogged drains, and burst pipes for homes and businesses." Natural language. Conversational tone. 2. Answer questions directly Structure content as Q&A: "How much does it cost to fix a leaky faucet?" "Most repairs cost between $150-300 depending on severity." Both voice search and AI can extract this clearly. 3. Use long-tail, conversational keywords Old keyword: "plumber" Voice/AI keyword: "emergency plumber available today" Old: "lawyer" Voice/AI: "do I need a lawyer for a car accident" 4. Focus on local + immediate intent Voice search users want: "Near me" "Open now" "Available today" AI search users want the same immediate, local answers. Optimize for urgency and location. 5. Create content that answers "why" and "how" Voice queries are often: "Why is my sink draining slowly?" "How do I unclog a toilet?" AI search queries are similar: "Why do I need SEO for my local business?" "How does AI search work?" Answer these thoroughly. The businesses that prepared for voice search in 2018? They're ahead for AI search in 2025. Because they already optimized for: Natural language Question-based content Conversational tone Intent matching Everyone else is scrambling to catch up. The pattern was there. Most people just didn't see it. If you optimized for voice search, you're already optimized for AI search. If you didn't, start now. Same principles. Same strategy. Are you still optimizing for typed keywords or conversational queries?

The Voice Stack is improving rapidly. Systems that interact with users via speaking and listening will drive many new applications. Over the past year, I’ve been working closely with DeepLearning.AI, AI Fund, and several collaborators on voice-based applications, and I will share best practices I’ve learned in this and future posts. Foundation models that are trained to directly input, and often also directly generate, audio have contributed to this growth, but they are only part of the story. OpenAI’s RealTime API makes it easy for developers to write prompts to develop systems that deliver voice-in, voice-out experiences. This is great for building quick-and-dirty prototypes, and it also works well for low-stakes conversations where making an occasional mistake is okay. I encourage you to try it! However, compared to text-based generation, it is still hard to control the output of voice-in voice-out models. In contrast to directly generating audio, when we use an LLM to generate text, we have many tools for building guardrails, and we can double-check the output before showing it to users. We can also use sophisticated agentic reasoning workflows to compute high-quality outputs. Before a customer-service agent shows a user the message, “Sure, I’m happy to issue a refund,” we can make sure that (i) issuing the refund is consistent with our business policy and (ii) we will call the API to issue the refund (and not just promise a refund without issuing it). In contrast, the tools to prevent a voice-in, voice-out model from making such mistakes are much less mature. In my experience, the reasoning capability of voice models also seems inferior to text-based models, and they give less sophisticated answers. (Perhaps this is because voice responses have to be more brief, leaving less room for chain-of-thought reasoning to get to a more thoughtful answer.) When building applications where I need a more control over the output, I use agentic workflows to reason at length about the user’s input. In voice applications, this means I end up using a pipeline that includes speech-to-text (STT) to transcribe the user’s words, then processes the text using one or more LLM calls, and finally returns an audio response to the user via TTS (text-to-speech). This, where the reasoning is done in text, allows for more accurate responses. However, this process introduces latency, and users of voice applications are very sensitive to latency. When DeepLearning.AI worked with RealAvatar (an AI Fund portfolio company led by Jeff Daniel) to build an avatar of me, we found that getting TTS to generate a voice that sounded like me was not very hard, but getting it to respond to questions using words similar to those I would choose was. Even after much tuning, it remains a work in progress. You can play with it at https://lnkd.in/gcZ66yGM [At length limit. Full text, including latency reduction technique: https://lnkd.in/gjzjiVwx ]

I’ve open-sourced a key component of one of my latest projects: Voice Lab, a comprehensive testing framework that removes the guesswork from building and optimizing voice agents across language models, prompts, and personas. Speech is increasingly becoming a prominent modality companies employ to enable user interaction with their products, yet the AI community is still figuring out systematic evaluation for such applications. Key features: (1) Metrics and analysis – define custom metrics like brevity or helpfulness in JSON format and evaluate them using LLM-as-a-Judge. No more manual reviews. (2) Model migration and cost optimization – confidently switch between models (e.g., from GPT-4 to smaller models) while evaluating performance and cost trade-offs. (3) Prompt and performance testing – systematically test multiple prompt variations and simulate diverse user interactions to fine-tune agent responses. (4) Testing different agent personas, from an angry United Airlines representative to a hotel receptionist who tries to jailbreak your agent to book all available rooms. While designed for voice agents, Voice Lab is versatile and can evaluate any LLM-based agent. ⭐️ I invite the community to contribute and would highly appreciate your support by starring the repo to make it more discoverable for others. GitHub repo (commercially permissive) https://lnkd.in/gAaZ-tkA

How LLMs Are Powering the Next-Gen IoT Interfaces LLMs aren’t just about chatbots anymore, they’re being wired deep into devices. I've seen this firsthand on projects - it's a big step forward for user interaction, autonomy, and decision-making in IoT. Here’s how: ➞ Define the Use Case Before the Tech Start with clarity. What’s the device meant to do with the LLM - answer queries, interpret commands, or analyze environments? ➞ Pick the Right Role for Your LLM Is it summarizing sensor logs? Acting as a chat interface? Or making autonomous decisions? Match model purpose to user flow. ➞ Decide: Edge, Cloud, or Hybrid? Choose deployment based on latency, power, memory, and privacy. Edge for speed, cloud for scale, hybrid for balance. ➞ Get the Data Channels Ready Microphones, sensors, system logs - your device needs structured, timely input before LLMs can even think. ➞ Preprocess Input Like a Pro Turn voice into text. Normalize sensor data. Align formats and timestamps. Garbage in still means garbage out. ➞ Inject Local Context Into Prompts Don’t just send raw data. Add metadata like device location, mode, or user identity - for smarter outputs. ➞ Build Flexible Prompt Templates Good prompts aren’t static. They adapt to use cases with variables, constraints, and fallback rules to avoid failure. ➞ Plug Into Your LLM of Choice Use hosted APIs like OpenAI or Claude for cloud jobs. Lightweight models like Llama or Gemma for edge tasks. ➞ Optimize for Real-Time & Offline Latency matters. Use compression, batching, and caching to ensure the LLM runs fast and works even when offline. ➞ Log Every Input & Output Capture interactions for traceability, debugging, and compliance. Logs are the unsung heroes of AI ops. ➞ Let Devices Parse and Act on Output LLMs generate suggestions, not commands. Use rules or classifiers to convert text into safe, structured device actions. ➞ Map Intents to Device APIs Translate parsed LLM outputs into real device actions using standard APIs (MQTT, CoAP, REST, etc.). ➞ Guardrails = Safety Net for AI Enforce guardrails with context windows, rate limits, and override checks to keep autonomy safe and aligned. ➞ Keep Context Fresh & Relevant Short-term memory helps continuity. Long-term patterns help personalization. Combine both for rich, evolving UX. ➞ Test, Tune, and Ship Continuously Validate accuracy, run simulations, gather feedback, and keep iterating. LLM-IoT success isn’t built in one shot. Want to build context-aware, intelligent devices? Start engineering LLMs like systems, not just APIs. ♻️ Repost if this helped you understand LLMs in IoT better ➕ Follow me, Nick Tudor, for deeper dives into real-world AI + IoT architectures

We built insanely powerful AI… and then slowed it down with a keyboard. Most people type at around 40 to 50 words per minute. Most people speak at 130 to 160 words per minute. That alone explains why voice AI is about to matter a lot. Models are now strong enough to handle messy input. They no longer require perfect grammar or carefully crafted prompts. They respond best to speed, intent, and context, and voice gives them all three. We are already seeing voice-first habits emerging: > People dictating ideas, notes, and code instead of typing > Teams capturing thoughts while walking between meetings > Companies caring more about audio quality than typing efficiency For founders building in voice AI, a few things really matter: 1/ Prioritise instant response. If the voice feels slow or waits for long processing, the user will go back to typing. Speed is the whole value proposition. 2/ Build for context tracking, not perfect transcription. The win isn't turning speech into flawless text. It's understanding intent across a conversation, remembering tasks, and continuing without being prompted every time. 3/ Start with private workflows where voice is genuinely faster than typing. Brainstorms, quick notes, task capture, journaling, thinking out loud. Voice shines where ideas flow faster than fingers. 4/ Create interfaces that feel natural, ambient, and always available. Not a feature you "open," but something that listens, helps, and fits into daily life. The best voice tools won’t feel like a tool, they’ll feel like a companion that moves with you. Voice is not just another interface. It is the missing bridge between how fast humans think and how fast AI can respond. The next generation of AI products will be built around that speed.

Most voice AI builders are tweaking TTS settings randomly, hoping something improves. Here's the systematic approach that actually works. After building voice agents for dozens of clients, I've noticed the same pattern: builders get frustrated with voice quality, start adjusting random settings, and often make things worse. The problem isn't the settings, it's the lack of a diagnostic framework. Here's what actually matters: Voice model selection sets your baseline. ElevenLabs Turbo V2 for quality, Flash V2 for speed, V2.5 only for multilingual. Each has different latency and cost implications. Speed isn't just about talking faster. It interacts directly with interruption sensitivity. Change one without considering the other, and your conversation flow breaks. Stability vs expression is a trade-off. High stability = consistent voice but less emotion. Low stability = more expressive but can sound inconsistent. Your use case determines the right balance. The diagnostic framework: Robotic sound → Check voice model, stability, and style exaggeration Interruption issues → Adjust interruption sensitivity AND response delay together Inconsistent voice → Increase stability and similarity boost Awkward pacing → Speed interacts with interruption settings—adjust both Here's what this approach enables: Instead of copying someone else's "perfect settings," you understand the system. When something sounds off in production, you can diagnose and fix it independently. This is the difference between builders who struggle with voice quality for weeks and those who optimize confidently. The framework isn't complicated. It just requires understanding what each setting actually controls and how they work together. What's been your biggest challenge with voice agent optimization?

Spent the morning reviewing voice AI response times across different implementations. Here's what most vendors won't tell you: sub-300ms response time doesn't matter if your system integration adds 2 seconds of latency. The conversation flow is only as fast as your slowest API call. This is why we obsess over the unglamorous stuff. Data pipelines. Webhook optimization. Cache strategies. The infrastructure nobody wants to talk about in sales meetings. But it's the difference between a demo that impresses and a production system that actually works at scale. When you're testing AI tools, don't just test the AI. Test the entire system under load. That's where you'll find out if it actually works. What are your thoughts? . . . #AIinCX #CX

Optimising for latency in voice bots. Latency is one of the biggest factor for a UX of a voice bot. It's unreal to see companies like SynthFlow and Vapi getting latency in 500 - 700 ms order. Having worked on multiple projects building voice bot and optimising the pipelines, here are some techniques to build a low-latency voicebot - Use Websockets and Streaming APIs Instead of HTTP APIs: Websockets and streaming APIs enable continuous data transmission, reducing latency compared to the request-response model of HTTP APIs. - Reduce calls to third party services: Even with web sockets, if you are using third party text-to-speech, llm and speech-to-text. There is added latency of these calls. Switch to local models for one or more of these services. - Optimize the LLM Layer: Use small LLMs or skip LLM and use other sentence models for classification + replying with a pre-saved response - Speed up the Synthesizer Layer: Generating audio on-the-fly for every response can be slow. Pre-recording common responses, especially initial greetings, can reduce response times significantly. - Reduce Payload Size of Audio: Use better encodings, quantize data type and reduce sample rate - Utilize Filler Sentences / Acknowledgments: When receiving audio input, responding with filler sentences (e.g., "Let me check that for you...") can give the impression of lower latency while the system processes the user's request. - Take Decisions on Partial User Transcript: For some questions, decisions can be made on partial transcripts, allowing for quicker responses without waiting for the full input, thus reducing perceived latency. [HARD] What are some other optimisation's related to voice bot you know?

You've done your macro optimizations and still can't get your 𝘝𝘰𝘪𝘤𝘦 𝘈𝘐 to an acceptable latency. (Read until the end for my bonus tip.) We all know latency is critical for smooth Voice AI experiences. ✅ 𝗬𝗼𝘂'𝘃𝗲: ⟡ Found a stream provider w/ good response time   ⟡ All your components are in the same region ⟡ Partnered with top speech providers  So what's next? For pipelined Voice AI, think of your flow like this:  𝑠𝑡𝑟𝑒𝑎𝑚 → 𝑠𝑝𝑒𝑒𝑐ℎ → 𝑖𝑛𝑓𝑒𝑟𝑒𝑛𝑐𝑒 → 𝑠𝑝𝑒𝑒𝑐ℎ 🎙️ 𝗦𝗽𝗲𝗲𝗰𝗵-𝘁𝗼-𝗧𝗲𝘅𝘁 (𝗦𝗧𝗧) Here are a few ways to improve latency at this stage: ⟡ If using Deepgram careful with "smart formatting" ⟡ Stream audio, don't wait for full recordings  ⟡ Use the latest model from your provider   ⟡ Set up endpointing to a low value For me, fine-tuning configuration made the biggest impact here. 🧠 𝗜𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 Lots to optimize, but start here: ↳ Stream the input/output whenever possible   ↳ Use caching to speed up repeated requests   ↳ Use tools like LiteLLM for latency-based routing   ↳ Pick the fastest providers/models (e.g, Groq/Gemini) Summary: Stream everything and use the fastest models available. 🔊 𝗧𝗲𝘅𝘁-𝘁𝗼-𝗦𝗽𝗲𝗲𝗰𝗵 (𝗧𝗧𝗦) TTS can be one of the biggest contributors to latency. To minimize: ⟡ Start streaming audio as soon as it's available   ⟡ Chunk your input at the first natural pause ⟡ Avoid unnecessary transcoding In my experience, chunking the audio at the first natural pause and queuing the rest gave the biggest win. 💡 Bonus Tip If your use-case involves tool calling try routing general inference to a faster/smaller model, and the larger models for actual tool use. There you have it—my best tips to improve latency in Voice AI. ▶ Want to join Fonoster early access program? Check this link: https://dub.sh/rphKJ8O