Multilingual Voice Interface Solutions

Cartesia Sonic-3 is the first AI voice model I’ve seen that nails Hindi perfectly. For years, even the best text-to-speech (TTS) models struggled with Hindi. The rhythm, tonality, and emotional micro-expressions just didn’t sound human and the accent was inaccurate. This model doesn’t just translate Hindi. It is specially trained for it, with precise control over pacing, expressions and  tonality, all rendered in real time. Under the hood, Sonic-3 is engineered for low-latency voice generation optimized for conversational AI agents, clocking in 3–5x faster than OpenAI’s TTS while maintaining superior transcript fidelity. What makes it stand out technically: → 𝗚𝗿𝗮𝗻𝘂𝗹𝗮𝗿 𝗰𝗼𝗻𝘁𝗿𝗼𝗹 𝘁𝗮𝗴𝘀 let developers dynamically modulate speed, volume, and emotion inside the transcript itself. ("Can you repeat that slower?" now works in production.) → 𝟰𝟮-𝗹𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗺𝘂𝗹𝘁𝗶𝗹𝗶𝗻𝗴𝘂𝗮𝗹 𝗺𝗼𝗱𝗲𝗹 built on a single unified speaker embedding, so one voice can switch between languages like Hindi, Tamil, and English natively while maintaining accent continuity. → 𝟯-𝘀𝗲𝗰𝗼𝗻𝗱 𝘃𝗼𝗶𝗰𝗲 𝗰𝗹𝗼𝗻𝗶𝗻𝗴 powered by a low-sample adaptive cloning pipeline that enables instant personalization at scale. → 𝗥𝗲𝗮𝗹-𝘁𝗶𝗺𝗲 𝗶𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝘀𝘁𝗮𝗰𝗸 achieving sub-300 ms end-to-end latency at p90, tuned for live interactions like support agents, NPCs, and healthcare assistants. → 𝗙𝗶𝗻𝗲-𝗴𝗿𝗮𝗶𝗻𝗲𝗱 𝘁𝗿𝗮𝗻𝘀𝗰𝗿𝗶𝗽𝘁 𝗮𝗹𝗶𝗴𝗻𝗺𝗲𝗻𝘁 that handles heteronyms, acronyms, and structured text (emails, IDs, phone numbers) which usually break realism in production systems. 🎧 Here is example of me trying Sonic-3’s Hindi. You have to hear it to believe it. If you’re building voice agents, conversational AI, or multimodal assistants, keep an eye on Cartesia. They’ve raised $100M to build the most human-sounding voice models in the world, and Sonic-3 just set a new benchmark for multilingual voice AI. #CartesiaPartner

India just got its own multilingual AI stack Not a demo. A real platform. Most AI still speaks English first. India does not. We keep talking about AI scale. But ignore language reality. Sarvam AI just shipped something important. An open-source foundational model suite built for 10 Indian languages and designed voice-first. That changes who AI is for. Here’s what stands out to me: India’s first open-source 2B Indic LLM trained on ~4 trillion tokens Voice agents deployable via phone WhatsApp and in-app workflows Speech → text → translation → synthesis in a single Indic stack Legal AI workbench for drafting redaction and regulatory Q&A Pricing that starts around ₹1 per minute for multilingual agents This is not chasing Silicon Valley scale. It’s solving Indian constraints. Smaller efficient models that run where India actually is Voice interfaces for users who skip keyboards Agentic workflows not just chat responses And the quiet but big idea: Sovereign AI infrastructure. Data stays local. Models align with Indian regulation. Control stays domestic. That matters for BFSI, legal, telecom and any sector touching sensitive data. The real unlock is inclusion. AI that works in Hindi, Tamil, Telugu Malayalam, Punjabi, Odia Gujarati, Marathi, Kannada, Bengali AI that listens before it types We keep saying India will be an AI market. This is India building AI rails. Open-source, voice-first, enterprise-ready That combination is rare. If this ecosystem compounds India does not just consume AI It exports it. Watching this space closely. Local language AI is the next growth curve. What sectors do you think adopt first?

Most voice AI is just a chatbot with a microphone. One company was purpose-built for real phone calls from day one. The architecture lesson most AI builders learn too late: Everyone builds text agents first. Winners build voice first. Text agents get retries. Formatting. Autocorrect. Voice agents get one shot. Real-time. No edits. Then production hits ↓ → Latency: Model takes 3 seconds. Customer hangs up. → Context: "Uh, yeah, so I need to, wait — can you also check my..." → Interruptions: Humans talk over each other. Chat agents break. → Compliance: Every voice interaction is regulated differently. Two traps I see teams fall into: Trap 1: Bolt STT onto a chat agent. Add TTS on output. Call it "voice AI." That's a wrapper. Wrappers break in production. Trap 2: Build your own with Pipecat, LiveKit, Vapi. 6 months later you're managing STT providers, TTS rate limits, LLM deprecations, infrastructure scaling, compliance audits. You wanted a voice assistant. Now you're a voice infrastructure company. PolyAI solved this differently. Full stack built for voice since 2017: → Proprietary ASR + LLM trained on real customer service transactions → 45+ languages. 24/7. Unlimited scale. → Handles surges instantly — storms, outages, promos — zero staffing panic Not just handling calls — generating revenue: → Turning bookings into room upgrades → Enrolling callers into rewards mid-conversation → QA Agents scoring every call automatically → Analyst Agents surfacing patterns no human team catches One healthcare company found fewer complaints from the AI than human reps — on the hardest, most emotional calls. Marriott. FedEx. Caesars. PG&E. 25+ countries. 391% ROI. $10.3M average savings. Payback under 6 months. The companies still running "press 1 for sales, press 2 for support"? Not behind on technology. Behind on architecture. That gap compounds every quarter. My stress test for any voice AI: → Noisy environment → Regional accent → Language switch mid-sentence → Multi-step transaction → Worth 15 minutes if you're evaluating: https://poly.ai/gordon Build or buy — what's your current approach to voice?

Built an AI that makes real phone calls in Hindi and English — and yes, it broke my brain (many times). 😅 4 sleepless nights. 847 failed API calls. Dozens of “why is this not working” moments at 3 AM. But it finally works. And it talks like a human. 💡 What I built: → An AI Phone Agent that actually speaks and listens in real time → Handles B2B sales & real estate calls with full context awareness → Speaks fluently in Hindi + English (more Indian languages coming 🇮🇳) → Converts speech → smart response → natural voice in under 2 seconds → Logs every call locally for data analysis & follow-ups ⚙️ Tech Stack: 🔹 Azure OpenAI Services – multilingual conversational intelligence 🔹 Murf AI – hyper-realistic voice synthesis 🔹 Twilio – call handling + telephony backbone 🔹 FastAPI + WebSocket streaming – real-time performance 🔹 Custom language-detection module – instant language switching 🎯 Built for: Murf Coding Challenge 5 – pushing how far conversational AI can go in real Indian business scenarios. 🙌 Grateful for: Murf AI, Microsoft Azure & Microsoft Learn Student Ambassadors community , Twilio and ngrok — their tools & support made this project possible. ⚙️ Version 1 Insights: It works — but there’s room to optimize. Current latency comes from: → Free-tier Twilio network → ngrok tunneling overhead → Azure OpenAI API round-trip delay → Murf voice synthesis lag (already reduced via WebSockets) Even with that, the system performs impressively — and once fine-tuned, it could power AI-driven customer communication across India. 🧠 What I learned: → Natural-sounding AI is hard (especially multilingual) → Real-time latency is the real villain → Debugging voice AI = chaos × caffeine → Indian users love bilingual agents → Simplicity + reliability > fancy complexity 💬 Breakthrough moment: When my AI agent “Shaan” switched mid-call from English to Hindi — and nailed a real-estate inquiry flawlessly. That’s when I knew this could actually change how businesses communicate. ⚡ Imagine this: An AI that can 📞 make 100+ calls/day in Hindi, English & regional languages 💬 handle objections like a human ⚡ never gets tired 💰 costs less than hiring one rep Would that help your business? 👇 Let me know: 1️⃣ What’s your biggest challenge in phone sales? 2️⃣ Would you try (or pay for) this AI? 3️⃣ Which language should I build next? Your feedback means the world — and your likes, comments & shares can help this innovation reach founders, AI enthusiasts & builders across India. 💙 #AI #VoiceAI #Innovation #MachineLearning #Startup #SalesTech #MurfAI #Azure #Twilio #MicrosoftLearnStudentAmbassadors #GenerativeAI #IndianStartups #RealEstateTech #B2BSales #ArtificialIntelligence #Engineering

We at EkStep Foundation brought together 24 Voice AI builders, cloud telephony experts, AI services providers, and model builders last week — 12 in the room at EkStep, 12 joining virtually. . The question: How do we build Voice AI for national helplines that works in more than 10 languages seamlessly? . Not whether or when to do it. How to do it now. . Here's what the room concluded: . 1. You can't rely on technology alone to detect what language a caller speaks Guessing based on the caller's telecom circle is imprecise. Automated detection struggles when someone speaks for less than 1.5 seconds — which happens constantly on phone calls. The simplest solution works best: just ask. A well-designed greeting where the caller picks their language beats any smart detection system. . 2. Start the call in two or three languages at once Greet the caller in the most likely languages simultaneously. Listen to their first response. Lock in their language from there. In Punjab for example — open with Punjabi, then Hindi, then English. Once you get the opening right, people rarely switch languages mid-call. . 3. Keep your language detection close to your speech recognition Don't build it as a separate system — it slows everything down. Run it alongside your speech recognition in real time. The orchestration layer — the brain of the system — should own the language decisions across the entire call. . 4. Avoid translating in real time if you can Adding a translation step on every single exchange slows the call and makes it feel unnatural. The better path: train your AI models directly in the target language. For rare and tribal languages that AI models don't yet support — this is the hardest and most urgent work ahead. . 5. This problem is solvable — but the real work is just beginning The technical architecture is coming together. The models are improving. What's ahead is testing in the field, fine-tuning for each language, and building the ecosystem — especially for languages that no AI model has ever been trained on. That's exactly what this community exists to do. . If you're working on Voice AI for public services in India — what's the biggest challenge you're running into? #VoiceAI #IndicLanguages #ConversationalAI #AIForGood Shankar Maruwada Nilesh Kumar Moksh Talreja Ankush Sabharwal Suman Gandham Kowshik Chilamkurthy Aditya Chhabra Manmeet Singh vedanta hatti Santosh Pawar Saurabh Yadav Shailendra Pal Singh Nikhilesh Kumar BalaKrishnan A, PMP® Pranava C Hiremath Nikhil Narasimhan Harish Shashidhar Kaimal Susan Mathew Ayush Anand Deep Desai

When a customer switches from English to Spanish mid-conversation, should your AI voice assistant immediately follow suit? One of the greatest challenges in the voice AI industry is to create a truly natural multilingual voice AI experience. It requires a balance of tech capabilities and conversational design. Typically, it’s an adaptive process. The autonomous voice agent’s language changes when the customer starts speaking another language. One of the simplest ways to approach this is to start with a multilingual ASR (speech-to-text system), then use an LLM, and, finally, the multilingual TTS (text-to-speech). This creates a seamless experience where the AI can switch languages while maintaining the same voice/persona. The multilingual ASR makes sure that no matter what language the customer speaks, the autonomous voice agent can understand it and reply. Then the LLM takes over and makes sense of what’s being said. The multilingual TTS answers in the right language, making it clear it's the same person talking as in the previous language. This is what it looks like: The customer starts the conversation in English, but switches to Spanish. The system calls the appropriate ASR and the autonomous voice agent also switches to Spanish with the same accent that was used for English. We’re often asked: “How good is your language detection and multilingual ASR capabilities?” But the overlooked human piece is: “how often do you want the multilingual voice AI to switch languages?” If a customer only utters one word in Spanish and returns to English, it might not make sense for it to switch to Spanish. It comes down to how human you want the conversation to be. In autonomous voice agents, the elephant in the room is latency: you want a reponse time within less than a second. But let's suppose it takes two seconds for your LLM, ASR, and TTS to put everything together. That's when intelligent filler words give the system time to respond. What language should those filler words be in? Spanish, English, or a mix of both? Some customers prefer to have explicit language switches: let the customer speak in any language and reply in the same language unless otherwise asked. Others prefer implicit switches. There’s no right answer. The goal is for real-time systems to answer fast enough that it flows like a real conversation between people. Ideally, you’re aiming for a high quality conversation that’s good for brand reputation and end outcomes. What that looks like will depend on the customer base, the segment you're targeting, the location, industry and the specific use case.

For the last 2 years, I’ve been living in the trenches of AI conversational agents—the messy reality behind the demos. And here’s the thing nobody tells you upfront: Great voice bots don’t fail because the model is “not smart enough.” They fail because the conversation doesn’t feel like a conversation. A 700ms pause. A clumsy handover. A language switch that breaks the flow. A “Sorry, I didn’t get that” at the exact moment a customer is already frustrated. That’s why I’m genuinely excited about BCG’s new strategic partnership with ElevenLabs—to help enterprises move beyond basic chatbots into enterprise-grade, multilingual conversational agents that can reason, act, and engage in natural voice. (And yes—this will be embedded into BCG X’s Deep Customer Engagement AI offering, which is a big deal for teams trying to scale CX impact, not just pilots.) On the tech side, ElevenLabs has consistently stood out for two things that matter in the real world: Latency (speed is empathy in voice) Language ability (especially when you’re not operating in “English-only” mode) In markets like Malaysia, we’ve been working with a voice bot that speaks both Malay and English—often in the same conversation. When that switch is seamless, it’s not a feature. It’s trust. This partnership feels like a step toward what customer experience should become: Less “press 1 for…” and more “tell me what you need—I’ve got it.” If you’re building voice agents in multilingual markets (SEA especially), I’d love to compare notes: What’s been the hardest part—latency, language, or integration into real ops? #ConversationalAI #VoiceAI #CustomerExperience #AgenticAI #BCGX #ElevenLabs #Malaysia #SEA

This was the week voice AI stopped looking like demos. Hyperscalers shipped models. Carriers shipped runtimes. Production tooling, multimodal UX, and nine-figure revenue runs all landed in the same seven days. Telnyx launched LiveKit on Telnyx, a hosted platform that runs LiveKit agents on Telnyx-owned infrastructure. 50 percent lower STT and TTS costs, sub-200ms round-trip time, native AMR-WB, and STIR/SHAKEN. Telephony-first voice AI is becoming its own category. xAI shipped standalone Grok speech-to-text and text-to-speech APIs. $0.10 per hour for batch STT and $4.20 per million characters for TTS, with word-level timestamps and multichannel. Same stack powering Tesla and Starlink support. Microsoft launched MAI-Transcribe-1 and MAI-Voice-1 on Microsoft Foundry. 25 languages, number one on the FLEURS word error rate benchmark, and 2.5x faster batch transcription than the prior Azure Fast offering. Already powering Copilot voice mode. DeepL entered voice with a real-time voice-to-voice suite plus API across 40+ languages, including Zoom and Teams add-ons. ElevenLabs had a massive week. A Razorpay partnership to run Hinglish outbound voice agents for millions of Indian merchants, reported $100M+ net new ARR in Q1 driven by telecom and fintech adoption, and full on-premise and on-device deployment for enterprise. Cloudflare shipped @cloudflare/voice, an experimental real-time voice pipeline for the Agents SDK running directly on Workers. Voice becomes a capability of the agent you already have, not a separate framework. Retell AI rolled out SMS during live calls with no A2P approval required. A small UX unlock with big implications for multimodal agent workflows. Sanas acquired Tomato AI to push real-time speech AI deeper into carriers and VoIP networks. Third acquisition in under two years. Sanas is at $62M ARR and on track for $130M. Speechmatics partnered with thymia to fuse speech-to-text with voice biomarkers, detecting stress and fatigue signals in real time. Voice is moving from interface to insight layer. The common thread: voice is now core business tech. Carriers own the rails, hyperscalers ship the models, observability and multimodal UX are table stakes, and the scale numbers are real.

Why I wake up every day to build Voice AI ? My grandmother couldn’t use a smartphone. No typing. No apps. No touchscreen. But she could speak—beautifully, in three languages. She’s not an exception. She’s the norm. Nearly 4 billion people are locked out of the digital economy because today’s interfaces demand: • Literacy • Typing • Visual navigation • English Voice changes that. Speech is the first interface every human learns. Not text. Not screens. When Voice AI is fast, accurate, and multilingual, it becomes the most inclusive interface we’ve ever built: → A farmer checks crop prices without reading → An elderly patient manages medication without apps → A visually impaired user navigates services independently → A non-English speaker accesses the digital world without translation friction At gnani.ai, 40+ languages isn’t a feature. It’s the baseline. Because access shouldn’t depend on the language you were born into. Voice AI isn’t just a business.It’s infrastructure for inclusion. A bridge to billions who’ve been unheard for too long. Who do you build for? Ganesh Gopalan Bharath Shankar Vasuta Agarwal Arunabh Biyani Ankur Agrawal Sharath Holla Thoshith S Avinash Benki Abhilash Sudhamshu Rajesh Rajyalakshmi Pantina #VoiceAI #Inclusion #AI #Technology #GnaniAI #Purpose

Interesting announcement from Tata Communications yesterday about #VoiceAI, specifically for the banking, finance & insurance sector (BFSI). It's a multilingual speech-to-speech offering that sits alongside its #cloud platforms, network assets and existing #CPaaS business. - #speechtospeech conversations with claimed latency under 500 ms - supports 40+ Indian / other languages, with context maintained across customer sessions. Importantly, it is dialect- and accent-aware, which is something I think localised cloudcos and telcos could find an advantage in - #agenticAI architecture which integrates with enterprise APIs and fintech systems to manage whole customer journeys (e.g. account setup, issue resolution) without forcing users to jump between channels - however, it is also multi-channel capable if users *prefer* to jump - based on Tata’s AI Cloud and global voice/network infrastructure One thing it reminded me about was a recent article I've read, noting that speech-based access to AI tools (and voice assistants more broadly) is already very common in India. Thinking about it further, it made me realise that the value-chain for AI voice in future verticalised (and some horizontal) applications is likely to expand. Depending on the specific functions involved, delivery platforms (eg app vs. calls), pain points such as regulatory awareness or network latency, or integration with other capabilities, there will be several options: - Direct path through a cloud platform such as OpenAI, Google, Microsoft, Amazon etc - Preference for conventional CPaaS, especially if international numbering or multi-channel is necessary - Telco channels where there is a suitably flexible voice/comms platform, plus route to market such as into SMBs There will also be hybrids and partnerships here - for instance where a telco or MSP partners with a cloud platform for better developer access and support, or (as here) where a CPaaS provider works with an AI or cloud infrastructure partner or close affiliate. The chart below is a fresh "epiphany in the shower", "not available from AI", illustration of this, and I'll probably refine it over coming weeks. Expect to see a version of this, if you come to the Nov 6th #UnthinkableLab workshop on Voice & AI. Links to the Tata announcement and workshop in the comments.