Analyzing User Behavior in Voice-Activated Systems

I’ve open-sourced a key component of one of my latest projects: Voice Lab, a comprehensive testing framework that removes the guesswork from building and optimizing voice agents across language models, prompts, and personas. Speech is increasingly becoming a prominent modality companies employ to enable user interaction with their products, yet the AI community is still figuring out systematic evaluation for such applications. Key features: (1) Metrics and analysis – define custom metrics like brevity or helpfulness in JSON format and evaluate them using LLM-as-a-Judge. No more manual reviews. (2) Model migration and cost optimization – confidently switch between models (e.g., from GPT-4 to smaller models) while evaluating performance and cost trade-offs. (3) Prompt and performance testing – systematically test multiple prompt variations and simulate diverse user interactions to fine-tune agent responses. (4) Testing different agent personas, from an angry United Airlines representative to a hotel receptionist who tries to jailbreak your agent to book all available rooms. While designed for voice agents, Voice Lab is versatile and can evaluate any LLM-based agent. ⭐️ I invite the community to contribute and would highly appreciate your support by starring the repo to make it more discoverable for others. GitHub repo (commercially permissive) https://lnkd.in/gAaZ-tkA

Anthropic introduced Clio, a new system that reveals patterns in how people actually use #AI assistants worldwide, providing detailed insights into real-world AI adoption while maintaining user privacy. AI assistants are becoming increasingly integrated into our daily lives, but each person leverages them in a different way — making this a fascinating window into how the tech is being used. Understanding the dominant real-world use cases can both help improve user experience and align development with actual user needs. Clio analyzes millions of conversations by summarizing and clustering them while removing identifying information in a secure environment. The system then organizes these clusters into hierarchies, allowing researchers to explore patterns in usage without needing access to sensitive data. - Analysis of 1M Claude conversations showed that coding and business use cases dominate, with web development representing over 10% of interactions. - The system also uncovered unexpected use cases like dream interpretation, soccer match analysis, and tabletop gaming assistance. - Usage patterns vary significantly by language and region, such as a higher prevalence of economic and social issue chats in non-English conversations.

There’s a KPI most voice AI teams still aren’t tracking - and it explains why users hang up the second they realize it’s a bot. Bot recognition drop-off rate. And here’s the uncomfortable part: it has almost nothing to do with how human your AI sounds. Because in reality, you usually have to disclose it’s an AI anyway. Laws are moving that direction, and most enterprises already do it by policy. So the idea that we can “hide the bot” long enough for users not to notice… that ship has sailed. The real question is: once users know it’s AI, why do they stay — or leave? The data from production deployments is pretty clear. People don’t hang up because it’s a bot. They hang up because it’s not useful fast enough. The teams with the lowest drop-off rates aren’t obsessing over voices or accents. They’re obsessing over the first five seconds. Delivering value before the user has time to think, ugh, a bot. Think about how good consumer apps do support. DoorDash doesn’t open with “How can I help you today?” It opens with your last order — because there’s a very good chance that’s why you’re there. Voice AI should work the same way. “Hi Melissa, are you calling about your Chipotle order arriving in 10 minutes?” beats “How can I help you?” every single time. You disclosed it’s AI. You used context. You predicted the reason. You moved the user forward instantly. That’s the shift happening now: from audio engineering to business logic engineering. From trying to sound human… to trying to be immediately helpful. And once you start measuring bot recognition drop-off rate, this becomes impossible to ignore. Context-aware openings outperform generic ones. Prediction accuracy matters. Speed to value matters. Voice naturalness? Diminishing returns. We dig into the data, the patterns, and the implementation playbook in our Voice AI 2026 report: The Year of Systematic Deployment.

I’ve been building and testing voice agents lately and noticed a pattern that keeps repeating. The moment people realize they’re talking to an AI, they try to break it. They say random things. They test edge cases. They push the system to behave outside its purpose. These are testers, beta users, even real customers. Maybe it’s the hacker instinct in all of us. Maybe it’s curiosity about how a nondeterministic system will react. Either way, it’s a real behavior pattern and it shapes how these products should be built. As builders, we can’t ignore this. Voice agents need to set context fast and remind users why they’re talking to the system, especially when the conversation drifts into unproductive territory. This is a new human-computer interaction challenge, and designing for it feels different from anything we’ve done before. How do you think voice AI should handle moments when users intentionally try to break the experience?

A voice- and text-enabled conversational agent was primarily used for health information, casual interactions, and clinical data entry — yet over half of users discontinued after a single session, underscoring barriers to sustained digital engagement. 1️⃣ Among 24,537 users of the Albert Health app, 58% engaged in only one session. 2️⃣ The most frequent intents were health information (32%), small talk (20%), and clinical parameter logging (16%). 3️⃣ Voice input dominated casual (64%) and medication-related (53%) interactions; screen-based input was preferred for clinical tasks (61%). 4️⃣ Participants in disease-specific programs exhibited higher sustained engagement than general health users (OR = 0.67). 5️⃣ A higher proportion of voice-based interactions was positively associated with continued use (OR = 1.005); screen-based interaction predicted attrition (OR = 0.994). 6️⃣ Engagement was more likely to be sustained when users employed a balanced mix of clinical and non-clinical intents (OR = 1.56). 7️⃣ Unexpectedly, higher system confidence scores in chatbot responses were associated with reduced user retention (OR = 0.43). 8️⃣ Users aged 15–45 were less likely to sustain engagement compared to pediatric or older adult cohorts. 9️⃣ Fall-back responses (13% of interactions) were frequently due to non-standard speech, slang, or recognition errors, highlighting limitations in natural language processing. 🔟 A modest engagement peak on day 8 aligned with reminder notifications, but overall retention remained low beyond initial use. ✍🏻 Selahattin Colakoglu, Mustafa Durmus, Zeynep Pelin Polat, Asli Yildiz, Emre Sezgin. User Engagement with A Multimodal Conversational Agent for Self-Care and Chronic Disease Management: A Retrospective Analysis. Journal of Medical Systems. 2025. DOI: 10.1007/s10916-025-02202-2