Evaluating User Experience in Innovation Tools

So many product teams work on new features they believe will be a game-changer for users. But how do you really know if a feature will be adopted by users? This is where UX research comes in. As UX researchers, we can help identify the probability of feature adoption by digging deep into user needs, behaviors, and expectations. Here are some ways we measure and predict feature adoption: 1. User Interviews and Surveys: By speaking directly to users, we can gauge their interest in a new feature. Through surveys or interviews, we explore how they might use the feature, what problems it would solve for them, and how it fits into their current workflows. These qualitative insights give us an early understanding of potential adoption barriers. 2. Usability Testing: A feature may seem like a great idea on paper, but how do users actually interact with it? Conducting usability tests on prototypes allows us to see whether users understand the feature, how intuitive it is, and where they might get stuck. If the feature feels cumbersome, adoption rates will likely be lower. 3. Task Success Rate: This metric allows us to measure how easily users can complete tasks using the new feature. A low success rate indicates friction, and users are less likely to adopt a feature if it doesn’t make their experience easier. 4. User Journey Mapping: By mapping out the user journey, we can see where the new feature fits into the overall user experience. Does it make sense within the flow of their tasks? Are there unnecessary steps or points of confusion? A smooth, integrated feature is more likely to be adopted. 5. A/B Testing: Once a feature is live, we can run A/B tests to see if it’s driving the desired behavior. Does the feature increase engagement or task completion compared to the previous version? These quantitative insights allow us to measure real-world adoption and refine the feature based on user interactions. 6. Feature Feedback: After a feature is released, gathering feedback is key. By monitoring user comments, satisfaction scores, and support tickets, we can understand how users feel about the feature. Are they using it as intended? Are there any pain points that need addressing? As UX researchers, our role is to validate whether a feature truly meets user needs and fits within their daily tasks. We can predict adoption rates, identify potential issues early, and help product teams make informed decisions before launching a feature. How do you measure feature adoption in your research?

AI changes how we measure UX. We’ve been thinking and iterating on how we track user experiences with AI. In our open Glare framework, we use a mix of attitudinal, behavioral, and performance metrics. AI tools open the door to customizing metrics based on how people use each experience. I’d love to hear who else is exploring this. To measure UX in AI tools, it helps to follow the user journey and match the right metrics to each step. Here's a simple way to break it down: 1. Before using the tool Start by understanding what users expect and how confident they feel. This gives you a sense of their goals and trust levels. 2. While prompting  Track how easily users explain what they want. Look at how much effort it takes and whether the first result is useful. 3. While refining the output Measure how smoothly users improve or adjust the results. Count retries, check how well they understand the output, and watch for moments when the tool really surprises or delights them. 4. After seeing the results Check if the result is actually helpful. Time-to-value and satisfaction ratings show whether the tool delivered on its promise. 5. After the session ends See what users do next. Do they leave, return, or keep using it? This helps you understand the lasting value of the experience. We need sharper ways to measure how people use AI. Clicks can’t tell the whole story. But getting this data is not easy. What matters is whether the experience builds trust, sparks creativity, and delivers something users feel good about. These are the signals that show us if the tool is working, not just technically, but emotionally and practically. How are you thinking about this? #productdesign #uxmetrics #productdiscovery #uxresearch

Product managers & designers working with AI face a unique challenge: designing a delightful product experience that cannot fully be predicted. Traditionally, product development followed a linear path. A PM defines the problem, a designer draws the solution, and the software teams code the product. The outcome was largely predictable, and the user experience was consistent. However, with AI, the rules have changed. Non-deterministic ML models introduce uncertainty & chaotic behavior. The same question asked four times produces different outputs. Asking the same question in different ways - even just an extra space in the question - elicits different results. How does one design a product experience in the fog of AI? The answer lies in embracing the unpredictable nature of AI and adapting your design approach. Here are a few strategies to consider: 1. Fast feedback loops : Great machine learning products elicit user feedback passively. Just click on the first result of a Google search and come back to the second one. That’s a great signal for Google to know that the first result is not optimal - without tying a word. 2. Evaluation : before products launch, it’s critical to run the machine learning systems through a battery of tests to understand in the most likely use cases, how the LLM will respond. 3. Over-measurement : It’s unclear what will matter in product experiences today, so measuring as much as possible in the user experience, whether it’s session times, conversation topic analysis, sentiment scores, or other numbers. 4. Couple with deterministic systems : Some startups are using large language models to suggest ideas that are evaluated with deterministic or classic machine learning systems. This design pattern can quash some of the chaotic and non-deterministic nature of LLMs. 5. Smaller models : smaller models that are tuned or optimized for use cases will produce narrower output, controlling the experience. The goal is not to eliminate unpredictability altogether but to design a product that can adapt and learn alongside its users. Just as much as the technology has changed products, our design processes must evolve as well.

Every few years, it feels like the CX industry latches onto a new acronym (CX+BX=TX anyone?), yet most “next big things” are just incremental builds on what's already there. Innovation is lacking, but the notion of UX 3.0 feels different. A recent arXiv paper, “UX 3.0: Experience as Interface,” posits the customer journey is a living system rather than a set of screens, proposing products should read what people are doing, sense how they feel, and reshape themselves in real time. A companion study, “Multi-Layered Human-Centered AI,” explains how to wire three layers together: the model that does the work, an explanation layer that chooses how to talk about it, and a feedback loop that learns from every interaction. Why is this a big deal? Because most of today’s “personalization” is really a flowchart diguised as a personalized experience. Like a chatbot greeting you with the same menu at 11 p.m. that it shows at noon; it's a polite automation that shouldn't be considered personalization. With UX 3.0, the system recognizes intent and emotion, picks the next best step, and adjusts response tone and depth for whoever is on the other side. Picture a service app that senses rising frustration and surfaces a human back channel without being asked. Or a mortgage portal that notices a customer is on a slow mobile connection and removes heavyweight content until the signal improves. That is the sort of moment-to-moment orchestration the new research is pushing toward. The implications for CX teams are practical and, frankly, within reach. First, design reviews can no longer focus only on the screen. They must map the invisible flows: what data feeds the model, how explanations adapt to a new versus a power user, and what signals trigger a course correction. Second, explainability is a product feature. A customer should be able to ask, “Why did you recommend this?” and receive an answer specific to them. So plain language for most of us, but deeper logic for an auditor or a regulator. Third, iteration cycles need to tighten. A product that learns live can't wait for UX research; it needs in-context telemetry and a governance plan that keeps those changes and the teams that deliver them on a tight leash. For large platforms like Qualtrics, PG Forsta, Medallia, UserTesting, or even Genesys, Verint, and NiCE, I think this shift threatens the comfort of dashboards. A true experience-led layer belongs closer to the data plane, with fast feedback and version control. Interestingly, the research community is already open-sourcing prototypes (check out the paper). So UX 3.0 is less about a new coat of paint and more about teaching our products to listen, explain themselves, and grow alongside the people they serve. My friend and colleague, Mike Debnar, and I have been talking about products talking to each other for years. Perhaps we will finally see it come together. Mike, what do you think? #customerexperience #design #ux #ai #future #technology

We do not experience the world in neat, discrete categories, yet much of UX research still measures behavior as if we do. Real experiences exist in the gray zone where satisfaction, trust, confusion, effort, and motivation overlap rather than fall into clean categories. When we compress this psychological complexity into Likert scales or binary outcomes, we lose the intensity and uncertainty that often signal early friction and churn. Most classic UX metrics summarize what users select, not what they actually feel. A single satisfaction score can hide hesitation, mixed emotions, and declining confidence, even though these blended states drive real behavioral change. By forcing fluid cognition into rigid buckets, we frame experience as static when in reality it is continuously evolving. Fuzzy logic approaches UX measurement differently by modeling experience as degrees of membership instead of fixed categories. Using membership functions, telemetry and survey inputs become graded psychological states in which multiple conditions coexist at once. Cognitive load, trust, frustration, and engagement are not treated as on–off switches but as overlapping mental states, allowing UX researchers to detect subtle tensions long before they appear as abandonment or negative feedback. Traditional regression assumes linear relationships and independence between variables, while ANOVA struggles to integrate many experiential dimensions into a single coherent signal. Fuzzy inference systems naturally combine correlated inputs into holistic experience indices, and through defuzzification these blended psychological states become continuous, actionable metrics such as friction levels or churn risk scores that support proportionate design responses instead of blunt thresholds. You might think Likert scales already work like fuzzy logic because they use graded numbers, but they are fundamentally different. Likert forces users to choose a single category, compressing mixed emotions into one number. When we later average scores or run regressions, we treat those values as if they represent continuous psychological intensity, even though the underlying uncertainty has already been removed at the moment of response. Fuzzy logic does the opposite. It preserves uncertainty instead of eliminating it, allowing users to belong partially to multiple psychological states at the same time. A person can be modeled as 70% satisfied, 20% neutral, and 10% confused simultaneously, rather than being forced into selecting whichever single box feels closest. Fuzzy logic does not replace traditional statistics, but it fills the gap where human psychology is layered, nonlinear, and ambiguous. Likert tells us which box users pick, classical statistics compare group averages, but fuzzy logic models how experience actually unfolds inside the mind, enabling UX research to move from static description toward psychologically grounded prediction and adaptive design.

Imagine you are working on a grocery planning app. Users enter the ingredients they already have, and the app suggests recipes. You want to improve the experience, but the problem is not just one feature. You have a few hypotheses in mind. If the app is easier to use, personalization might feel stronger. If personalization feels right, users may be more satisfied. And if users are satisfied, they are more likely to keep using the app. The challenge is that all of these things are connected. Testing them one by one does not really reflect how people experience products. This is where SEM becomes useful. SEM is a way to test an entire system of relationships at the same time. Instead of asking whether ease of use predicts satisfaction in isolation, you can test how ease of use affects personalization, how personalization affects satisfaction, and how satisfaction influences continued use all within a single model. Another reason SEM fits UX so well is that many of the things we care about are not directly observable. Trust, satisfaction, perceived usefulness, and enjoyment are psychological constructs. You do not measure them with a single number. You infer them from patterns in survey responses. SEM is designed for this. It separates what you observe, like ratings or task success, from what you are actually trying to understand, like satisfaction or trust. This matters because UX is rarely about one variable. People’s behavior emerges from a network of influences. SEM lets you model that network. You can see which effects are direct, which ones are indirect, and where the strongest leverage points are. You can also account for measurement error, which most simpler analyses quietly ignore. If you want to compare different user groups, such as new users versus experienced users, SEM can handle that too. Back to the grocery app example. You might collect data on perceived ease of use, perceived personalization, overall satisfaction, and intention to keep using the app. SEM allows you to test whether ease of use improves personalization, whether personalization increases satisfaction, and whether satisfaction actually drives retention. The output is not just a set of coefficients. It is a structured explanation of how your product experience works.

AI in healthcare isn’t just about intelligence—it’s about trust, usability, and collaboration. The best AI tools don’t feel like black boxes, they feel like partners. Transparency builds trust. Users shouldn’t have to guess why AI made a decision. The best tools show their work step by step, let users ask, “Why did AI do that?” and use visuals to explain decisions. Advancing AI is important, but so is improving how humans and AI work together. The best experiences help users guide AI, not just receive its output. That means providing multiple ways to interact and designing AI that helps refine inputs before execution. AI should work with you, not just for you. Collaboration beats automation. The best AI tools feel interactive, not one-and-done. They offer different collaboration modes and let users refine and iterate on results. Users should see and edit AI’s impact before it’s final. Trust grows when people stay in control. That means previewing changes before committing, offering undo options when needed, and creating a try-before-you-buy experience, often without needing an account. AI should fit into workflows, not disrupt them. Good AI feels seamless. The best designs let users quickly accept or reject AI suggestions, make transitions between AI and manual work effortless, and keep the user’s context in focus without unnecessary interruptions. AI alone isn’t the differentiator anymore. Great user experience is.

A proposed qualitative evaluation framework for Generative AI writing tools: This post is my first draft of an evaluation framework for assessing generative AI tools (e.g. Claude, ChatGPT, Gemini). It's something I’ve been working on with Ryan Low — originally in the interest of selecting the best option for Rotational. At some point we realized sharing these ideas might help us and others out there trying to pick the best AI solution for your company's writing needs. We want to be clear that this is not another LLM benchmarking tool. It's not about picking the solution that can count the r's in strawberry or repeatably do long division. This is more about the everyday human experience of using AI tools for our jobs, doing the kinds of things we do all day solving our customers' problems 🙂. We're trying to zoom in on things that directly impact our productivity, efficiency, and creativity. Do these resonate with anyone else out there? Has anyone else tried to do something like this? What other things would you add? Proposed Qualitative Evaluation Criteria 1 - Trust and Accuracy Do I trust it? How often does it say things that I know to be incorrect?  Do I feel safe? Do I understand how my data is being used when I interact with it? 2 - Autonomous Capabilities How much work will it do on my behalf? What kinds of research and summarization tasks will it do for me? Will it research candidates for me and draft targeted emails? Will it read documents from our corporate document drive and use the content to help us develop proposals? Will it review a technical paper, provided a URL? 3 - Context Management and Continuity How well does the tool maintain our conversation context? Not to sound silly, but does the tool remember me? Is it caching stuff? Is there a way for me to upload information about myself into the user interface so that I don’t have to continually reintroduce myself? Does it offer a way to group our conversations by project or my train of thought? Does it remember our past conversations? How far back? Can I get it to understand time from my perspective? 4 - User Experience Does the user interface feel intuitive? 5 - Images How does it do with images? Is it good at creating the kind of images that I need? Can the images it generates be used as-is or do they require modification? 6 - Integrations Does it integrate with our other tools (e.g. for project management, for video conferences, for storing documents, for sales, etc)? 7 - Trajectory Is it getting better? Does the tool seem to be improving based on community feedback? Am I getting better at using it?

Innovation means existing technology necessarily changes over time, which results in design changes. Some of the population can easily handle technological change - most of the time, us geeks can readily adapt. But for a large portion of the population, design changes can be inconvenient, break existing workflows and mental models, and otherwise disrupt people's lives. When technology inevitably changes, we can help build user trust by helping them through the process: 1) Set expectations/WIIFM: tell users well in advance what will be changing, why it is happening, and how they may benefit 2) Provide a clear overview of changes: a well-designed information visualization that shows design changes can help draw users into the documentation; avoid large blocks of text that convey changes 3) Provide a repeatable walk-through: in the new design, show users primary interactions that have moved; allow them to repeat the walk-through 4) Allow a delay: software often updates at critical times for the user (giving a presentation, teaching a class, etc.); notify users of the upcoming change so they can set a convenient time for it to happen 5) Provide a preview: allow users to toggle between the existing and updated interface so they can get comfortable with the update over time

Most UX failures start the same way: Teams skip mapping, rely on assumptions, and build without clear user insight. Effective products, in contrast, are based on structured understanding. Here are the key methods used: 1. 𝗘𝗺𝗽𝗮𝘁𝗵𝘆 𝗠𝗮𝗽 • Captures what users say, think, feel, and do • Builds a shared understanding of user behavior 2. 𝗖𝘂𝘀𝘁𝗼𝗺𝗲𝗿 𝗝𝗼𝘂𝗿𝗻𝗲𝘆 𝗠𝗮𝗽 • Maps how users interact across stages • Highlights friction points and drop-offs 3. 𝗘𝘅𝗽𝗲𝗿𝗶𝗲𝗻𝗰𝗲 𝗠𝗮𝗽 • Provides a broad view of the full experience • Identifies key moments and interactions 4. 𝗦𝗲𝗿𝘃𝗶𝗰𝗲 𝗕𝗹𝘂𝗲𝗽𝗿𝗶𝗻𝘁 • Connects user experience with internal processes • Aligns front-stage and back-stage activities 5. 𝗔𝗳𝗳𝗶𝗻𝗶𝘁𝘆 𝗠𝗮𝗽 • Organizes research into patterns • Reveals core insights and issues 6. 𝗔𝘀𝘀𝘂𝗺𝗽𝘁𝗶𝗼𝗻 𝗠𝗮𝗽 • Separates assumptions from validated facts • Helps reduce risk early 7. 𝗘𝗰𝗼𝘀𝘆𝘀𝘁𝗲𝗺 𝗠𝗮𝗽 • Shows the wider network of people, systems, and services • Adds context to user interactions 8. 𝗦𝗰𝗲𝗻𝗮𝗿𝗶𝗼 𝗠𝗮𝗽 • Illustrates how users achieve specific goals • Frames real-life usage situations 9. 𝗖𝗼𝗴𝗻𝗶𝘁𝗶𝘃𝗲 𝗠𝗮𝗽 • Reflects how users mentally structure information • Supports better information design 10. 𝗦𝗶𝘁𝗲 𝗠𝗮𝗽 • Defines content structure and hierarchy • Improves navigation clarity 11.𝗙𝗹𝗼𝘄 𝗠𝗮𝗽 • Outlines user paths through a product • Clarifies steps and decision points 12. 𝗥𝗼𝗮𝗱𝗺𝗮𝗽 • Aligns goals, priorities, and timelines • Guides product development UX mapping reduces guesswork and improves decisions. Which method do you use most? #UXDesign #UserExperience #UXResearch #ProductDesign #UXStrategy #DesignThinking #CustomerExperience #imenmlika