Evaluating User Experience in SaaS Usability Testing

How do you figure out what truly matters to users when you’ve got a long list of features, benefits, or design options - but only a limited sample size and even less time? A lot of UX researchers use Best-Worst Scaling (or MaxDiff) to tackle this. It’s a great method: simple for participants, easy to analyze, and far better than traditional rating scales. But when the research question goes beyond basic prioritization - like understanding user segments, handling optional features, factoring in pricing, or capturing uncertainty - MaxDiff starts to show its limits. That’s when more advanced methods come in, and they’re often more accessible than people think. For example, Anchored MaxDiff adds a must-have vs. nice-to-have dimension that turns relative rankings into more actionable insights. Adaptive Choice-Based Conjoint goes further by learning what matters most to each respondent and adapting the questions accordingly - ideal when you're juggling 10+ attributes. Menu-Based Conjoint works especially well for products with flexible options or bundles, like SaaS platforms or modular hardware, helping you see what users are likely to select together. If you suspect different mental models among your users, Latent Class Models can uncover hidden segments by clustering users based on their underlying choice patterns. TURF analysis is a lifesaver when you need to pick a few features that will have the widest reach across your audience, often used in roadmap planning. And if you're trying to account for how confident or honest people are in their responses, Bayesian Truth Serum adds a layer of statistical correction that can help de-bias sensitive data. Want to tie preferences to price? Gabor-Granger techniques and price-anchored conjoint models give you insight into willingness-to-pay without running a full pricing study. These methods all work well with small-to-medium sample sizes, especially when paired with Hierarchical Bayes or latent class estimation, making them a perfect fit for fast-paced UX environments where stakes are high and clarity matters.

Recently, someone shared results from a UX test they were proud of. A new onboarding flow had reduced task time, based on a very small handful of users per variant. The result wasn’t statistically significant, but they were already drafting rollout plans and asked what I thought of their “victory.” I wasn’t sure whether to critique the method or send flowers for the funeral of statistical rigor. Here’s the issue. With such a small sample, the numbers are swimming in noise. A couple of fast users, one slow device, someone who clicked through by accident... any of these can distort the outcome. Sampling variability means each group tells a slightly different story. That’s normal. But basing decisions on a single, underpowered test skips an important step: asking whether the effect is strong enough to trust. This is where statistical significance comes in. It helps you judge whether a difference is likely to reflect something real or whether it could have happened by chance. But even before that, there’s a more basic question to ask: does the difference matter? This is the role of Minimum Detectable Effect, or MDE. MDE is the smallest change you would consider meaningful, something worth acting on. It draws the line between what is interesting and what is useful. If a design change reduces task time by half a second but has no impact on satisfaction or behavior, then it does not meet that bar. If it noticeably improves user experience or moves key metrics, it might. Defining your MDE before running the test ensures that your study is built to detect changes that actually matter. MDE also helps you plan your sample size. Small effects require more data. If you skip this step, you risk running a study that cannot answer the question you care about, no matter how clean the execution looks. If you are running UX tests, begin with clarity. Define what kind of difference would justify action. Set your MDE. Plan your sample size accordingly. When the test is done, report the effect size, the uncertainty, and whether the result is both statistically and practically meaningful. And if it is not, accept that. Call it a maybe, not a win. Then refine your approach and try again with sharper focus.

When we run usability tests, we often focus on the qualitative stuff — what people say, where they struggle, why they behave a certain way. But we forget there’s a quantitative side to usability testing too. Each task in your test can be measured for: 1. Effectiveness — can people complete the task? → Success rate: What % of users completed the task? (80% is solid. 100% might mean your task was too easy.) → Error rate: How often do users make mistakes — and how severe are they? 2. Efficiency — how quickly do they complete the task? → Time on task: Average time spent per task. → Relative efficiency: How much of that time is spent by people who succeed at the task? 3. Satisfaction — how do they feel about it? → Post-task satisfaction: A quick rating (1–5) after each task. → Overall system usability: SUS scores or other validated scales after the full session. These metrics help you go beyond opinions and actually track improvements over time. They're especially helpful for benchmarking, stakeholder alignment, and testing design changes. We want our products to feel good, but they also need to perform well. And if you need some help, i've got a nice template for this! (see the comments) Do you use these kinds of metrics in your usability testing? UXR Study

"I don't know where to look first." "So I don't look at all." One user said this about a SaaS dashboard. And it explained everything. The team couldn't understand why engagement was tanking. "The dashboard has everything users need." That was the problem. I watched 40 user sessions. Same pattern. Every. Single. Time. Open dashboard. Eyes dart corner to corner. Scroll up. Scroll down. Close tab. 14 metrics on one screen. Zero decisions made. More information. Less clarity. No action. So we flipped the approach. Instead of showing everything: → 1 primary metric (the decision they came to make) → 3 supporting metrics (context only) → Everything else collapsed below Same data. Radically different focus. 3 weeks later: Session time up 23%. Decision speed up 18%. But the best metric? A user wrote in: "Finally. I can actually think." Here's what most dashboards get wrong: Users don't come for information. They come for clarity. Information is everywhere. Clarity is rare. And the moment you show everything... You help with nothing. Count the metrics above the fold on your dashboard right now. If it's more than 4, you're not informing users. You're paralyzing them.

Most interviews stay surface-level. You get polite answers. You get user stories. But you don’t get the kind of thinking that changes roadmaps. Here’s what most researchers miss: Every question triggers a type of thinking. If your question is basic, the answer will be too. Bloom’s Taxonomy breaks cognitive effort into 6 levels and the best UXRs shape their questions around them. Here’s how to spot which one you’re aiming for (and what to ask instead): 𝗪𝗮𝗻𝘁 𝗿𝗲𝗰𝗮𝗹𝗹? 𝗔𝘀𝗸 𝗳𝗼𝗿 𝗸𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲. Great for timelines and habits. Not insight. ↳ “When did you last use it?” ↳ “Who was involved?” ↳ “Where were you?” Use when you’re mapping the what, not the why. 𝗪𝗮𝗻𝘁 𝗰𝗹𝗮𝗿𝗶𝘁𝘆? 𝗔𝘀𝗸 𝗳𝗼𝗿 𝗰𝗼𝗺𝗽𝗿𝗲𝗵𝗲𝗻𝘀𝗶𝗼𝗻. Perfect for mental models, first-use tests, confusion points. ↳ “Walk me through how you would explain this to a teammate” ↳ “Explain what you think this does.” Use when you want to catch mismatched expectations. 𝗪𝗮𝗻𝘁 𝘁𝗼 𝘁𝗲𝘀𝘁 𝘂𝘀𝗮𝗯𝗶𝗹𝗶𝘁𝘆? 𝗔𝘀𝗸 𝗳𝗼𝗿 𝗮𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻. This is real-world behavior, not theory. ↳ “Show me how you’d complete this.” ↳ “Tell me about what you did the last time X happened.” Use when you need to see the friction. 𝗪𝗮𝗻𝘁 𝗶𝗻𝘀𝗶𝗴𝗵𝘁 𝗶𝗻𝘁𝗼 𝗱𝗲𝗰𝗶𝘀𝗶𝗼𝗻-𝗺𝗮𝗸𝗶𝗻𝗴? 𝗔𝘀𝗸 𝗳𝗼𝗿 𝗮𝗻𝗮𝗹𝘆𝘀𝗶𝘀. This is where users break things down. ↳ “Describe what matters most when deciding.” ↳ “Explain where you get stuck.” Use when you’re mapping criteria and tradeoffs. 𝗪𝗮𝗻𝘁 𝘁𝗼 𝘀𝗽𝗮𝗿𝗸 𝗰𝗿𝗲𝗮𝘁𝗶𝘃𝗶𝘁𝘆? 𝗔𝘀𝗸 𝗳𝗼𝗿 𝘀𝘆𝗻𝘁𝗵𝗲𝘀𝗶𝘀. (my fave!!!) Co-creation starts here. ↳ "Tell me about a time when a tool or product really impressed you. Describe what made it stand out.” ↳ “If you’ve ever hacked together your own version of this, walk me through what you did.” Use when you’re in the generative zone. 𝗪𝗮𝗻𝘁 𝘁𝗼 𝘂𝗻𝗰𝗼𝘃𝗲𝗿 𝘃𝗮𝗹𝘂𝗲𝘀? 𝗔𝘀𝗸 𝗳𝗼𝗿 𝗲𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻. This is judgment, reasoning, preference. ↳ “Talk me through what you would recommend.” ↳ “Explain why you chose that one?” Use when you’re surfacing what matters most. If your research isn’t getting deep enough, it’s not the user, it’s the question. Want to rewrite your interview guide using Bloom’s? I’ve made a cheat sheet with prompts for every domain: https://lnkd.in/eSuNdv_V

💡 System Usability Scale (SUS): A Simple Yet Powerful Tool for Measuring Usability The System Usability Scale (SUS) is a quick, efficient, and cost-effective method for evaluating product usability from the user's perspective. Developed by John Brooke in 1986, SUS has been extensively tested for nearly 30 years and remains a trusted industry standard for assessing user experience (UX) across various systems.  1️⃣ Collecting user feedback Collect responses from users who have interacted with your product to gain meaningful insights using the SUS questionnaire, which consists of 10 alternating positive and negative statements, each rated on a 5-point Likert scale from "Strongly Disagree" (1) to "Strongly Agree" (5). 📌 Important: The SUS questionnaire can be customised, but whether it should be is a debated topic. The IxDF - Interaction Design Foundation suggests customisation to better fit specific contexts, while NNGroup recommends using the standard version, as research supports its validity, reliability, and sensitivity. 2️⃣ Calculation To calculate the SUS score for each respondent:   • For positive (odd-numbered) statements, subtract 1 from the user’s response.   • For negative (even-numbered) statements, subtract the response from 5.   • Sum all scores and multiply by 2.5 to convert them to a 0-100 scale.  3️⃣ Interpreting the Results • Scores above 85 indicate an excellent usability, • Scores above 70 - good usability, • Scores below 68 may suggest potential usability issues that need to be addressed. 🔎 Pros & Cons of Using SUS ✳️ Advantages:  • Valid & Reliable – it provides consistent results across studies, even with small samples, and is valid because it accurately measures perceived usability.  • Quick & Easy – requires no complex setup, takes only 1-2 minutes to complete. • Correlates with Other Metrics – works alongside NPS and other UX measures.    • Widely respected and used - a trusted usability metric since 1986, backed by research, industry benchmarks, and extensive real-world application across various domains. ❌ Disadvantages: • SUS was not intended to diagnose usability problems – it provides only a single overall score, which may not give enough insight into specific aspects of the interface or user interaction. • Subjective User Perception – it measures how users subjectively feel about a system's ease of use and overall experience, rather than objective performance metrics.  • Interpretation Challenges – If users haven’t interacted with the product for long enough, their perception may be inaccurate or limited. • Cultural and language biases can affect SUS results, as users from different backgrounds may interpret questions differently or have varying levels of familiarity with the system, influencing their responses. 💬 What are your thoughts? Check references in the comments! 👇  #UX #metrics #uxdesign #productdesign #SUS

Usability testing is the ultimate insurance policy for your product—protecting you from wasting time, money, and effort on features that users don’t need, won’t use, or might outright hate. Being a UX Designer, Product Owner, or SME might make you knowledgeable about a product, but it does NOT automatically make you a good User Researcher. 1. You’re Too Close to the Product (a.k.a. Designer’s Curse) 2. You Think You Know What Users Want (But You Don’t)  3. You’re Looking for Validation, Not Truth  4. You Can’t Turn Off “Problem-Solving Mode 5. You Don’t Have the Right Research Methods I’ve watched the slow decline of proper user testing, and the consequences are costly—bad or misleading data will steer your software product down the wrong (and very expensive) path. First let's start with: Usability / User Testing: What it is and What it isn't. What IS usability testing? ✅ Observing real users in action without interfering, so you can learn what’s working (and what’s painful). ✅ Asking open-ended “why” questions to uncover real usability issues, not just surface opinions. ✅ Testing with one user at a time to avoid bias, influence, and groupthink disasters. ✅ Encouraging users to think out loud so you understand their decision-making process. ✅ Creating realistic tasks that match how users would naturally interact with your product. ✅ Letting the data speak for itself—not defending design choices in real-time. What ISN’T usability testing? ❌ Running a focus group where one outspoken person shapes the entire narrative. (This happens often and is NOT User Testing and leads to Groupthink - the quietest person may have the best ideas but won't share it). ❌ Asking leading questions like “Wouldn’t this be easier if we just added more buttons?” ❌ Turning the test into a tutorial—if you have to explain how to use it, it’s already broken. ❌ Looking for validation instead of insights—“Do you like this?” is not a usability test. ❌ Gathering opinions instead of behaviors—what people say they’ll do ≠ what they actually do. ❌ Ignoring uncomfortable feedback because “they just don’t get it”—users are always right (even when it hurts). Tune in later for Part 2...