UX Researcher at PUX Lab | Human-AI Interaction Researcher at UALR
Benchmarking is one of the most direct ways to answer a question every UX team faces at some point: is the design meeting expectations or just looking good by chance? A benchmark might be an industry standard like a System Usability Scale score of 68 or higher, an internal performance target such as a 90 percent task completion rate, or the performance of a previous product version that you are trying to improve upon. The way you compare your data to that benchmark depends on the type of metric you have and the size of your sample. Getting that match right matters because the wrong method can give you either false confidence or unwarranted doubt. If your metric is binary such as pass or fail, yes or no, completed or not completed, and your sample size is small, you should be using an exact binomial test. This calculates the exact probability of seeing your result if the true rate was exactly equal to your benchmark, without relying on large-sample assumptions. For example, if seven out of eight users succeed at a task and your benchmark is 70 percent, the exact binomial test will tell you if that observed 87.5 percent is statistically above your target. When you have binary data with a large sample, you can switch to a z-test for proportions. This uses the normal distribution to compare your observed proportion to the benchmark, and it works well when you expect at least five successes and five failures. In practice, you might have 820 completions out of 1000 attempts and want to know if that 82 percent is higher than an 80 percent target. For continuous measures such as task times, SUS scores, or satisfaction ratings, the right approach is a one-sample t-test. This compares your sample mean to the benchmark mean while taking into account the variation in your data. For example, you might have a SUS score of 75 and want to see if it is significantly higher than the benchmark of 68. Some continuous measures, like task times, come with their own challenge. Time data are often right-skewed: most people finish quickly but a few take much longer, pulling the average up. If you run a t-test on the raw times, these extreme values can distort your conclusion. One fix is to log-transform the times, run the t-test on the transformed data, and then exponentiate the mean to get the geometric mean. This gives a more realistic “typical” time. Another fix is to use the median instead of the mean and compare it to the benchmark using a confidence interval for the median, which is robust to extreme outliers. There are also cases where you start with continuous data but really want to compare proportions. For example, you might collect ratings on a 5-point scale but your reporting goal is to know whether at least 75 percent of users agreed or strongly agreed with a statement. In this case, you set a cut-off score, recode the ratings into agree versus not agree, and then use an exact binomial or z-test for proportions.