Evaluating Usability in Educational Technology

Working on instructional invariants today and the idea that evaluability is far more important than feedback. In fact, feedback is not an invariant at all. An instructional invariant is a non-negotiable design condition that must hold for learning to occur reliably. If violated, it causes learning to fail — even if everything else appears to be working. Instructional invariants are constraints on learning environments that prevent predictable failure. Feedback is something the system does. Evaluability is something the learner does. There’s a massive difference between the two. You can drown a learner in feedback and still leave them saying: “I’m trying, but I don’t know what I’m doing wrong.” Evaluability means the learner can tell whether their response is correct or incorrect in a way that allows them to actually do something about it. In other words, they can evaluate what is going on. It needs to be: detectable, localisable, interpretable and actionable. If learners cannot evaluate their performance, learning becomes unreliable — regardless of how much feedback is provided. Put simply: feedback ≠ evaluability. This matters because learning is not the default outcome of exposure. Learners will always choose the cheapest way to reduce error. If they can’t evaluate their performance, they switch strategies or disengage. From a design perspective, this is why “more feedback” often fails. It increases noise without sharpening the error signal. In other words, the problem isn’t quantity. It’s precision. Evaluability is a core instructional invariant. Learning can fail even when learners are active, motivated, and using the right skill. If they don’t know what’s wrong and how to fix it, feedback is useless. So the design question isn’t: “Did we give feedback?” It’s: “Did we make correctness and error visible, interpretable, and actionable?” I think this matters even more with edtech because, again, learning is not the default outcome of exposure. In apps and AI systems, learners will almost always take the cheapest path to reduce error — guessing, pattern-matching, prompt-surfing, etc. — if evaluability is weak. Ultimately, feedback is a method, not a condition. You can have: lots of feedback detailed feedback well-intentioned feedback even “high-quality” feedback …and still leave the learner unable to answer: Was I right? What exactly was wrong? What do I do differently next time? That means learning stalls, even though “feedback” occurred. So feedback fails the invariant test: Learning can fail even when feedback is present. Therefore, feedback cannot be an instructional invariant.

AI products do more than introduce a new interface pattern. They reshape the interaction itself. In traditional systems, people gradually learn the rules, form expectations, and usually become more efficient with repeated use. AI changes that rhythm. A system may feel highly capable while still being inconsistent, opaque, overly persuasive, or confidently wrong in ways users do not catch right away. For that reason, evaluating AI through the same lens we use for ordinary digital products leaves out too much. In many teams, evaluation still centers on familiar questions. Is the system usable? Do people enjoy it? Can they complete the task? Those questions still matter, but they do not capture the full experience. An AI feature can feel polished and still lead users toward overtrust. An assistant can seem fast and impressive while actually increasing effort because people have to verify outputs, manage uncertainty, and fix errors. A product can feel smooth on the surface while still producing unfair outcomes or nudging people toward poor decisions. Human AI evaluation needs a wider and more grounded scope. Usability remains essential because a confusing interface can undermine everything else. But beyond that, teams need to examine whether the system is truly useful, whether it improves judgment, whether people understand how it behaves, and whether trust is appropriately calibrated. The goal is not simply to make users feel confident. The goal is to help them rely on the system when it is appropriate and question it when needed. Mental models, perceived control, and collaboration also deserve much more attention. Many AI systems are framed as assistants, copilots, or partners, which means the relationship between person and system becomes part of the user experience. Researchers need to ask whether the AI strengthens human judgment or gradually displaces it, whether it reduces effort or merely shifts effort into hidden checking and correction work. In many AI products, these dynamics are central to the experience rather than secondary concerns. The more difficult side of evaluation matters just as much. Fairness, safety, accountability, and recovery from failure cannot be treated as edge cases. AI systems will fail at times. What matters is whether users can detect those failures, respond effectively, and recover without losing orientation, performance, or trust. A strong AI experience is not defined by the absence of mistakes. It is defined by how well the system supports people when mistakes happen. That is why AI evaluation should extend well beyond usability and satisfaction. It should also address usefulness, trust calibration, explainability, agency, cognitive burden, fairness, safety, resilience, and emotional fit.

Selecting the right AI tool can be challenging when new products appear almost daily. This guide helps you cut through the noise with a clear, structured process for testing, evaluating, and integrating AI tools in the classroom. It introduces a practical framework built around three pillars: usability, pedagogy, and ethics. Each is broken into a checklist of focused questions to help educators quickly determine whether a tool fits their curriculum, supports deep learning, and meets privacy standards. The guide also includes tips for piloting tools with a small group, gathering student feedback, and reflecting on results. This guide is informed by key resources, including aiEDU’s AI Readiness Framework, ISTE’s Teacher Ready Edtech Product Evaluation Guide, the U.S. Department of Education’s AI Integration Toolkit, and UNESCO’s Recommendation on the Ethics of AI. These references shaped the usability, pedagogy, and ethics checklists to keep the framework practical and research-based. #AIinEducation #EdTech #TeachingWithAI #TeacherTools #AIforTeachers #EdLeaders #ClassroomInnovation #DigitalLearning #AIIntegration #EducationTechnology

Most product teams kill great ideas with the wrong evaluation process. They either: → Ship MVPs that never evolve → Over-engineer before validating → Miss the signal in noisy metrics After managing 50+ product launches across FinTech, EdTech, & Logistics, I've refined a framework that works across every stage: MVP → MMP → MAP. 𝗧𝗵𝗲 𝗣𝗿𝗼𝗱𝘂𝗰𝘁 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 (𝗣𝗘𝗙) Six dimensions.  One decision framework. 1. 𝘝𝘢𝘭𝘶𝘦 𝘍𝘪𝘵 — 𝘋𝘰𝘦𝘴 𝘪𝘵 𝘮𝘢𝘵𝘵𝘦𝘳? Does this solve a real problem worth solving? → Problem-solution fit validated → Clear user pain addressed → Expected business outcome (ROI, revenue, cost savings) → Early adopter feedback collected Score: Low / Medium / High 2. 𝘜𝘴𝘢𝘣𝘪𝘭𝘪𝘵𝘺 𝘍𝘪𝘵 — 𝘊𝘢𝘯 𝘱𝘦𝘰𝘱𝘭𝘦 𝘢𝘤𝘵𝘶𝘢𝘭𝘭𝘺 𝘶𝘴𝘦 𝘪𝘵? Beautiful features mean nothing if users struggle. → UX simplicity tested → Accessibility compliance checked → Key flow completion rates measured → Support tickets analyzed Score: Poor / Acceptable / Excellent 3. 𝘍𝘦𝘢𝘴𝘪𝘣𝘪𝘭𝘪𝘵𝘺 𝘍𝘪𝘵 — 𝘊𝘢𝘯 𝘸𝘦 𝘥𝘦𝘭𝘪𝘷𝘦𝘳 𝘪𝘵 𝘳𝘦𝘭𝘪𝘢𝘣𝘭𝘺? Technical stability determines long-term success. → Architecture scalability reviewed → Performance benchmarks met → Security & compliance validated → Tech debt quantified → Maintainability assessed Score: Red / Yellow / Green 4. 𝘔𝘢𝘳𝘬𝘦𝘵 𝘍𝘪𝘵 — 𝘞𝘪𝘭𝘭 𝘪𝘵 𝘨𝘳𝘰𝘸? Early traction isn't the same as sustainable growth. → Market size & demand validated → Competitive differentiation clear → Adoption trend positive → Expansion potential identified → Retention probability high Score: Low / Moderate / Strong 5. 𝘍𝘪𝘯𝘢𝘯𝘤𝘪𝘢𝘭 𝘍𝘪𝘵 — 𝘐𝘴 𝘪𝘵 𝘸𝘰𝘳𝘵𝘩 𝘵𝘩𝘦 𝘪𝘯𝘷𝘦𝘴𝘵𝘮𝘦𝘯𝘵? The brutal question every CFO asks. → CAC vs LTV healthy → Cost of build vs cost of delay calculated → Pricing model validated → Revenue forecast realistic → Ongoing OPEX sustainable Score: Not Viable / Viable / Highly Viable 6. 𝘖𝘱𝘦𝘳𝘢𝘵𝘪𝘰𝘯𝘢𝘭 𝘍𝘪𝘵 — 𝘊𝘢𝘯 𝘵𝘩𝘦 𝘣𝘶𝘴𝘪𝘯𝘦𝘴𝘴 𝘴𝘶𝘱𝘱𝘰𝘳𝘵 𝘪𝘵? Great products fail when operations aren't ready. → Sales enablement complete → Support team trained → Documentation ready → SLAs defined → Monitoring in place Score: Not Ready / Partially Ready / Fully Ready 𝗧𝗵𝗲 𝗗𝗲𝗰𝗶𝘀𝗶𝗼𝗻 𝗠𝗮𝘁𝗿𝗶𝘅: Calculate your Evaluation Score = (Value + Usability + Feasibility + Market + Financial + Operational) / 6 Then decide: ✅ Go → Move to MMP or MAP 🔄 Grow → Improve & iterate ⏸ Pause → Fix critical blockers 🛑 Stop → Pivot or sunset Why this works: - Most teams evaluate products emotionally or politically. - This framework forces objective, multi-dimensional assessment before you burn budget & morale on the wrong bet. Your turn: What’s the most overlooked dimension? 📌 Follow Santonu Mukherjee for 𝗚𝗲𝗻𝗔𝗜-𝗱𝗿𝗶𝘃𝗲𝗻 digital 𝘁𝗿𝗮𝗻𝘀𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻 stories. 🔄 Repost & 👍 Like if you enjoyed reading. #ProductManagement #DigitalTransformation #ProductStrategy #Innovation #GenAI #Leadership

UK Government news about AI tutoring for disadvantaged pupils, Following my earlier post, I want to share some more specific evidence that should inform how these tools are designed. Because... Not all "AI tutors" are equal. In policy/procurement conversations, the term is often used as a catch-all for very different systems. A tool that drafts feedback or generates practice questions is not the same as a system that can reliably diagnose misconceptions, plan pedagogical moves over time, and produce measurable learning gains. Both may be useful. But conflating them leads to unrealistic expectations. Recent benchmarking research is sobering. TutorBench, designed to evaluate tutoring capabilities rather than generic question-answering, tested frontier models across 1,490 expert-curated samples. No model exceeded approximately 56% overall performance on tutoring-specific rubrics. Performance on adapting explanations to student confusion averaged just 47%. Strong reasoning for solutions does not automatically translate into strong pedagogy in dialogue. Why this matters for the government initiative Because transformer-based language models face fundamental challenges when it comes to effective tutoring. They can infer a learner's state locally, but sustaining a reliable student model over time is not a native capability. They can generate fluent responses, but maintaining a multi-step instructional strategy across turns remains brittle. The good news is that hybrid approaches work. Systems combining conversational AI with explicit learner modelling, domain verification, and scaffolding can retain usability while meeting the reliability requirements that tutoring demands. Research on hybrid human-AI tutoring shows improved outcomes when AI supports human tutors. This has direct implications for design: 1. procurement should distinguish between tools supporting tutoring tasks and systems delivering genuine adaptive tutoring. Both have value, but require different evaluation approaches. 2. demand evidence of learning outcomes with tool-withdrawal checks. The question is not whether students get answers right while using the tool, but whether they can solve comparable problems independently afterwards. 3. engagement matters. Implementation research describes a "5% problem": only a small subset of students achieve recommended usage levels, and these are often students who would likely succeed anyway. Design must address how to reach the other 95% through teacher dashboards, structured practice, and shared responsibility for engagement. What I am listening to – "Higher Ground" by Stevie Wonder What I am reading – Lessons in Chemistry What I am baking – Chocolate Mousse https://lnkd.in/eSvE6iyw See you in the kitchen. Prof Rose Luckin UCL and EVR Ltd #AIEducation #AITutoring #EdTech #LearningScience #EvidenceBasedPolicy #AIED #EducationalEquity

Every teacher has been there. You find a digital tool that looks promising. It's engaging, it ticks the curriculum boxes, so you bring it into your classroom. But was it truly pedagogically sound? Accessible to every learner? Legally compliant? Built to last? Most classroom practitioners in Europe have never had a structured way to answer those questions — until now. During 2024 and 2025, I had the privilege of serving as a Senior Expert contributing to the European Commission's newly-published guidelines: "Making Informed Choices on Digital Education Content." This isn't a checklist. It's a genuine framework for how educators can evaluate the digital content they bring into their classrooms . It covers pedagogical alignment, accessibility and inclusion, reliability, legal compliance, interoperability, and long-term sustainability. Because digital education content is not just "textbooks in digital form." It shapes how students learn, who gets included and what data leaves the room. If you work in education, policy or #EdTech — I encourage you to read, use, and share these guidelines with every teacher and school leader in your network. Link to the guidelines - https://lnkd.in/dNyH4B-s

I built a vocabulary app for my child—and treated it like a real product. The problem statement was simple: Kids can “know” a word and still not understand it. So instead of flashcards, I designed around the Frayer Model—definition, characteristics, examples, and non-examples—because it forces clarity of thought, not memorisation. From a product perspective, a few deliberate choices: • Feedback over scoring: the app gives constructive, specific feedback instead of right/wrong labels • Explainability built in: it shows where understanding breaks down (missing examples, vague characteristics, weak boundaries) • Progress as insight: improvement is visible as clearer thinking, not higher scores The most interesting insight? When users (even 6-year-olds) understand why something needs improvement, engagement goes up without gamification. This started as something I built for one user with extremely high expectations. Turns out, that’s not a bad way to build products. #ProductThinking #LearningExperience #UX #EdTech #BuildingInPublic #FrayerModel #FeedbackLoops

When we run usability tests, we often focus on the qualitative stuff — what people say, where they struggle, why they behave a certain way. But we forget there’s a quantitative side to usability testing too. Each task in your test can be measured for: 1. Effectiveness — can people complete the task? → Success rate: What % of users completed the task? (80% is solid. 100% might mean your task was too easy.) → Error rate: How often do users make mistakes — and how severe are they? 2. Efficiency — how quickly do they complete the task? → Time on task: Average time spent per task. → Relative efficiency: How much of that time is spent by people who succeed at the task? 3. Satisfaction — how do they feel about it? → Post-task satisfaction: A quick rating (1–5) after each task. → Overall system usability: SUS scores or other validated scales after the full session. These metrics help you go beyond opinions and actually track improvements over time. They're especially helpful for benchmarking, stakeholder alignment, and testing design changes. We want our products to feel good, but they also need to perform well. And if you need some help, i've got a nice template for this! (see the comments) Do you use these kinds of metrics in your usability testing? UXR Study