How Educators Evaluate New Tools

Selecting the right AI tool can be challenging when new products appear almost daily. This guide helps you cut through the noise with a clear, structured process for testing, evaluating, and integrating AI tools in the classroom. It introduces a practical framework built around three pillars: usability, pedagogy, and ethics. Each is broken into a checklist of focused questions to help educators quickly determine whether a tool fits their curriculum, supports deep learning, and meets privacy standards. The guide also includes tips for piloting tools with a small group, gathering student feedback, and reflecting on results. This guide is informed by key resources, including aiEDU’s AI Readiness Framework, ISTE’s Teacher Ready Edtech Product Evaluation Guide, the U.S. Department of Education’s AI Integration Toolkit, and UNESCO’s Recommendation on the Ethics of AI. These references shaped the usability, pedagogy, and ethics checklists to keep the framework practical and research-based. #AIinEducation #EdTech #TeachingWithAI #TeacherTools #AIforTeachers #EdLeaders #ClassroomInnovation #DigitalLearning #AIIntegration #EducationTechnology

Common Sense Media recently released a comprehensive risk assessment of AI teacher assistants/lesson planning tools. Their findings reveal that while these tools promise increased productivity and creative support, they're also creating "invisible influencers" that could fundamentally undermine educational quality. Unlike GenAI foundation model chatbots, these tools are specifically designed for instructional planning and classroom use and are rapidly being adopted across districts. Key Concerns from their report: • "Invisible Influencers" in Student Learning: AI-generated content directly shapes what students learn through potentially biased perspectives and historical inaccuracies that teachers may miss; evidence also shows these tools suggest different approaches and responses based on student race/gender • “Outsourced Thinking" Problem: Tools make it dangerously easy to push unreviewed AI instructional content straight to classrooms, while novice teachers lack experience to spot subtle errors and biasses • High-Stakes Outputs: IEP and behavior plan generators create official-looking documents that could impact student educational trajectories even though these plans should be human-generated (and in the case of IEP goals are mandated to be human generated) • Undermining High-Quality Instructional Materials: Without proper integration, these tools fragment learning and can undermine coherent, research-backed curricula Recommendations from the report: • Experienced educator oversight required for all AI-generated educational content • Clear district policies and guidelines for AI teacher assistant implementation • Integration with existing high-quality curricula rather than replacement of established materials • Robust teacher training on identifying bias and evaluating AI outputs • Careful oversight of real-time AI feedback tools that interact directly with students We'd also recommend foundational AI literacy for teachers before they begin using GenAI teacher assistants, so that they are aware of the potential limitations. While AI teacher assistants aren't inherently problematic, they require the same careful implementation and oversight we'd expect for any tool that directly impacts student learning. The potential for enhanced productivity is real, but so are the risks to educational equity and quality. This report underscores the urgent need for GenAI EdTech tool makers to provide evidence of how their tools mitigate these issues along with evidence-based policies and professional development to help educators navigate AI tools responsibly. All of which underline how important AI Literacy is for the 2025-2026 school year. Link in the comments to check out the full report. Also check out our 5 Questions to Ask GenAI EdTech Providers resource in the comments if you are planning to implement any of these tools in your school or district. #AIinEducation #ailiteracy #Education #K12 AI for Education

A new study from the University of Washington offers something we rarely see in education AI discourse: actual classroom evidence. For seven weeks this spring, 21 teachers across five Washington State school districts used AI directly in instruction with more than 600 students in grades 6-12. The research team tested four tools: a Teaching Aide for structured student-AI conversations, AI-assisted assessment and grading, an AI Tutor, and a dashboard called Student Growth Insights. A few findings stood out to me. Teachers who had prior experience with AI and believed it could be useful designed more effective activities. They framed the tool well for students and reported confidence in using it again. Teachers without that background often ran one session and walked away. AI functioned well as what the researchers call a "third agent" in the classroom, but only when teachers provided clear framing and stayed present. Without scaffolding, student conversations drifted or became overwhelming. Here's the tension worth noting. Teachers valued the narrative feedback AI generated. It helped surface misconceptions and guide revision. But the automated scores were inconsistent. Misaligned with rubrics. Teachers ignored them and retained final grading authority. Developmental differences mattered too. Middle school students needed shorter outputs and explicit structure. High schoolers could handle longer exchanges with more ambiguity. The most useful feature, according to teachers, was Student Growth Insights. It aggregated common struggles across a class and saved hours of manual review. What does this suggest for implementation? Investing in teacher AI competency may be the single highest-leverage move a district can make. The tools can extend instructional reach. But teachers have to design the conditions under which that happens. https://lnkd.in/gvXT7ZDz

Is AI truly helping students learn better, or are we measuring the wrong things? If you are a leader at a school or university, you are likely hearing a lot of claims about how "AI improves results." However, many of these claims come from studies that might sound rigorous but aren't designed well enough to measure whether students are truly learning over the long term. Here are some common mistakes to look out for when you are evaluating new AI programs: - Relying on personal feelings: Some studies focus on things like how satisfied students feel or how they rate their own learning. This only measures subjective variables, not the actual process of learning or the final knowledge gained. - Confusing supported performance with real learning: Just because a student performs well while using the AI tool doesn't mean they've actually learned the material. You need to see if they can remember and use that information without the AI support later on. - Comparing AI only to doing nothing: When the control group—the group not using AI—receives no extra support at all, the study only proves that AI is better than nothing. It doesn't prove that AI is better than a great teacher or peer learning. Leaders need to be able to separate the hype from the reality for AI effectiveness in education. Bauer and colleagues offer a useful framework to classify what AI is really doing to the learning process—it's called Inversion, Substitution, Augmentation, and Redefinition (ISAR). - Inversion: Did the AI tool make the task too easy, causing students to put in less mental effort? For example, providing too many hints might lead to a superficial understanding. In this case, we might be sacrificing deep learning for convenience. - Substitution: Does the AI achieve the same learning results as a non-AI method, like standard electronic feedback, but save time or money? This can be a positive step for efficiency, even if the learning outcomes themselves don't change. - Augmentation: Does the AI add extra cognitive supports, such as timely hints, helpful examples, or spacing out practice, which improve the instruction without completely changing the task? Here, we expect to see slightly better results compared to the method without the AI. - Redefinition: Does the AI completely change the assignment to encourage deeper, more interactive, or constructive learning—like working through arguments with structured critique—in ways that wouldn't have been possible before? This is the scenario where we are most likely to see lasting, significant improvements in learning. By recognizing common pitfalls and using the ISAR framework to classify the effects, leaders can make better decisions on how to effectively integrate AI. How can teachers and students help analyze results to ensure decisions fit real-world teaching? What guardrails can ensure that AI augments human judgement (e.g. valuable teacher feedback) instead of replacing it? #AI #Education #EdTech #ISAR 

Just finished reading "Design and assessment of AI-based learning tools in higher education: a systematic review" by Luo et al.. This is a synthesis of 63 peer-reviewed studies examining how AI tools are being designed and deployed in higher education effectively, and more important, responsibly. Employing Kraiger et al. (1993)'s framework to assess three learning outcome dimensions (cognitive, skill-based, and affective), they revealed a fascinating pattern: while AI-based learning tools excel at enhancing cognitive knowledge acquisition and affective learning outcomes (enhanced motivation, engagement, and self-efficacy), their impact on higher-order thinking and skill development were mixed. Three key insights I found very intriguing: 1. The black box problem persists Unlike traditional instructional tools with predefined rules, many AI tools operate opaquely, obscuring decision-making processes. This opacity particularly hinders complex reasoning in mathematics, physics, and medicine. 2. Design matters more than we think The finding about AI-enabled personalised video recommendations is insightful. It only benefited moderately motivated learners, as high achievers had already mastered the content, while less motivated ones remained disengaged. Perhaps it is a calibration issue that invites the concept of Flow? 3. The human element is irreplaceable Current AI tools excel at providing instant, contextual answers but often lack the strategic pedagogical depth of expert human tutors. The review warns of declining critical thinking and growing AI dependency: concerns that align with recent research on metacognition and cognitive offloading. The authors propose a "design-to-evaluation" framework emphasising five principles:  - human-centered design that incorporates learner traits beyond performance metrics - multimodal content strategically tailored to learning objectives - transparent decision-making processes - inclusive design for marginalized students - ethical safeguards for privacy and bias This review, to me, reinforces the notion that AI tools work best when they complement, rather than replace, human expertise. Continuous teacher calibration, metacognitive scaffolding, digital literacy (the SCAN framework that Alina and I developed: https://lnkd.in/eanDnGbm), and strategic task assignment and application of multimodal approaches tailored to specific learning objectives and student needs remain essential. Many thanks to Jihao Luo, Chenxu Zheng, Jiamin Yin, and Hock Hai Teo for this insightful work that pushes us toward more intentional, human-centered AI design in higher education. As we race to integrate AI in education, we need equal rigor in understanding how and when these tools genuinely enhance learning. Link: https://lnkd.in/e7x2S2f7 #AIinEducation #HigherEducation #EdTech #ArtificialIntelligence #LearningScience #EducationalTechnology #PedagogicalInnovation #FutureOfLearning