Evaluating Educational Innovations through Data

Your programme works. You have data to prove it. Then the hard questions came: 'How do you KNOW it was YOUR intervention?' 'Which parts must stay the same when we replicate this in 12 countries?' 'Why did it work in the first place?' Silence. You're not alone in not having the answers. Most programme (innovative or traditional) can't answer these questions because they collected activity data, not evidence for scale. Here's what you should be measuring at each stage instead: 📍 Early stage (Pilot): Don't just count participants. Measure: Did it work? Was it feasible? Do users actually want this? 📍 Mid-stage (Acceleration): Don't just report more numbers. Measure: What are the core elements that CAN'T change? What CAN flex for different contexts? 📍 Scale stage: Don't just show reach. Measure: Can you prove YOUR intervention caused the change? Can others sustain it without you? UNICEF's Innovation MEL Toolbox breaks down exactly what evidence you need at each stage (from ideation to scale) including practical tools like: →Theory of Change for different stages →Contribution Analysis (when RCTs aren't possible) →Fidelity & Adaptation Monitoring →Scaling Approach frameworks Whether you're testing something new, expanding what works, or adapting proven approaches to new contexts, this document is for you. 🔥 If this resonated, follow me. I break down Monitoring and Evaluation (M&E) concepts daily with practical, implementable tips that are grounded in facilitation experience across sectors. #MonitoringAndEvaluation

Unpacking the impact of digital technologies in Education This report presents a literature review that analyses the impact of digital technologies in compulsory education. While EU policy recognizes the importance of digital technologies in enabling quality and inclusive education, robust evidence on the impact of these technologies is limited especially due to its dependency from the context of use. To address this challenge, the literature review presented here, analyses the focus, methodologies, and results of 92 papers. The report concludes by proposing an assessment framework that emphasizes self-reflection tools, as they are essential for promoting the digital transformation of schools. The literature review on the impact of digital technologies in education revealed several key findings: - Digital technologies influence various aspects of education, including teaching, learning, school operations, and communication. - Factors like digital competencies, teacher characteristics, infrastructure, and socioeconomic background influence the effectiveness of digital technologies. - The impact of digital tools on learning outcomes is context-dependent and influenced by multiple factors. - Existing evidence on the impact of digital tools in education lacks robustness and consistency. The assessment framework proposed in the report offers a structured approach to evaluating the effectiveness of digital technologies in education: 1. Identify contextual factors influencing technology impact. 2. Map stakeholders and their characteristics. 3. Assess integration into learning processes and practices. 4. Utilize self-reflection tools like the Theory of Change. 5. Provide evaluation criteria aligned with the framework. 6. Adapt existing tools for technology assessment. 7. Consider digital competence frameworks for organizational maturity. Implications and recommendations for policymakers and educators based on the report findings include: - Recognizing the contextual nature of technology use. - Focusing on creating rich learning environments. - Adopting a systems approach to studying technology impact. - Ensuring quality implementation and professional development. - Developing policies for monitoring and evaluation. - Encouraging further research on technology impact. By following these recommendations, stakeholders can leverage digital technologies effectively to improve teaching and learning outcomes in educational settings. https://lnkd.in/eBEN5XQg

Program evaluation serves as a cornerstone for improving implementation, measuring outcomes, and enhancing accountability in programs across diverse sectors. This comprehensive Program Evaluation Toolkit, crafted with contributions from the Regional Educational Laboratory at Marzano Research, offers a step-by-step framework designed to support evaluators at every stage of the evaluation process. From planning and logic models to data collection, analysis, and dissemination of findings, this guide equips practitioners with the tools and resources needed to drive evidence-based decisions. Emphasizing both the practical and theoretical aspects of evaluation, the toolkit aligns its methodologies with internationally recognized standards, ensuring rigor and applicability across local, state, and federal programs. Each module is designed to build the capacity of users, guiding them through crafting measurable evaluation questions, identifying quality data sources, selecting robust designs, and interpreting findings in meaningful ways that address key stakeholder needs. Designed for program managers, policymakers, and evaluators, this toolkit transforms evaluation from a compliance exercise into a strategic tool for learning and improvement. By leveraging its structured approach, users can not only assess program effectiveness but also identify pathways for innovation and sustainability, ultimately fostering greater impact in the communities they serve.

How to quickly assess if an AI EdTech product is actually worth your time: Try using a simple 2x2 mental model I call the "REAL Framework." When approached by a sales team or evaluating a partnership, asking these four questions can be helpful: R – Real-world data: Does the product use authentic, diverse, and up-to-date learning data—or is it trained on generic internet content? E – Educational value: Does it measurably improve learning outcomes—or is it just a repackaged chatbot? A – Adoption friction: How easily can it integrate into existing systems, workflows, or curriculum? Will teachers actually use it? L – Learner-centric design: Is the AI aligned with how students learn best—or does it automate for the sake of automation? If the answer is weak in 2+ quadrants, consider passing. Rate each dimension 1–5: 1 = Not at all 5 = Well designed Score every AI product out of 20: 16–20 = Strong candidate 11–15 = Promising, but needs scrutiny <10 = Likely not worth your time Use this in vendor evals, build vs. buy discussions, or internal prioritization. Save this if you evaluate tools. Share it with someone building one. #ProductStrategy #EdTech #AIinEducation #ProductDevelopment

#LargeScale #𝐄𝐝𝐮𝐜𝐚𝐭𝐢𝐨𝐧𝐚𝐥𝐀𝐬𝐬𝐞𝐬𝐬𝐦𝐞𝐧𝐭𝐬: 𝐇𝐨𝐰 𝐏𝐀𝐑𝐀𝐊𝐇 𝐑𝐚𝐬𝐡𝐭𝐫𝐢𝐲𝐚 𝐒𝐚𝐫𝐯𝐞𝐤𝐬𝐡𝐚𝐧 𝟐𝟎𝟐𝟒 𝐢𝐬 𝐋𝐞𝐚𝐝𝐢𝐧𝐠 𝐭𝐡𝐞 𝐖𝐚𝐲! Educational assessments have evolved, and so has India’s approach! With PARAKH Rashtriya Sarvekshan 2024, we're introducing innovations that set a new benchmark in evaluating learning outcomes. Ever wondered how PARAKH Rashtriya Sarvekshan 2024 is setting itself apart? Here's a quick breakdown: 1️⃣ 𝐒𝐜𝐨𝐩𝐞 𝐑𝐞𝐝𝐞𝐟𝐢𝐧𝐞𝐝 While NAS 2021 covered Grades 3, 5, 8, and 10, PARAKH focuses on Grades 3, 6, and 9, aligning with the competency-based approach of NEP 2020 and National Curriculum Frameworks 2️⃣ 𝐄𝐧𝐡𝐚𝐧𝐜𝐞𝐝 𝐂𝐨𝐯𝐞𝐫𝐚𝐠𝐞 PARAKH reaches 782 districts and samples over 22 lakh students from unique schools—ensuring a more representative and insightful evaluation. 3️⃣ 𝐁𝐚𝐥𝐚𝐧𝐜𝐞𝐝 𝐈𝐧𝐜𝐨𝐦𝐩𝐥𝐞𝐭𝐞 𝐁𝐥𝐨𝐜𝐤 (𝐁𝐈𝐁) 𝐃𝐞𝐬𝐢𝐠𝐧𝐬 BIB designs are highly efficient for the estimation of summary statistics. They boost efficiency while accommodating India’s unique linguistic and logistical needs. Imagine managing assessments in 20 languages with 12 distinct booklets! 4️⃣ 𝐏𝐥𝐚𝐮𝐬𝐢𝐛𝐥𝐞 𝐕𝐚𝐥𝐮𝐞𝐬 (𝐏𝐕𝐬) PARAKH Rashtriya Sarvekshan 2024 uses plausible values for reporting results instead of weighted maximum likelihood estimates (WLEs), that were used to report the results for NAS 2021. PVs offer a nuanced understanding of student performance, capturing uncertainties and enabling in-depth group-level analysis. This shift aligns us with global best practices like NAEP and PIRLS. 💡 𝐖𝐡𝐚𝐭’𝐬 𝐲𝐨𝐮𝐫 𝐭𝐚𝐤𝐞 𝐨𝐧 𝐭𝐡𝐞𝐬𝐞 𝐜𝐡𝐚𝐧𝐠𝐞𝐬? Drop a comment below with your thoughts or share how you think such advancements can shape India’s education system! We’d love to hear from educators, parents, policymakers, and all education enthusiasts. #DepartmentOfSchoolEducationAndLiteracy, #MoE, conducts the #PARAKHRashtriyaSarvekshan2024 Implementation of #NEP2020 in #SchoolEducation

Updated and latest Monitoring and Evaluation (M&E) methods and techniques: Quantitative Methods: 1. Data Analytics: Using statistical software (e.g., R, Python) for data analysis. 2. Machine Learning: Applying algorithms for predictive modeling. 3. Big Data Analysis: Handling large datasets for insights. 4. Survey Methods: Online surveys, mobile-based surveys. 5. GIS Mapping: Geospatial analysis for spatial planning. Qualitative Methods: 1. Participatory Rural Appraisal (PRA) 2. Focus Group Discussions (FGDs) 3. Key Informant Interviews (KIIs) 4. Case Studies 5. Narrative Analysis Mixed-Methods Approaches: 1. Integrating quantitative and qualitative data 2. Triangulation: Combining multiple methods for validation 3. Meta-Analysis: Synthesizing findings from multiple studies Real-Time Monitoring: 1. Mobile-based data collection 2. Remote sensing and satellite imaging 3. Social media monitoring 4. Sentinel Site Surveillance Impact Evaluation Methods: 1. Randomized Controlled Trials (RCTs) 2. Quasi-Experimental Designs (QEDs) 3. Counterfactual Analysis 4. Propensity Score Matching (PSM) Participatory and Collaborative M&E: 1. Participatory M&E (PM&E) 2. Collaborative, Learning, and Adapting (CLA) approach 3. Empowerment Evaluation 4. Community-Based M&E Technology-Enabled M&E: 1. Mobile apps for data collection (e.g., ODK, SurveyCTO) 2. Online M&E platforms (e.g., DevInfo, TolaData) 3. Data visualization tools (e.g., Tableau, Power BI) 4. Artificial Intelligence (AI) for data analysis Other Innovative Methods: 1. Theory of Change (ToC) approach 2. Outcome Mapping 3. Most Significant Change (MSC) technique 4. Social Network Analysis (SNA) Stay updated on the latest M&E methods and techniques through: 1. American Evaluation Association (AEA) 2. International Development Evaluation Association (IDEAS) 3. Evaluation Capacity Development (ECD) Group 4. BetterEvaluation website 5. M&E journals and publications (e.g., Journal of MultiDisciplinary Evaluation)

Andy Brock’s latest Re Education newletter carries an interview with Brad Olsen that is worth reading, and the argument I keep returning to is Brad’s insistence that most scaling conversations in education point at the wrong target. Too much of the debate fixates on growing the innovation itself, moving a programme from two hundred schools to two thousand, when the real question is whether impact grows alongside reach. The innovation is only the means. The end is the change in children’s learning, or in teacher agency, or in access for girls, or whatever the programme was set up to deliver, and the honest test is whether that impact holds as the reach expands or fades away as conditions change. Pilots almost always look better than they are, because participants have opted in, resources are concentrated, attention is high, and the teachers involved tend to be the most motivated in the system, and all of this inflates the proof of concept. Move into places where teachers are required rather than volunteering, where reform fatigue is real and conditions are harder, and results often flatten or decline. Without data asking whether impact is keeping pace with reach, the programme gets bigger while the point of it drifts. The second question Brad presses, and one funders often avoid, is whether the innovation is actually scalable in the first place, and this is where his Velcro metaphor for partnership matters. Outside actors bring resources, evidence and a wider perspective. Government brings proximity to the people, local ownership, and an understanding of the contours of the system the innovation has to survive in. Neither side on its own can honestly answer whether something will scale, and planning for scale has to begin at the beginning, not after five years of proof-of-concept work, because there is little sense investing heavily in something that cannot survive outside the conditions of the pilot. His example of a digital learning device in Guatemala makes the point clearly, because the question is not whether it works in the classrooms where it has been tested, but whether it can function across three quarters of the country, where electricity is unreliable, connectivity is patchy, humidity is extreme and the rubber cabling is often eaten by local rodents. Taken together, these two questions sharpen what funders, governments and partners should be asking before a single additional pound goes into replication. Is this worth scaling, and can it scale at all. Subscribe to Andy’s newsletter for free, and read the full interview, via the link in the comments.

🚀 Unlocking the Potential of Data in Education: From Data-Driven to Data-Informed 📊✨ Do you think you're ready to elevate your approach to school improvement? My latest article dives into the often blurred lines between "data-driven" and "data-informed" decision-making and their profound educational implications. 🔍 Key Highlights: Data-Driven vs. Data-Informed: Understand the distinct differences and why it matters. Five-Level Hierarchy: Learn the stages from basic data collection to integrating R&D for innovation. Practical Examples: Real-world scenarios from schools and districts that illustrate each level. 📚 Levels of Transition: Data Collection and Basic Analysis: Reactive decision-making based on primary data. Descriptive Analytics: Identifying trends to inform improvement. Diagnostic Analytics: Understanding the root causes of trends and issues. Predictive Analytics: Forecasting outcomes for proactive planning. Prescriptive Analytics and R&D Integration: Driving innovation through evidence-based strategies. 👩🏫 Transformative Practices: Discover how transitioning to a data-informed approach can revolutionize school improvement, leading to more strategic, proactive, and innovative solutions. Dive into the full article to explore how these transformative practices can set the foundation for continuous educational growth and excellence. #Education #SchoolImprovement #DataDriven #DataInformed #Innovation #R&D #Analytics #EducationalLeadership #ContinuousImprovement

Edtech is often criticised for poor quality, misuse of student data and limited learning impact (I’ve voiced those concerns myself several times). But we can’t hold systems accountable without first showing what good or exceptional performance looks like. Once that’s clear, we can create competitive pressure and drive improvement.  ⬇️ Excited to finally share our paper in HSCC Springer Nature that outlines key benchmark criteria for high-quality EdTech. The paper summarises the work our research group has been doing over the past three years. It focuses on educational impact and edtech’s added value for students’ learning. 📚 After an extensive literature review and cross-sector consultations, we’ve developed a multidimensional framework grounded in the “5Es” — efficacy, effectiveness, ethics, equity, and environment.  Efficacy and Effectiveness combine experimental evidence with process-focused metrics and pedagogical implementation studies. Broader metrics focus on ethical data processing, inclusive and equitable approaches and edtech’s environmental impact. 👇 The fifteen tiered impact indicators already guide a comprehensive and flexible evaluation process of international policymakers, educators, EdTech developers and certification bodies (see EduEvidence - The International Certification of Evidence of Impact in Education and our case studies). 🙏 Huge thanks to all who contributed, especially through our participatory Delphi process. Your insights were invaluable! Nicola Pitchford Anna Lindroos Cermakova Olav Schewe Janine Campbell /Rhys Spence Jakub Labun Samuel Kembou, PhD Tal Havivi/ Ayça Atabey Dr. Yenda Prado Sofia Shengjergji, PhD Parker Van Nostrand David Dockterman Stephen Cory Robinson Andra Siibak Petra Vackova Stef Mills Michael H. Levine  #EdTech #ImpactMeasurement #5Es #EdTechQuality #EdTechStandards 👇 Read here or download from:

📊 Only 5 percent of genAI pilots deliver fast revenue gains. The other 95 percent do not move the P&L. The question is not does AI work, it is are we setting it up to work. 🧩 MIT’s new analysis shows heavy investment with light returns, especially when projects stay at the demo stage. The winners embed AI into real workflows, adapt systems over time, and measure business outcomes, not novelty. 🎓 For education and EdTech this matters even more. If revenue led use cases struggle to show quick wins, learning led use cases will need patient design, teacher training, strong data governance, and clear guardrails. Quick demos do not equal durable classroom impact. 👩🏫 As EdTech Specialist & AI Lead, I focus on long term value. I am building AI literacy pathways for staff and students, running practical PD tied to lessons, and aligning tools with GDPR and the EU AI Act. We track time saved, feedback quality, and student outcomes, not hype. 💡 Short term metrics can underprice long term transformation. The real gains show up in better feedback loops, improved planning, and consistent assessment, plus safer data practices that unlock responsible innovation. That takes strategy, not just spend. 💬 How are you balancing quick wins with long term AI investment in your school or organisation. Which 2 or 3 metrics prove value in the first 12 months without chasing vanity numbers. Share your approach below!