Educational Program Assessment

“Show outcomes, not outputs!” I’ve given (and received) this feedback more times than I can count while helping organizations tell their impact stories. And listen, it’s technically right…but it can also feel completely unfair. We love to say things like: ✅ 100 teachers trained ✅ 10,000 learners reached ✅ 500 handwashing stations installed But funders (and most payers) want to know: 𝘞𝘩𝘢𝘵 𝘢𝘤𝘵𝘶𝘢𝘭𝘭𝘺 𝘤𝘩𝘢𝘯𝘨𝘦𝘥 𝘣𝘦𝘤𝘢𝘶𝘴𝘦 𝘰𝘧 𝘢𝘭𝘭 𝘵𝘩𝘢𝘵? That’s the outcomes vs outputs gap: ➡️ Output: 100 teachers trained ➡️ Outcome: Teachers who received training scored 15% higher on evaluations than those who didn’t The second tells a story of change. But measuring outcomes can be 𝗲𝘅𝗽𝗲𝗻𝘀𝗶𝘃𝗲. It’s easy to count the number of people who showed up. It’s costly to prove their lives got better because of it. And that creates a brutal inequality. Well-funded organizations with substantial M&E budgets continue to win. Meanwhile, incredible community-led organizations get sidelined for not having “evidence”- even when the change is happening right in front of us. So what can organizations with limited resources do? 𝗟𝗲𝘃𝗲𝗿𝗮𝗴𝗲 𝗲𝘅𝗶𝘀𝘁𝗶𝗻𝗴 𝗿𝗲𝘀𝗲𝗮𝗿𝗰𝗵: That study from Daystar University showing teacher training improved learning by 10% in India? Use it. If your intervention is similar, cite their methodology and results as supporting evidence. 𝗗𝗲𝘀𝗶𝗴𝗻 𝘀𝗶𝗺𝗽𝗹𝗲𝗿 𝘀𝘁𝘂𝗱𝗶𝗲𝘀: Baseline and end-line surveys aren't perfect, but they're better than nothing. Self-reported confidence levels have limitations, but "85% of teachers reported feeling significantly more confident in their teaching abilities," tells a story. 𝗣𝗮𝗿𝘁𝗻𝗲𝗿 𝘄𝗶𝘁𝗵 𝗹𝗼𝗰𝗮𝗹 𝗶𝗻𝘀𝘁𝗶𝘁𝘂𝘁𝗶𝗼𝗻𝘀: Universities need research projects. Find one studying similar interventions and collaborate. Share costs, share data, share credit. 𝗨𝘀𝗲 𝗽𝗿𝗼𝘅𝘆 𝗶𝗻𝗱𝗶𝗰𝗮𝘁𝗼𝗿𝘀: Can't afford a 5-year longitudinal study? Track intermediate outcomes that research shows correlate with long-term impact. 𝗧𝗿𝘆 𝗽𝗮𝗿𝘁𝗶𝗰𝗶𝗽𝗮𝘁𝗼𝗿𝘆 𝗲𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻: Let beneficiaries help design and conduct evaluations. It's cost-effective and often reveals insights that traditional methods miss. For example, train teachers to interview each other about your training program. And funders? Y’all have homework too. Some are already offering evaluation support (bless you). But let’s make it the rule, not the exception. What if 10-15% of every grant was earmarked for outcome measurement? What if we moved beyond gold-standard-only thinking? 𝗟𝗮𝗰𝗸 𝗼𝗳 𝗮 𝗰𝗲𝗿𝘁𝗮𝗶𝗻 𝗸𝗶𝗻𝗱 𝗼𝗳 𝗲𝘃𝗶𝗱𝗲𝗻𝗰𝗲 𝗱𝗼𝗲𝘀𝗻’𝘁 𝗺𝗲𝗮𝗻 “𝗻𝗼𝘁 𝗶𝗺𝗽𝗮𝗰𝘁𝗳𝘂𝗹”. We need outcomes. But we also need equity. How are you navigating this tension? What creative ways have you used to show impact without burning out your team or budget? #internationaldevelopment #FundingAfrica #fundraising #NonprofitLeadership #nonprofitafrica

MEAL/MERL/ MEL/ M&E/ MERLA The evolution of project management frameworks, particularly in the international development and non-profit sectors, shows a steady shift from simple data collection to complex, people-centered systems. The Evolution of Monitoring, Evaluation, and Learning (MEL)** 1. M&E: Monitoring and Evaluation** Focus: Tracking Results Definition: The foundation of the framework. Monitoring is the continuous collection of data to see if a project is on track; Evaluation is the periodic assessment of the project’s overall impact and relevance. 2. MEL: Monitoring, Evaluation, and Learning Focus: Learning from Results Definition: Adds a "Learning" component to ensure that the data collected in M&E isn't just filed away. It emphasizes using data to improve current and future project decision-making. MEAL: Monitoring, Evaluation, Accountability, and Learning** Focus: Accountability to Communities Definition: This introduces "Accountability," shifting the focus to the stakeholders. It ensures there are mechanisms for beneficiaries to provide feedback and that the organization is answerable to the people it serves. 4. PMEL: Planning, Monitoring, Evaluation, and Learning Focus: Planning with Measurement in Mind Definition: Explicitly integrates "Planning" into the cycle. It highlights that effective monitoring and evaluation cannot happen unless the project is designed from day one with measurable indicators. 5. MERL: Monitoring, Evaluation, Research, and Learning Focus: Research-Informed Programming *Definition: Introduces "Research" as a formal pillar. This approach uses rigorous scientific methods or deep-dive studies to understand the "why" behind trends, rather than just tracking the "what." 6. MERLA: Monitoring, Evaluation, Research, Learning, and Adapting Focus: Adapting Based on Evidence Definition: Adds "Adapting" to create a circular feedback loop. It’s not enough to learn; the organization must have the agility to change its strategy mid-course based on what the evidence suggests. 7. MEALK: Monitoring, Evaluation, Accountability, Learning, and Knowledge Management Focus: Knowledge Management & Learning Definition: Adds "Knowledge Management" to ensure that the insights gained are documented, stored, and shared across the entire organization or sector, preventing "reinventing the ....

The evolution of Monitoring and Evaluation (M&E) in practice reflects the growing demands of development. While the terminology may appear similar, the focus is increasingly on impact, learning, and accountability. 1. M&E (Monitoring & Evaluation) - Tracking results: Monitoring activities and evaluating outcomes to measure performance. 2. MEL (Monitoring, Evaluation & Learning) - Learning from results: Using evidence to continuously improve programs and interventions. 3. MEAL (Monitoring, Evaluation, Accountability & Learning) - Accountability to communities: Ensuring beneficiary feedback and accountability shape program decisions. 4. PMEL (Planning, Monitoring, Evaluation & Learning) - Planning with measurement in mind: Designing programs with indicators and monitoring systems from the beginning. 5. MERL (Monitoring, Evaluation, Research & Learning) - Research-informed programming: Integrating research to deepen understanding and inform policy and practice. 6. MERLA (Monitoring, Evaluation, Research, Learning & Adaptation) - Adapting based on evidence: Adjusting programs as contexts and evidence change. 7. MEALK (Monitoring, Evaluation, Accountability, Learning & Knowledge Management) - Preserving and sharing knowledge: Capturing learning so organizations build institutional memory and future impact. #Discussion: Although these terms belong to the same family, their priorities are evolving. Where does your organization currently sit in this M&E evolution

Every product manager is racing to ship AI features. But here's what nobody talks about: most ship broken, get fixed quietly, or die slowly. The difference between shipping and shipping something that works? Evals. An eval = systematic way to measure if your AI output is actually good. If you want an AI feature that actually works for real users (not just in demos), evals are the most important thing you need to learn. These insight comes from Hamza Husein (ex-OpenAI, ex-Airbnb) and Shrea Shanker (ex-Atlassian, ex-GitHub), two of the sharpest minds in AI product management. Here’s a simple 5-step framework to get started 👇 1️/ Start with Error Analysis Generate 50 diverse outputs For each answer this: "Would I ship this? Yes or No?" For every "No," write why in 1-2 sentences Output: A list of 5 -10 recurring failure patterns. 2️/ Find Your Failure Modes Group similar errors together. Give each a clear name and note how often it appears. Example: Hallucination (12), Wrong Tone (18), Missing Context (8) Stop when you’ve reviewed around 20 more outputs without discovering any new failure types. Output: 3-5 named failure modes with counts 3/ Build Binary Rubrics Turn your top 3 failure modes into clear rubrics For each, define: → A pass/fail rule (no 1–5 ratings) → 3 examples of PASS → 3 examples of FAIL Example - Hallucination: PASS: Every fact is verifiable or clearly marked as inference. FAIL: Any unverifiable or made-up fact. Output: 3 rubrics with examples that define your quality bar. 4/ Test for Alignment Take 20 new outputs. You and a teammate score them independently using your pass/fail rules. Then calculate → (number of agreements) / 20. Target: 80 % + agreement. Below that? Your rubric is unclear. Refine the definitions or examples and test again. Output: Rubrics you can trust across the team. 5/ Diagnose & Fix with the Three Gulfs Now that you know your failure modes, it’s time to diagnose why they’re happening. There are only three reasons your AI feature isn’t working and each needs a completely different fix: Gulf #1 — Specification Problem → Fix with better prompting (days to fix) Gulf #2 — Knowledge Problem → Fix with RAG or retrieval (weeks to fix) Gulf #3 — Capability Problem → Fix with better models or fine-tuning (months to fix) Most teams reach for the wrong solution. In reality, 80% of problems are Gulf #1 (specification) but teams jump straight to Gulf #3 (fine-tuning) way too early. I’ll break down the complete Three Gulfs Framework with detailed examples and fixes in my upcomig posts. It’s dense enough to deserve its own deep dive. Liked this breakdown? Follow + Save for more no-fluff posts on how to build AI features that actually work.

What makes a blended learning solution truly impactful? If you're looking for inspiration, this Learning Uncut episode with Millie Law from ANZ Bank is a masterclass in business-aligned learning design. The initiative emerged from ANZ's strategic workforce planning, identifying business development as a critical capability need for their home lending specialists. The L&D team partnered closely with the business to understand both performance gaps and mindset barriers that were holding back home lending specialists from proactive customer engagement. The solution beautifully combined self-paced learning, peer workshops facilitated by respected internal coaches, and practical application with real customers. By addressing both skill development and mindset shifts around customer engagement, the program achieved sustained improvements of 20-25% in key business metrics. The success in Australia led to adaptation for New Zealand operations and integration into ANZ's core lending curriculum. What struck me most was how well-crafted this initiative was from end to end - from understanding business needs and connecting with individuals, to engaging influencers and measuring impact. It's such a thorough example of professional L&D practice. Thank you Millie for joining me to share this outstanding work that rightfully won the 2024 AITD award for best blended learning solution. Join us as we explore how the Better BD program achieved such significant outcomes. This episode is filled with useful insights to help L&D professionals create high-impact solutions. Listen on your favourite podcast app or go to the episode landing page to listen and access additional resources: https://lnkd.in/eyCkAPPY #LearningUncut#LearningAndDevelopment #BlendedLearning #CapabilityDevelopment #PerformanceConsulting

If AI can now produce competent answers in seconds, what exactly are we assessing in our degrees? AI is already embedded in how students learn, think, and produce work. So, the question is no longer about its use. Rather, the real question is whether assessment is designed to treat AI as a liability to be controlled or as a resource to be used well. AI-integrated assessment does not mean looking the other way when students use AI. It means designing tasks where AI use is expected, visible, and evaluated. The shift is subtle but fundamental: from policing outputs to assessing judgment. Several practical design principles follow. First, assess decisions rather than artefacts. In an AI-rich environment, polished outputs are cheap. What remains scarce is the ability to frame problems well, choose appropriate tools, test assumptions, and decide when not to trust an AI response. Assessment can require students to justify how AI was used, why particular prompts were chosen, and how outputs were validated against disciplinary knowledge. Second, make the process evidence assessable. Short AI logs, annotated iterations, or structured commentaries can document how thinking evolved through interaction with AI. This is forensic reasoning about choices made, alternatives rejected, and risks managed. Used well, it turns AI from a shortcut into a cognitive amplifier. Third, build in authentic constraints. In professional settings, AI is used within limits, including ethical rules, organisational policies, incomplete data, and reputational risk. Assessment can simulate these conditions through ambiguous briefs, imperfect datasets, or explicit governance boundaries. Students are evaluated on how they navigate trade-offs, not how elegant the final output appears. Fourth, reintroduce dialogue selectively. Ask for recorded walkthroughs or live critiques, which allow students to explain how AI shaped their reasoning. The purpose is not detection but sense-checking judgment. Weak understanding surfaces quickly when students must articulate why they trusted or rejected an AI-generated insight. Finally, reward responsible AI use explicitly. Rubrics should recognise transparency, validation, ethical awareness, and the integration of AI output with human judgement. When expectations are clear, students learn how to use AI well rather than how to hide it. This approach develops genuinely transferable skills such as judgment under uncertainty, learning agility, ethical reasoning, and accountability. It prepares students for workplace realities where AI is normal, governed, and consequential. It fosters better feedback and stronger academic relationships by shifting conversations from suspicion to reasoned discussion. The irony is that AI-integrated assessment is not easier. It is harder. It raises the bar. We need to shift our thinking from compliance to using assessment to develop graduates who not only know how to use AI, but also when, why, and to what effect.

We're measuring learning at the wrong time. And it's costing us real impact. Most learning providers measure before and after their programs. But here's what I've discovered after years of analyzing client outcomes: when we measure should be 100% determined by what we hope will happen AFTER learning, not during it. With this idea in mind, our measurement strategies change significantly: Compliance programs? Don't wait until deadlines to measure. Measure weekly so clients can support their people in actually becoming compliant. Skills development? If learners apply those skills daily, measure daily. If weekly, measure weekly. The breakthrough happens when we shift from measuring around learning experiences to measuring around desired workplace results. Here's how I've been thinking about when to measure, and it's made a real difference in the quality of the data I receive from my measurement efforts! For compliance programs: Design measurement that helps organizations support their people in meeting requirements, not just tracking completion. For behavior change programs: Match measurement frequency to how often learners have opportunities to apply what they learned. Answering "when to measure" is actually the secret backdoor to figuring out "what to measure." The simple take-away? Stop measuring your programs. Start measuring new behaviors participants are applying in the flow of work. Here's a simple flow chart to help you get started: https://lnkd.in/gB5Yh8nm What's been your experience with measurement timing? Have you found that when you measure changes the results you can demonstrate? #learningproviders #measurementmethods #datastrategy

Unpacking the impact of digital technologies in Education This report presents a literature review that analyses the impact of digital technologies in compulsory education. While EU policy recognizes the importance of digital technologies in enabling quality and inclusive education, robust evidence on the impact of these technologies is limited especially due to its dependency from the context of use. To address this challenge, the literature review presented here, analyses the focus, methodologies, and results of 92 papers. The report concludes by proposing an assessment framework that emphasizes self-reflection tools, as they are essential for promoting the digital transformation of schools. The literature review on the impact of digital technologies in education revealed several key findings: - Digital technologies influence various aspects of education, including teaching, learning, school operations, and communication. - Factors like digital competencies, teacher characteristics, infrastructure, and socioeconomic background influence the effectiveness of digital technologies. - The impact of digital tools on learning outcomes is context-dependent and influenced by multiple factors. - Existing evidence on the impact of digital tools in education lacks robustness and consistency. The assessment framework proposed in the report offers a structured approach to evaluating the effectiveness of digital technologies in education: 1. Identify contextual factors influencing technology impact. 2. Map stakeholders and their characteristics. 3. Assess integration into learning processes and practices. 4. Utilize self-reflection tools like the Theory of Change. 5. Provide evaluation criteria aligned with the framework. 6. Adapt existing tools for technology assessment. 7. Consider digital competence frameworks for organizational maturity. Implications and recommendations for policymakers and educators based on the report findings include: - Recognizing the contextual nature of technology use. - Focusing on creating rich learning environments. - Adopting a systems approach to studying technology impact. - Ensuring quality implementation and professional development. - Developing policies for monitoring and evaluation. - Encouraging further research on technology impact. By following these recommendations, stakeholders can leverage digital technologies effectively to improve teaching and learning outcomes in educational settings. https://lnkd.in/eBEN5XQg

So, OpenAI just got into the Learning Measurement game. Here's what L&D teams should know: → The Learning Outcomes Measurement Suite is a framework to understand how AI shapes learning and outcomes across different contexts. It's OpenAI's answer to a problem L&D knows well: most learning measurement is shallow. Test scores and completion rates don't tell you whether someone has actually developed (shocking, I know!) LOMS is designed to fix that tracking how AI interactions shape cognitive development over time, not just performance on a single assessment. It was built with Estonia's University of Tartu and Stanford's SCALE Initiative, and is currently being validated with nearly 20,000 students in Estonia. - - - 👀 How it works LOMS is built on three signals: 1/ Is the AI behaving like a good teacher? 2/ Is the individual genuinely engaging? 3/ Are their cognitive capabilities improving over time? - - - Five components answer those questions: 1️⃣ System instructions: Rules that shape how ChatGPT behaves, aligning it to sound learning principles rather than just giving answers. 2️⃣ Interaction classifiers: Automatically scan real conversations and flag moments where meaningful learning is or isn't happening. 3️⃣ Quality graders: Score each learning moment against teaching standards. Did the individual get there? Did the AI guide them well? 4️⃣ Longitudinal graders: Track the same learner over time, monitoring changes in engagement, persistence, and self-regulation. 5️⃣ Standardised assessments: Validated assessments for critical thinking, creativity, and memory, run before, during, and after AI use to measure genuine capability shifts. - - - Still with me? Good... - - - Beyond test scores, LOMS measures five deeper capabilities: → Autonomous motivation: are learners self-directing, or dependent on the AI? → Productive engagement: quality and variety of learning interactions, not just volume →Task persistence: do they push through hard problems or default to the AI? → Metacognition: are they planning, reflecting, and monitoring their own learning? → Recall: can they accurately remember content from previous sessions? - - - Damn, did you get all that?... I'll break down more on this in an upcoming Steal These Thoughts! newsletter (link in comments, obvs). Plus, link to the OpenAI announcement in comments too.

Showing Return on Investment is becoming a critical factor for L&D teams to show their worth within a business, but how can they do this effectively? Utilising such frameworks as the 'V Model' from the ROI Institute Canada is an excellent way for L&D teams to demonstrate the ROI of their learning initiatives. This model provides a structured approach that aligns with business goals and focuses on evaluating the effectiveness and impact of training programs. Here's how the V Model can be utilized by L&D teams: 👍 Needs Assessment: At the beginning of the V Model, L&D teams conduct a thorough needs assessment to understand the specific skills or knowledge gaps within the organization. This step involves identifying performance issues, analyzing business objectives, and determining the desired outcomes of the learning initiative. 👍Design and Development: Based on the needs assessment, L&D teams design and develop targeted learning solutions. This phase involves creating learning objectives, selecting appropriate instructional methods and materials, and designing evaluation strategies to measure the effectiveness of the training program. 👍Implementation: The next stage is the implementation of the training program. L&D teams deliver the learning content to employees through various methods such as classroom training, e-learning modules, workshops, or on-the-job training. 👍Evaluation: After the training is delivered, the evaluation phase of the V Model begins. L&D teams assess the effectiveness of the learning initiative by measuring learning outcomes, gathering feedback from participants, and evaluating whether the training has resulted in improved performance and behavior change. 👍Impact and ROI Analysis: The final step of the V Model focuses on measuring the impact of the training program on business results. L&D teams use data and metrics to calculate the ROI by comparing the costs of the training initiative to the benefits, such as increased productivity, reduced errors, higher employee engagement, or improved customer satisfaction. TL:DR; The key areas that this focuses on: ⭐ Align training with business objectives. ⭐Use data for informed decision-making. ⭐Demonstrate value through ROI analysis. ⭐Focus on continuous improvement based on evaluation feedback. ⭐Ensure training initiatives drive measurable business outcomes. Hope that helps! #ROI #learningdesign #instructionaldesign