Learner Engagement Measurement

Over the past 10 years, I've seen the full spectrum of attempts to make corporate training "more engaging" - from points to virtual reality. Gamification can be a powerful tool when used thoughtfully, but it's just 1 piece of engagement. True engagement in learning & development is an INDEX: 1. Relevance to role and career goals 2. Alignment with company mission and values 3. Quality and depth of content 4. Opportunities for practical application 5. Peer and leadership support A quick case study. We worked with a multi-unit restaurant group in TX that switched from a gamified learning app. Initial participation spiked, but long-term behavior change and skill development were minimal. Opus provided them a new approach, integrating gamification elements into a more 360 degree plan. - Implementing a coaching program alongside digital - Creating "feedback opportunities" for peer support and collaboration - Applying new training and skills dev to company initiatives The result? 80% ongoing engagement for the past 12 months. At Opus, our approach leverages the best aspects of gamification - like immediate feedback and a sense of progress - while addressing deeper motivational factors. Here's where I want your thoughts: How does your organization measure the impact of training beyond completion rates?

How do we measure beyond attendance and satisfaction? This question lands in my inbox weekly. Here's a formula that makes it simple. You're already tracking the basics—attendance, completion, satisfaction scores. But you know there's more to your impact story. The question isn't WHETHER you're making a difference. It's HOW to capture the full picture of your influence. In my many years as a measurement practitioner I've found that measurement becomes intuitive when you have the right formula. Just like calculating area (length × width) or velocity (distance/time), we can leverage many different formulas to calculate learning outcomes. It's simply a matter of finding the one that fits your needs. For those of us who are trying to figure out where to begin, measuring more than just the basics, here's my suggestion: Start by articulating your realistic influence. The immediate influence of investments in training and learning show up in people—specifically changes in their attitudes and behaviors. Not just their knowledge. Your training intake process already contains the measurement gold you're looking for. When someone requests training, the problem they're trying to solve reveals exactly what you should be measuring. The simple shift: Instead of starting with goals or learning objectives, start by clarifying: "What problem are we solving for our target audience through training?" These data points help us to craft a realistic influence statement: "Our [training topic] will help [target audience] to [solve specific problem]." What this unlocks: Clear metrics around the attitudes and behaviors that solve that problem—measured before, during, and after your program. You're not just delivering training. You're solving performance problems. And now you can prove it. I've mapped out three different intake protocols based on your stakeholder relationships, plus the exact questions that help reveal your measurement opportunities. Check it out in the latest edition of The Weekly Measure: https://lnkd.in/gDVjqVzM #learninganddevelopment #trainingstrategy #measurementstrategy

A few years ago, I worked with an online education platform facing challenges with student engagement. While they had a significant number of users enrolling in courses, they struggled with low participation rates in course discussions and activities, leading to a decline in course completion rates. The platform needed to identify the causes behind low engagement and implement strategies to encourage more active participation. Improving Student Engagement Using Data Analytics 1️⃣ Analyzing Engagement Data We began by analyzing user interaction data, focusing on metrics such as time spent on the platform, participation in discussions, video completion rates, and quiz scores. Using SQL, we aggregated the data to identify patterns and pinpoint where students were losing interest. SELECT student_id, course_id, AVG(time_spent) AS avg_time_spent, COUNT(discussion_post_id) AS posts_made, AVG(quiz_score) AS avg_quiz_score FROM student_activity GROUP BY student_id, course_id; 🔹 Insight: We identified that students who interacted with course discussions and quizzes had higher completion rates, while others dropped off quickly. 2️⃣ Building a Predictive Model We then created a predictive model to determine which students were at risk of disengaging based on their activity patterns. The model incorporated features such as time spent on the platform, participation in discussions, and progress through the course material. # Pseudocode for Predictive Model def predict_student_engagement(student_data): model = train_engagement_model(student_data) predictions = model.predict(student_data) return predictions 🔹 Insight: This model helped us flag students who were likely to disengage early, allowing for timely interventions. 3️⃣ Implementing Engagement Strategies Based on insights from the model, we implemented strategies such as sending personalized emails with reminders, offering incentives for completing activities, and increasing interaction opportunities through live Q&A sessions. # Pseudocode for Engagement Follow-Up def send_engagement_reminder(student_data): if model.predict(student_data) == 'at_risk': send_email_reminder(student_data) 🔹 Insight: Personalized engagement and incentives led to an increase in student participation. Challenges Faced Identifying meaningful engagement metrics that were predictive of success. Finding the right balance between engaging students without overwhelming them. Business Impact ✔ Student engagement improved, leading to higher completion rates. ✔ Retention rates increased, as more students continued with courses. ✔ Revenue grew, driven by more active and satisfied students. Key Takeaway: By analyzing user activity and leveraging predictive analytics, businesses can identify disengaged customers early and implement strategies to improve engagement and retention.

Stop measuring attendance and start measuring impact. We have analyzed, designed, developed, and implemented. Now comes the moment of truth: Evaluation. In the traditional ADDIE model, this phase is often reduced to "smile sheets." We ask learners if they liked the course, if the room was cold, or if the instructor was engaging. We gather data that tells us how they felt, but rarely how they will perform. In ADDIE 2.0, AI turns Evaluation into business intelligence. We no longer have to rely on manual surveys or disjointed spreadsheets. AI tools can ingest vast amounts of unstructured data—from chat logs to open-text survey responses—and identify patterns that a human eye might miss. It bridges the gap between "learning" and "doing." Here are three ways to revolutionize your Evaluation phase today: ✅ Ditch the 1-5 scale for sentiment analysis. Stop looking at average scores. Take all your open-text feedback and run it through a Large Language Model (LLM). Ask it to identify the top three friction points and the top three "aha!" moments. You will get a nuanced report on learner sentiment that goes far beyond a simple satisfaction score. ✅ Correlate learning with performance. This used to require a data scientist. Now you can upload anonymized training completion data alongside sales or productivity metrics into a tool like ChatGPT’s Data Analyst or Microsoft Copilot. Ask it to find correlations. Did the reps who completed the negotiation module actually close more deals next quarter? AI can help you prove that link. ✅ Automate the "Forgetting Curve" check. Evaluation should not end when the course closes. Configure an AI agent or chatbot to message learners 30 days later. Have it ask a simple question: "How have you used the negotiation framework this month?" The AI can collect and categorize these real-world stories, giving you qualitative evidence of behavior change. Why does this matter to the C-Suite? ROI. When you can show that a learning intervention directly correlates with a 15% increase in efficiency or revenue, L&D stops being a cost center and starts being a strategic partner. AI gives you the evidence you need to defend your budget and prove your value. Series Wrap-Up: We have walked through the entire ADDIE model. Analysis: Using data to find the real gaps. Design: Blueprinting faster with AI assistants. Development: Generating assets at scale. Implementation: Personalizing the delivery. Evaluation: Measuring real-world impact. The ADDIE model is not dead. It just got a massive upgrade. I want to hear from you: Which phase of the new ADDIE do you think offers the biggest opportunity for your team? Let’s discuss in the comments. -------- Resources: Kirkpatrick Model vs. Phillips ROI Methodology in the Age of AI, "The AI-Enabled Learning Leader," xAPI and Learning Analytics. -------- #ADDIE #LearningAndDevelopment #AIinLearning #PerformanceSupport #InstructionalDesign

When Fortinet piloted IMMERSE with 75+ employees across three languages, they measured results using the Kirkpatrick Model - evaluating Reaction, Learning, Behavior, and Results. The outcomes across all four levels: 𝗘𝗻𝗴𝗮𝗴𝗲𝗺𝗲𝗻𝘁 & 𝗦𝗮𝘁𝗶𝘀𝗳𝗮𝗰𝘁𝗶𝗼𝗻: • 91% felt more confident speaking at work • 90% reported improved speaking skills and reduced anxiety • 100% wanted IMMERSE to become their permanent learning platform 𝗦𝗸𝗶𝗹𝗹𝘀 𝗗𝗲𝘃𝗲𝗹𝗼𝗽𝗺𝗲𝗻𝘁: • 88% improved in at least one skill (pronunciation, vocabulary, fluency, sentence complexity) • 98% said IMMERSE helped them learn more efficiently than other providers 𝗪𝗼𝗿𝗸𝗽𝗹𝗮𝗰𝗲 𝗔𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻: • 88% applied new language skills directly to their work • Real impact: "I traveled to São Paulo two weeks ago. I went from not speaking the language at all to having conversations with native speakers." 𝗕𝘂𝘀𝗶𝗻𝗲𝘀𝘀 𝗜𝗺𝗽𝗮𝗰𝘁: • 88% of learners save an estimated 15 minutes per day through more effective communication • 60 hours saved per employee annually • For 77 participants: 4,640 hours recovered—equivalent to 2.5 full-time employees • Estimated ROI: 3x return on investment Needless to say, when 88% of employees apply new skills at work within 90 days, the ROI stops being theoretical.