Data-Driven Training Evaluation

The DOJ consistently says that compliance programs should be effective, data-driven, and focused on whether employees are actually learning. Yet... The standard training "data" is literally just completion data! Imagine if I asked a revenue leader how their sales team was doing and the leader said, "100% of our sales reps came to work today." I'd be furious! How can I assess effectiveness if all I have is an attendance list? Compliance leaders I chat with want to move to a data-driven approach but change management is hard, especially with clunky tech. Plus, it's tricky to know where to start– you often can't go from 0 to 60 in a quarter. In case this serves as inspiration, here are a few things Ethena customers are doing to make their compliance programs data-driven and learning-focused: 1. Employee-driven learning: One customer is asking, at the beginning of their code of conduct training, "Which topic do you want to learn more about?" and then offering a list. Employees get different training based on their selection...and no, "No training pls!" is not an option. The compliance team gets to see what issues are top of mind and then they can focus on those topics throughout the year. 2. Targeted training: Another customer is asking, "How confident are you raising bribery concerns in your team," and then analyzing the data based on department and country. They've identified the top 10 teams they are focusing their ABAC training and communications on, because prioritization is key. You don't need to move from the traditional, completion-focused model to a data-driven program all at once. But take incremental steps to layer on data that surfaces risks and lets you prioritize your efforts. And your vendor should be your thought partner, not the obstacle, in this journey! I've seen Ethena's team work magic in terms of navigating concerns like PII and LMS limitations – it can be done!

𝐓𝐡𝐞 𝐒𝐞𝐜𝐫𝐞𝐭 𝐭𝐨 𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐓𝐡𝐚𝐭 𝐀𝐜𝐭𝐮𝐚𝐥𝐥𝐲 𝐖𝐨𝐫𝐤𝐬? 𝐒𝐭𝐚𝐫𝐭 𝐚𝐭 𝐭𝐡𝐞 𝐄𝐧𝐝. 🏁 I used to think my job as an L&D professional started with a syllabus. I was wrong. Recently, I was tasked with building a learning solution for our Talent Acquisition (TA) team. The goal wasn’t just to "train recruiters"—it was to solve a business problem. Instead of looking at what they needed to know (Level 2), I started with what the business needed to achieve (Kirkpatrick Level 4). The "Reverse" Approach I didn’t start with slides. I started by analyzing Voice of the Customer (VOC) survey results, focusing on various metrics from both Hiring Managers and Candidates. Working Backwards: ✅ Level 4 (Results): I defined the business KPI. ✅ Level 3 (Behavior): Based on the VOC metrics, I identified the specific actions recruiters needed to change—specifically around "Precision Intake" and "Candidate Experience Management." ✅ Level 2 & 1 (Learning & Reaction): Only then did I design the actual training content that addressed those specific behavior gaps. The Result? The training didn't feel like a chore; it felt like a solution. Because I built it based on the actual metrics revealed in the VOC surveys, the TA team saw immediate value, and the business saw a measurable shift in hiring efficiency. The Lesson: If you want your learning solutions to be more than just "check-the-box" exercises, stop asking "What should we teach?" and start asking "What does the data say I need to solve?" How do you use VOC data to shape your enablement programs? 👇 #LearningAndDevelopment #InstructionalDesign #TalentAcquisition #KirkpatrickModel #Enablement #DataDrivenLD #BusinessImpact

From chatbots that personalize microlearning to systems that predict who’s likely to disengage, artificial intelligence (AI) is changing how we train and learn. AI opens new opportunities to improve on some of the challenges with traditional training models such as scalability, personalization and real-time feedback. Core AI applications in the L&D space can be broken down into four categories: Artificial Intelligence (AI) Platforms: These tools tailor difficulty, pacing and topics in real time. An AI-enhanced platform can tailor the content to the learner based on their performance trends. Natural Language Tools: These are used to summarize content, create quizzes and provide conversational coaching. These applications can reduce time spent on administrative tasks and increase the focus on building relationships and delivering value. Predictive Analytics: This category of tools help learning leaders identify skills gaps and forecast learner success. Virtual Coaches and Chatbots: These tools reinforce knowledge through spaced repetition and feedback loops. AI-Powered Learning: A Case Study Streamline Services is a fifth-generation plumbing, electrical and HVAC company that handles up to 200 calls a day and serves thousands of customers each month. The company is using AI to not only coach employees but also identify areas where the team needs skills development or training. Streamline adopted an AI-powered virtual ride along platform to help transform everyday customer interactions — both in the field and in the call center — into powerful, data-driven learning opportunities. Traditionally, managers and trainers could only coach based on a handful of ride alongs or recorded calls each month. With AI, every service visit and customer conversation has become searchable, analyzable and coachable. AI highlights key themes including customer concerns, missed opportunities and tone shifts, allowing trainers to see real patterns instead of isolated incidents. The training team and managers use this knowledge to design training and structure coaching for individual needs. Because AI is deepening Streamline’s understanding of customer needs, the L&D team can develop targeted training that improves customer service and empathy across the company. Streamline’s experience illustrates how AI is fundamentally changing the learning process — from reactive coaching based on limited observation to proactive, personalized development powered by real data. This case study showcases how technology can elevate human performance rather than replace it. AI offers the ability to provide more learning opportunities and personalized learning across roles and industries. L&D professionals need to embrace this change and evolve alongside the technology. The future of learning isn’t artificial — it’s intelligently human. #LearningandDevelopment #AI #FutureofLearning

I was reviewing quarterly reports with a client last month when they asked me a question that stopped me in my tracks: "Scott, we have all this learning data, but I still don't know which programs are actually improving performance." After 12 years as CEO of Continu, I've seen firsthand how organizations struggle with this exact problem. You're collecting mountains of learning data, but traditional analytics only tell you what happened - not why it matters. Here's what we've learned working with thousands of organizations: The real value isn't in completion rates or assessment scores. It's in the connections between those data points that remain invisible without the power of tools like AI. One of our financial services clients was tracking 14 different metrics across their onboarding program. Despite all that data, they couldn't explain why certain regions consistently outperformed others. When we implemented our AI analytics engine, the answer emerged within days: specific learning sequences created knowledge gaps that weren't visible in their traditional reports. This isn't just about better reporting - it's about actionable intelligence: - AI identifies which learning experiences actually drive on-the-job performance - It spots engagement patterns before completion rates drop - It recognizes content effectiveness across different learning styles Most importantly, it connects learning directly to business outcomes - the holy grail for any L&D leader trying to demonstrate ROI. What's your biggest challenge with learning data? Are you getting the insights you need or just more reports to review? #LearningAnalytics #AIinELearning #WorkforceDevelopment #DataDrivenLearning

Training without measurement is like running blind—you might be moving, but are you heading in the right direction? Our Learning and Development (L&D)/ Training programs must be backed by data to drive business impact. Tracking key performance indicators ensures that training is not just happening but actually making a difference. What questions can we ask to ensure that we are getting the measurements we need to demonstrate a course's value? ✅ Alignment Always ✅ How is this course aligned with the business? How SHOULD it impact the business outcomes? (i.e., more sales, reduced risk, speed, or efficiency) Do we have access to performance metrics that show this information? ✅ Getting to Good ✅ What is the goal we are trying to achieve? Are we creating more empathetic managers? Creating better communicators? Reducing the time to competency of our front line? ✅ Needed Knowledge ✅ Do we know what they know right now? Should we conduct a pre and post-assessment of knowledge, skills, or abilities? ✅ Data Discovery ✅ Where is the performance data stored? Who has access to it? Can automated reports be sent to the team monthly to determine the impact of the training? We all know the standard metrics - participation, completion, satisfaction - but let's go beyond the basics. Measuring learning isn’t about checking a box—it’s about ensuring training works. What questions do you ask - to get the data you need - to prove your work has an awesome impact?? Let’s discuss! 👇 #LearningMetrics #TrainingEffectiveness #TalentDevelopment #ContinuousLearning #WorkplaceAnalytics #LeadershipDevelopment #BusinessGrowth #LeadershipTraining #TalentDevelopment #LearningAndDevelopment #TalentManagement #Training #OrganizationalDevelopment

❗ Only 12% of employees apply new skills learned in L&D programs to their jobs (HBR).  ❗ Are you confident that your Learning and Development initiatives are part of that 12%? And do you have the data to back it up?  ❗ L&D professionals who can track the business results of their programs report having a higher satisfaction with their services, more executive support and continued and increased resources for L&D investments.    Learning is always specific to each employee and requires personal context. Evaluating training effectiveness shows you how useful your current training offerings are and how you can improve them in the future. What’s more, effective training leads to higher employee performance and satisfaction, boosts team morale, and increases your return on investment (ROI). As a business, you’re investing valuable resources in your training programs, so it’s imperative that you regularly identify what’s working, what’s not, why, and how to keep improving. To identify the Right Employee Training Metrics for Your Training Program, here are a few important pointers: ✅ Consult with key stakeholders – before development, on the metrics they care about. Make sure to use your L&D expertise to inform your collaboration. ✅Avoid using L&D jargon when collaborating with stakeholders – Modify your language to suit the audience. ✅Determine the value of measuring the effectiveness of a training program. It takes effort to evaluate training effectiveness, and those that support key strategic outcomes should be the focus of your training metrics. ✅Avoid highlighting low-level metrics, such as enrollment and completion rates. 9 Examples of Commonly Used Training Metrics and L&D Metrics 📌 Completion Rates: The percentage of employees who successfully complete the training program. 📌Knowledge Retention: Measured through pre- and post-training assessments to evaluate how much information participants have retained. 📌Skill Improvement: Assessed through practical tests or simulations to determine how effectively the training has improved specific skills. 📌Behavioral Changes: Observing changes in employee behavior in the workplace that can be attributed to the training. 📌Employee Engagement: Employee feedback and surveys post-training to assess their engagement and satisfaction with the training. 📌Return on Investment (ROI): Calculating the financial return on investment from the training, considering costs vs. benefits. 📌Application of Skills: Evaluating how effectively employees are applying new skills or knowledge in their day-to-day work. 📌Training Cost per Employee: Calculating the total cost of training per participant. 📌Employee Turnover Rates: Assessing whether the training has an impact on employee retention and turnover rates. Let's discuss in comments which training metrics are you using and your experience of using it. #MeetaMeraki #Trainingeffectiveness

Training isn’t the goal. Impact is ⬇️ Training doesn’t end with the session. It ends with results. Most companies track training attendance. But few measure what really matters, impact. The Kirkpatrick-Phillips Model helps you do just that. It moves beyond completion rates to ask: Did learning change behaviour? Did it drive results? Was it worth the investment? Here’s how the 5 levels break down: ✅ Level 1 – Reaction ↳ Was the training relevant, engaging, and useful? ✅ Level 2 – Learning ↳ Did participants gain new knowledge or skills? ✅ Level 3 – Behaviour ↳ Are they applying what they learned on the job? ✅ Level 4 – Results ↳ Are we seeing improvements in performance, productivity, or quality? ✅ Level 5 – ROI ↳ Did the business gain more value than it spent? To apply this model well: Start with the end in mind ↳ Define clear business outcomes before designing training. Link each level ↳ Show how learning leads to behavioural change and how that drives results. Use real data ↳ Track both qualitative and quantitative outcomes across all five levels. Involve managers ↳ Bring them into the process early, they’re key to learning transfer. Be selective and focused ↳ Avoid tracking everything. Focus on what truly moves the needle. Tell a clear story ↳ Use the data to tell a results-focused narrative that shows the full value of training. 🧠 Remember: Great training isn’t just delivered. It’s measured, proven, and improved over time. Which level do you think L&D teams struggle with the most? -------------------------- ♻️ Repost to help others in your network. ➕ And follow me at Sean McPheat for more.

We love to focus on models and algorithms, but data quality makes the real difference when training LLMs! Here’s a practical guide for debugging your LLM’s training dataset… Developing an LLM. When training an LLM, we follow an iterative, two-step process: 1. Train our model 2. Evaluate our model Each time we train a new model, we perform some intervention. Usually, this intervention is data-related. We keep everything the same, tweak our data, and see if performance improves. Data curation strategies. There are two ways we can approach tweaking our data: - Data-focused curation: directly look at the data and analyze its properties to find (and debug) existing issues. - Model-focused curation: train an LLM over our data, find issues in its output and use these issues to find corresponding problems in the data. Data-focused curation does not require training a model, which makes it useful in the early phases of developing an LLM. But, we should use both of these strategies in tandem. Data-focused curation. To gain a deep understanding of our data, we need to start with manual inpsection. Although this process is tedious, it’s extremely important and done by all effective researchers–the more data you manually inspect the better. As we inspect, we will begin to notice—and fix in some cases—issues and patterns in our data. To scale this curation process beyond our own judgement, however, we use automated techniques based either upon heuristics or other machine learning models (e.g., fastText models or LLM-as-a-Judge-style models). Model-focused curation. Once we have started training LLMs over our data, we can use these LLMs to debug issues in the dataset. The idea of model-focused curation is simple, we just: - Identify problematic or incorrect outputs produced by our model. - Search for instances of training data that may lead to these outputs. The identification of problematic outputs is handled through our evaluation system. We can have humans (even ourselves!) identify poor outputs via manual inspection or efficiently find low-scoring outputs via our automatic evaluation setup. OLMoTrace. Once problematic outputs are identified, we can use standard search techniques to match outputs to training data. However, researchers have developed specialized techniques for this purpose as well. For example, OLMoTrace uses a specialized span matching algorithm to efficiently trace model outputs over pre-training-scale data.