Measuring Employee Training Effectiveness

I want to show you a clever trick you didn't know before. Imagine you have six months' worth of data. You want to build a model, so you take the first five months to train it. Then, you use the last month to test it. This is a common approach for building machine learning models. Unfortunately, you may find out your model works well with the train data but sucks on the test data. Overfitting is not weird. We've all been there. But often, the worst you can do is try and fix it before understanding why it’s happening. Ask anyone about this, and they will give you their favorite step-by-step guide on regularizing a model. They will jump right in and try to fix overfitting. Don’t do this. There's a different way. A better way. Here is the question I want you to answer before you start racking your brain trying to fix a model: Do your test and training data come from the same distribution? When building a model, we assume the train and test come from the same place. Unfortunately, this is not always the case. Here is where the trick I promised comes in: 1. Put your train and test set together. 2. Get rid of the target column. 3. Create a new binary feature, and set every sample from your train set to 0 and every sample from the test set to 1. This feature will be the new target. Now, train a simple binary classification model on this new dataset. The goal of this model is to predict whether a sample comes from the train or the test split. The intuition behind this idea is simple: If all your data comes from the same distribution, this model won't work. But if the data comes from different distributions, the model will learn to separate it. After you build a model, you can use the ROC-AUC to evaluate it. If the AUC is close to 0.5, your model can't separate the samples. This means your training and test data come from the same distribution. If the AUC is closer to 1.0, your model learned to differentiate the samples. Your training and test data come from different distributions. This technique is called Adversarial Validation. It's a clever, fast way to determine whether two datasets come from the same source. If your splits come from different distributions, you won't get anywhere. You can't out-train bad data. But there's more! You can also use Adversarial Validation to identify where the problem is coming from: 1. Compute the importance of each feature. 2. Remove the most important one from the data. 3. Rebuild the adversarial model. 4. Recompute the ROC-AUC again. You can repeat this process until the ROC-AUC is close to 0.5 and the model can’t differentiate between training and test samples. Adversarial Validation is especially useful in production applications to identify distribution shifts. Low investment with a high return.

Kirkpatrick is often criticized. But rarely fully understood. Let's change this 👇 The model is simple. It describes four levels of evaluating learning impact: Level 1 — Reaction How participants experience the learning. Level 2 — Learning What knowledge and skills they acquire. Level 3 — Behavior How their on-the-job behavior changes. Level 4 — Results What organizational outcomes improve. That’s it. Four levels. And yet, it is frequently dismissed as outdated or simplistic. Why? Because we often treat it as a measurement checklist, instead of a design framework. Kirkpatrick is not just about evaluating training. It’s about thinking in cause-and-effect logic. Instead of asking, “Was the training good?” we should be asking a sequence of strategic questions. When designing: – What business outcome must change? – What behavior must shift to deliver that outcome? – What knowledge and skills are required? – What learning experience will enable mastery? And when evaluating: – How did participants evaluate the experience? – How well did they acquire the knowledge and skills? – How did behavior change at work? – What changed in the targeted business indicators? Planning must start from the top (Results). Measurement must begin from the bottom (Reaction). Think forward. Measure backward. Of course, the model has nuances - leading and lagging indicators, performance environment, manager accountability, isolation factors. But beneath the complexity lies a simple and powerful logic. The pyramid is not a hierarchy of surveys. It’s a chain of impact. That’s why I created this visual, to show the model not as theory, but as a practical thinking framework. How do you approach Kirkpatrick in your projects? #designforclarity #LearningAndDevelopment #InstructionalDesign #LearningStrategy #Kirkpatrick #LearningImpact #LXD #CorporateLearning

𝗪𝗵𝗮𝘁 𝗠𝗼𝘀𝘁 𝗢𝗿𝗴𝗮𝗻𝗶𝘀𝗮𝘁𝗶𝗼𝗻𝘀 𝗔𝘀𝗸 𝗟&𝗗 𝗧𝗼 𝗗𝗼 𝗩𝘀 𝗪𝗵𝗮𝘁 𝗛𝗶𝗴𝗵-𝗜𝗺𝗽𝗮𝗰𝘁 𝗟&𝗗 𝗔𝗰𝘁𝘂𝗮𝗹𝗹𝘆 𝗗𝗼𝗲𝘀 Most organisations still approach Learning & Development with a simple request: “Can you run a training on this?” And on the surface, it sounds reasonable. But the truth is… That question is often the beginning of the wrong solution. Because high-impact L&D does not start with training. It starts with clarity. ▶️ What exactly is not working? ▶️ Where is the performance breaking down? ▶️ Is it a skill gap… or something deeper? ▶️ What outcome are we trying to change? Most of the time, the issue is not: -Lack of knowledge -Lack of content -Lack of courses It is: -Misaligned expectations -Broken processes -Weak manager capability -No reinforcement after learning And this is where the role of L&D changes. From: 👉 Delivering programmes To: 👉 Diagnosing problems 👉 Challenging assumptions 👉 Recommending the right intervention (even if it is NOT training) 👉 Connecting learning directly to performance and business outcomes Sometimes, the most valuable thing L&D can do is pause and say: “Let us not jump to training yet.” Because real impact does not come from how many sessions you deliver. It comes from what actually changes after the learning. 👉 Do people behave differently? 👉 Do managers lead differently? 👉 Do results improve? If the answer is no… then learning did not happen. Only activity did. If you work in L&D, here is a simple reflection for you: The next time someone asks for training… Will you design a programme? Or will you diagnose the problem? Because that choice… quietly defines your career. What do you think? Is L&D still seen as a training function in your organisation… or is it evolving into a performance partner? #LearningAndDevelopment #LearningStrategy #WorkplaceLearning #TalentDevelopment #FutureOfWork

𝐒𝐭𝐫𝐞𝐬𝐬 𝐄𝐱𝐩𝐨𝐬𝐮𝐫𝐞 𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐚𝐭 𝐭𝐡𝐞 𝟐𝟎𝟐𝟑 𝐑𝐮𝐠𝐛𝐲 𝐖𝐨𝐫𝐥𝐝 𝐂𝐮𝐩 In this video, South African Rugby Union (SA Rugby) head coach Rassie Erasmus walks around playing the French national anthem during training. This was in preparation for the quarter-final match against the FFR - Fédération Française de Rugby— during the Rugby World Cup France 2023. This was to prepare the players for the loud noise and the home crowd, and is a form of stress exposure training. 𝗪𝗵𝗮𝘁 𝗶𝘀 𝗦𝘁𝗿𝗲𝘀𝘀 𝗘𝘅𝗽𝗼𝘀𝘂𝗿𝗲 𝗧𝗿𝗮𝗶𝗻𝗶𝗻𝗴? According to Driskell et al. (2008), this form of training involves exposing individuals to demands that may be present in a given task setting: ✅Noise ✅ Threat ✅ Time pressure ✅ Fatigue ✅ Other environmental demands 𝗪𝗵𝗮𝘁 𝗗𝗼𝗲𝘀 𝗦𝘁𝗿𝗲𝘀𝘀 𝗘𝘅𝗽𝗼𝘀𝘂𝗿𝗲 𝗧𝗿𝗮𝗶𝗻𝗶𝗻𝗴 𝗜𝗻𝘃𝗼𝗹𝘃𝗲? Driskell and Johnston (1998) stated that this form of training involves 3 distinct phases: 1️⃣ 𝙀𝙣𝙝𝙖𝙣𝙘𝙞𝙣𝙜 𝙁𝙖𝙢𝙞𝙡𝙞𝙖𝙧𝙞𝙩𝙮 𝙒𝙞𝙩𝙝 𝙩𝙝𝙚 𝙏𝙖𝙨𝙠 𝙀𝙣𝙫𝙞𝙧𝙤𝙣𝙢𝙚𝙣𝙩 During this part of training, individuals are provided with information on what stress is, common symptoms that people can experience when experiencing stress, and the effects of stress on performance in a pressurised setting. We 2️⃣ 𝙄𝙢𝙥𝙖𝙧𝙩 𝙃𝙞𝙜𝙝-𝙋𝙚𝙧𝙛𝙤𝙧𝙢𝙖𝙣𝙘𝙚 𝙎𝙠𝙞𝙡𝙡𝙨 During Phase 2, individuals learn the skills that are required in the specific task setting (i.e., rugby match, military, law enforcement). 3️⃣ 𝙋𝙧𝙖𝙘𝙩𝙞𝙘𝙚 𝙎𝙠𝙞𝙡𝙡𝙨 𝙖𝙣𝙙 𝘽𝙪𝙞𝙡𝙙 𝘾𝙤𝙣𝙛𝙞𝙙𝙚𝙣𝙘𝙚 Practice should involve exposure to realistic conditions to build the confidence of individuals. This is what we can see in the video. The South African team are training under noisy conditions to enhance their confidence at performing under such conditions. 𝗗𝗼𝗲𝘀 𝗦𝘁𝗿𝗲𝘀𝘀 𝗘𝘅𝗽𝗼𝘀𝘂𝗿𝗲 𝗧𝗿𝗮𝗶𝗻𝗶𝗻𝗴 𝗪𝗼𝗿𝗸? Driskell et al. (2001) found that stress exposure training enhanced performance during stressful settings and reduced stress perceptions. The Federal Law Enforcement Training Center (2004) has also used stress exposure training to simulate stress among law enforcement officers and improve decision-making under stress (Norris & Wollert, 2011). 𝗪𝗵𝗮𝘁 𝗮𝗯𝗼𝘂𝘁 𝗖𝗼𝗽𝗶𝗻𝗴 𝗦𝗸𝗶𝗹𝗹𝘀? In addition to exposing individuals to demanding situations, I think it's very important to teach coping strategies that can be deployed and practised under stressful training conditions because we know that coping is associated with performance across many sports (Nicholls et al., 2016). That is, give athletes the tools to be able to cope when they are exposed to different demanding environments and allow them time to practice these coping strategies because we know it can take time for coping to develop and become more effective (Nicholls, 2007).

Many teams overlook critical data issues and, in turn, waste precious time tweaking hyper-parameters and adjusting model architectures that don't address the root cause. Hidden problems within datasets are often the silent saboteurs, undermining model performance. To counter these inefficiencies, a systematic data-centric approach is needed. By systematically identifying quality issues, you can shift from guessing what's wrong with your data to taking informed, strategic actions. Creating a continuous feedback loop between your dataset and your model performance allows you to spend more time analyzing your data. This proactive approach helps detect and correct problems before they escalate into significant model failures. Here's a comprehensive four-step data quality feedback loop that you can adopt: Step One: Understand Your Model's Struggles Start by identifying where your model encounters challenges. Focus on hard samples in your dataset that consistently lead to errors. Step Two: Interpret Evaluation Results Analyze your evaluation results to discover patterns in errors and weaknesses in model performance. This step is vital for understanding where model improvement is most needed. Step Three: Identify Data Quality Issues Examine your data closely for quality issues such as labeling errors, class imbalances, and other biases influencing model performance. Step Four: Enhance Your Dataset Based on the insights gained from your exploration, begin cleaning, correcting, and enhancing your dataset. This improvement process is crucial for refining your model's accuracy and reliability. Further Learning: Dive Deeper into Data-Centric AI For those eager to delve deeper into this systematic approach, my Coursera course offers an opportunity to get hands-on with data-centric visual AI. You can audit the course for free and learn my process for building and curating better datasets. There's a link in the comments below—check it out and start transforming your data evaluation and improvement processes today. By adopting these steps and focusing on data quality, you can unlock your models' full potential and ensure they perform at their best. Remember, your model's power rests not just in its architecture but also in the quality of the data it learns from. #data #deeplearning #computervision #artificialintelligence

Measuring Success: How Competency-Based Assessments Can Accelerate Your Leadership If it’s you who feels stuck in your career despite putting in the effort. To help you gain measurable progress, one can use competency-based assessments to track skills development over time. 💢Why Competency-Based Assessments Matter: They provide measurable insights into where you stand, which areas you need improvement, and how to create a focused growth plan. This clarity can break through #career stagnation and ensure continuous development. 💡 Key Action Points: ⚜️Take Competency-Based Assessments: Track your skills and performance against defined standards. ⚜️Review Metrics Regularly: Ensure you’re making continuous progress in key areas. ⚜️Act on Feedback: Focus on areas that need development and take actionable steps for growth. 💢Recommended Assessments for Leadership Growth: For leaders looking to transition from Team Leader (TL) to Assistant Manager (AM) roles, here are some assessments that can help: 💥Hogan Leadership Assessment – Measures leadership potential, strengths, and areas for development. 💥Emotional Intelligence (EQ-i 2.0) – Evaluates emotional intelligence, crucial for leadership and collaboration. 💥DISC Personality Assessment – Focuses on behavior and communication styles, helping leaders understand team dynamics and improve collaboration. 💥Gallup CliftonStrengths – Identifies your top strengths and how to leverage them for leadership growth. 💥360-Degree Feedback Assessment – A holistic approach that gathers feedback from peers, managers, and subordinates to give you a well-rounded view of your leadership abilities. By using these tools, leaders can see where they excel and where they need development, providing a clear path toward promotion and career growth. Start tracking your progress with these competency-based assessments and unlock your full potential. #CompetencyAssessment #LeadershipGrowth #CareerDevelopment #LeadershipSkills

Training a Large Language Model (LLM) involves more than just scaling up data and compute. It requires a disciplined approach across multiple layers of the ML lifecycle to ensure performance, efficiency, safety, and adaptability. This visual framework outlines eight critical pillars necessary for successful LLM training, each with a defined workflow to guide implementation: 𝟭. 𝗛𝗶𝗴𝗵-𝗤𝘂𝗮𝗹𝗶𝘁𝘆 𝗗𝗮𝘁𝗮 𝗖𝘂𝗿𝗮𝘁𝗶𝗼𝗻: Use diverse, clean, and domain-relevant datasets. Deduplicate, normalize, filter low-quality samples, and tokenize effectively before formatting for training. 𝟮. 𝗦𝗰𝗮𝗹𝗮𝗯𝗹𝗲 𝗗𝗮𝘁𝗮 𝗣𝗿𝗲𝗽𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴: Design efficient preprocessing pipelines—tokenization consistency, padding, caching, and batch streaming to GPU must be optimized for scale. 𝟯. 𝗠𝗼𝗱𝗲𝗹 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 𝗗𝗲𝘀𝗶𝗴𝗻: Select architectures based on task requirements. Configure embeddings, attention heads, and regularization, and then conduct mock tests to validate the architectural choices. 𝟰. 𝗧𝗿𝗮𝗶𝗻𝗶𝗻𝗴 𝗦𝘁𝗮𝗯𝗶𝗹𝗶𝘁𝘆 and 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗮𝘁𝗶𝗼𝗻: Ensure convergence using techniques such as FP16 precision, gradient clipping, batch size tuning, and adaptive learning rate scheduling. Loss monitoring and checkpointing are crucial for long-running processes. 𝟱. 𝗖𝗼𝗺𝗽𝘂𝘁𝗲 & 𝗠𝗲𝗺𝗼𝗿𝘆 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗮𝘁𝗶𝗼𝗻: Leverage distributed training, efficient attention mechanisms, and pipeline parallelism. Profile usage, compress checkpoints, and enable auto-resume for robustness. 𝟲. 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 & 𝗩𝗮𝗹𝗶𝗱𝗮𝘁𝗶𝗼𝗻: Regularly evaluate using defined metrics and baseline comparisons. Test with few-shot prompts, review model outputs, and track performance metrics to prevent drift and overfitting. 𝟳. 𝗘𝘁𝗵𝗶𝗰𝗮𝗹 𝗮𝗻𝗱 𝗦𝗮𝗳𝗲𝘁𝘆 𝗖𝗵𝗲𝗰𝗸𝘀: Mitigate model risks by applying adversarial testing, output filtering, decoding constraints, and incorporating user feedback. Audit results to ensure responsible outputs. 🔸 𝟴. 𝗙𝗶𝗻𝗲-𝗧𝘂𝗻𝗶𝗻𝗴 & 𝗗𝗼𝗺𝗮𝗶𝗻 𝗔𝗱𝗮𝗽𝘁𝗮𝘁𝗶𝗼𝗻: Adapt models for specific domains using techniques like LoRA/PEFT and controlled learning rates. Monitor overfitting, evaluate continuously, and deploy with confidence. These principles form a unified blueprint for building robust, efficient, and production-ready LLMs—whether training from scratch or adapting pre-trained models.

Machine Learning students try more complex ML models when they wanna improve their results. So they miss the elephant in the room 🐘 ↓ A Machine Learning model is like a cake, with 2 main ingredients: → a dataset → an ML algorithm, for example, linear regression, or XGBoost. And the thing is, no matter what algorithm you choose, 𝘁𝗵𝗲 𝗿𝗲𝘀𝘂𝗹𝘁𝗶𝗻𝗴 𝗠𝗟 𝗺𝗼𝗱𝗲𝗹 𝗰𝗮𝗻 𝗼𝗻𝗹𝘆 𝗯𝗲 𝗮𝘀 𝗴𝗼𝗼𝗱 𝗮𝘀 𝘁𝗵𝗲 𝗱𝗮𝘁𝗮𝘀𝗲𝘁 𝘆𝗼𝘂 𝘂𝘀𝗲𝗱 𝘁𝗼 𝘁𝗿𝗮𝗶𝗻 𝗶𝘁. The problem is that in online courses, and ML competitions, you work with a 𝗳𝗶𝘅𝗲𝗱 dataset that someone has generated for you. In real-world projects, there is no dataset waiting for you. Instead, you need to 𝗰𝗿𝗲𝗮𝘁𝗲 it. And this is the most critical step in the whole project. Most ML problems in the real world are solved in a supervised manner, which means your dataset contains: → a collection of 𝗳𝗲𝗮𝘁𝘂𝗿𝗲𝘀, that serve as inputs to your model → a 𝘁𝗮𝗿𝗴𝗲𝘁 metric you want to predict, aka the model output. ✅ Useful features bring information and signal relevant to the target you want to predict. ❌ Useless features are just noise, and add no value to your ML model, no matter how complex your algorithm is. → Adding a useful feature to your model is the best way to improve it. 🏆 → Adding two useful features works even better. 🏆🏆 → And having 3 of them is a blessing. 🏆🏆🏆 To add new useful features, you need to → think beyond the data available right now at the data warehouse. → talk to senior colleagues who have context about the business. → think outside of the box you put yourself into after 3 weeks of working on the model. You often find pieces of information, relevant to the problem, that are scattered in the company's IT systems, or maybe outside on a third-party vendor, which will greatly help your model. 𝗧𝗼 𝘀𝘂𝗺 𝘂𝗽: → in real-world ML, the dataset is not set in stone. YOU have the power to expand it. → adding useful features to your dataset is the best way to improve your model. → improving ML models in the real world is more about data engineering than fancy ML models. ---- Hi there! It's Pau Labarta Bajo 👋 Every day I share free, hands-on content, on production-grade ML, to help you build real-world ML products. 𝗙𝗼𝗹𝗹𝗼𝘄 𝗺𝗲 and 𝗰𝗹𝗶𝗰𝗸 𝗼𝗻 𝘁𝗵𝗲 🔔 so you don't miss what's coming next #machinelearning #mlops #realworldml

Ever felt that post-workshop high? But you wonder if it translates to lasting change? Here's a 5 step process for real impact We've been there. You finish a workshop. Everyone leaves buzzing. Your feedback scores are through the roof. But was it... A "sugar rush" or "nutrient rich" experience? In the 21 years of running sessions in different contexts, I've realised there is a way to deliver energising workshops AND provide lasting value. → 𝗦𝘂𝗴𝗮𝗿 𝗥𝘂𝘀𝗵 𝗪𝗼𝗿𝗸𝘀𝗵𝗼𝗽𝘀 Participants leave excited. High feedback scores. Temporary motivation. No real change in behaviour. → 𝗡𝘂𝘁𝗿𝗶𝗲𝗻𝘁 𝗥𝗶𝗰𝗵 𝗖𝗼𝗻𝘀𝘂𝗹𝘁𝗮𝗻𝗰𝗶𝗲𝘀 Participants leave with a plan. Lower immediate excitement (perhaps). Content is processed. Lasting behaviour change. We want to the latter. And here's how: 1️⃣ SET THE CONTEXT ↳ Uncover challenges and hopes ahead of time. Meet people where they're at to unfold what happens next. 2️⃣ ENGAGE DEEPLY ↳ Ensure participants are not just passive listeners. Design for interactivity and cater of different styles 3️⃣ PLAN FOR ACTION ↳ Help them develop a concrete plan to implement what they've learned. Conduct debriefs. Give an action plan. 4️⃣ FOLLOW UP ↳ Provide post-workshop support and resources. Pre-design with the sponsor even if you're not involved in the implementation. 5️⃣ MEASURE IMPACT ↳ Go beyond feedback forms. Capture a baseline, collect evidence in sessions & track outcomes over time. Remember, the true measure of success is not how high your feedback scores are. It's the lasting impact you have on your participants. Let's move away from sugar-rush workshops and towards nutrient-rich consultancies. ~~ ✍️ What do you do to ensure your workshops have a lasting impact? ♻️ Reshare if you found this useful

Just read a groundbreaking paper on image retrieval model training that every computer vision practitioner should know about! "All You Need to Know About Training Image Retrieval Models" provides comprehensive insights into optimizing image retrieval systems - the backbone of visual search engines and content-based recommendations we use daily. The researchers conducted tens of thousands of training runs to analyze how various factors impact retrieval accuracy across multiple datasets (Cars196, CUB-200-2011, iNaturalist 2018, and Stanford Online Products). Key technical findings: - Model architecture: DINO-v2's CLS features outperform other architectures - Optimization: Adam optimizer with 1e-6 learning rate yields best results when fine-tuning all layers - Loss functions: Two distinct categories perform differently based on resources:  -- High-resource settings: Contrastive losses (ThresholdConsistentMargin, Multi-Similarity) with online miners excel with larger batch sizes (256+)  -- Resource-constrained: Classification losses (CosFace, ArcFace) perform better with smaller batches - Batch composition: For contrastive losses, 2-4 images per class works best; for classification losses, 1 image per class is optimal - Learning rate tuning: Critical to set separate learning rates for model (1e-6) and classifier (around 1.0) - using the same rate for both can cause 10%+ accuracy drops - Feature dimensionality: Direct use of CLS token (768-dimensional for DINO-v2-base) achieves optimal results - Dataset strategy: All metric learning losses are robust to annotation errors, suggesting resources are better spent collecting more data than ensuring perfect labeling The paper provides practical guidance for balancing accuracy, computational resources, and data annotation strategies in image retrieval systems. Kudos to the researchers from Polytechnic of Turin and Setta.dev for this valuable contribution to the field!