Learn Smart

A Quantitative Approach to Learning Impact Measurement

As companies are increasingly attentive to their bottom lines, and demand evidence-based approaches to impact measurement of investment, learning development has been no exception to the trend. Executives aren’t satisfied with positive participant reactions, and survey-based approaches are viewed with increasing skepticism as selection, response bias, and other issues with surveys and polling create an impression of unreliability. Learning goals of ‘increasing awareness’ are viewed as being nearly synonymous with ‘wasting time.’ Whether the learning program is about company diversity or how to safely use a tool, the first question many learning content creators, learning managers, and learning staff hear is “What is the business impact of this?” Unfortunately, many of the standards for measuring    learning impact have not kept pace with the rest of how businesses measure and retain data in nearly every other aspect of operations.

From Client/Customer Relationship Management through market analysis, to Just in Time logistics, and even into Human Resources screening and selection processes, businesses have increasingly moved to a data-driven mentality. Measuring everything from when sales are made to how many frames of TV time a logo gets during a sporting event, data has become not just how business store their history, but how they measure impact of nearly everything they do. It is time for learning to do the same, so business can brag not just about how much learning they make available to their employees, or how many hours their employees spend trying to get smarter every year, but so they can relay to investors the actual value returned on their learning investments. More consulting firms now than ever are operating to try to help companies   measure their learning program impact… and more consulting firms now than ever are doing it wrong.

When Alexander Pope wrote ‘A little learning is a dangerous thing; drink deep, or taste not the   Pierian spring,’ he probably wasn’t considering just how easily people would confuse correlation with  causality, nor was he considering how to properly instrument a variable or construct a time-series      analysis. Yet, here we are, in a world with highly-paid consultants peddling their digital snake-oil in the form of analytics that they don’t understand. Many of these companies claiming to measure impact will throw the data into an Ordinary Least Squares model that isn’t customized to the task at hand, which measures naught but correlation, and declare that they have measured impact. Others, who have just a little learning will follow up by asking about R-squared values… all while missing the fact that correlation tautologically cannot measure impact (causality). In their zeal to move past surveys, which largely   measure the quality of donuts served at the learning event more than the quality of learning itself, many learning experts have been lulled into a false sense of security by people who tell them that analytics and data science are easy. Bad analytics are easy. Throwing data into a pre-existing model without even bothering to check for data quality takes no time at all, and very little effort. Good analytics require an ongoing commitment, but once the pieces are in place can offer not only analysis of existing learning, but also suggestions on how to make existing learning better and what learning may be needed in the future. 

Here’s How:

 Foundation: Operational and Transactional Reporting

                 How do we count this? It seems like an easy question. We count things every day. How long until my next meeting? How many pages will this take if I print it? Counting seems like it should be one of the easiest parts of the process, which is why so many people forget just how much it needs governance, and how easy it is to get wrong. Is this four-part class one item, or is it actually four separate learning events? Are we measuring time in hours, or in minutes? Is it possible for a learning item to take zero time to consume? Do we actually know who attended this    learning event, or just how many people were in the room? Did all of these people actually show up, or did they just enroll? If they showed up for three of the four parts, is it still complete? These are just the tip of the iceberg when it comes to questions your company will face when answering the seemingly simple question of how to track learning. Without good data governance so you can know who attended which courses and when, none of the rest of the analytics will yield mathematically valid results.

The pyramid above lists out the basis for all of the rest as “Operational/Transactional Reporting.” In simple terms, that data is the foundation for all that follows. Though it will (as part of other steps) be combined with other data, depending on which impacts you’re trying to measure, it is the linchpin to the entire process. A key part of developing governance surrounding this step is a vision of the future and consideration of scalability. Though storing spreadsheets on instructors laptops may seem like a viable option for a small or medium-sized business, a centralized governance, storage, and maintenance is the only way to ensure that your company will be ready to continue the same process no matter how successful (and big) your company becomes over time. Regardless of which database your company chooses, it is very important to distinguish a database (S/4HANA, SQL/Access, Oracle, Sybase, etc.) from a spreadsheet (Excel, SmartSheet, etc.) and utilize people who understand how to build a database in a scalable, efficient manner that not only meets the business needs now, but will be able to interface with your other systems (CRM, HR data, etc.). This is the time to develop a comprehensive strategy,  coming back later to change which database you use will not only be time-consuming, but expensive. Bringing in SME’s to develop a strategy now may seem expensive, especially for a small business, but the cost of coming back to revisit these decisions later will be exponentially more expensive.

This is also the time to employ the metrics-oriented-mindset amongst learning content creators on your team. Every learning program should start with the same question “What metric(s) are we trying to move, and in what direction?” Vague or ethereal statements like “understanding,” “visibility,” or “awareness” are answers that might have been acceptable before, but they no longer work. Your diversity program should seek to influence measurable behavior, whether it’s reducing the number of claims filed, or the number of HR complaints, the key is that it should be something measurable. Whether it’s increasing sales, reducing wait-times, or reducing worker’s compensation claims, if its regularly measured, and that data is stored somewhere, we’ll be able to measure the impact. If the goal can’t be measured, neither can the outcome, which will raise the question amongst stakeholders as to why the content should be developed/invested in at all.

Initial Implementation: Advanced Reporting

                 Now that the learning data is all in one place, we can start             combining it with other data to answer some basic questions. Have all of the people who have jobs that entail handling hazardous materials taken the most current version of the training for that? Are employees outside of that group taking the same learning? How long does the average employee spend on learning? Do managers spend more time learning than other     employees? Combining these two data sets, so we know not only who took learning, but we know more about the kind of person who took the learning (job title, manager type/level, geographic location, sub-business, etc.), gives us the ability to answer some of the most fundamental learning    questions (How much? When? Where?). These answers and descriptors will become invaluable as we move from reporting into analytics.

Why do so many businesses stop at this step? For many of the    consulting firms, and many businesses, this is the first, last, and only step. As mentioned earlier, doing bad analytics is easy, and doesn’t take nearly as much time. Companies will look at baseline information for a target group (e.g. Sales), deliver training, then either survey participants to ask them if the training was helpful, or they’ll re-sample the baseline and simply attribute the change to the training. This fundamentally fails to account for several different factors, including:

· Was the participant group representative of the target population at large?

· Was the participant group already trending up/down prior to the training?

· Did participants also participate in other learning events that could have been responsible?

· Are the results from training, or just a changing economy on the outside?

· Is the sample size large enough to be statistically significant?

Depending on your organizational structure and size, this may be the point where you want to consider either hiring or contracting a Statistician, Econometrician, or Data Scientist. Moving forward to the advanced analytics steps, it becomes increasingly easy to make mistakes in data handling and interpretation. There is, unfortunately, often a high degree of pressure from the business to “show results,” so take great care to find someone with a strong enough sense of integrity that they’re willing to say “the data don’t support any conclusions at this time” when such is the case.

Intermediate Implementation: Advanced Analytics

So how do we know if an outcome is really correlated with having participated in a learning experience? We bring in even more data. By testing for statistical significance (and including those that are significant) of control variables, from outside economic conditions through gender, or things like time in company, we can account for the impact those things are having, and account for as much as possible when assessing both whether participation is correlated with a  desired outcome, and what magnitude of metric change is associated with that participation. The two most common tests we can do at this stage of analytics are A|B tests and Ordinary Least Squares (Correlative) Analysis.

An A|B test means that there’s one group of people who participated in the learning experience, and a separate group of people who did not participate in that learning experience, but the two groups are as similar as possible other than that one event (similar gender composition, time in company, manager levels/composition, etc., similar baseline metrics, similar learning participation other than that single learning event for which we are testing). By comparing what happens with the participant group with what happened with the non-participant (control) group, we can determine what changes are likely related to participating in the learning event. If we see a metric go up for both groups, it’s reasonable to say that the change wasn’t associated with the event, but when we see a change in one group and not the other, we can assert that there’s a relationship that may warrant further investigation.

The other type of test, an OLS-test, can also tell us some about what outcomes are correlated with participation in a learning event, but again, it cannot speak to causality. An OLS model has one variable (the outcome for which we’re testing) on the left, we’ll call it ‘Y’, then on the right, we have all the variables we’ve tested and found to be significant, we’ll call them X1, X2, and X3 for this example. The model will assign values to the different magnitudes to which those variables affect the outcome, so what we’ll have is a model that looks like: Y = AX1+BX2+CX3+D. To make this a bit more understandable for those not mathematically inclined, lets say we make widgets. We’ll say X1 is our variable for whether or not a person participated in the learning event, X2 is going to be defined as salary, and X3will be time in their current job. In easier terms, the formula means: Number of Widgets produced = A*class participation+ B*salary+ C*time in position+D. A, B, and C tell us the magnitude to which each of those variables affects the number of widgets produced, and D tells us the base number that we would expect to be produced regardless of those other things.

 Advanced Implementation: Predictive Analytics

 As we shift from correlation to causal relationships, definitively stating that an outcome is the result of participation in a learning event, it becomes more necessary that these systems be able to work with each other in a more integrated manner. This integration allows for significant efficiency gains and time savings, but also allows for better recommendation engines (to be discussed later in the page) to be built.

By having all of the performance data over time, including historic data, all of the HR variables like gender, age, time in company, region, all of the other internal data, and all of the external data scraped or otherwise automated for intake, we can now build models that tell us historically how people were doing, how they were trending, how the variables all interact with each other, and we can quantify the actual impacts of participating in the learning program. This process, called Time-Series Analysis, also allows us to model what would happen if more people participated, or the program was made available to other similar audiences. When applying for learning awards or other recognition, this is the level of mathematical rigor that is needed to be considered proof that your program actually did/does what you claim it does. Beyond that, when you’re at this level of analytics, you can quantify those returns in dollar values. Taking this program leads to Y% increase in widget-making productivity, which means the company got Z dollars in returns. Quantifying returns on learning investments in an objective, data-backed manner is what turns perceptions of learning organizations from cost-centers to investments. Additionally, depending on the data, we can measure clustered effects, to see not only how a class affects the participant, but how it may be affecting their peers or subordinates.

This level of data integration can also, with the help of a qualified Data Scientist, experienced in building recommendation engines, enable you to build systems that not only demonstrate the efficacy of your programs, but help employees find the programs that will most impact people like them. If an employee is a widget maker, it doesn’t necessarily follow that their end-goal is to be a widget maker,  maybe they want to be a salesperson. What courses should an employee take that would help them   transition to the salesperson job? Does the order in which they take those courses matter? By using a fully– integrated data set, we can make useful recommendations to employees to help them not only do better in their current role, but to develop into other roles.

 Operational/ Run Phase : Prescriptive Analytics

Predictive Analytics tells us what will happen if we provide a scenario. Prescriptive analytics tells us what’s going to happen on the current course, and helps us understand what we should do about it. Let’s say in our hypothetical widget factory, every widget needs sprockets. 8 years ago we had 10 sprocket makers, 7 years ago, we had 9, 6 years ago 8, and so on. Now we’re down to 2 sprocket makers. They’ve gotten pretty good over the years, so everything is ok right now, but a good prescriptive system would let us know that maybe it’s time to develop a sprocket making course, so we have some people with the needed skills on hand, just in case the trend of losing one sprocket maker a year continues.

The data-driven decision-making part is still up to the managers in charge; it is they who will be choosing whether to hire sprocket makers from the outside, gauge interest in sprocket making from search histories in the recommender from the previous step, or make product changes such that the sprockets aren’t necessary at all any more. That said, the prescriptive analytics systems can let them know that there’s an issue ahead, and the decisions can be made about how to avoid those issues, keeping your company at peak efficiency.

Then comes the not-so-secret part of analytics… it never ends. The factory is up and running, some people are taking the sprocket making class, and now we want to know: “who took the class?”, “were they managers or assembly line people?”, “was participation in the class associated with positive outcomes?”, “did taking the class result in financial gains for the company?”, and lastly… the most important question of them all: What’s next?

About the Author

Kenneth Lord is a Data Scientist currently working for SAP’s Learning Center of Excellence, he has led numerous learning impact studies, and projects to help determine learning modality effectiveness and automatically tailor learning experiences to the modalities that work best for an individual learner. Kenneth holds an MA in Economics from the City University of New York.