You and your organization invested time and money to design, implement and conduct training for a group of employees. Now comes the hard part: Determining whether the program had any impact on the business. At this point, many training managers break into a cold sweat. A business, not to mention the world, is complex. Any number of variables can affect outcomes. A training program is just one of them. But there's no reason to panic as there are many tools to help the training manager and stakeholders determine whether a program was worth the investment. One key challenge, and the focus of this post, is isolating the impact of the learning. This involves understanding the goals of a program and what other factors may be influencing the result. For example, a training session on sales techniques may happen to correlate with a 36% increase in sales. The training, of course, may have been the reason. But for a complete analysis, you'll want to investigate what was happening with the economy, business rivals, your own firm's product offerings and even the weather. For the sake of argument, let's say the economy remained stagnant, your product mix was unchanged and the weather was not a factor for the period. You may also find no change in business at the rival firm. If this is the case, you can examine the impact of your training with more confidence. Even if you identify changes in other variables, you can isolate these, too. The job is generally easier if you have quantitative data available, such as the percentage decline in overall business activity. There are a number of analysis techniques for isolating the impact of training, some of which require advance planning. My three favorites are: 1. Plotting a trend line that illustrates how sales changed before and after the training. You can plot other lines as well, including economic stagnation or even the competitor's sales collapse. 2. Using a control group. This takes some advance planning and involves not training a subset of employees. After the program is finished, you can compare results. 3. Using matched pairs. This involves comparing the performance of each individual before and after the training. If there are significant differences, you can say that the training had measurable impact. Of course, you will likely encounter training programs where isolating business impact is impossible or not appropriate. In one of my books on training evaluation, I used the example of machine operators who were trained on quality-improvement techniques. They not only learned about working as teams but also how to use a document-management system. Their incentive program was revised as well. All of these components form a single, integrated change program. Another example: Training the sales team on a new customer relationship management (CRM) system. In this case, it makes no sense to measure the effect of training versus the impact of the new CRM system. Why? -precisely because employees can't apply the training without the new CRM being installed. If you're having no luck finding quantitative numbers to measure the impact of training, you can always ask stakeholders to estimate the effect of the program. This is not ideal and introduces a significant subjective element, but the information may be all that's available to do any analysis at all.
2 Responses
Conservative estimating?
When the effect is not easily quantifiable, I recommend Kent Barnett’s and Jeffery Berk’s conservative estimating technique (source: Barnett, K., Berk, J. (2007) Human Capital Analytics). By asking the participants to (a) estimate the change (e.g. productivity), (b) estimate the effect due specifically to the training event, (c) estimate how confident they are in their own estimate, and then (d) assign a cost to the overall estimate, it is possible to obtain enough data for ROI. It can be a simple as (a)x(b)x(c)x(d) and you may be surprised at what you can ‘conservatively’ quantify.
-or you could always try this ‘expert’ approach:
Step 1
Step 2
Step 3
When to use ROI estimates
Hello Martin,
I love the Dilbert cartoons. I’m very wary of using personal estimates in quantifying benefits and ROI. If we need to rely on personal estimates, my first question is always: "Why is there no data?" Was no baseline set? Are there no measures? Was the training program design short on quantifiable objectives? All of these are signals that we need to do better next time when we do our TNA and measurement plan (if one existed in the first place).
Les Allan Author: Training Evaluation Toolkit
Twitter: http://www.businessperform.com/twitter