9 Responses

1. How?
   Also let me add the supplementary question of “How?”

2. defence vs offence
   Martin, you pose probably the greatest unanswered question in this business. Defensive training, that which is realted to (say) compliance or some form of regulation is a given; time should not be invested in even attempting any kind of ROI calc because it simply isn’t relevant. However for pretty much all other stuff, some attempt should be made to demonstrate that value is being derived from the spend. The big question is – can this be satisfied by ONLY going as far as Level 3, or should you push this just one stage further?

3. When there’s benefit
   You should conduct an evaluation at level 3 and 4 when there’s a genuine benefit of doing so.

   There’s not always an ROI on statutory training. There’s not always an ROI on preventative training – you might be able to guesstimate what you have saved and then turn it into ROI but where’s the value in that.

   Otherwise ROI should be calculated at some point into a programme after the initial evaluations have taken place and shown to have some benefit.

   If after all everyone felt that a course was lousy and none of them put the skills into place for whatever reason it’s easy enough to assume that your ROI was nothing.

   But it’s also important if at all possible to time your intervention to measure ROI, if it’s a company wide roll out you might not want to wait until all x-thousand staff have been trained (because if there’s no real value you’ll have wasted a lot of money) but you might want more than a handful trained (so you can gauge relative success in different areas of the business).

   It’s a great question this one and I’d love to hear the end results not just of this debate but also of a simple “rule of thumb” calculation for ROI which allows you to make the decision whether to invest further time and resource in calculating the accurate bottom line benefits of training.

4. Some more thoughts…
   Some great comments – thank you!

   Nik also suggests that it would be useful to have a rule of thumb to help decide when to do a full-blown L4 or L5/ROI calculation.

   The general wisdom espoused by Jack Phillips et al is to only ROI/L4 some 5-10% of programmes, specifically those that are expensive, public, political, and/or where the consequences of failure may be expensive (as opposed to physically dangerous).

5. Isolating effects of training
   Martin
   Well, you are not asking much are you! But here goes.
   When and why – like you have said, it should be when the training (or development) is expensive, high profile or when investigation may highlight how to further add value. In a sense we need to make a judgement upfront on whether the return on investment in the evaluation will be worth it. I liken it to a set of gears. Pretty much all training should be subject to some basic evaluation. But we should be able to go up a gear, and then up another gear, as necessary. Maybe only 10% will be subject to top gear at any one time but I would be reluctant to suggest that a particular percentage be typical or a target.
   How – (how long have you got?) as you know there is no easy answer but we do a number of things like predicting desired results, using control groups, getting participant, line manager and senior manager estimations, and use trend line analysis. On the calculatory side there is calculating the economic value of staff, the cost of preformance deficit (the cost of performing below par), the cost of not training, and the identification of tangible and non-tangible benefits.
   For some forms of training that are interwoven in to a change initiative or are as part of a wide OD intervention, then isolating the training component is too hard. Evalauting the whole is probably more sensible.
   Isolating specific effects, I think, is most useful not as part of some post-event justification but as part of ongoing learning within training. If we can learn to have more of the right effect, more often then maybe one day no one will be interested in evaluation. The impact will be self evident.
   Well, we can dream.
   Hope this helps contribute to the debate.
   Graham

6. What’s the isolated impact of a light bulb?
   Hi,

   I am glad you ask this question, Martin. I find this aspect (isolating the impact) of the Phillips ROI methodology to be highly questionable. The fact that it is presented as one of the “Guiding Principles” in such terms as “at least one method must be used to isolate…” doesn’t make it any easier for me to swallow…

   In fact, if pressed to choose between “you must isolate” and “don’t ever isolate”, I would go for the later!

   One of the key problems that corporate training functions have faced (and still do) is precisely its isolation from other strategic business functions. If our ambition is to truly partner with these other functions in achieving meaningful business goals, attempts at disaggregated value calculations may well play us a disservice.

   The challenge is to convincingly show that training was one of the key contributors (along with others) in achieving the goals and that without training it most probably would have failed. Intricate confidence level factors and exact (isolated) ROI figures probably invite more suspicion than they dispel.

   Furthermore, the isolation game blinds us to the systemic and holistic nature of the environment we work in.

   That’s why I like to make the absurd analogy of applying the ROI methodology to measuring the value of a light bulb. Depending on the nature of the activities of the room the bulb is in, you can probably show ROI results of several hundred million percent. But is it convincing?

   Lars Kjellander
   Product Manager Learning Solutions,
   Ericsson AB

7. Isolation of Training???
   Martin, This is a completely rhetoical question as it is only part of a larger effect. You can never isolate ‘training’ for whatever purpose. The event of training is simply part of the larger concern of the businesss, leading from a decision to perform a TNA through to the line managers’ support of the learners in their implementation and, (rarely mentioned) arrangem,ents for the sharing of the learning with others in the department/section/team. This has nothing to do with ROI whether all you can do (95% of the time) is ensure that the objectives of the TNA are realised or in the lesser range, put an ‘accurate’ monetary figure to it. You know me – I could go on, but I won’t!!
   Leslie

8. ROI on parenting courses
   Ok guys, you’re not really helping me here!! I am doing an evaluation of a training programme aimed at improving the parenting skills of parents, I have to include the ROI, as part of the bid to continue the budget required to support and develop the training. I’m using pre- and post training assessments tools from the Dept. of Health, its a well researched manualised training programme(its a 2 hour x 10 week course) to see if there is a significant change in parenting behaviour, and a control group to try to reduce the effects of TV programmes like “Little Angels”, I’m also looking at Social Services intervention rates post training, and planning a 6 month post-training follow up. What else can I be doing to get this right?

9. Who are the stakeholders and what do they value?
   Hi Shirley & thanks for presenting a real-world problem!

   Let me ask you for some clarification & extra info here:

   You are doing an evaluation on a training programme. Who is the evaluation for and what do they value?

   You state that you have to include the ROI as part of the bid process. This suggests that some financial officer may be involved in bid scrutiny. You need to know the baseline performance level, and you need to know what improvement the training will provide, measured in the same terms as your baseline.

   I assume you have to forecast the performance improvement and present it with some credibility and confidence if it is to be believed? The empahsis on forecasting here is because you are bidding to deliver the training – it hasn’t been delivered yet?

   In order to forecast with this kind of credibility and confidence don’t you need to KNOW what the drivers are behind the trainees exhibiting the right behaviours? And what the drivers are that prevent the right behaviours being exhibited – or that develop the wrong behaviours as well?

   It strikes me that a root cause analysis is needed of the reasons why the groups of parents coming on this course have poor parenting skills. I suspect from my own short experience of working with Social Services and touching on the edges of child protection that there are many root causes, and they are very diverse. Also that what looks like a root cause probably isn’t.

   You need to find the evidence, the PROOF that takes you from the behaviours of these parents before training, though the training, its objectives and delivery methods, and on to the new behaviours required. Where and what are the causal links? What are the main factors for success? The main barriers? How does the training address each of these? How does the improved parenting skills reduce or eliminate child protection issues, or educational issues etc, such that the agencies involved don’t need to direct so many resources to the consequences of poor parenting skills? Where’s the evidence for this?

   Get this data and you will have a very powerful case for your training and the budget it needs. The difficulty is it may take time, may require some subterfuge, and a bit of office politics (at least it did when I worked with Social Services, health trusts etc!)

   We could go round in circles here, so I go back to my initial questions about who wants the evaluation, why, and what do they value. You may have to do separate evaluations for each stakeholder, and present the value of the programme in terms that matter to each. Including ROI for the finance people. Even though it may not be meaningful! (See http://www.5boxes.com/5boxes/resources/5%20Boxes%20White%20Paper%20-%20When%20Is%20It%20Meaningful%20To%20Evaluate%20ROI%20Of%20Training.pdf)

   I hope this helps in some way!?

   Good luck,

   Martin