In my last blog I talked about my personal introduction to the world of evaluation with the New World Kirkpatrick Model. Having learnt all about the Four Levels Model and how to effectively apply this theory to display return on expectations, I was really eager to learn more about the other evaluation methodologies out there.
The second programme I attended took a much more quantitative view of evaluation than I had experienced with the Kirkpatrick Certification, although the Phillips Return on Investment (RoI) programme had a similar 2-day, learning-then-doing format. I’d never thought about learning and development from a purely financial view before; for me, it’s very much about soft skills acquisition, building a team and growing as an organisation, so it was interesting to see it all broken down into numbers and financial returns.
For some areas and in certain industries, RoI works really well and can be a great way to justify that you are getting back the money you’re spending on development. One problem I found with it though; RoI can’t really go into the inherent value of customer service, branding, and all those other things that are not as easily quantifiable as ‘time saved’. It’s these intangibles that, for me, are the basis of Learning and Development- that essential foundation, not particularly quantifiable but nonetheless vital to success, to up-skilling - and ultimately to the bottom line.
An interesting way to look at successful RoI is through the classic call centre training environment- think more efficient phone calls, time saved, all that good stuff. In a situation like this, there are plenty of things that can be measured; and these things all cost money, so you can do an effective RoI on how much money the training has cost vs. the amount of money saved or the increase in revenue (or hopefully, both). Okay, so let’s think about call centres for a minute. Living in an era of technology, getting to actually speak to a real person is increasingly difficult a task. And, by the time you do get through… you might be a little less than cheerful. So, imagine then the relief when you’re put through to someone who is just incredibly good at their job. Friendly, helpful, knowledgeable, going the extra mile so that you leave that call with an overwhelmingly positive image of the organisation. You’d mention that to people, wouldn’t you? They might be looking for a new service provider. They remember your conversation, and decide to go with the same company as you because they can be assured that if something goes wrong, it will be resolved. They have a positive experience themselves. They mention it to people in their own circles. And, just like that, it snowballs into a fantastic reputation and great corporate success.
Now, that organisation may well have spent massive amounts of money on training its call centre staff. And it is true that part of the return on this is financially quantifiable efficiency; each employee can now answer more calls, save more time, make the company more money and provide a return on investment for the training. The effect of that training on overall organisational branding, however – this isn’t quantifiable. But it is nevertheless massively important. And it can make you a lot of money.
This was really my main niggle with the Phillips RoI programme, and the gaps that the Kirkpatrick programme filled in for me; it was great learning how to mathematically demonstrate RoI in certain circumstances, but sometimes you need to justify your learning and development programmes by stepping back and taking a look at the bigger picture. Training and subsequent evaluation should be a consultative process; there should be a collaborative approach to ensuring the right solution is selected and delivered so that there is a genuine return on expectations. For me, interesting as it is to add a mathematical element to evaluation, I find that in using RoI exclusively, the roles of consultation and collaboration in effective evaluation are massively underplayed.
Do you evaluate your training?
Which evaluation methodologies deliver for you - and why?
Get in touch, I'd love to hear more about your own experiences of evaluation and evaluation training programmes!
5 Responses
Vague In – Vague Out
Hi Lucy,
Hi Lucy,
I use a variety of tools to evaluate my learning solutions and I have to say the one which I find works the most effectively is the Phillips ROI methodology.
With this tool (as you know), you start the process of 'evaluating' before you even determine what solution you are going to implement – it almost becomes part of your TNA process. It is vital that you are clear on what you are looking to achieve from any learning intervention prior to making any decisions, otherwise you will never get the right results (a point I think Garry also made).
Not only does this clarity help you determine what to actually design and implement, but it also helps you demonstrate to the business/organisation how this investment will have a positive impact. For learning and development to remain a key player in organisations/businesses, it is imperative that the function supports it's strategy and objectives – I find this ROI tool helps me map that out.
F.
Working Within – Or Working For?
Hi Fiona and Garry,
I'd like to first off thank you both for taking the time to feedback on this, as I mentioned I'm new to the world of evaluation methodology, and indeed to L&D in general, so it's really interesting to hear other opinions on the validity of programmes I have attended and methodologies I have encountered so far.
Fiona, I agree completely that evaluation needs to be considered before putting together a training programme – it's fundamentally just a vital part of starting with the end in mind, and without this foresight any comprehensive evaluation is going to be challenging.
Something interesting I learnt on the ROI programme which I perhaps did not make explicit within my blog was that it is recognised that there are circumstances whereby level 5, the ROI itself, is not a practical evaluation method; in these (numerable) circumstances, it's levels 1 – 4 (and not even always all of these!) which are encouraged. These four levels are those outlined originally by Don Kirkpatrick, and create the core of the methodologies which came afterwards.
Garry, I apologise if I was ambiguous with my terminology; specifically, my definition of 'branding' within a business environment. You asked 'What do you mean? Branding? What is it? Why is it an issue? What is it you hope to achieve?’ Branding in this context refers to organisational reputation, image within their sector (and further afield) – even beyond this, the overarching ‘ethos’ of a company as illustrated by a trusted position in the market. That is, not only knowing of a company – but trusting that they are the best at what they do and how they do it. I'm sure you can see, then, how his would have a positive financial impact on an organisation's success, but that it is not specifically 'trained' for so cannot be included in an ROI?
I believe that there is a fundamental difference here between working within an organisation and working for an organisation. I wholeheartedly believe that L&D should be an integral part of a business- but moreover, an integrated part of the organisation as a whole. This is where I would disagree with an independent consultant such as yourself who is interested in ‘existing or future performance issues’ rather than in the organisational branding (as defined above) as a whole.
I do agree that L&D can let itself down with an over-emphasis on ‘fluffy’ skills; but that said, it is my belief and my experience throughout my journey as a learner in various sectors that learning, and more importantly behaviour change and results, are best achieved when the learning is provided collaboratively, as a part of something rather than as a distinctly separate entity.
Referring back to your comment ‘The moment I read training as contributing to 'empowerment', 'engagement', 'effectiveness' et al I don't dismiss it but I certainly want to dig beneath and determine what is really meant’ – It struck me as slightly dismissive of the importance of intangible skills training, and I’d like us to discuss that in a bit more depth. I noticed your ‘Brinkerhoff’ comment on the ‘Evaluation- Capturing Impact’ blog (a very interesting read for a newcomer to the industry)… and I was wondering how you would conduct a monetary ROI on Diversity training, something you mention in your ‘Brinkerhoff’ comment? Doubtless this is something that some organisations feel the need to undertake, not to fill a hard-skills gap but from an ethical (and even potentially legal) point of view. Essentially – it is soft skills training at its finest. I can see how an ROE (Return on Expectation) can be derived from evaluation of this training – how would you as a trainer then take that and convert it into a fully-fledged monetary-based ROI?
Intangible Skills Training?
Lucy, here are my responses to the issues you raised:
Lucy: “I'm sure you can see, then, how his would have a positive financial impact on an organisation's success, but that it is not specifically 'trained' for so cannot be included in an ROI?”
I agree, if we’re not training it we can’t analyse or claim any ROI.
Lucy: “This is where I would disagree with an independent consultant such as yourself who is interested in ‘existing or future performance issues’ rather than in the organisational branding (as defined above) as a whole.”
I can’t speak for other consultants only myself and I work for the EEF and am not an independent consultant. When I work with clients I am concerned about their reputation, standing and ‘branding’. For fairly simple reasons:
My approach however to helping clients sustain or improve their reputation, standing and ‘branding’ is to follow back through a series of chained elements:
Experience of customers and internal and external stakeholders which in turn develops;
Reputation, standing and ‘branding’
So I work on behaviours and actions which ultimately lead to reputation and standing. I can absolutely see that extent of my investment is likely to be signifcantly less than those employed directly by the company.
Lucy: “Referring back to your comment ‘The moment I read training as contributing to 'empowerment', 'engagement', 'effectiveness' et al I don't dismiss it but I certainly want to dig beneath and determine what is really meant’ – It struck me as slightly dismissive of the importance of intangible skills training,-“
Well, it’s as I said, I don’t dismiss it but I certainly want to identify what they mean by it and exactly what they are going to deliver. What I do dismiss is ‘intangible skills training’; I almost fell of my chair when I read that phrase. If it is truly intangible then I don’t see how it can be trained, unless we are interpreting the word intangible in different ways?
Lucy: “ – I was wondering how you would conduct a monetary ROI on Diversity training, something you mention in your ‘Brinkerhoff’ comment? Doubtless this is something that some organisations feel the need to undertake, not to fill a hard-skills gap but from an ethical (and even potentially legal) point of view. Essentially – it is soft skills training at its finest. I can see how an ROE (Return on Expectation) can be derived from evaluation of this training – how would you as a trainer then take that and convert it into a fully-fledged monetary-based ROI?”
For any readers who are not familiar with the discussion referenced by Lucy it’s here: https://www.trainingzone.co.uk/comment/184867#comment-184867
How would I undertake ROI for Diversity training? It would depend on numerous factors: Why are they doing it? What do they hope to achieve? What’s the current situation? What is being experienced? Is it a problem? What are the critical success factors? (Example: Diversity issues in the Metropolitan Police are going to be completely different from those in the Citizens Advice Bureau, so one standard ROI assessment methodology will not fit all.) Against these I would search for appropriate metrics and baseline measures. In the case of the German client I listed the methods they were employing but it cannot be said that these are universal as each case is different. Here is the relevant section:
‘I have one major German owned client who is currently focusing some attention on ‘Diversity’. Hardly operational, but they have some very clear metrics against which they are going measure success, some financial some not. Metrics such as:
+ Frequency of related disciplinary events and associated costs. (Financial)
+ Numbers of complaints and reported incidents with follow up investigations and associated costs. (Financial)
+ Staff survey figures. (Expectation)
+ Position and movement of ethnic minorities in management positions. (Expectation)
+ Position and movement of genders in management positions (Expectation).
Now that to me seems like a simple and fair combination of factors to determine whether Diversity training is actually worth it financially (leaving aside the ethical issues) and delivers.’
Hi Garry,
Hi Garry,
Thank you so much for your detailed response! As a newcomer to any industry, it's invaluable to have the insight of someone experienced, so thank you for taking the time out to discuss. You've definitely given me a lot to think about!
Lucy