Why evaluate? Kenneth Fee and Dr Alasdair Rutherford provide the community with another great piece about the E word.
"Training evaluation is a bit like eating five portions of fruit a day: everyone knows they are supposed to do it, everyone says that they are planning to do it better in future, and few people admit to having it right." [1]
Everyone agrees that evaluation is a high priority for HR and L&D professionals – in principle. The trouble is, hardly anyone puts that principle into practice.
Allison Rossett of the American Society for Training and Development (ASTD) recently said "we all speak fluent Kirkpatrick...but knowing is not doing, not even close". ASTD has found that almost all training is measured at Kirkpatrick level one (learner reactions), about a third at level two (assessment of learning), only 13% at level three (behavioural change) and merely 3% at level four (business results) [2]. Closer to home, and irrespective of evaluation methodology (i.e. it's not just Kirkpatrick), surveys by the European Union [3], the CIPD [4], among others [5], have produced similar findings.
How can we explain the mis-match between our intentions and our actions, and more importantly, what can we do about it?
We can identify five reasons why people put off learning evaluation.
It's too difficult
Many people think it's too difficult to measure the outcomes of learning and their contribution to business performance, but as Peter Drucker said "if you can't measure it, you can't manage it". Some managers' fears derive from unfamiliarity with research methods and statistical analysis – but this specialised help is readily available. A more sophisticated version of this argument is that people development is hard to measure because it's behavioural and doesn't lend itself to quantification. Yet qualitative data is also measurable, with just a little more thought.
It's too expensive
This is an absurd generalisation, akin to claims that all training is expensive. It can be, but equally it can be managed economically. Online surveys can be distributed cheaply; telephone interviews are usually much cheaper than face-to-face, and just as useful; judicious sampling can significantly reduce costs; and regular practice makes evaluation quicker and easier. The alternative is much more expensive – continuously repeating training that doesn't pay. The reality is that lack of evaluation is wasteful, and not evaluating is too expensive.
It's not wanted
HR and L&D professionals who claim their senior management don't care about evaluation are taking a huge gamble. Senior management should care. At some point, they almost certainly will. And when they do, there had better be a case for L&D ready in response - otherwise it'll be cut. Just because senior management may have a temporary blind spot doesn't justify this lazy approach. All managers need foresight and planning, and if HR managers aren't measuring the cost, value, and business contribution of L&D, they're not doing their job properly.
It's not needed
It's a truism that all learning is valuable. But so are many things organisations do, and ultimately learning is competing for scarce resources. It's not enough to be A Good Thing, it has to be demonstrably more valuable than alternative investments. Certainly, there can be debate about how to measure value, and economic/financial considerations may not be the only ones, perhaps not even the most important, but measures still need to be made.
Procrastination
Learning evaluation is rarely seen as a pressing priority, and no matter how important it is judged to be, may continue to be put off till 'tomorrow' indefinitely. This is a dangerous way to operate, and more enlightened organisations will weigh urgency and importance to determine when to allocate time and resources to evaluation (and other things that never seem urgent). The antidote to procrastination is to plan and to budget: allocate a proportion of L&D budgets (both time and money) to evaluation, and ensure that it is used.
If we acknowledge that these barriers to evaluation, or rather these excuses for indefinite postponement of evaluation, are mistaken, what can we do instead? We can identify five sets of actions to counter them.
- Become more value aware
Most organisations are cost aware to a reasonable extent, but too few consider all the value added by L&D. Total Value Add is a philosophy for identifying all the value added by learning, and using a range of appropriate evaluation methods to measure it. If we're weighing value against costs, we must be as confident of the measures of value – no matter how hard it seems at first, repeated practice of this will make a big difference, and is the only way to yield credible cost/benefit analyses, cost effectiveness analyses, and ROI calculations.
- Plan to measure the effects of L&D
Learning needs to be integrated with business performance, which means aligning L&D with performance improvement, and tracking what happens when learning is applied. Learning initiatives should always be conceived and designed with a view to measuring the effects they have, whether in transfer to work for immediate gains, or in enhanced understanding and capability for future benefits. There should also be a means to capture additional, unexpected, or unwelcome effects.
- Link to business outcomes
If learning is not making a clear, measurable, and significant contribution to business outcomes – the things the organisation exists to accomplish – then it is at best a marginal activity, and at worst a needless luxury. Instead learning should be a key means of leveraging competitive advantage, and the way to ensure this is to plan to have an impact on the most important business metrics. If there is no clear link in place, then Business Impact Modelling can help change that.
- Acquire research expertise
This addresses the barrier of difficulty. It is rare for HR or L&D practitioners to have expertise in sample selection, or regression analysis, or the use of software tools like SPSS or NVivo. But there may be other people in the organisation with those skills and knowledge, and if not, there are plenty of consultants available. Organisations need to ensure they can draw upon this expertise, or else run the risk of devising surveys that aren't fit for purpose, collecting data that doesn't provide answers, and drawing mistaken conclusions when analysing findings – all of which may lead to wrong decisions about learning investments.
- Budget properly
This addresses the barrier of how expensive it seems. If any business activity doesn't have a budget heading, then any cost arising from it will always seem too much, when the real problem lies in the planning. Allocate a set proportion of L&D budgets to learning evaluation – the usual figure is 10 - 15%, but initially it may have to be more – and ensure that it is used to optimum effect. It will soon become apparent that this investment is worthwhile.
What can we conclude? Organisations need to find ways to make learning evaluation a higher priority – in practice as well as intent. They need to build it into their strategies and plans, they need to back that commitment up with sufficient budgets of time, finance, and resources (not least human resources), and they need to see it through. Individuals in HR and L&D roles need to take responsibility for it, and act now. Evaluation has to move up the agenda: what are you going to do about learning evaluation in the next six months?
This is article first appeared on the Airthrey website
Kenneth Fee and Dr Alasdair Rutherford are the founding directors of learning evaluation firm Airthrey Ltd. Ken is a career learning and development professional, whose latest book, 101 Learning & Development Tools, deals with evaluation among other topics. Alasdair is an evaluation and econometrics specialist, and a Research Fellow at the University of Stirling
[1] Tamkin, P, Yarnall, J & Kerrin, M, Kirkpatrick and beyond: a review of models of training evaluation, Institute of Employment Studies, 2003
[2]http://www.astd.org/Publications/Blogs/Learning-Technologies-Blog/2012/05/Evaluation-Words-Into-Action.aspx
[3] http://www.cedefop.europa.eu/EN/articles/4410.aspx
[4] http://www.cipd.co.uk/research/_learning-talent-development
[5] http://www.trainingreference.co.uk/news/gn040223.htm