In the third of four articles in this series, Martin Schmalenbach looks at when, and when, not to evaluate. Read part one, part two and part four.
Setting aside any personal views about always evaluating training and performance enhancing interventions, there are times when you perhaps should NOT evaluate. The following covers most situations for when you should and shouldn’t evaluate. Training may need to take place in a hurry, for whatever good reasons, to the extent that the elapsed time needed to do the evaluation properly is too long to be able to directly influence the decision as to whether to commit resources to the training or not. There is one good reason to do the evaluation anyway, and that is to develop some data and that can help to validate the decision taken and so support organisational learning about how to respond to such scenarios in the future.
| When to evaluate | When NOT to evaluate |
| Political Necessity The training in question is in such a bright political spotlight that it needs to be seen to be value adding. The method for evaluating this "value-add" therefore needs some credibility & robustness to stand up to closer scrutiny. Having a 'pre training' baseline will be very important in demonstrating robustly that the 'after' is better than the 'before'. This baseline will need to be firmly anchored on a solid root cause analysis to ensure credibility. A trainer or training department should associate itself clearly with value-adding activities if it is to ensure it does not get 'downsized' the next time resources become even more scarce. | Regulatory Or 'Must Have' Training The training is required either as part of a statutory requirement or other legal instrument (e.g. health and safety related), or the organisation believes within its core values that the training is the right thing to do (e.g. induction). The training can still be validated though - did the training do what "it says on the tin"? and perhaps also "was this the most efficient way of delivering the training?" |
| Client Requirement Obviously if a client (internal or external) asks for it, then do it, or more appropriately, help them to do it themselves - it's their resources being spent! Again, having a robust and rigorous process really helps. The client may already have one, but it won't hurt to challenge it in the sense of making sure it can actually answer the key questions the client is likely to have. | "Luxury" Training The training is not required to add to the bottom line or otherwise move forward the performance of the individual/team/organisation, i.e. it's a 'luxury' so deploying limited resources to evaluate or even validate the training is not good use of them. An obvious example is non-work related training that some organisations provide to employees in the form of say £100 per year to cover things like pottery classes or sports/arts/crafts at night school. |
| Decisions Surrounding Limited Resources Any manager responsible for deciding where, when and how to deploy limited resources would like to know in advance the likely benefits of each option so he or she can more easily make the decisions that lead to success. This should apply to training, after all employees are diverted from their 'day jobs' to attend or participate, money and other resources are deployed in order to make sure the training takes place and is exploited. This is one case where evaluating in advance is going to be a huge help to managers. | When There Is No Time To Do It Properly Training may need to take place in a hurry, for whatever good reasons, to the extent that the elapsed time needed to do the evaluation properly is too long to be able to directly influence the decision as to whether to commit resources to the training or not. There is one good reason to do the evaluation anyway, and that is to develop some data and that can help to validate the decision taken and so support organisational learning about how to respond to such scenarios in the future. |
| Client Relations A training function should ensure that its clients are completely clear about the aims, outcomes and expectations arising from the intervention. In doing so it has a better chance of actually meeting the client's requirements and managing effectively any expectations. This is part of the art of good customer or client relations, and can do wonders for bottom line performance for both client and training function, as well as encourage future repeat business. Such a situation is unlikely to lead to future downsizing on the grounds of limited resources! | When Not Allowed Or Able To Develop A Baseline If you can't develop (for any reason) a credible baseline of performance of key indicators, including those relating to any root cause analysis, you really have nothing to compare the new performance against. Any evaluation you do can not be judged as objective, is likely to lose credibility as a result, so why waste the effort and heartache of a full evaluation? You can certainly develop a subjective view of those involved, basically by asking "in what way was it worth it?" |
| Buy-In From Staff & Line Managers When buy-in is needed from line managers and their staff who will be attending any training or participating in associated interventions, it helps if they know what is expected, why, their part to play in it and how the training and interventions help them do their jobs better/quicker/cheaper - and to some (great!) extent, answers the "what's in it for me" question as well. | When the reasons for doing the intervention cannot be expressed in terms of strategic objectives and/or key performance measures If you can't measure the performance issue and/or explicitly and credibly link the activity to the strategic objectives, not only should you consider NOT evaluating, you should also consider NOT doing the intervention at all! |
| After The Fact - i.e. After Training If you are asked to conduct an evaluation there are generally 2 things that can come out of it: lessons to learn for the future, and political activities designed to show the worth or otherwise of a training programme or function that is either falling out of favour, or is the target for losing resources because all other departments have been able to defend their resource allocations and historically the training function has not been good at this. If you can develop a baseline with supporting root cause analysis, even after the fact, then you can do a reasonable evaluation. Either way you can state what has and hasn't happened in the past, and how things will be in the future, starting immediately. It's a chance to show a reliable, robust and credible process and your part in it, and how the combination will contribute positively to the bottom line in the future. It may get you a reprieve! | |
| Academic and other research | Any approach you want, as required by the research! |
| Audience | Approach or Model To Use |
| Trainer/Training Dept | Kirkpatrick Level 1 - useful feedback on environmental issues, pace of delivery etc and specifics for the actual trainer. ALWAYS. Kirkpatrick Level 2 - this will indicate if the training is being learned - if there are future problems this will help to eliminate or otherwise pinpoint if training methods and objectives are at fault. Ideally ALWAYS do this. ROI-focused approach - this will demonstrate the value-adding contribution of the training function. ALWAYS do this. If you can't, put it in writing WHY, and share this with the client and higher authority. |
| Manager/Client | ROI-focused approach - this is the business case they need to get behind the intervention and fully support it in resourcing and priorities. Do this ALWAYS at the programme level unless the manager/client waives this in writing, in which case ensure they know the consequences of this, and that changing their minds in the future does not make for a credible evaluation. DO NOT do this at the individual level - let line managers make that judgement, with your "guidance". Kirkpatrick Level 3 - behaviours need to change if performance is to change, so ALWAYS do this at the programme level, and if possible, for each course and individual too. |
| Shareholder/Owner | ROI-focused approach - this is the business case for having and keeping the training function. Do this ALWAYS, but not necessarily for each programme, but certainly for the function as a whole in any major reporting cycle, at least quarterly. |
| Delegate/Employee | CIRO / CIPP as these look at the process, outcomes or product, inputs etc - much closer to the operational or 'shop floor' end of the organisation. Help delegates to do this for themselves ALWAYS, ideally with involvement from their line managers. |
| Academic and other research | Any approach you want, as required by the research! |
