Stuck with where to begin on your route to evaluation nirvana? Let Kenneth Fee and Dr Alasdair Rutherford be your guide.
There was once an old sailor my grandfather knew, Who had so many things that he wanted to do, That, whenever he thought it was time to begin, He couldn’t because of the state he was in.[1]
People often wonder where the best place is to start in learning evaluation. They often wonder so long they never actually get around to doing any evaluation. Like AA Milne’s Old Sailor, they end up doing nothing at all. Alternatively, they do the wrong things, or perhaps they attempt the right things, but underestimate what’s involved and so don’t evaluate properly. It needn’t be so confusing. There are just six things to think about when embarking upon learning evaluation.
Make a commitment
This is not just a question of having a firm resolve, although that certainly helps. It’s a matter of getting the organisation collectively to commit to doing more evaluation, agreeing the benefits that will accrue, and putting that commitment into practice. Securing commitments from relevant colleagues is really all about stakeholder consultation. This means identifying all those who have an interest in learning and its outcomes and its contribution to the business, and agreeing with them what needs to be done and the extent to which they need to be involved. Bear in mind that some of these stakeholders will probably be outside your organisation.
So think about who your stakeholders are, define the nature of their stakeholding, weigh their importance, and based on that weighting, decide how much to ask of each stakeholder. However the bottom line is that they all need to be on board.
"It’s a matter of getting the organisation collectively to commit to doing more evaluation, agreeing the benefits that will accrue, and putting that commitment into practice."
So think about who your stakeholders are, define the nature of their stakeholding, weigh their importance, and based on that weighting, decide how much to ask of each stakeholder. However the bottom line is that they all need to be on board.
"It’s a matter of getting the organisation collectively to commit to doing more evaluation, agreeing the benefits that will accrue, and putting that commitment into practice."In practical terms, making a commitment is also about allocating resources, planning time, and setting an adequate budget (at least 10% of your L&D budget). Learning evaluation is rarely urgent, but it is usually very important, which means you need to plan ahead how you are going to tackle it, and make sure it has its place on everyone’s agenda.
Think value
Most people in most modern organisations are cost-aware, but surprisingly few are as value-aware. Costs amount to reasons not to undertake learning and development, but value is about what we gain from it; costs are usually easy to express numerically, but value may be expressed qualitatively and may be hard to measure against costs. Nevertheless, this difficulty is not a reason to give up, but a reason to start making estimates. Educated guesswork may start out wide of the mark, but with experience will become more accurate and more precise.
Total Value Add is the technique for identifying all of the value generated by learning – all the forms of that value, not just economic or financial – and deciding whether and how to measure it. It’s also about taking a situational view, and choosing pertinent evaluation techniques on the basis of the given situation and business need.
Total Value Add is the technique for identifying all of the value generated by learning – all the forms of that value, not just economic or financial – and deciding whether and how to measure it. It’s also about taking a situational view, and choosing pertinent evaluation techniques on the basis of the given situation and business need.
Clarify your purpose
Clarifying your evaluation purpose means thinking about what matters to your organisation about the learning and development:
-
Is it about controlling large-scale training delivery, to ensure consistency or provide quality assurance?
-
Is it about finding ways to improve the training and make it more effective or efficient (formative evaluation)?
-
Is it about proving training has made a difference, measuring results or impact (summative evaluation)?
-
Is there are a broader purpose for the organisation to learn from the evaluation?
There needs to be a shared understanding of your purpose throughout your organisation, and in general terms your purpose should be one or more of the four above – controlling, improving, proving or learning. The one purpose it should not be about is routine or habit – simply doing it because that’s the way it’s always been done, or that’s what people expect. The mass distribution of reaction sheets often falls into this trap.
Clarifying your general purpose should help in focusing in on your more specific purposes.
Clarifying your general purpose should help in focusing in on your more specific purposes.
Determine your evaluation strategy
A strategy is not something grand, and it’s not just for senior management; a strategy is what you need to act logically and move forward. Your strategy should include three considerations: a diagnosis of the problem/s, challenge/s or opportunity/ies you face; a guiding policy for how to deal with them; and a set of actions to implement that policy[2]. Contextually, your learning evaluation strategy should be part of your learning and development strategy, which in turn should be part of your overall business strategy.
Setting out this strategy will help ensure that you and your colleagues stay true to your purpose.
Setting out this strategy will help ensure that you and your colleagues stay true to your purpose.
Select your approach
There are two broad sets of approaches to learning evaluation, and they are not usually compatible. The two are goal-based and system-based approaches.
If you want to track the quality of the learning experience, what learners actually learn, what they transfer to work, and the results for the organisation, you probably need to take a goal-based approach. Alternatively, if you want to conduct a more objective review of what actually happens with a learning and development initiative, regardless of intentions or objectives, you probably need to take a system-based approach (if you don’t understand the choice, you need help – see point six).
If you want to track the quality of the learning experience, what learners actually learn, what they transfer to work, and the results for the organisation, you probably need to take a goal-based approach. Alternatively, if you want to conduct a more objective review of what actually happens with a learning and development initiative, regardless of intentions or objectives, you probably need to take a system-based approach (if you don’t understand the choice, you need help – see point six).
"Never underestimate the importance of an independent perspective, both to help see the wood despite the trees, and to guard against in-built bias."
Whichever approach you choose, there are common factors you need to consider, including establishing your baseline measurement – the situation before the learning intervention, which provides a point for comparison afterwards. You also need to consider what sort of evidence you’re looking for, and what will constitute a sufficient measure of success.
Get help
Finally, you need to decide what help you need. Can you conduct your evaluation using just internal staff and resources, or do you need external support? In particular, do you have the knowledge and skills within your organisation to ensure an effective and meaningful evaluation, or do you need help with those? One area where organisations often need help is with research expertise, which is essential for method selection, question-setting, data analysis, clear reporting and more. And never underestimate the importance of an independent perspective, both to help see the wood despite the trees, and to guard against in-built bias.
Do you fully understand how to implement techniques like Total Value Add, Business Impact Modelling, Return on Investment, and the Success Case Method? If not, you need someone who can help you master the techniques that are important for you.
There are many learning providers, consultants and academics who claim they can help with learning evaluation, but too often they want to champion one specific approach or method, and denigrate everything else. Forget them and their dogma. Instead, think about these six things, and find people who can help you take it from there.
Do you fully understand how to implement techniques like Total Value Add, Business Impact Modelling, Return on Investment, and the Success Case Method? If not, you need someone who can help you master the techniques that are important for you.
There are many learning providers, consultants and academics who claim they can help with learning evaluation, but too often they want to champion one specific approach or method, and denigrate everything else. Forget them and their dogma. Instead, think about these six things, and find people who can help you take it from there.
[1]Milne, A.A., (1927) ‘The Old Sailor’, Now We Are Six, Methuen
[2]Rumelt, R, (2011) Good Strategy/Bad Strategy: the difference and why it matters, Profile Books
This article first appeared on the Airthrey website
Kenneth Fee and Dr Alasdair Rutherford are the founding directors of learning evaluation firm Airthrey Ltd. Ken is a career learning and development professional, whose latest book, 101 Learning & Development Tools, deals with evaluation among other topics. Alasdair is an evaluation and econometrics specialist, and a Research Fellow at the University of Stirling
Stuck with where to begin on your route to evaluation nirvana? Let Kenneth Fee and Dr Alasdair Rutherford be your guide.
There was once an old sailor my grandfather knew, Who had so many things that he wanted to do, That, whenever he thought it was time to begin, He couldn't because of the state he was in.[1]
People often wonder where the best place is to start in learning evaluation. They often wonder so long they never actually get around to doing any evaluation. Like AA Milne's Old Sailor, they end up doing nothing at all. Alternatively, they do the wrong things, or perhaps they attempt the right things, but underestimate what's involved and so don't evaluate properly. It needn’t be so confusing. There are just six things to think about when embarking upon learning evaluation.
Make a commitment
This is not just a question of having a firm resolve, although that certainly helps. It's a matter of getting the organisation collectively to commit to doing more evaluation, agreeing the benefits that will accrue, and putting that commitment into practice. Securing commitments from relevant colleagues is really all about stakeholder consultation. This means identifying all those who have an interest in learning and its outcomes and its contribution to the business, and agreeing with them what needs to be done and the extent to which they need to be involved. Bear in mind that some of these stakeholders will probably be outside your organisation.
So think about who your stakeholders are, define the nature of their stakeholding, weigh their importance, and based on that weighting, decide how much to ask of each stakeholder. However the bottom line is that they all need to be on board.
"It's a matter of getting the organisation collectively to commit to doing more evaluation, agreeing the benefits that will accrue, and putting that commitment into practice."
In practical terms, making a commitment is also about allocating resources, planning time, and setting an adequate budget (at least 10% of your L&D budget). Learning evaluation is rarely urgent, but it is usually very important, which means you need to plan ahead how you are going to tackle it, and make sure it has its place on everyone's agenda.
So think about who your stakeholders are, define the nature of their stakeholding, weigh their importance, and based on that weighting, decide how much to ask of each stakeholder. However the bottom line is that they all need to be on board.
"It's a matter of getting the organisation collectively to commit to doing more evaluation, agreeing the benefits that will accrue, and putting that commitment into practice."In practical terms, making a commitment is also about allocating resources, planning time, and setting an adequate budget (at least 10% of your L&D budget). Learning evaluation is rarely urgent, but it is usually very important, which means you need to plan ahead how you are going to tackle it, and make sure it has its place on everyone's agenda.
Think value
Most people in most modern organisations are cost-aware, but surprisingly few are as value-aware. Costs amount to reasons not to undertake learning and development, but value is about what we gain from it; costs are usually easy to express numerically, but value may be expressed qualitatively and may be hard to measure against costs. Nevertheless, this difficulty is not a reason to give up, but a reason to start making estimates. Educated guesswork may start out wide of the mark, but with experience will become more accurate and more precise.
Total Value Add is the technique for identifying all of the value generated by learning – all the forms of that value, not just economic or financial – and deciding whether and how to measure it. It’s also about taking a situational view, and choosing pertinent evaluation techniques on the basis of the given situation and business need.
Total Value Add is the technique for identifying all of the value generated by learning – all the forms of that value, not just economic or financial – and deciding whether and how to measure it. It’s also about taking a situational view, and choosing pertinent evaluation techniques on the basis of the given situation and business need.
Clarify your purpose
Clarifying your evaluation purpose means thinking about what matters to your organisation about the learning and development:
- Is it about controlling large-scale training delivery, to ensure consistency or provide quality assurance?
- Is it about finding ways to improve the training and make it more effective or efficient (formative evaluation)?
- Is it about proving training has made a difference, measuring results or impact (summative evaluation)?
- Is there are a broader purpose for the organisation to learn from the evaluation?
There needs to be a shared understanding of your purpose throughout your organisation, and in general terms your purpose should be one or more of the four above – controlling, improving, proving or learning. The one purpose it should not be about is routine or habit – simply doing it because that's the way it's always been done, or that's what people expect. The mass distribution of reaction sheets often falls into this trap.
Clarifying your general purpose should help in focusing in on your more specific purposes.
Clarifying your general purpose should help in focusing in on your more specific purposes.
Determine your evaluation strategy
A strategy is not something grand, and it's not just for senior management; a strategy is what you need to act logically and move forward. Your strategy should include three considerations: a diagnosis of the problem/s, challenge/s or opportunity/ies you face; a guiding policy for how to deal with them; and a set of actions to implement that policy[2]. Contextually, your learning evaluation strategy should be part of your learning and development strategy, which in turn should be part of your overall business strategy.
Setting out this strategy will help ensure that you and your colleagues stay true to your purpose.
Setting out this strategy will help ensure that you and your colleagues stay true to your purpose.
Select your approach
There are two broad sets of approaches to learning evaluation, and they are not usually compatible. The two are goal-based and system-based approaches.
If you want to track the quality of the learning experience, what learners actually learn, what they transfer to work, and the results for the organisation, you probably need to take a goal-based approach. Alternatively, if you want to conduct a more objective review of what actually happens with a learning and development initiative, regardless of intentions or objectives, you probably need to take a system-based approach (if you don't understand the choice, you need help – see point six).
Whichever approach you choose, there are common factors you need to consider, including establishing your baseline measurement – the situation before the learning intervention, which provides a point for comparison afterwards. You also need to consider what sort of evidence you're looking for, and what will constitute a sufficient measure of success.
If you want to track the quality of the learning experience, what learners actually learn, what they transfer to work, and the results for the organisation, you probably need to take a goal-based approach. Alternatively, if you want to conduct a more objective review of what actually happens with a learning and development initiative, regardless of intentions or objectives, you probably need to take a system-based approach (if you don't understand the choice, you need help – see point six).
"Never underestimate the importance of an independent perspective, both to help see the wood despite the trees, and to guard against in-built bias."
Whichever approach you choose, there are common factors you need to consider, including establishing your baseline measurement – the situation before the learning intervention, which provides a point for comparison afterwards. You also need to consider what sort of evidence you're looking for, and what will constitute a sufficient measure of success.
Get help
Finally, you need to decide what help you need. Can you conduct your evaluation using just internal staff and resources, or do you need external support? In particular, do you have the knowledge and skills within your organisation to ensure an effective and meaningful evaluation, or do you need help with those? One area where organisations often need help is with research expertise, which is essential for method selection, question-setting, data analysis, clear reporting and more. And never underestimate the importance of an independent perspective, both to help see the wood despite the trees, and to guard against in-built bias.
Do you fully understand how to implement techniques like Total Value Add, Business Impact Modelling, Return on Investment, and the Success Case Method? If not, you need someone who can help you master the techniques that are important for you.
There are many learning providers, consultants and academics who claim they can help with learning evaluation, but too often they want to champion one specific approach or method, and denigrate everything else. Forget them and their dogma. Instead, think about these six things, and find people who can help you take it from there.
[1]Milne, A.A., (1927) 'The Old Sailor', Now We Are Six, Methuen
Do you fully understand how to implement techniques like Total Value Add, Business Impact Modelling, Return on Investment, and the Success Case Method? If not, you need someone who can help you master the techniques that are important for you.
There are many learning providers, consultants and academics who claim they can help with learning evaluation, but too often they want to champion one specific approach or method, and denigrate everything else. Forget them and their dogma. Instead, think about these six things, and find people who can help you take it from there.
[1]Milne, A.A., (1927) 'The Old Sailor', Now We Are Six, Methuen
[2]Rumelt, R, (2011) Good Strategy/Bad Strategy: the difference and why it matters, Profile Books
This article first appeared on the Airthrey website
Kenneth Fee and Dr Alasdair Rutherford are the founding directors of learning evaluation firm Airthrey Ltd. Ken is a career learning and development professional, whose latest book, 101 Learning & Development Tools, deals with evaluation among other topics. Alasdair is an evaluation and econometrics specialist, and a Research Fellow at the University of Stirling
