I work in a large organization, with a workforce in excess of 20,000 staff. Evaluating learning at Kirkpatrick's Level 2 is rather straightforward with tests, quizzes etc. However, with such a large staff numbers, evaluating learning at Level 3 proves to be more challenging for behavioural type courses, e.g. Aggression Management Techniques. On-the-job observation of 20,000+ staff to determine how the learning is being applied on-the-job is time consuming, costly and just not practical.
I would love to hear your suggestions about how best to undertake Level 3 evaluation for behavioural courses in such a large organization. Thank you.
5 Responses
Use people from the Operation….
…. your challenge is one that I've heard a number of times so you aren't alone. The danger is that the expectation is that someone from L & D will be carrying out this activity where in actual fact if your learning journey is fully aligned to the business, then people from the operation will be helping with the evluation.
So, simply put, people out at the 'coal face' should be undertaking observations and evaluation activity whether it be managers or performance coaches or whatever roles you have in your structire that are working in the operation. To fully embed learning as an aid to performance improvement; people from the operation should also be involved.
Hope that helps
Sign-on not sign-off
Thanks Clive,
That's a very helpful reminder. I recall reading something a while back from Kirkpatricks (I think) about getting stakeholders to 'sign-on' rather than 'sign-off'.
So instead of stakeholders signing-off on a training program, which they then leave in the hands of the training dept. to implement, stakeholders sign-on to their active and continuing role in implementing, and embedding the learning and evaluation activities.
I guess it's about only committing to this level of evaluation where the resources are available to undertake the evaluation properly and the business is prepared to partner with us to do it.
Business results instead of happy sheets
Stop measuring short term "successes" through happy sheets and focus on longer term business results. Much easier said than done, but it's a question of changing the mindset of what learning initiatives should be for large organizations.
1) Engage key stakeholders throughout the organization, as high as possible (CEO, CFO, COO, CMO, Sales, Logistics, etc)
2) Define together with them performance criteria (KPIs, etc.)
3) Establish baseline measurements
4) Link learning initiatives to the criteria
5) Establish clearly defined benchmarks and transparent, clearly communicated measurements
6) Link personnel evaluations to benchmarks.
If you do this in close cooperation with the stakeholders, not only are you getting a more accurate picture of what learning really means for the organization, you are also building commitment throughout the company by linking learning to business results.
leading indicators
Hi there just to add some more food for thought and just to support what Leo has said is that learning activity should be a pertnership and it's value will be better recognised if it is linked to a hard business target, for example a specific % increase in sales, reduction in call answering time, increased customer satisfaction etc.
Another point is to have an agreed set of leading indicators. They are things that need to happen along the way in order for the learning 'event' to be a success. One of the things that is a must have in terms of leading indicators is for business managers to be involved in evlauation activity. In our world we have a contractual agreement where we and the operation agree what the end result will be (a business measure) and what support will be required along the way. If it is critical to teh success of the learning activity and it's written in the contract it is easier to identify where the successes lie. As Leo says you do need to engage and have a senior stakeholder who will communicate the importance of this so that people take notice. Hope that helps.
Brinkerhoff might be useful?
http://www.kenblanchard.com/img/pub/newsletter_brinkerhoff.pdf