I have just joined a new organisation of some 10,000 staff as the lead for L+D. The organisation needs to go back to basics and this includes evaluation. At present they think of happy sheets as being all the evaluation you need (dont laugh).
My issue is that I need an evaluation process that can be used by L+D, staff and managers which meets all the levels of evaluation and is simple to use and not time consuming so that the managers will buy into it. It also needs to be structure so that RoI can be extrapolated from it.
So.... over to you, anyone got a system i could steal? ideas? suggestions? im all ears....
craig mitchell
10 Responses
Nirvana it is, you seek!
Craig
You aren’t going to find a simple system that isn’t time consuming, meets all 4 levels for L&D, line and staff and provides all the answers to RoI.
“Quality is never an accident, it is the result of intelligent effort”
Sorry to be a downer
Rus
Thanks for that
Hi Rus, thanks for that. I kind of knew that really but the issue is that if its time consuming and complicated then managers just dont want to know. Any ideas?
May the force be with you…
Hi Craig,
Even though I recently finished a 10,000 word dissertation on evaluation I’m afraid I can’t help. My quest was to find exactly what you are looking for, and I had no sucess in finding something. ROI is very tricky to measure (every book I read said so)but I suppose if your business considers it that important they will give you a chance to research it properly.
On the brightside if anyone can come up with an easy workable system they have probably made their first million!
Something called The Evaluation Toolkit by David Simmonds may be helpful to you though, it has some really useful stuff in it.
Best of luck,
Nikki
evaluation
Hi Craig,
I have found evaluation to be tricky as well. I’ve just implemented a process that helps make the “happy sheet” more useful. It’s not too time consuming, but does require some oragnisation and effort.
I have been going back to training attendees 3 months after the event and asking 3 simple questions:
What happened and why did it happen that way? (for classroom based training you can just attach an agenda, for on the job training an explanation of why the plan may not have been followed is required)
What were the main learning points from this experience? (this is to find out what knowledge they have picked up from the training)
How have you used the things you learnt in your job?
(this is to ensure the knowledge has been used)
From the responses you can determine business benefits and use the info to get your ROI – if your training objectives are tight that is. I am an internal trainer for a catering company. Most of my learners don’t have access to computers at work, so I have been conducting thse refection interviews – it only takes about 15 minutes to half an hour with each person and I have been pleasantly surprised by the results. If your people can respond via email the process can be quicker, but people might need prompting for examples re the last question.
Kind Regards,
Claire
No easy answer…
Hi Craig,
As the others have already mentioned, there is no easy answer.
We have devised a way of summarising the feedback from Happy Sheets (both quantitaive and qualitative) on a spreadsheet and then sorting these by course and/or trainer to get a big picture overview of opnions on the course. This gives a balanced level of feedback as prevents one ‘bad’ course effecting an unnecessary change.
We’ve also found through experince that if you want a job doing well, sometimes you just have to do it yourself, and so we have undertaken an initiative to hold focus groups post-course with delegates 4-8 weeks into their roles. This makes sense for us as our training is principally a very intense 3 or 4 days, whereas this would probably be very difficult for short courses or huge attendee numbers. You could of course take a sample of delegates to get their feedback…
Finally, to tie in our work and gain some sort of ROI measure, we have built relationships with those who report the ‘quality’ of the work produced in the area. This allows us to build training to counteract any downturns in quality and take credit (though obviously and unfortunately not exclusively) when things go well!
I wish you the best of luck!
Helen
The blogosphere
This is an issue that comes up quite often on the blogosphere. In fact, quite recently there was a discussion on this very point – Check out this post: http://elearningtech.blogspot.com/2007/01/state-of-assessment-by-e-learning.html Alternatively, you could just google Kirkpatrick and ROI…
One of your steepest hills, though (in my experience) is going to be the middle managers who often can’t be bothered to co-operate with you as you try to move towards more meaningful monitoring and evaluation (let us know if one of them ever tells you with a knowing smile “Don’t worry – you don’t need to prove ROI – your job is secure!”
Real evaluation
Foregt Happy Sheets – they are not evaluation instruments as all they do is express the mood, attitudes and feelings of the completer at that moment in time (he/she may be suffering from a hangover!).
I make no apologies in recommending my best-selling book that contains all you wish to know about effective but not trainer time-consuming evaluation. ‘Assessing the Value of your Training – The evaluation process from training needs to the report to the Board’. Leslie Rae. Gower, 2002.
Maybe this could help you some of the way at least
Hi Craig
I work for a company developing blended learning. I’m not trying to advertise here (so no mention of my company etc), but part of what we do is allow online (internet or intranet based) feedback to evaluate both workshops and the implementation of skills and action plans. Our system allows reporting on both measured and non-measured (i.e. textual) responses, which can be pasted into MS-Excel for reformatting and showing to appropriate managers. We use a mixture of the sort of questions that Claire talks of, which prompt textual answers, plus some that ask a user to rate effectiveness, how well they feel they have implemented skills learned etc – these can then produce stats, which impress some managers.
I’m always looking for opportunities to get our own feedback (!) to help our product development (which is what I am responsible for), so if you were interested, and depending on the time you’d need the facility for, I could look to give you (and your personnel) access to an online system in return for feedback and (if you were happy) acting as a reference. Of course, my hope is that you would be so satisfied and like us so much that you’d want to purchase from us in the future…!
The answer? maybe a help!
Visit Helpdesk and have a look at our downloads page.
On there you’ll find some automated survey forms under ‘Training Analysis: Feedback Forms and Spreadsheet Analysis’.
The forms are in Word, and the analysis sheets are in Excel, and work hand-in-hand
Data input is fast, as it’s a grid, and analysis is done for you.
For help using or customising them, call +618 9279 9779, or email mark@helpdesk.net.au.
We can help…
Hi Craig, I’m sure that we can help you out. Check out our evaluation white paper ‘getting the value out of evaluation’ (http://www.knowledgepool.com/downloads/home.html) as a starting point and then feel free to give us a call on 0870 320 9015. Good luck! Sarah