I had some fabulous responses when I used this forum last time. Once qualified and hoovering up experience, I promise to give back severalfold to the community.
This time, I'd like your ideas on how to administer and also analyse level I (reactionnaires) evaluations and your experiences of the various methods of 2 through to 4/5 (tests, questionnaires,interviews,observation [Hawthorne effect et al]). I had some great, innovative responses, so looking forward to hearing about some new, creative methods of evaluating training.
13 Responses
Get a Monkey!
I use the on-line survet tool, Survey Monkey. Our organisation has moved away from paper based evaluation and now use this for evaluation at levels 0-4 (Kirkpatrick).
The survey tool does all the data gathering but the hard part of analysis stll requires my input. Its pretty cheap too £200 professional subscription per year, our copying bill was more than that plus we had admin staff doing hours of data input so this has helped free up time for other things.
Get a Monkey!
I use the on-line survet tool, Survey Monkey. Our organisation has moved away from paper based evaluation and now use this for evaluation at levels 0-4 (Kirkpatrick).
The survey tool does all the data gathering but the hard part of analysis stll requires my input. Its pretty cheap too £200 professional subscription per year, our copying bill was more than that plus we had admin staff doing hours of data input so this has helped free up time for other things.
First Response – Great! Thank you.
Hello Martin,
Thanks for being the first to respons.
Survey Monkey is great (I use the basic package for my assignments), so it is good to know how much the pro pack is.
How do you conduct level 2 and 3 evaluations using a survey? Are they in the form of tests? If so, do you tailor make them for each course or job spec?
How do you analyse and act on the results?
Best.
— ‘Desola
Evaluation using SM at Levels 2-3
We looked again at our strategy after I met Jim Kirkpatrick at the last training zone live event. His book Training on Trial is worth a read, if you can make it to one of his lectures he might help you decide on a simmilar path to my organisations.
We evaluate learning (level 2) a little later than we did previously. We now do this 2-4 weeks after the training has taken place. The questions are about confirming knowledge, adaptation of skills and behaviour etc begins here.
Level 3 we use a self assessment for the learner and a mirror version of the questions for the line manager to get a balanced opinion on the perception of application of the learning, (typically 6-8 months later) then we build further assessment into normal performance management processes.
We use the data from Survey Monkey to inform on-going quality assurance, making changes as determined by either feedback or organisational changes to policy, legislation, procedure values etc
E Mail
Hello Desola – If you drop me an e mail to gplatt(a)eef.org.uk I’ll happily send you a set of word documents with an accompanying explanation and outline of a process which can go a long way if implemented correctly to achieving what you are asking about.
Evaluating effectiveness of training
We use an online survey tool for level 2 upwards. At the moment we are continuing to use paper reactionnares available at events because we have found the response rate to surveys to be very low.
I’d love to know how others achieve a good response rates.
Evaluation
We use a survey tool for first stage evaluation, but the response rate is very low. would be good to hear how others manage to get a good response. We have used paper forms at the end of the session but found that attendees didn’t have time to reflect.
The next stages we conduct through call monitoring, other performance measures and quizzes
We’ve added a trainer evaluation
Not of the trainer, but by the trainer. As we use a number of internal and external facilitators, we felt it would be useful to ask how they had felt the session had been. We ask about latecomers and non-attendees for audit purposes, how far they felt the objectives were received by the group, any issues they had within the delivery and their suggestions for change. It’s helped to provide a contrast to the L1 data that the participants provide and has improved our service delivery through suggestions and comments made.
survey
Survey monkey has proved succesful for us simple to use once set up. We Have a number of external trainers that are grateful for the feedback
Dave
Evalution Tools
Interesting subject as I am currently completing my CIPD Foundation in Learning & Development practice and we have just looked at Evaluation.
If anyone has any examples they can send me, they would be great (scott.cullen(a)southernrailway.com)
Regards
Scott
Practical ways to Evaluate
Hi,
I’m not sure if this is what you are looking for, but I’ve put together some ideas for practical evaluation at the 4 different levels here. http://www.keystonedevelopment.co.uk/pdf_files/Evaluating_Training_Programmes.pdf
For me the problem is getting commitment to ‘proper’ evaluation. At lot of L&D departments are too keen to move on to the next initiative rather than really analyse the impact of the last one. I did manage to do this one (a long time ago), and was absolutely delighted with the results. A very worth-while activity!
Sheridan Webb
Keystone Development
evaluation
Is there any chance that you could send copies of the survey monkey forms that you have used. We’re using SNAP to try to do the same kind of thingand I have managed to make up PDFs of the SNAP forms.
Thanks
Clare
Evaluation Examples
Hello Scott
Sorry I haven’t responded before now, but I’ve been rather occupied with putting my project to bed.
If you e-mail me on desol@sky.com, I could cobble together relevant abstracts from it which might be of interest (it was awarded a distinction).
— ‘Desola