We recently ran an interview with Donald Kirkpatrick the 'father' of evaluation - where we gave members a chance to ask questions of the great man himself. Here are his first set of answers to your evaluation concerns.
Question: It was great fun reading your interview with TrainingZone.co.uk. I have been in the corporate training industry for some time and evaluation of trainings is quite a concern area due to individualistic opinions and pressures from the HRs of concerned trainees.
I have often noticed that many participants are more eager to evaluate than learn or get trained from a training session. To the extent that some even refuse to let concepts sink in if the training medium or delivery is not what they think is right! What to do with people who try to influence evaluation and how to correct these needless and fruitless habits?
Thanks in advance.
Ravichandran Jv
Answer: You are right that some learners are more interested in the way the material is presented than in the content. I have an answer for that: don't blame the learner. Blame the trainer!
I recognise this is the situation in all, or nearly all, programs. Some learners have been forced to attend or are lukewarm on the content. This is the challenge for all presenters whether they be trainers, speakers, teachers, preachers, etc.
I use an approach that I call PIE. It stands for Practical (meets the needs or wants of the listeners); Interesting (holds their attention); and Enjoyable (they like the stories, illustrations, activities, aids, etc.) Try it!
Question: What do you need to do to ensure that the evaluation of training delivered is undertaken as objectively as possible and that the result of the evaluation has a constructive impact?
What does a training evaluation system that works look like? What are the important steps/rules to follow?
How do we confidently eliminate/isolate other factors that may have had an impact on any positive change and pinpoint that any change is definitely down to the training intervention?
Sharon Abbott
Answer: The first question. In my acronym PIE, the word Practical says it all. The second question. My four levels (Reaction, Learning, Behavior and Results) say it all.
The third question is more complicated. One way is to use
experimental groups (receive the training) and control groups (do not receive the training). This has a number of flaws, the main one being sure that the groups are identical on all factors that might be involved. Another possible problem is that the experimental groups communicate what they learned to their co-workers who are in the control groups. Let me suggest another way.
In one organization, the level of turnover among new employees was 5% or 6% per month. The training people felt that it was due to the poor job that supervisors were doing in orienting and training new employees. They decided to train them on how to orient and train new employees, and they set a goal of reducing turnover to 1% or 2% per month. They succeeded and began to pat themselves on the back for accomplishing their objective. A manager asked: 'Are you sure that the turnover was due to the lack of training?' And the answer was something like 'quite sure'.
When we look at this case study, we would consider what other factors might have caused the reduction in turnover. Three possibilities are: the employment people were hiring mature people instead of kids right out of high school; the company had just increased the pay for new employees; and the economy has gone bad and there are no other jobs available.
It would have been pretty easy to check and see if any of these are true and thereby eliminate them from a possible reason for the turnover.
Question: My organization seems to think that the model no longer has validity as it has no reference to ROI. Thoughts?
Maxine Clark
Answer: I feel sorry for you if your organization feels that way. And I would like to tell them how wrong they are!
First of all, ROI has many flaws. Without the four levels as the basis, it would not be able to show any relationship between improvements and the reason for them, whether through training or some other intervention. Jack Phillips, the originator of ROI, even calls it Level 5! This means, of course that my four levels form the basis for it.
I never called ROI Level 5 because I don't think it is a level above Level 4. It is one way of attempting to measure results. I like to refer to this kind of measure as ROE, Return on Expectations. And the expectations would be determined by the jury – top management.
Question: I found your comparison of top executives as jury very apt because from my experience (as an award entry writer) it is all too often the case that it is awards judges (following the legal analogy) that inspire people to do level 4 evaluations when they are pot hunting. It is still very rare in FTSE 100 and SMEs alike for executives to demand or even suggest that training departments prove business impact. So my question is this, surely the challenge in evaluation still lies in educating top executives to make them make their training departments more accountable, so are you taking your crusade to those who develop MBA and executive development programs to change behaviours there?
Chris Robinson
Answer: I agree that not many executives – the jury – suggest (or demand) that training departments prove business impact. More and more of them, however, are looking for more evidence than just the number of programs, the number of learners, the reaction to the programs, and learning that takes place in the classroom are sufficient evidence to justify the training budget. And the day of reckoning will come in every organization in case it hasn't already come. And that is why measuring behavior and results become more urgent.
Regarding your comment about MBA and executive development programs, I have not tried to sell my four levels to them. However, I hear from many graduates that the Kirkpatrick Model is often taught in these programs.
Question: Prof. Kurt Kraiger has made some comments about Prof. Kirkpatrick's evaluation model over the years. How does Prof. Kirkpatrick respond to these comments and how does he see his model developing?
All the best
Paul Whitehead
Answer: I am sorry, Paul, I am not familiar with the comments that Kurt Kraiger has made. In fact, I have never heard his name.
Regarding how is my model, I don't like your word 'developing' because the model remains the same as it was when I first used it in my dissertation. There are many things that have been added to it by my son Jim and others including the concept of a 'business partnership model'. Also, many new ideas have been added to my original examples, such as focus groups and success stories. But the model has stood the test of time (over 50 years) and has been translated into four languages and is currently used all over the world. The only changes to the model have been used by Phillips who has substituted the word 'implement' for behavior and 'impact' for results.
Question: There isn't a week that goes by without me explaining to delegates or colleagues your principles of evaluation. All my learning and development colleagues do the same and we probably say the name 'Kirkpatrick' at least every other day in the office! When you came up with the model did you instantly know how popular it was going to be? Was it a eureka moment that you knew would change the face of training evaluation as people knew it?
Ian Francis
Answer: Thank you, Ian for your kind words.
When I did my dissertation, I had done some research and decided to measure the effectiveness of a leadership program I was teaching at the University of Wisconsin Management Institute in Madison. I decided to measure reaction, learning, behavior and results. I did the research, completed my dissertation and received my Ph.D. And I went back to work teaching the courses.
Five years later, Bob Craig, editor of the Journal for the American Society for Training and Development (ASTD) called and asked me to write an article on evaluation because he had heard about my dissertation. Instead of writing one article, I wrote four, one for each word I had used.
The word spread like wildfire and I was overwhelmed with invitations to present at national training programs (ASTD, Training magazine, IQPC) and conduct workshops in many organizations. At a workshop at Ford, one of the trainers said that I had come up with four practical terms that all could relate to when they evaluated.
Trainers began to talk about the Kirkpatrick Model and the Four Levels. (I never called them that). Articles were written by organizations that had implemented their training programs at one or more of the four levels. Articles appeared such as How Do You Evaluate Level 3?
It wasn't until 30 years later that I wrote the first book when a friend of mine, Jane Holcomb suggested it.
I had no idea that it would 'change the face of Training Evaluation' as you have put it. Perhaps the Lord was in it and wanted me to do it so that others could benefit from the gift He gave me.
Our great thanks to Donald Kirkpatrick for taking the time to answer members' questions. He has tackled some of the many emailed ones first and we hope to have more answers to those that were posed on site in the future.
Please note that we have deliberately left the American spellings in!
Read Annie Hayes's original interview with Donald Kirkpatrick
