This is the transcript from the EvaluationZone workshop held on Friday 1 March 2002 on the subject of the Kirkpatrick model of evaluation.
EvaluationZone Admin: Hi Paul Kearns here - do you want to say hello?
Stephanie Phillips: Hi Paul. Is that Linda there?
TrainingZONE test: Hi, yes . It's Linda Tate.
EvaluationZone Admin: Hi Linda.
Stephanie Phillips: Paul, Linda is reviewing EvaluationZone. Linda, do you want to ask Paul anything about the Zone while we wait for other subscribers?
TrainingZONE test: I suppose as it's the first time I am unsure about what is likely to be the format for the event. Can you explain that please?
EvaluationZone Admin: Basically this is an opportunity to just air certain issues. This format does not lend itself to in-depth discussion. For that, we would use other means, but people can ask specific questions.
EvaluationZone Admin: One interesting point - 900 people read the intro to this workshop.
Stephanie Phillips: Linda - what do you think to the Zone so far, having had a look around?
TrainingZONE test: It certainly shows the amount of interest - if 900 sign in there could be a lot of discussion. Do you anticipate it will be about the methods used to evaluate or the actual evaluation?
EvaluationZone Admin: It's mainly about methods. The reason evaluation is so problematic is because the old methods don't do the job.
TrainingZONE test: About the Zone - I am still looking about. On methods have you found IIP has influenced change?
EvaluationZone Admin: IiP put evaluation right at the top of agenda
Stephanie Phillips: Hi again Bill!
EvaluationZone Admin: Hi Bill.
Bill Barker: Hi Stephanie and Admin - only us?
EvaluationZone Admin: I suggest we wait a couple of more minutes.
Stephanie Phillips: Yes, so far, but Linda has come to look around too - that's her under TrainingZONE test.
EvaluationZone Admin: Bill I prefer Paul to Admin!
Bill Barker: OK - how are you both?
EvaluationZone Admin: Raring to go.
Stephanie Phillips: Good, thanks. We got a lot of interest in this workshop - 900 people read the intro. Hopefully we'll get a few of them today.
EvaluationZone Admin: Maybe this time we'll slow the pace down a bit Bill.
Bill Barker: I hope so - it was my first try last time.
Stephanie Phillips: How did you get on with the 3 box system stuff after the last session?
Bill Barker: I have tried to evaluate by the 3 box system and found it useful.
EvaluationZone Admin: Great.
EvaluationZone Admin: Shall we kick off?
Stephanie Phillips: Yes - let's. We'll see if anyone else comes along in a bit.
Bill Barker: Fine by me.
EvaluationZone Admin: I'll try to address the questions raised in the intro article. Kirkpatrick is normally seen as a 4 level model, but actually it isn't, it's a 5 level model. Even Kirkpatrick realised that pre-training measures are very important, but he did not emphasise the baseline measures as a level. On EvaluationZone, we emphasise the baseline, because you cannot measure improvement without having a pre measure. The question is what pre measures do we need - any ideas?
Bill Barker: Performance, ability?
EvaluationZone Admin: Yes - particularly performance. This is easy in sales - sales per person. In many other areas it is more difficult but the principle is sound. So we need a good performance measurement system. Very few trainers have this, what do you have Bill?
Bill Barker: PRA for senior posts. For non-management we have sales measures, fault clearance levels - more...engineering is as easy as sales I guess, some admin posts cause problems...
EvaluationZone Admin: Do you design training to directly impact on these?
Bill Barker: Yes we do design training to address.
EvaluationZone Admin: Okay, the next big question missed by Kirkpatrick is value. What value would a 1% increase by engineers be worth?
Bill Barker:Hard to say - a faster clearance rate = more equipment in use to generate sales / profit.
EvaluationZone Admin: That's the point. We may know in theory but we need to articulate in reality. Do engineers know what their own training is worth? How can we use this to design training?
Bill Barker:It's difficult...
EvaluationZone Admin: Difficult maybe, but without it we may as well forget evaluation. Let's look at more issues. What level do you evaluate at?
Bill Barker: I'm not sure what you mean?
EvaluationZone Admin: Happy sheets, tests?
Bill Barker: Both of those.
EvaluationZone Admin: These are helpful but they don't tell us that the training worked.
Bill Barker:True - we use line managers to tell is things are better afterwards.
EvaluationZone Admin: So you do level 3 - observation after the training?
Bill Barker:Yes.
EvaluationZone Admin: Do you ask managers, before the training, what 'better' would look like?
Bill Barker:Not always, but often.
EvaluationZone Admin: The baseline tells us we should do this as often as possible, because it checks the managers' commitment, and it makes them accountable.
Bill Barker:Okay.
EvaluationZone Admin: Have you done any level 4?
Bill Barker:Sorry, I have had little time to pre-read.
EvaluationZone Admin: Okay. Level 4 means organisational impact - profit, cost, revenue. It's very difficult.
Bill Barker: We refer to that as the balanced scorecard.
EvaluationZone Admin: The scorecard is Okay, but does it link one trainee to one performance measure?
Bill Barker:No - global issues.
EvaluationZone Admin: Exactly. We can't train 'globals', we only train individuals. We need performance measures for each individual. Back to the performance measurement problem.
Bill Barker: Yes.
EvaluationZone Admin: Afterwards look at the performance measurement pages. There are some simple ideas to start you off. What about evaluation at individual and team level?
Bill Barker:Thanks, I will.
EvaluationZone Admin: This can confuse - even if we have a team performance target we can only train individuals to help achieve it. They all have to own performance measures. This leads to another question - continuous improvement. Kirkpatrick is not seen as a cyclical, continuous process, but evaluation has to be continuous. We feedback bad and good results. This changes behaviour - the whole reason for training. Do you use continuous improvement systems?
Bill Barker:No, we don't.
EvaluationZone Admin: This should be a priority. Have you heard of the PDCA cycle?
Bill Barker:No.
EvaluationZone Admin: It stands for Plan, Do, Check, Act. It is simple and powerful. Take one of your engineers. Ask them to choose a performance measure to focus on, then ask them to plan to improve it. When they have 'done' something ask them to check (evaluate) whether it worked.
EvaluationZone Admin: Depending on the result they have to Act on this (or review it). It is very much like Kolb's learning cycle.
Bill Barker:It is the systematic approach - cyclical. It applies to most things, not just training/evaluation.
EvaluationZone Admin: We learn from experience. Yes it applies to everything. Kolb, PDCA and the baseline model go hand in hand. I will put some stuff on the site to help you.
Bill Barker:Thanks Paul.
EvaluationZone Admin: How do people at your co handle feedback?
Bill Barker:Well if addressed correctly - constructive.
EvaluationZone Admin: And if not?....
Bill Barker:That can be destructive. We try to train on delivery.
EvaluationZone Admin:Evaluation and blame cultures do not work well together. If we are not constructive evaluation will be used as a big stick. I suggest you introduce evaluation efforts with managers you like and trust first. Do you have one in mind?
Bill Barker: I have one such who I am working with.
EvaluationZone Admin: What job title?
Bill Barker: Business Solution Provision Head.
EvaluationZone Admin: Sounds a good place to start. Ask them what value means!
Bill Barker: Oh he knows!
EvaluationZone Admin: Good. Now ask him how he thinks training can add value - you may need to help him. Try by asking about individual performance figures - commission, sales, etc. Then ask who needs training - the best or the worst? What would you go for?
Bill Barker: A lot is currently just "feelings" and intuition. We would go for the worst.
EvaluationZone Admin: Hey, we don't want intuition - I can't evaluate intuition, it's too fickle!! I would usually train the worst first too - any good reasons why?
Bill Barker: It's a better common denominator if they are closer together?
EvaluationZone Admin: That's one. How about they have a good reason to improve as well! And, training them will have a knock on benefit.
Bill Barker: True, there's personal buy-in.
EvaluationZone Admin: Good performers see standards being raised, but we still need individual performance measures first.
Bill Barker: True.
EvaluationZone Admin: Ask your manager to score people he knows - subjectively, On a 1 to 10 scale.
Bill Barker: We can do that, I am sure.
EvaluationZone Admin: ...where 1 is awful and 10 brilliant. When he has scored them, ask why they got that score, and what they need to do to get a higher score. Then, you are starting to look at possible training interventions.
Bill Barker: Got it.
EvaluationZone Admin: Okay, any other questions on evaluation models?
Bill Barker: Not at the moment - I need to spend more time learning.
EvaluationZone Admin: Okay, but watch out for the level 5.
Bill Barker: Which is...?
EvaluationZone Admin: This is something started in the US by Jack Phillips. It refers to ROI. He says Level 4 has to be converted to ROI at an extra level. This is a red herring. Level 4 should also be good enough for level 5. Okay?
Bill Barker: 50/50.
EvaluationZone Admin: OK we'll cover it at another time. I hope you like the personal tuition!
Stephanie Phillips: Hopefully we'll get a few more subscribers logging on next time!
EvaluationZone Admin: Feel free to post any questions on the Forum and we'll answer them or raise a discussion.
Bill Barker: I do enjoy it. My problem is knowing when you expect a response - I start one and you have carried on.
EvaluationZone Admin: Thats Okay. We're all getting used to it.
Stephanie Phillips: It does definitely take a bit of getting used to!
EvaluationZone Admin: Any questions Linda finally?
TrainingZONE test: No. It was good. It sounded like Bill got a lot out of the discussion. It was a shame more did not login but I gathered there were a lot last time.
EvaluationZone Admin: Yes, obviously time is precious, but I'm sure we will continue to grow.
TrainingZONE test: Thanks.
EvaluationZone Admin: Okay, bye everyone for now.
Bill Barker: OK Bye and thanks.
Stephanie Phillips: Thanks all - bye.
TrainingZONE test: Yes bye.
