This is the transcript of the TrainingZONE online workshop led by George Edwards, development officer with the Institute for Supervision and Management on 2 May 2000:-
george edwards: Welcome to this workshop on training zone. I am "in the chair", more as a mderator than an expert, I hasten to say! However, to kick the thing off, can I suggest a natty distinction to be made between "validation", and "evaluation"? VAlidation is the business of whether the training worked in training terms, whether it "did the job" against the objectives, and whether it tells us anything. "Evaluation" is the side that deals with cost benefits. Was it "value"? Tis is the nub of many "problems"; deciding on how to value the results of trainig, and determining what are the factors in a cost-benefit equation.
george edwards: There is in this very little place for the "happy sheet", as it is unlikely the aprticiapnts can objectively determine if objectives set have been met, and even if they can, this won't really show "value" - that will come along post-course with business benefits
george edwards: Well, I had put some words of wisdom up earlier on to welcome you all, but they've been zapped!
george edwards: However, to get the conference rolling, let me say that a lot of the queries I see on UK HRD and in other places are caused by the confusion of "validation" and "evaluation"
george edwards: "Validation" is about whether the trainig delivered the trainnig objectives. These may or may not be directly linked to the business objectives on which validation is based.
george edwards: Sorry, "on which EVALUATION is based!
george edwards: "Evaluation" is about defining the business benfits gained as a result of the traininig, and attempting to make a judgement as to the value that the training delivered. Sometimes its easy, sometimes its damn near impossible......
george edwards: : Well, I had put some words of wisdom up earlier on to welcome you all, but they've been zapped!
george edwards: Hello! Anyone out there?
Ken Hare: Hi
george edwards: Now, Twice I've put words of wisdom up to get this thing going
Stephanie Phillips: Hi George - thanks for agreeing to run today's workshop. I think we should wait a minute or two to see whether anyone else comes along.
george edwards: and twice they've been zapped by the system!
george edwards: The theme I was hoping we could develop was the distinction between VALIDATION and EVALUATION
Trevor Sherman: Trevor Sherman joining from Northampton. First TrainingZone workshop. Good afternoon
george edwards: and the cases where there are no clear cut cost-benefits to be measured
Stephanie Phillips: Hi Trevor, Ken, Theresa - does anyone want to set the ball rolling on these issues?
george edwards: Or where the training can't lay claim to being the sole cause of the cost benefits...
Theresa Gallagher: Hi Theresa joining from Core Attribute Development.
Ken Hare: My group has always had an easier time validating things like books. Evaluating them was more obscure.
george edwards: Thought I'd just set ball rolling. (Shrugs)
george edwards: Sorry ken, validating books?
Stephanie Phillips: Yes, sorry - bad choice of words!
Ken Hare: repair manuals or class materials. If they are "validated" they are accurate. If they are "effective" behaviors have been modified.
Trevor Sherman: I think you've touched an important point George - it's sometimes difficult to "keep other things constant" when you're evaluating
george edwards: On the theme of validity and these manuals, validity would be about whether they were complete, accurate, up to date, and likely to provide what the user needed
Ken Hare: correct
george edwards: The resultant behaviours contribute to the "value"
george edwards: IHowever, even here other things have abn effect. Changes in tooling say, new premises, lots of things that impact on the conversion of a valid learning process into a value
Ken Hare: yes, if a class has been successful, there has been a value added. But the degree is hard to measure at times.
george edwards: Only if you believe that the "successful class" predicts that they will implement better practices, and make a business gain.
george edwards: Essentially this is one of the main problems - does your successful traing event predict future performance?
Trevor Sherman: Which means the "gains" should have been modelled/predicted before the class
george edwards: And the traing objectives designed to make it more probable that learning will translate into better performance IN THe JOB!
Stephanie Phillips: Pre-course preparation I have used always asks for objectives to be set along these lines.
Ken Hare: Many times, there are semantics involved. Higher ups can see numbers of students, etc. When students return to their places of business, we can't tell if they are more successful
george edwards: This is why it is oftten the case that trainng is not evaluated at all -
Trevor Sherman: Pre-course objectives also need an "owner" who will sign up to them and take part responsibility for JOB performance (they are often the budget owners!)
george edwards: Even in company traing depts find it hard to gain access to the info necessary to evaluate
Stephanie Phillips: There often seem to be administrative problems to overcome - people don't like form-filling.
Trevor Sherman: Why would this info not be available to them George?
george edwards: The line managers don't necessarily work with trainers to show value gained
george edwards: And of course, trainers are often ot aware that the value gained is only partly due to traing given
george edwards: Anyone lurking want to add to this?
Trevor Sherman: Is this because training is often seen as a bolt-on? Something line managers don't think they need to be involved in?
george edwards: Yes. Big mistake of trainig world no 14. Forgetting its the line manager's duty to translate a trained employee into a experienced employee
Stephanie Phillips: Quite often training is seen as a quick fix , regardless of whether it covers hard or soft skills, I would agree that managers don't think they need to follow up the training undertaken.
Trevor Sherman: Value gain - do you think the evaluation process extends to measuring the effectiveness of the contract between the trainer and the line manager?
Liz Speake: Hi there! I can't get into the transcript, so can't see have you've got from evaluation into this!
george edwards: Training CAN be a quick fix. If a half day on correct procedures results in fewer errors, there is a real gain and a quick fix. But soft skills ... how to evaluate this... it may fix, but it usually ain't quaick
Stephanie Phillips: Can we discuss evaluating soft skills? This has always been a thorny issue for me!
Trevor Sherman: If soft skills have job based assignments then it involves three parties in achieving the gain - the trainer, the student and the line manager (as coach)
george edwards: you've got from evaluation into this! ----- Well, because evaluation is a business judgenment, based on business gains. Often the trainers don't get to see or hear of them.
george edwards: Lets put it this way. What would the evaluation criteria be for a programme that delivered coaching skiulls to supervisors?
Ken Hare: Employees who don't shoot up the place???
Trevor Sherman: A suggestion - confidence in the coachee
george edwards: Now put a value on it, and ask what percentage of the value derives from the training directly - or indirectly but attributable?
Trevor Sherman: To put a value on it you'll need a before and after situation - either, generally or for individual supervisors
george edwards: For example, no longer having to buy soft skills traing for some staff, because the coaches are doing it, presents a cash figure
Trevor Sherman: Not losing staff who are not "cared for"
george edwards: But there will also be a cluster of gains around job satisfaction, betteer relationships, etc. Harder to put a value on these.
Stephanie Phillips: Yes, that's clear - how would you set your timescales for the after situation though?
Ken Hare: True, we are becomming more concerned with tallent preservation...
Trevor Sherman: If the coaching has immediate application to the supervisor/staff, the "after" can be within 2 weeks of training - what do you think?
george edwards: Knowledeg nmanagement is current, but how does one put real value to it? Evaluating training is not about the sort of speculation that dot com enjoys - to be credible trainers need to say, "Here are the fiugures that show the spend was worth it"
george edwards: You can't show that sort of thing - for the example given - in a fortnifght
george edwards: You CAN in a simpler case
george edwards: Such as the "processing errors" quick fix example earlier
Trevor Sherman: George - now we're returning to your theme of cost-benefit analysis - what tools are avaialble for doing this?
george edwards: Your best "tools" are the cost of labour, and the time a process takes.
george edwards: This works for service delivery too
george edwards: Who is lurking now?
Ken Hare: Can't it also be divided among the studnets reached?
Trevor Sherman: Another tool could be cost associated with errors - in service delivery this would be lost customers and their value (or cost to replace)
george edwards: Yes. But "reach" has to mean that they then DO something that can have a value put to it
george edwards: Absolutely Trevor.
george edwards: And all service companies should be able to estimate the cost to gain a customer
george edwards: In for eg, visits, adverts, PR effort, etc. T
george edwards: There will be a figure... it just has to be got to.
Trevor Sherman: Thanks for the name-check George - its my first TZ online Workshop and your name on the screen gives a buz!!
george edwards: So reduce the time to gain a customer, and claim the proportion due (probably) to better trained staff
Stephanie Phillips: I can see there are a few people who haven't added anything to the discussion yet - please don't hesitate to join in at any stage.
george edwards: Flattered!
Trevor Sherman: The customer lifetime value calculations are fightening. I've done some in retail motor trade and figures of £250,000 can apply over 10 years
george edwards: Stephanie - editor, M'aaam, I keep asking the lurkers to come on in, but hey...
george edwards: Quote f £250,000 can apply over 10 years - so let's get to the effect trainng has on that figure, not just in feel good factors
Ettie Mccormack: Cost benefit analysis is not the same thing as ROI, and being able to evaluate success v performance is one of the hardest things, if not the hardest, that any training provider can quantify in their delivery objectives
george edwards: But trainign does need to show ROI
george edwards: Cause it is an "I"
george edwards: Until it produces an "R"
Trevor Sherman: ROI is the upside - the other side is often about financial risk of "not doing"
george edwards: OK, good. NOT TRAINING
george edwards: What will happen if you don't? Definitely?
george edwards: Or probably?
george edwards: Or just possibly?
Ken Hare: Cost avoidance factors seem hard to pin down also. They are infinite...
george edwards: And is the risk associated with the just poissibly (death?) going to be justified if you are later asked why the training stopped?
red: it appears easier to assess it training results than soft-skills?
george edwards: Try suspending legislatively driven training ...
george edwards: Hi Red. At Last!
george edwards: IT is a special case, but only in as much as new skills should tranlate into faster/cheaper/better performance by OIT users -
george edwards: OK, what do you say ruth?
red: i thought all training should result in faster/cheaper/better performance
Trevor Sherman: Summary point - isn't a lot of this about who is paying for the training? The basis of assessment needs to meet their needs (ROI/risk/compliance/etc.)
george edwards: Yep
george edwards: red's point is that all trainining SHOULD result in etcS
george edwards: The organisations evaluation neds to show it does
george edwards: Or why are they paying at all?
Trevor Sherman: What's OIT pls?
george edwards: mistyped IT
Trevor Sherman: Thanks
george edwards: ken, you stll there?
Elaine England: Hi but often the clients won't pay for the time and effort you need for evaluating any type of results. They want them but won't pay!
Ken Hare: Yep
red: I find the organisation's aims and the delegates reason to attend can be 2 completely diff reasons
george edwards: Especially in IT, eh?
red: not just IT - a day off rom work in some cases!!
george edwards: "Hey man, let me get that certificate and I'm outta here!"
red: correct
Trevor Sherman: So - are we still agreed cost-benefit is hard, but it must be done?
george edwards: That is the same as other trainig, just moves faster in IT!
george edwards: Not MUST
george edwards: There can be cases where it isn't done, as a policy
Trevor Sherman: For example?
red: agreed cost benefits be done - but who does it for who?
Ken Hare: I guess some safety training myst be done no matter the expense.
george edwards: For example, it may be felt that the results of some training (stress) are self evident, and too hard to value to be worth actually bottom lining
george edwards: There's no point in spending more on evaluating than on trainng!
Trevor Sherman: So, you're saying another form of evaluation is done - e.g. survey, observation, etc - with no financial?
george edwards: But a lot of training can show quick and dirty cost benefits
Elaine England: Interesting point re how much to spend on each
george edwards: NO FINNACIAL - is a policy dcision, but may be right in a lot of cases.
george edwards: Its false to claim that there are ever NO cost benefits to find, but how useful is it to know them?
Trevor Sherman: Bottom line on stress can be judged from cost of days off, cost of employee conselling, errors, knoock on to other staff, etc.
george edwards: Best sales tool of freelance trainers is to be able to say, look universal international did this programme, and they reckoned it saved them a hundred thousand in a year..."
Stephanie Phillips: We'll have to wind things up shortly I'm afraid - any final comments?
Trevor Sherman: That was a short 45 minutes. Thanks a lot George and others, I'll come back again whn
george edwards: Thanks Stephanie
george edwards: Lets try and get a few more in next time!
Trevor Sherman: ... when i can (sorry pressed enter too soon)
Stephanie Phillips: Many thanks George - a very stimulating discussion, lots of input - great!
george edwards: Thanks to all who came along. See you in cyberspace again! Off
