5 Responses

1. evaluating the business effect
   Hi Jayne

   with regard to the evaluation of the business impact there are two fundamental things you need which, as you are starting after you have finished, you may not have.

   Firstly you have to have accurate benchmarks of “where we were” so that you can now establish “where we are”, so you can work out “how far we’ve come” and assess whether “you have travelled in the right direction” and whether it was “worth the effort”.

   Secondly, and related to this, you need to be sure that you can attribute the whole of/limitation of the journey travelled to the training you initiated.

   Sorry that this is theoretical rather than experiential, I hope it helps
   Rus

2. level 3 & 4 after the event
   Hi Jayne
   Some years ago when I was a newly appointed TM I was asked to evaluate a major organisational wide customer care programme. I was given limited objectives and no needs analysis data. The only way that I could evaluate even a part of the programme was to use our external client satisfaction reports – it showed that the programme was ineffective nothing else. Since then I have always set the evaluation criteria up front of a programme.

   So to your situation…
   To effectively measure level 3 & 4 you need to set the criteria for success before the training starts, along with the measurement process. This will also depend upon the nature of the material (training content) and the participant’s job role.

   While this will now prove very difficult there is some ‘reverse engineering’ activity you can do to add value.

   1) go back to the original TNA and cost the losses that were stated
   2) re-run the TNA and measure the difference in value

   Depending on the type of training activity you can also measure:
   a) Reduced number of dropped calls
   b) Average time on a call
   c) Customer satisfaction scores (assuming you externally measure these)
   d) Service level Agreement performance
   e) Staff satisfaction scores
   f) Changes in staff turnover
   g) Reduced lateness
   h) The capability of the organisation/ team now and cost what it would be to buy in those skills.

   The type of evaluation that you do at levels 3 & 4 will to a certain extent depend upon the use of the data you collect.

   Prior to starting to define these measures think carefully why you want to do this & what you will do with the data.

   Happy to talk more off line if it will help

   Mike Morison

3. evaluating programmes retrospectively
   Hi Jayne

   Can’t use the best evidence ie before and after using any number of evaluation techniques, I can only suggest some 360 evaluation with, if possible,comparison to a matched control group that did not undertake the programme. Alternatively look at achievement v. objectives of the training -are there hard measures such as attrition, absenteesim, appraisal ratings to compare?

4. Case Study
   Jayne – I have a case study of some work we did with a client just recently which I have written up and it is to be published in Training Journal. if you drop me an e mail I’ll be happy to forward a copy to you.

   My e mail is garry.platt(at)wgrange.com

5. success case method
   Hi Jayne,

   If I understand correctly, the training has already taken place?

   If not, then Mike’s suggestion about agreeing the criteria and measurement process upfront is what I’d go for.

   If the training has not already taken place, I’d suggest that

   (a) for future interventions, as Mike says, sort this out up front.

   (b) take a look at Robert Brinkerhoff’s Success Case Method (book available on amazon for not too much). It’s simple, transparent and tested, and is specifically designed for ‘after the training’ and has practical tools to help determine the criteria to evaluate against. It embodies in one sense the sentiments behind the 360 degree approach suggested by Ken.

   I’d also take up Gary’s offer – he’s done some stirling work in this area.

   I have a number of articles printed over the past 18 months in Training Journal and Train The Trainer, both by Fenman, that I’d be happy to email over to you. I’m at martins(at)kaizen-training.com

   Finally, make sure if you do use control groups, that you have credibly discounted all other factors as being significant – generally (though not always – can’t get in to the theory here – look up Taguchi on the web!) you need 2 to the power of ‘n’ groups, where ‘n’ is the number of factors, to properly and fully isolate and quantify the contribution the presence (or not) of each factor makes to the outcome – way too unwieldy and expensive in practice!

   Good luck!

   Cheers

   Martin