10 Responses

1. this is going to be virtually impossible unless…..

   Hi Leon

   Unless you are given access to commercially sensitive data you are not going to be able to meaningfully carry out any evaluation at this level.

   As an external consultant you are usually limited to level one evaluation

   If you have the opportunity to carry out follw on workshops/coaching sessions you may well be able to assess level two outcomes.  You are unlikely to be able to observe level three but you may, if you have a close partnership relationship with the client, get some level three reports.  Very few clients will let you have the opportunity to see any level four outcomes.  This may be possible IF you trained company wide and are able to see a declared improvement in business results.

   Rus

2. Evaluating training

   Hi Leon

    

   It is a challenge to get longer term evidence of improvement if you only have a short training session with individuals.  I was in a similar position, not within the travel sector, but similar circumstances. We got the the instant feedback from the course and then also did follow calls to participants, with their permission to see how they were implementing what they’d learned.  On the course feedback form we asked them what they were going to do with what they’d learned and could we contact them in 6 months. Most people said yes.

   I know it would be indirect but if you could get sales and marketing trends for performance you might be able to plot when you started training with the company and see if there are any improvements.  Otherwise it is just about the feedback on the day.

   I don’t know how the training contract is agreed but would there be any opening to build in follow up sessions with a cross section of delegates.  You could do this either direct with the delegate or through their line manager as part of their PDP possibly.

    

   All the best

    

   Anne

    

    

3. Pre & Post

   In my experience as working as a third party supplier on open programmes where the participants come from virtually any background, unless you have undertaken individual upstream reviews of why they are attending and what they hope to get it is exceedingly difficult (if not impossible) to ensure that ‘Money Out’ equals ‘Money In’ back in the workplace. However, it is not always impossible and there are two different ways of looking at ROI. The return might be financial, but it might equally be specified behavioural change; a behavioural return that is not necessarily quantified in terms financial impact but behavioural outcomes and changes in measurable results.

    One of the strategies we have adopted is to get line managers and participants to review what the business case is for attending the programme. This process of reviewing has to be guided, but essentially the questions are: What will be done more efficiently or effectively? (This must be defined in terms of measurable outcomes and results. Vague ambiguous responses almost always indicate there has been no real identification of any performance gap and whether this performance gap is indeed a developmental need. In which case ROI will return a negative.) Where however there is a clearly defined response you have a metric that can be base lined and followed up. After the developmental process line managers are contacted and reminded of the programmes content and what the business case was for submitting the individual for development, they are adviced and given a process for setting targets and tracking progress. 2 months later a 10% follow up is undertaken to review progress and results.

    I have materials and systems that I can gladly share with you provided they are not forwarded to third parties.

    

4. Evaluation

   The evaluation author, Leslie Rae, sadly passed away a couple of years ago. Right to the end he argued firmly that it is the investor who should evaluate the return – in other words, it is the companies or sponsors who paid for people to attend the training that should make sure they got value for money, they should pusue the transfer of learning and, if they wish, measure any ROI. He felt that trainers have a responsibility to evaluate the training – the quality, relevance, etc.

   I argued that in today’s world a trainer has to think beyond the point of delivery. They should negotiate, where feasible, in advance of any training the success factors and how they might subsequently be pursued and measured. I argued that even with people coming from different organisations to a pre-designed open programme this is possible through things like pre-course questionnaires that involve the line manager and post course follow ups by phone, for example. Surely this is now accepted good practice, I argued, and that his position is now old fashioned. (I immediately regretted this rather perjorative term).

   Leslie acknowledged my views with taking umbrage. He simply said that this was fine in principle but who pays for all this work, especially when then is no gaurantee that those who can do something about the results – typically the managers of those attending – may not pay any attention. Unless each and every sponsoring organisation also sponsors the evaluation then you risk wasting a lot of energy and time, even if you can get decent evidence. That is today’s reality, he said. On top of that, even if you can pursue the performance of people across different organisations, it will be hard to identify consistent impacts. And even if you can get access to all those organisations, getting sufficient responses will be very difficult.

   I am still convinced that trainers ought to do as much as they can to drive evaluation. But, maybe Leslie was right too, the practicalities for this type of situation mean that you have to be pragmatic about what is possible.

   Graham

5. Leslie Rae Dead?

   Les Rae is dead? I use to work with him at Ranmoor Hall in Sheffield. He used to come round the offices and try out his latest material on us.

   I remember he was scheduled to do a session on Time Management on one of the events I was running and deliberately turned up late to show how annoying it was when people didn’t meet deadlines, only Les could do that and get away with it.

6. ROI

   Hi Leon

   I agree with the comment above that ROI is essentially the remit of the host organisation – because there are so many other things that need to be in place by the host organisation for the learning or training to be successful. Having said that, I do think that in certain situations a trainer can demonstrate some clear gains.

   Regarding ROI itself my own view is that the gains have to be measured financially to be ROI – this can either be by, for example, by a person producing more of their products per hour or a change in behaviour or skill that leads to a reduced time to deal with something.

   I wonder whether you could incorporate aspects of the following example into what you do – perhaps by suggesting to the clients the benefits of being able to quantify the successes of what you deliver? … 

   I recently delivered a 11 day (spread over 6 months) Management and Leadership Programme to newly appointed Contact Centre first line managers. The final day was presentations to senior management as to what they were doing differently as a result of the learning. I had requested that I be able to include some ROI measurement in their presentations, and this was agreed. Not all were able to show any, but some showed a potential first year ROI of up to 1000%, although that was a one-off and the average was around 300%. This was demonstrated through reduced call times for staff they were managing, reduced call transfer rate and more effective management of teams and their workloads, etc. The data used was verified by contract managers and the financial returns were ‘signed off’ by finance staff.

   Regards,

   Paul

    

    

7. thanks

   Hi Rus

   Many thanks on your straightforward answer. This has really helped me to position my argument.

   Thanks and much appreciated

   Leon

8. thanks

   Hi Anne. Thanks a lot for your reply. It all makes total sense, and you were just confirming what I had suspected all along ! 

   This is great info though in helping me position my argument for my assignment.

   Many thanks

   Leon

    

9. interesting repsonse !

   Hi Graham

   I really liked what you wrote. It has turned a lot of what I have learned so far on a theoretical basis, completely on its head ! 

   Great food for thought though – thanks a mill

   Leon

10. ROI

    Hi Paul. Thanks very much for your interesting reply. I can see the effectiveness of your ROI plan there. The only thing with me is that

    I am dealing with over 300 different ‘clients’. I.e. I can be called in at any given time to train a small or large group of travel agents on

    many different products/areas of learning. The logistics of this in terms of following up and agreeing evaluation with line managers

    and senior staff would be a nightmare – I think buy-in and commitment on their behalf would be the biggest barrier.

    The crux of the problem is that I deliver short sharp sessions that are very much sales/promotion based in content, aimed at increasing

    travel agents’ awareness of our products as well as motivating them to sell more.

    Anyway I really appreciate your response and will consider your comments when making my arguments ! 

    Cheers

    Leon