7 Responses

1. It’s not the process…
   Hi Gary,

   It’s hard to simplify an already simple process (I’m making assumptions here!).

   I suspect the reason why you are having to chase managers for this data is that they don’t value the evaluation process enough to give it attention, not thet there is something wrong per se with the process.

   Tackle this situation as if you would tackle any performance challenge:

   1. describe the actual problem and desired outcomes, in detail. Use measures and observable behaviours that are relevant to the situation. This will be your baseline.

   2. determine the root causes for the current behaviours and situation.

   3. select enough of the biggest contributing root causes that, if tackled, would enable you to just achieve the required outcomes described in step 1.

   4. develop interventions that will reduce/eliminate these root causes (and not introduce additional root causes!). You can do this in part by building an impact model, working backwards from the measures and behaviours that describe the required outcome, indentifying the skills, knowledge, behaviours, attitudes, processes, policies etc needed to deliver the outcome.

   5. deploy your interventions!

   6. revisit the work you did in step 1, where you built your baseline, to see the extent the required outcomes have been achieved. Any shortfall should be investigated in a similar way, as a continuous improvement project. Any ‘over achievement’ should also be investigated for future learning to see if the overachievement was worth the investment – could the minimum required outcomes have been achieved for less? Use this to evolve the whole process towards being more efficient in use of resources…

   I can happily send you an article describing these stages and the tools needed for each. Just email me at martin [at] p-nrg.com

   Good luck & best wishes,

   Martin

2. Try using the right model
   You seem to have missed a step at the beginning – the Baseline. This would ensure you won’t have a problem with level 3

3. What’s In It For Me?
   This is a problem I have come across many times. Level 3 requires that follow-up with the manager at a later date and often for those managers it is hard to see the link between the training and today. When managers do reply it can be so vague that it is difficult to truly evaluate the impact.

   In my experience it is important for the manager to be able to answer the age old question “what’s in it for me?”. If they cannot see why they should complete the survey they won’t do it. Showing them what is in it for them can take many forms – support from their manager to you, budget impact, ‘free’ training, or better still – tangible results based on previous Level 3 input that has helped to shape learning interventions.

   I personally find that constant and consistent contact with the manager from the L&D department acting as a partner is the most effective. If the manager believes the L&D dept to be a partner and friend rather than simply a training provider, they are more likely to respond to a request to complete a form asking for input.

   It is also important to have the right expectations. If you are looking for 100% completion rates for Level 3 surveys you will always be disappointed. Based on the size of the audience work out what a valid and relevant number of returns is and focus on that target.

4. Many Thanks
   Martin, Paul & Keith, thank you for your comments.

   Sometimes you get so caught up with something that it’s hard to see the wood for the trees and you’ve allowed me to find these again.

   We are now currently reviewing our evaluation policy and are very much looking to engage the parties involved from the outset.

   Thank you again.

5. Level 3 challenges
   Gentlemen,
   It’s interesting to see that you are dealing with similar challenges to evaluating learning as we are here in Canada. I work for the Canadian Federal Government and we have recently formed a training measurement and evaluation Community of Practice which is dedicated to sharing best practices and lessons learned on evaluation and measurement of learning.

   Level 3 is becoming most appealing to those of us in the Public Sector because new ROI methodologies (Phillips’ Level 5) are often not very applicable to us.

   My biggest recommendation in Level 3 studies is to use follow-up face-to-face meetings or telephone calls to get the results you need. Managers/supervisors are often a lot more willing to share information over the phone or while having a coffee. Naturally, this is assuming that you don’t have a huge sample population to target…

   Cheers!

6. Best practice is the Baseline
   Balou Antia – If you put your emphasis and most of your effort into establishing clear Baseline measures and other criteria of success (see https://www.trainingzone.co.uk/cgi-bin/item.cgi?id=122638 )then Level 3 is rarely required at all, I do this for a living and rarely recommend that trainers have to do any level 3.
   Kirkpatrick and Phillips make evaluation a bureaucratic exercise and seem to lose sight of the fact that the whole purpose is learning.
   Regarding ROI and the ‘public sector’ – the principles can be applied just as easily to your sector but again, Philips model doesn’t help.

7. Kirkpatrick Level 3 ideas
   Dear Garry,

   Hello, I am the daughter-in-law of Don Kirkpatrick, who created the 4 Levels. I recently became aware of this site and I am happy to be a contributor!

   One idea I have for you related to Level 3 is to consider how much information you need from the managers of the training participants, and how much you could gather on your own or from the participants themselves.

   At levels 2-4 we recommend obtaining data from 2 sources whenever possible. At level 3, this could be the survey you are currently sending to managers, and perhaps a survey to the training participants themselves.

   It doesn’t have to make it hard!

   Best regards,
   Wendy Kirkpatrick