Whenever there is a discussion about evaluation, almost the same words and phrases pop up. I’m talking about things like ROI, Kirkpatricks, Phillips and Success Case Method. What this implies is that we are not short of ideas when it comes to evaluation, we just don’t do it because it’s hard. Did I just say it’s hard? Well, no I don’t think it’s hard, we just have to think about it differently. Just today I was listening to a GoodPractice podcast titled, the Business Impact of L&D. The guest on the show who is an L&D practitioner, but also an experienced business person tore L&D apart on the subject of ROI. One of the things he made clear was, when most of us in l&d talk about ROI, we really don’t know what we are talking about (he’s an accountant, what do you expect). His advice is, leave ROI alone because it will probably never be a viable way of evaluating the impact of l&d on the business. There are lots of other sensible things he said, which I can’t relate here. I do recommend listening to the podcast on the GoodPractice website, you can access it here.
What about Kirkpatricks? Lots of l&d people criticize it as being outdated and not useful. I wouldn’t go that far. Here’s why, to start with Kirkpatrick’s is not an active model, by that I mean, it doesn’t show you how to do anything. Rather it points out the various levels at which we should be evaluating learning. Now whether you agree with the Kirkpatrick levels or not is another story altogether. I prefer the Success Case Method from Brinkerhoff which focuses on both qualitative and quantitative analysis of programmes to identify their impact. As you must have deduced from my statement, it is not restricted to learning and if you read the book on Success Case Method, you will see the framework applied to domains beyond learning and development. You can read a review of the book here.
But whether it’s Kirkpatricks or Success Case Method that you intend to adopt, they both need one major thing to work and that is data. When we talk about evaluation being difficult, what we are really saying is, I don’t know which data to collect or I don’t have time to collect data or I collected the wrong data. At its core evaluation is about collecting data and not frameworks. To evaluate you need to collect data that validates the change or impact your learning intervention is trying to achieve and that is where we start the conversation of evaluation from, not from ROI or which evaluation model works best.
