In response to Donald H Taylor's article Why Training ROI Doesn't Matter, Garry Platt argues against ignoring this level of evaluation.
Donald H Taylor of Infobasis recently had a featured article published on TrainingZone entitled: "Why Training ROI Doesn’t Matter".
The title immediately grabbed my attention and the content proved equally interesting. The essential thrust of Mr Taylor’s diatribe against ROI is that the entire process is simply not worth it. In the author’s view ROI is very difficult and once completed the results are questionable due to so many exogenous variables affecting the results.
What quickly becomes apparent from reading the article is that the Mr Taylor has a narrow view of ROI and does not appear to be familiar with the already broad church which exists in this area. Nor does he appear to be familiar with the work of Paul Kearns. But despite this apparently small knowledge base he condemns the whole concept out of hand.
Let me state clearly my own position in relation to the matter of ROI:
I view Jack Philips approach to ROI as worse than chronic.
Baseline
Throughout the article Mr Taylor seems to be using Jack Philip’s approach as the metric of best practise, I don’t believe it is. I am familiar with but don’t use it or its equivalents. I employ the Baseline Method which I learnt from Paul Kearns. I see it as far and away the best and most practical approach I have encountered. It negates the complexity argument and adds a valuable stratagem to the trainer’s arsenal. The author I think should have at least familiarised himself with the Kearns model before putting pen to paper and then perhaps he might have changed the argument and direction of his critique?
So, let me to deal with some of the issues I have with this article. The author states: “Apart from training professionals, nobody really cares about training ROI.” This is a general and sweeping statement. Toyota of Derby, Virgin Atlantic Airlines in Crawley, Assa Abloy, FedEx, Scottish Water, I could go on, but you get the point, all these organisations and not just the training functions are focussed on ROI and committed to it as an evaluation strategy. This is not a collective I would choose to call ‘nobody’.
Complexity and compromise
Mr Taylor claims that there are only two reasons why anyone asks for an ROI study. The first being that the individual making the request doesn’t know how complicated it is and so requests it. So, not knowing how complicated something is causes them to ask for it? Does anyone else follow the logic of this argument because I can’t? The second reason is apparently because it ‘”ensures a long period of quiet while that pesky training person goes away and tries to sort it out”. I have encountered numerous reasons why clients ask for an ROI analysis, neither of the reasons listed above have featured in any of them. In reality there are more than two reasons for asking for an ROI analysis, the majority in my experience are genuine.
The author consistently emphasises that an ROI analysis of any worth is overly complex and the results can be compromised by lack of absolute certainty. As his prime example he cites the work of Neil Rackham who he describes as a “scientifically rigorous research psychologist”. Rackham was studying human behaviour in relation to sales and selling in the 1970’s. I question why we are reviewing work done so long ago that the Prime Minister at the time was Ted Heath? Is there nothing contemporary to support the author’s hypothesis? The Huthwaite Corporation during this period was involved in a project which led to studies of more than 35,000 sales calls over a period of 12 years commencing in 1969 and with a final cost of more than $1,000,000. All this was in order to establish the proprietary SPIN® selling system. The cited Motorola report wasn’t an ROI analysis as we know it, it was an independent evaluation of the SPIN selling system conducted by a Martha A Silliman, a third party consultant with a background in measurement and evaluation.
In his article Mr Taylor provides what he views as a damming quote from Rackham: “There are even more tests I'd like to carry out before I'll be totally satisfied that the ideas I've described in this book will significantly improve the results of major sales.” But, this quote is not condemning Silliman’s evaluation project. Read it again - he is actually questioning his own SPIN® system and not invalidating this particular case study, but this is implied by Mr Taylor when he adds: “If he (Rackham) felt less than totally convinced by his ROI study, this is surely a lesson for the rest of us.” Rackham’s quote has in my view been taken out of context. Anyone who wishes to see if they share Mr Taylor’s concerns about Silliman’s research should read the case study for themselves, particularly page 5 & 6:
http://www.huthwaite.se/OmHuthwaite/SPIN-motorola,HRG.pdf
The control group
If I were conducting research into a new cure for cancer the scientific and medical community would insists at some point on control group trials and double blind testing to verify the research. As managers we are not working in a laboratory but rather a commercial and business context and a practical and appropriate level of certainty is quite sufficient. Consequently we can dispense with the control group; it is frequently an irrelevance. I
I have yet to encounter any manager who has insisted on this level of clinical pedantry and if I did it would probably be an accountant who used personality as a form of birth control. Throughout the article Mr Taylor keeps returning to this issue as though without it all is lost, it isn’t. Ask yourself what is ‘reasonable’ in a business context (not a clinical hospital trial) and then apply some common sense. A control group may be appropriate for a scientific paper to prove beyond all quantifiable doubt a point or issue. But if we required this level of proof for all management decisions the entire business world would come to a grinding halt in a cloud of suffocating paralysis. It doesn’t because managers employ prudent judgement based upon realistic and relevant research which doesn’t extend to reports presenting calculations to fifteen decimal places.
External (exogenous) factors
Mr Taylor emphasises in his article how the results of an ROI investigation can be compromised by the impact of external factors influencing the results of the research. He lists such influences as: pre-existing trends; improvement of internal systems which were extraneous to the training; bankruptcy of a competitor; natural demand increases. The list I suggest is virtually endless.
Competent managers are capable of interpreting and understanding finance reports, which positively steam with exogenous factors. Similarly they can read and interpret sales reports which are probably more affected by external influences than any other element in business and yet they can still make sense of them. Marketing intelligence in the form of papers and reports can likewise be reviewed and analysed for their worth and value. Does Mr Taylor believe this ability deserts managers when it comes to an ROI analysis? What phase the moon was in when the training was done or whether Pluto was in conjunction with Uranus has never been a real problem with any ROI analysis I have undertaken because those involved posses a reasonable level of awareness and judgement
The Hawthorn Effect
Mr Taylor raises the issue of the Hawthorne Effect - ie studying someone changes their performance. This is not the first time I have encountered this criticism. It is true any action taken by anybody might influence anything, is this sufficient reason to hold the results invalid or take extreme steps to try and negate the impacts? Probably not. Once again, am I conducting a scientific research project or a simple business analysis? Example: Mr Taylor’s own company; Infobasis presents several case studies on their web site which outlines the positive impacts of their interventions but I don’t see any mention of the results being contaminated by the Hawthorne Effect nor indeed the wider exogenous factors which no doubt played a part. Why? Because presumably they employ a modicum of common sense and expect their readers to do the same. A reasonable assumption which Mr Taylor seems to think might be inappropriate in relation to ROI?
End note
It was with a growing sense of frustration that I read Donald Taylor’s article. Especially when viewed in the context of a recent article in the Times: http://www.timesonline.co.uk/tol/life_and_style/career_and_jobs/senior_executive/article1294799.ece
ROI as Donald Taylor knows it and describes it is a less than efficient system in my opinion, that is the reason so few people do it. But ROI using the Kearns Baseline model is alive and well and thriving. Alas, the backward facing rump of trainers and developers in this country who do not or cannot see a link between training and its financial impact on an organisation will view Taylor’s article as clear supporting evidence of their position. I don’t think this is what he wanted to achieve but that is most certainly what he will accomplish.
I spent much of my weekend writing this response because I think it is vitally important that as trainers and developers we do not see ROI as the enemy but rather an important influence and useful tool. Mr Taylor and his ilk do us no favours trying to promote the opposite.
