Hey all,
Could anyone direct me to some sites or send me some information about the CIRO model concerining evaluations. Links to the Hamblins 5 levels of evalutaions would also be greatly appreciated ..
Br,
Kris
Kris Meuleman
Menu
Menu
17 Responses
See Bitesize 4
Both of these models are, to all intents and purposes, obsolete – see the Baseline Model on the Bitesize Business Partners page
These models are not obsolete
These models are not obsolete and, like Kirkpatrick, can be very useful in gaining evidence about the effectiveness of training interventions. I appreciate Paul’s interest in promoting his own model – and it is useful – but it is also impracticable in many training situations.
Interesting definition of obsolete
Actually Wayne’s right, he is quite entitled to drive around in a ’59 Chevvy and no doubt he still has an abacus on his desk!
I’m with Wayne
The baseline model quoted seems extraordinarily similar to KirkPatrick. The only difference appears to be that (1) you have to ‘evaluate’ up-front in order to have a comparision… with which I agree of course. In fact I think Kirkpatrick said that from the start: there can be no evaluation if you don’t know what you’ve changed from. That many training-folks fail to ‘baseline’ as Paul calls it doesn’t invalidate KirkPatrick, any more than the fact that many people don’t wash their hands after weeing doesn’t invalidate the need to do so; and
(2) ROI has been added in. ROI is controversial to say the least (I believe Kirkpatrick rejected the idea that it’s a fifth level; it could however be a component of the fourth). For my money, ho ho, ROI calculations are a fiction to be avoided – at least, in the psueduo-scientific format that most people try to do them. There are so many assumptions hidden by apparently “black and white” maths.
Kris, you wouldn’t go far wrong getting hold of a copy of Kirkpatrick’s book, rather than trying to learn about it via second sources. The man just got it right.
References
You need to get hold of Hamblin’s book, probably through a library, since it seems to be out of print at the moment:
Evaluation and Control of Training
Anthony Crandell Hamblin
McGraw-Hill, 1974
ISBN 0070844429
It’s quite widely cited, notably here:
http://www.businessballs.com/trainingprogramevaluation.htm
Read around things
Kirkpatrick is so popular (which model is referenced and talked about more?) because (IMHO) it was the first to be promoted in an easily digestable form – it could be understood by a very wide audience.
It carries very little detail on HOW to undertake respective steps at each level, though Kirkpratrick’s books do say WHAT to do, and also STATE that so-called Level 4 evaluations should only be done 5-10% of the time because they are so burdensom in terms of data gathering and analysis. Further there is a strong implication, again IMHO, running through his books that the real evaluation happens after the fact – he even states that AFTER the training you go and gather the data, and compare it with a CONTROL GROUP to see what has changed. THIS IS NOT A BASELINE METHOD – it’s comparing some people who go training with some who didn’t. It tells you evry little about the root causes of the performance issue that presumably the training in question is trying to address.
The CIRO and other frameworks have tried to expand the debate and here we aer, still debating it 45 years after Kirkpatrick’s first article!
Kris specifically asked for details about the CIRO model and links to Hamblin. I hope he’s got something, but equally there is nothing intrinsically wrong in others pointing out that, just in case Kris is going to use CIRO or Hamblin ‘in anger’ there are potential pitfalls to consider, including obselescence.
It is by no means universally agreed that Kirkpatrick got it right. Sure, read his book, but also read around the subject too. Kirkpatrick has its uses, as do many other models and approaches, including that of Paul Kearns. Use the one that is appropriate. I’d add that there are likely to be some core principles in evaluating, such as root cause analysis (my soap box), baselining (one of Paul’s I think), having data that is relevant, and considering who the audience of any evaluation is.
Kris – if you get some info on CIRO and Hamblin I’d appreciate anything you can email me ([email protected]) – I’m always looking to learn more. If you are interested I have some (incomplete) stuff of my own at http://www.trainerbase.co.uk/Useful/items.asp?CategoryID=15
Good luck in what ever you are doing!
A little unnecessary?
I think Paul, your comment is unnecessary and insulting.
Many of us take evaluation seriously, but the methods and metrics we use need to be right for both the type of project we are working on and the needs of the client. In my experience, the more resources, tools and methods we have at our disposal the better.
Let’s stick to the facts
Wayne – Perhaps you can suggest where it is impracticable. Any comments I make are based on over 25 years experience in training and 12 years as an evaluation specialist. We’ve had 40 years of contradictory and confusing advice from Kirkpatrick, Hamblin et al – that’s why so few trainers have any idea how to resolve the evaluation issue.
Wow! And elephants ahead?
Folks, thank you for your passion and enthusiasm. Wow!
I stand from afar and am delighted to learn from your collective wisdom.
As a businessman I naturally share your transparently honest concern for bottom-line results.
And as with many other such debates about ‘process’, I find them fascinating for all I search for ‘outcomes’ by personal preference.
So elephants? They may be hard to ‘define’ but are easy to recognise. We may surely know one when we see one! How? There’s the rub!
So also the value of training? Or spend on IT, leadership, ethical values, social justice, community responsibility and much else?
Sometimes – not always, of course – the business-need for a calculated ROI is absurdly misguided. Even in investment in much more tangible investments, of capital plant or acquisition, how often do we find this wide of the mark! There is so much more going on here isn’t there than pure financial analysis? For example, business sentiment, market feel, organisational well-being and morale, management will, leadership drive, ethical concern, visionary purpose? How do these get measured and factored in to ‘judgement’, ‘experience’, ‘corporate will’, ‘clarity of vision’, ‘shared values’ and the rest?
So might you usefully, in this context, take ‘learning’, rather than ‘training’, an even more abstract thought, and work from there?
Lesson in point? My own nascent company paid for two admin staff with no great secondary let alone tertiary educational qualifications to undertake a certificated course in Massage. It made no obvious, ‘calculated’ sense to us tho’ it didn’t cost a lot and they really enjoyed it, which was the only outcome we ever asked. No apparent ROI at the time, at all. As I recall, one even failed the detailed bio-medical theory at first taking – it was really stretching for her. But guess who subsequently volunteered to be our vital H&S reps and statutory First-Aiders when we realised we needed them as we grew?
And yet that particular outcome was the last thing on our minds at the time.
So to contribute usefully to this debate, is it just possible that the Holy Grail of ‘scientific management’ – which I must admit is my own educational heritage – may sometimes ignore the essence of the whole in defining its parts?
I think I recognise the origin of this debate. Much tactical training does indeed seem to be wasted. And when fighting for funds, whether for investment in people, equipment, systems, corporate promotion or whatever, it is terribly attractive (and important) to present a case based on a common and measurable language that discretionary fund holders may understand and use to make such awesome decisions.
So may your ROI efforts also embrace these non-financial yardsticks to define the *whole* elephant?
There’s a challenge for you tigers!
Kind regards
Jeremy
Whats the problem?
Paul, you seem to have several issues, that perhaps need to be addressed on a different website.
The question was can you direct kris to some information not tell him if they are obsolete or not and certainly not start a pointless debate about whether Wayne uses a calculator or not!!!
Masterclass Recommendations Sought
I’m looking for masterclasses in Facilitation and Advanced Presentation (this one for Business Development Manager of high value global sales). Any recommendations.
Online IT Assessment (Microsoft) needed
I am looking for a short to the point assessment for testing ability with various Microsoft Packages at interview.
A response
Paul – perhaps I’m taking your comments personally because they were.
I have been trained to use the baseline model – by you – and I found it useful.
However, as I stated before, there are situations where it is not necessarily the best on to use.
Being there isn’t being trained
Wayne, you might have been on a workshop but level 3 validation tells me you do not know how to apply the Baseline model. A training day is no measure of ‘training’.
Baseline Model
Paul,
I don’t think the purpose of this site is to slag off anyone who has a different point of view. One of the great things about this site is that it brings together people who have a range of different and similar experiences, perspectives and points of view.
Evaluation
With 40 years experience of training and evaluation and many years of evaluation, including the publication of at least 4 books on the subject, I echo the concerns about universal ROI via evaluation. Remember that Paul Kearns has written that if you can’t measure a learning event in financial ROI terms, then it wasn’t worth doing!!!!
Common ground?
As somebody who once lost their nicely paid job without warning as a direct consequence of not being able to demonstrate a financial return for the cash spent on training, I am naturally attracted strongly to the idea that training is seen as valued by an organisation more so than the cost of doing it.
Both Jeremy and Leslie have once again reminded me (thank you both!) that passion can be a double edged-sword!
I am working on a performance improvement project with a local authority. The successful completion of this project, which does involve some training – certainly learning and educating (to pick up one of Jeremy’s suggestions!) – will have no financial consequence in terms of more profit or notable cash savings (there may be several £100s saved over the year – nothing compared to the costs of the project) – but some of the benefits are considered very important by the organisation and are tied to strategic business objectives that are essential non-financial in nature, though could, at a (long!) stretch have financial consequences. In Paul Kearns’ language the benefits and hence reasons for doing any associated training/learning/educating (all 3 are involved) are in his ‘Box 1’.
For the dozen or so individuals who are living with the current process, the ‘improved version’ provides significant reductions in personal stress and frustration levels and offers a real opportunity for improving working relations with their internal customers. They can see this which is why they are very much engaged in the project.
The employees won’t get any extra money or bonus for doing this work, and neither will the organisation. So a meaningful, credible and financial ROI is very difficult to build in this case. And frankly, nobody cares. Why not? Because a meaningful and credible non-financial ROI has been built, by the organisation and the employees themselves. Like anybody else, they have both wanted an answer to the question “is it worth (for me) getting involved in this project and supporting it with some of my very limited resources?”. Nowhere are these limited resources at any level overtly financial in nature. This picks up I think on Leslie’s point that (my words) evaluation doesn’t always have to be about money, but it does have to be about (relative) value. Paul’s points that training/learning/educating needs to be explicitly linked to clear measures of what is important to an organiastion to be sure of seeing performance improve are IMHO different expressions of the same view.
It would be a shame if the medium of electronic text and the way so many of us use it were to cloud the many valuable learning opportunities this debate offers, but I don’t want this thread to end just yet – I feel there may be more opportunities for me to learn!
Best wishes