Real evaluation is a state of mind and has absolutely nothing to do with happy sheets, questionnaires or surveys. Arguments about whether ROI is the Holy Grail and declarations of a preference for one particular evaluation model over another completely miss the point. Paul Kearns discusses the benefits of real evaluations.

The point is no one knows whether learning is effective unless real evaluation takes place and real evaluation ensures that learning has a clear purpose, right from the start. In other words - learning and evaluation are indivisible, inextricably linked and completely inter-dependent.

A case in point is the plethora of ‘leadership’ programmes currently masquerading as learning in most organisations today. In the specific case of the NHS its Modernisation Agency Leadership Centre declared in 2004 that: -

“Improved leadership will:
- improve patient's care, treatment and experience
- improve the health of the population.”

This was quite a bold assumption, as they had no evidence to back up such a statement. More worryingly, it revealed a complete ignorance of real evaluation because it was a classic case of putting the cart before the horse. In fact, the only attempt at evaluation occurred well after this expensive edifice had been created (see the 'Office for Public Management Guide to Evaluating Leadership Development' produced for the Modernisation Agency in June 2005).

Now, regardless of the arguments for or against such an approach, it is worth noting that whatever the Leadership Centre was doing it was obviously not convincing anyone of its value: it was disbanded just as the evaluation guide was being published. The NHS did not even follow its own good medical practice; no post mortem was carried out, the body was just buried quietly without ceremony, hoping nobody would notice.

Of course, those who have no interest in real evaluation cannot face up to the mistakes they have made. This is precisely why serious problems, like the increasing incidence of MRSA in hospitals, are unlikely to be resolved. The NHS is not a learning organization and this should strike fear into the hearts of all future patients.

So what has the NHS learned from all of this? Nothing apparently, as the NHS just re-created the same body in a different guise, the Institute for Innovation and Improvement, with a 'leadership qualities framework' that is a theoretical model of what supposedly makes good leaders. Yet again though it fails to offer any clear evidence to back up its assertion. One thing we can say for certain though is that while the NHS is going round and round in circles it will never learn. It will therefore be unable to transform itself into the modern health service that we all want and for which we have already paid a great deal in taxes.

Real evaluation is about evidence-based learning. It is about having a crystal clear purpose, right at the outset, before any learning activity is even dreamt of, never mind delivered. It is about checking commitment to learning from the most senior levels of management and exposes those who are only too willing to pay lip service. It is about accountability as well as engaging the hearts and minds of the learner: surely the wellspring of all effective learning.

So is this just a problem for the NHS, or the public sector, or is it a more fundamental issue? Perhaps we can look at how seriously evaluation is taken in the technology-obsessed world of e-learning?

The brochure for Learning technologies 2008 has a session entitled 'Demonstrating and Measuring Value' but the underlying attitude to evaluation is made plain for all to see when it says: 'It seems that the question of associating value with training just will not go away'. The implicit message is that the technologists wish it would.

E-training has been around for long enough now for us to ask where is the evidence that the technology adds any value or even produces better or even more efficient ‘learning’ than face-to-face teaching. The lack of evaluation in the e-learning industry does not seem to have hindered its growth but it has certainly done nothing to convince us that any of this huge activity represents progress or transformation in the way we learn.

E-learning providers, by definition, have to prepare content, in advance, for individuals they are never likely to meet. This is contrary to everything that real learning professionals have been taught over the years. It tries to break the sacrosanct principles enshrined in the classic training cycle of individual training needs analysis, design, delivery and evaluation. Anyone who offers a pre-prepared ‘learning solution’ is as guilty of malpractice as a doctor who writes a prescription without properly diagnosing the patient’s ailment first. We would all be up in arms if our GP’s and medical consultants adopted this approach, so there is no reason why e-learning should think it can bypass such a rigorous diagnostic process.

‘E-learners’ may have to sit in front of a screen without anyone checking their commitment to learn or apply what they learn. In the worst cases, e-learning can be a solitary experience that is totally devoid of that all-important human touch. Moreover, it is often detached from the culture, the environment and the organisational context in which the learner is situated. Unless e-learning advocates start to demonstrate that they fully understand the experiential importance of real evaluation they are likely to go the way of the NHS Leadership Centre.

Not wanting to know what works in practice, through real evaluation, is a key indicator of organisational stasis and deterioration. Real evaluation, by definition, produces real learning and real learning sets up the next iterative cycle of evaluation. If this principle were applied to the increasing incidence of MRSA not only would the NHS learn how to deal with this particular problem better, it would start to transform its whole approach to learning and improvement, and by doing so might eventually become a world-leading healthcare institution once more.

Paul Kearns specialises in measuring the value of the human contribution to organisational success and teaches real evaluation around the world. He is the author of the CIPD’s best selling 'Evaluating the ROI from Learning'. Visit www.paulkearns.co.uk

This feature was first published in January 2008