Written by Moira Nicolson
Something is happening in the world of public policy and it ought to be happening within learning and development. Earlier this year, the government’s “Behavioural Insights Team” opened their report with something that should be learnt from and used to transform the way in which we evaluate training:
‘Randomised controlled trials (RCTs) are the best way of determining whether a policy is working…[but] RCTs are not routinely used to test the effectiveness of public policy interventions in the UK. We think that they should be.’
I’d like to second this and add that randomised control trials should be used to test the effectiveness of training interventions – whether online, in the classroom, virtual or otherwise. In learning and development, resources are tight and people need to know what works.
More and more we’re seeing clients, industry professionals and industry bodies like the Learning and Performance Institute requesting that vendors provide rigorous and demonstrable proof of the effectiveness of training. Quite rightly, there isn’t one category in either the E-learning Age or the LPI’s learning awards that doesn’t ask entrants to provide some measure of demonstrable performance improvement or return on investment.
Yet we also know how challenging measuring return on investment can be. Randomised trials are the best way to find out whether training will work.
What are randomised control trials and how can they help the L&D industry?
What makes RCTs different from other methods of evaluation is that they involve randomly assigned control groups – groups of people who aren’t given the training – which allows you to compare the effectiveness of a new intervention against what would have happened if you’d done nothing. In drug trials, the control group of people are often given a placebo or ‘sugar pill.’
So, imagine that you’ve introduced a new performance management scheme for people who are underperforming. How will you know whether those receiving the extra support might not have improved anyway? And how can you know that it wasn’t something else that improved their performance?
For example, one member of staff might be underperforming at work because they’ve been suffering from undiagnosed stress. Perhaps you introduce your performance management system at around the same time that they go to the doctor to get treatment for their stress. Their performance improves. How do you know that it was the new performance management system that caused the improvement in their performance? After all, it could have just been the help they got from the doctor.
In a randomised control trial, you would control for all the other factors which will undoubtedly also be affecting the performance of your staff.
In the fictitious example in the diagram, we can see that those staff members who participated in the performance management intervention were much more likely to improve in terms of their individual performance than those who didn’t (in this case, twice as likely, which is pretty good). Because we have a control group (a group of people who aren’t entered into the performance management intervention), we know that it’s the intervention that’s improving the performance of staff and not something else (like the fact that a poor manager was just replaced with a good one or that they just sought treatment for an underlying medical condition like stress or depression).
In short, randomised control trials are the best way to find out if your training, rather than something else, made the difference.
Why does it need to be random? And how easy is it to run one?
The answer to the first question will be covered in my Advance article next week. The answer to the second is: randomised control trials can be much cheaper and simpler to run than you might think, partly because of the reasons given above. It’s especially important in times like these when L&D budgets are being cut. I’ll elaborate on this as well as provide a three step plan for how to go about running your own randomised control trial in an Advance article next week.
For now, it would be great if people want to comment below. Let us know:
- Have you or your organisation run a randomised control trial to test or evaluate the success of any of your training programmes?
- If so, what were the results? Did you think it was worthwhile?
- Randomised control trials have been used for about 60 years to test the effectiveness of new medicines and are being used by the UK government to test policy interventions. Does anybody have any reservations about using this approach in the training industry?
