While the importance of informal training is widely recognised, methods for measuring its impact are far from established. Dawn Smith looks at the challenges involved in evaluating informal learning in the workplace.
A term first coined in the 1950s, informal learning has been shown by successive studies to account for a massive proportion of the learning that goes on in the workplace. In the US, a 1999 study by the Department of Labour estimated that 70% of workplace learning is done informally, while Norwegian research found that 60-70% of learning is done ‘on the job’.
In recent years, technology has increased the ways in which informal learning can take place, so that now ‘embedded’ informal learning is the hot trend. “People are increasingly learning in informal ways using the Internet, blogs, instant messaging…The learning provider community must now embrace informal learning by embedding its learning content into these places also in informal ways so that workers find learning an entirely natural experience,” says Alfred Remmits, CEO of LearningGuide, which supplies electronic performance support systems to help workers to resolve queries .
Challenges
But although employers increasingly recognise and encourage informal training, supporting it by various means – from web-based support systems to creating a networking culture – relatively few have got to grips with measuring the impact of their efforts. “The strategies for measuring informal learning are still in their infancy,” says Dr Saul Carliner, assistant professor of education at Concordia University in Montreal, whose research interests include assessing informal learning. “There's no Kirkpatrick model that is widely used.”
One of the challenges of measuring informal learning is that there are no events on which to base evaluations. “If people are learning informally, they may be asking a colleague how to do something, using a company support system, or using sat nav in their car. You can’t ask them what they’ve learned every time they they’ve used the resource,” says Rob Clark, Director of International Marketing at LearningGuide. “If I turn to my colleague and ask them how to do something, the company can’t send me a survey asking what I learned.”
The problems associated with measuring informal learning have led many organisations to conclude that it can’t be done at all. In a survey by US magazine Chief Learning Officer (CLO) in May 2006, 49% of respondents said that it is not possible to measure the impact that informal learning has on business.
A programmatic approach
But it can be done. According to Rob Clark among others, it is possible to apply some of the principles used for measuring structured learning and adapt these to assess informal training. The key is to evaluate the learning at “a programmatic level,” he says. “For example, after you’ve implemented an informal learning programme – such as a new online support system - you can conduct a survey, and then another one a few months down the line.”
While not religiously following Kirkpatric and Phillips, Clark says his company does adopt their methodology. “We ask learners if they like our systems, test them to see what they learned and then measure job impact,” he says. “You can also do a financial analysis. For example, you can measure how many calls the help desk gets compared to how many it got before. An online support system might reduce the number of queries.” LearningGuide quotes research that shows helpdesk calls cost an average of around £14 per call, so a reduction in queries would provide a measurable return on investment.
A programmatic approach is also taken by KnowledgeAdvisors, which provides technology and solutions to measure and analyse learning impacts. To measure informal learning it is necessary to build a ‘structured process of evaluation’ that systematically measures results, says Kent Barnett, the company’s founder and CEO. For example, KnowledgeAdvisors has been working with a large insurance company that has three leadership development programmes, all of which include formal and informal learning elements (coaching, mentoring, books, classroom work, e-learning, facilitated online learning, etc.). “We worked with them to evaluate the different types of learning over 9-12 months,” says Barnett, who adds that the most cost-effective evaluation method is to conduct periodic surveys.
When assessing learning programmes, KnowledgeAdvisors applies the “human capital contribution model” developed by the company. The first stage of this model starts long before any assessment is carried out, and consists of assessing the business goals of the learning programme. Says Barnett: “Do you want to increase revenue, increase quality, reduce employee turnover, etc? You need to know why you are doing it.”
At the assessment stage, the actual business results will be tracked in order to determine whether the benefits of the programme outweigh the costs, and whether the learning is impacting the financial results in the right way. “Advanced clients of ours are trying to find the right combination of formal and informal learning to drive the financial results,” adds Barnett.
Questions and response
When it comes to the nitty-gritty of what to include on surveys designed to evaluate informal learning, Saul Carliner has some adviceon how, when and where to send them. “Organisations must actively seek the input of participants,” he says. “I know of one organisation that puts a brief satisfaction survey on each page of its technical support website. In other cases, if an organisation has registered users of its website, it can send out a period poll.”
Response rates will not match those in the classroom, he warns, but if they match those for other types of direct response materials, they can be considered valid. “In practical terms, that means for 10,000 surveys sent, maybe 100 will be returned. For classroom trainers used to 100% response rates, that's kind of shocking. But 100 people out of 10,000 is about as valid as 100% of 15 participants -and perhaps more so, because the instructor isn't watching them filling out the form.”
On the forms, Carliner advises including questions about how much the learners believe they have learned. “Ask them how much they knew before and after reading the materials. Ask them to name what they learned (more time-intensive to analyse, but what could be more direct?)”
In terms of learning and transfer, he reckons that qualitative assessment using a small sample of participants, to find out what they're learning and how they're using the knowledge, is “definitely worth the investment on a first project.” He adds that organisations that are encouraging self-study through learning contracts can include some sort of assessment - a formal test of some sort - as well as a debriefing on what was learned, ensuring that the results are captured and reported onward.
Pitfalls
If evaluating informal learning is difficult but do-able, there still remains a debate over the extent to which organisations should try to measure something so amorphous, individual and spontaneous. In the CLO research this year, many respondents said that even if informal learning could be measured, the spirit of the activity would be lost if too many metrics were imposed. Talking to the magazine, Peter McStravick, senior research analyst for IDC’s Learning Services group, warned that “Too much attention might stifle the very thing that makes informal learning so successful”.
On the other hand, says Kent Barnett, “if you don’t try to analyse the value of informal learning you are going to leave a lot on the table.” While he accepts that there is a balance to be struck between measuring the results of informal learning and focussing too much attention on analysing it, he believes it’s important to systematically monitor any informal learning programme to ensure it’s driving the right kind of business results: “If not, then it’s very costly.”
More information
• Saul Carliner, PhD, CTDP
E-mail: saulcarliner@gmail.com
Web: http://education.concordia.ca/~scarliner
Dr Carliner will be discussing the subject of evaluating informal learning in the workplace at the Canadian Society for Training and Development Symposium in May 2007. For details see the website: www.cstd.ca
• KnowledgeAdvisors
Web: www.knowledge-adviser.com
• LearningGuide
Web: www.learningguide.co.uk
Related articles
