For many organisations, measuring learning starts and stops with completion rates:
How many people started a course?
- How many completed it?
- Did everyone pass?
These simple metrics are easy to capture and simple to report but they don’t tell us much about whether or not learning really made an impact.
New research from global people-development company Insights and the Association for Talent Development (ATD), which surveyed 445 talent development professionals and 471 learners, reveals just how widespread this issue remains – particularly when it comes to digital training.
For asynchronous digital learning, 68 per cent of organisations track completion rates, while only 36 per cent measure behavioural change.
For human facilitated learning solutions, the numbers are higher: 60 per cent measure behavioural change. The overall picture is that we’re still measuring activity over impact.
As learning experiences continue to evolve across digital, blended and human facilitated formats, this gap is becoming increasingly prevalent and problematic.
If we really want learning to support performance, adaptability and leadership capability, whatever the mode, we need to change how we measure it.
The overall picture is that we’re still measuring activity over impact
Why completion became the default for digital
In the past, learning was often viewed as an ‘event’. A course was delivered, attended and completed.
As such, learning platforms were initially designed to distribute content and track participation, which meant completion metrics tended to dominate. So, it’s not surprising that digital learning inherited this model.
However, behaviour change is so much more complex to measure and achieve. It also happens over time, requiring careful observation, awareness of the purpose and context and input from others.
Faced with this complexity, many organisations default to what’s simple and easy.
Completion doesn’t equal competence
Completion metrics tell us an event has occurred but not the nature of the event or whether it was impactful. This is especially problematic in digital environments.
Learners can move through content quickly without taking the essential time to apply the learning or to seek feedback. This means that it can remain theoretical, misunderstood or be lost altogether.
Real behaviour change requires more than exposure to information. It requires awareness, application and time.
For example, leaders don’t truly learn how to manage effectively without trying, listening to feedback and adapting as necessary. If we rely on completion metrics alone, we risk mistaking activity for progress.
Effective measurement should … reflect the nature of the learning experience
What should we measure instead?
Good learning starts with clarity about the behaviours we’re seeking to develop and associated learning outcomes.
So, to understand whether learning is effective, organisations need to be clear on what they’re measuring, why and the most effective way to do it.
Behavioural indicators are a strong starting point:
- Are leaders holding more effective conversations?
- Are teams using shared language from the learning?
- Are collaboration practices changing?
These signals provide insight into whether learning is influencing day-to-day behaviours. Performance metrics can also be useful when linked to learning objectives. For example, customer loyalty, colleague engagement and performance outcomes.
At Insights, we start with the reason for the training and then write ‘Think, Feel, Do’ learning outcomes.
Ensuring these are clear and clearly related to the reason for the training provides clarity about measurement. For well-designed programmes, measuring identified learning outcomes can become a powerful proxy for behaviour change.
Practical ways to measure behaviour change
Measuring behaviour doesn’t need to be complex, just intentional. Behavioural checklists can be a simple but highly effective tool, with behaviours that are clearly linked to the learning outcomes of the training.
Structured ‘follow ups’ can reinforce accountability. Observation and peer feedback are also important in team and leadership contexts. Elsewhere, pulse surveys can provide ongoing insight in a more formal way and 1-2-1/team ‘check ins’ can do this less formally.
Reflective data from the learners themselves is another important metric. When learning increases self-awareness, people tend to describe how they’re thinking or feeling or behaving differently.
If we go back to the ‘Think, Feel, Do’ model, Think objectives can be measured by knowledge tests or checks on understanding; Feel objectives can be measured by reflective surveying; and Do objectives can be measured by tracking behaviours, through observation, self-report or other report.
If we rely on completion metrics alone, we risk mistaking activity for progress
The role of digital platforms in better measurement
Digital platforms play a critical role in training, but they need to evolve beyond participation metrics.
Platforms that can capture signals of engagement and application, like practice activities, reflections, commitments to action and follow up ‘check ins’ can provide a more complete picture of how learners interact with learning materials.
They can also support measurement by prompting behavioural experiments, collecting feedback from managers or peers and enabling simple pulse diagnostics. In this way, they become part of the learning process itself, supporting awareness, experimentation and reflection.
Measuring across different modalities
Effective measurement should also reflect the nature of the learning experience. In facilitated learning, observation and reflection are particularly valuable. Facilitators can see behavioural shifts emerging in real time and guide discussion accordingly.
Blended learning creates multiple points for measurement. Engagement with digital components, participation in facilitated sessions and application in the workplace can all be tracked and connected.
For digital learning, the focus should move away from completion towards evidence of application. ‘Follow up’ reflections, action commitments and behavioural ‘check ins’ provide more meaningful insight into impact.
From product stacking to learning ecosystems
Many organisations approach learning as a collection of products. Courses, tools and platforms get added over time and the connection isn’t always clear. This ‘product stacking’ approach can result in fragmented learning that’s difficult to measure.
A learning ecosystem, however, takes a different approach. It creates a coherent journey where learners move from one element to the next based on their needs and supported by feedback and reflection.
In a learning ecosystem, measurement takes a different approach also. Instead of the number of courses completed, key performance indicators become behavioural adoption, team effectiveness and sustained engagement – the focus shifts from consumption to activation.
Learners can also consider setting personal goals – based on their individual needs – and measure their progress against that goal. Reporting can then capture individual level progress as well as wider team and/or organisational progress. This kind of approach can be transformative.
Improving learning measurement – particularly in the digital space – is a mindset shift.
If we continue to treat learning as an event, completion will remain the dominant measure.
But, if we recognise personalised learning as an ongoing process that supports behaviour change and performance, and measure accordingly, then learning can become a true driver of performance.
Your next read: Workplace training has a problem but L&D already knows the answer
