What is the positive impact of the many recent technological developments on evaluation? Dr Alasdair Rutherford and Kenneth Fee have a few ideas and examples.
Digital technology can provide the leverage to make learning evaluation more affordable, efficient and effective.
What passes for a joke in evaluation circles is that the return on investment (ROI) of calculating ROI isn't worth it. This reflects the widespread assumption that ROI, indeed every form of evaluation, is expensive. It can be so, but the generalisation that evaluation is expensive is no more accurate than comparable generalisations that training is expensive, or marketing is expensive, or any business activity is expensive. Whether it's too expensive depends on how it's done, and the other side of the coin is that not doing evaluation can be much more expensive, sometimes fatally so.
The advent of digital technology that is widely available, easy to use and readily adopted has eliminated much of the expensive/affordable debate in evaluation. However, digital technology can contribute much more. There are five distinct phases of learning evaluation that benefit significantly from the lever of digital technology, and these five phases form a cycle: evaluation planning leads to data collection, then storage and analysis, to reporting and from there to planning again.
What passes for a joke in evaluation circles is that the return on investment (ROI) of calculating ROI isn't worth it. This reflects the widespread assumption that ROI, indeed every form of evaluation, is expensive. It can be so, but the generalisation that evaluation is expensive is no more accurate than comparable generalisations that training is expensive, or marketing is expensive, or any business activity is expensive. Whether it's too expensive depends on how it's done, and the other side of the coin is that not doing evaluation can be much more expensive, sometimes fatally so.
The advent of digital technology that is widely available, easy to use and readily adopted has eliminated much of the expensive/affordable debate in evaluation. However, digital technology can contribute much more. There are five distinct phases of learning evaluation that benefit significantly from the lever of digital technology, and these five phases form a cycle: evaluation planning leads to data collection, then storage and analysis, to reporting and from there to planning again.
This cycle is depicted in the diagram below.
Evaluation planning
Online shared workspaces and planning templates are resources that can help involve stakeholders and enable linking of evaluation to learning objectives and business outcomes. They're not typically a feature of a Learning Management System (LMS), but some generic templates are available from niche providers. The question each organisation has to ask is whether a generic template has sufficient flexibility to accommodate all the unique issues for their organisation, and the variations for each training implementation; if not, creation of a bespoke solution is a viable option and needn't be prohibitively expensive.
The important thing to remember is that this is not just about efficiency, but the effectiveness of learning evaluation. Evaluation planning is the stage when organisations can devise strategies, accompanied by scorecards and dashboards to measure and manipulate them, and this is where digital technology comes into its own. This is a powerful way to secure buy-in, to brand the evaluation, and to tie it into other systems in the organisation.
"Whether [evaluation] is too expensive depends on how it's done, and the other side of the coin is that not doing [it] can be much more expensive, sometimes fatally so"
Examples of planning activities enhanced by digital technology include: linking learning objectives to business outcomes; enabling stakeholder participation.
The important thing to remember is that this is not just about efficiency, but the effectiveness of learning evaluation. Evaluation planning is the stage when organisations can devise strategies, accompanied by scorecards and dashboards to measure and manipulate them, and this is where digital technology comes into its own. This is a powerful way to secure buy-in, to brand the evaluation, and to tie it into other systems in the organisation.
"Whether [evaluation] is too expensive depends on how it's done, and the other side of the coin is that not doing [it] can be much more expensive, sometimes fatally so"Examples of planning activities enhanced by digital technology include: linking learning objectives to business outcomes; enabling stakeholder participation.
Data collection
Data collection is usually cheaper and easier when done digitally rather than manually. The gathering of data via questionnaires is an obvious instance where cost and time considerations can be driven down by online delivery. Virtual meeting software – even just Skype – can also enhance telephone interviews, raising them almost (but not quite) to the level of interaction of a face-to-face interview. Any paper-based method of data collection can be switched to a digital version to achieve savings and efficiencies. And this can include automated collection of data from a number of sources, such as different stakeholder inputs, identified together by learner, or by course, or by whatever other criterion is required. The other side of this coin is that linkage of paper-based data from various sources can be difficult and expensive.
Examples of data collection activities enhanced by digital technology include: distribution of surveys; audio and video recording; virtual meetings.
Examples of data collection activities enhanced by digital technology include: distribution of surveys; audio and video recording; virtual meetings.
Data storage
Like data collection, data storage is transparently more efficient when computerised. The difficulties of safe and secure retention of paperwork, perhaps over a lengthy period of time, are simplified if not eliminated by computerised storage. Whereas filing cabinets and archive boxes can quickly fill with reams of paper, digital files may be compressed and stored with relatively little digital memory, and easily retrieved when required. Multimedia data, including image files and video clips, can also be stored in the same place, for further simplification. An added advantage of this sort of digital concentration is compliance with data protection and ethical considerations, such as anonymity and confidentiality of records, which can be more easily secured.
Examples of data storage advantages from digital technology include: reduced storage space; greater security; different formats held in the same place.
Examples of data storage advantages from digital technology include: reduced storage space; greater security; different formats held in the same place.
Data analysis
Analysis is about asking questions of data, and making sense of it. Ultimately, making judgements is a human facility, but computers can help organise and present data better, and inform those human judgements. Digital technology does not just facilitate calculations, which themselves may be large and complex (and therefore time-consuming and costly), but helps pick out and project trends, and highlight common factors and other significant themes.
Examples of data analysis benefits from digital technology include: faster calculations; easier identification of themes and trends.
Examples of data analysis benefits from digital technology include: faster calculations; easier identification of themes and trends.
Reporting
Digital technology can generate reports far faster than any manual system, and vary those reports at the touch of a button. This much is straightforward enough, but the digital option is not just a quick way of collating quantitative data, it can yield far more sophisticated outputs, including cross-referencing of themes and ready production of visualisation (eg graphs and charts). Qualitative reporting may also be formatted graphically, pictorially, in video or audio clips, such as when articulating individual opinions, or in more prosaic text formats. Thus digital reporting can capture and communicate not just facts and figures, but examples, case studies and points of view, turning a dry report into something more lively and persuasive.
Examples of reporting benefits from digital technology include: faster report generation and distribution; better quantitative and qualitative reporting; multi-media capability.
Examples of reporting benefits from digital technology include: faster report generation and distribution; better quantitative and qualitative reporting; multi-media capability.
Potential drawbacks
Taking the cycle as a whole, there are two issues of potential concern to be aware of. One is cost - purchasing new software can be expensive, not least in terms of training staff in its use, and embedding it. The other is the issue of unfamiliarity - It can be tough enough securing staff engagement in evaluation activities without asking them to master new technology as well; wherever possible, it makes sense to stick to familiar systems. Some quick and simple – and small scale – evaluations may still work best with paper.
"As a general guideline, buyers should opt for the bespoke in preference to the generic"
Summary
Altogether, the capacity for digital technology to enhance each of the five phases of the cycle amounts to a convincing argument for deploying it whenever possible, and certainly for any learning evaluation of any significant scale. This leaves the question of which technologies to choose, and as a general guideline, buyers should opt for the bespoke in preference to the generic.
There are some good off-the-shelf digital tools available, but these are rarely integrated to address all of the five cycle phases described in this paper. And where they are, they usually constrain the user within the limits of their templates. A better option is to use the most generic software tools, such as Microsoft Access and Excel, or SurveyMonkey – the same everyday tools used for other business applications – and integrate these into corporate systems everyone in the organisation is already comfortable using. This may still mean some bespoke development, but this should then be affordable and lower risk.
Deployed in this way, digital technology does not just leverage more from your learning evaluation, it may add significant value.
There are some good off-the-shelf digital tools available, but these are rarely integrated to address all of the five cycle phases described in this paper. And where they are, they usually constrain the user within the limits of their templates. A better option is to use the most generic software tools, such as Microsoft Access and Excel, or SurveyMonkey – the same everyday tools used for other business applications – and integrate these into corporate systems everyone in the organisation is already comfortable using. This may still mean some bespoke development, but this should then be affordable and lower risk.
Deployed in this way, digital technology does not just leverage more from your learning evaluation, it may add significant value.
This article was first published on the Airthrey website
Kenneth Fee and Dr Alasdair Rutherford are the founding directors of learning evaluation firm Airthrey Ltd. Ken is a career learning and development professional, whose latest book, 101 Learning & Development Tools, deals with evaluation among other topics. Alasdair is an evaluation and econometrics specialist, and a Research Fellow at the University of Stirling
