In my research into validation of learning (or is that measuring the effectiveness of learning) I have come across any number of different terms used in the 'measurement' process. Are there any standard conventions as to what is actually meant by and when they should be used? eg, assessment, validation, verification, measurement, evaluation, etc?
Katherine Chapman
4 Responses
Terms
The e-learning world is a very divided one and there are many inconsistencies. Originally, ‘validation’ was used in reference to course material. It is a measure of whether a course delivers its objectives or not.
This means producing properly constructed behavioural objectives and a pre-/post-course test. Both these tests are exactly the same. As a pre-course test it identifies pre-knowledge and what can be guessed. As a post-course test it identifies what has been learned. By subtracting the raw scores for every question in the pre-test from those of the post-test for a pilot group, you can identify which elements of the course fail to meet the standards and make corrections accordingly.
Assessment of the course is done by the learner (the happy sheet). However, the learner may also be assessed by setting a test or providing a project that proves he or she is able to do what the behavioural objectives say should be done.
Hope the helps.
Brian Holley
Sherpa Integrated Learning Ltd.
http://www.sherpa,org.uk
ITOL Glossary
Katherine,
You are not alone in having this difficulty, it is a problem which has plagued our profession for some time. Which is why ITOL has published ‘A Glossary of UK Training and Occupational Learning Terms’. The Glossary, the result of extensive deliberations and virtual discussion by a working party formed from members of the Institute, contains definitions of almost 400 terms. The Glossary has been favourably reviewed by trainingzone (go to Community, Reviews and enter ITOL Glossary in the Search box).
Here are some examples of the definitions you asked for:
assessment: An exercise that seeks to measure a learner’s skills, performance, or knowledge in a subject area. This may be either prior to, during, or following their learning.
validation (of training): the measurement of whether the training achieved what it set out to achieve:
(1) Internal validation – a series of tests and assessments designed to ascertain whether a training programme has achieved the behavioural objectives specified.
(2) External validation – a series of tests and assessments designed to ascertain whether the behavioural objectives of an internally valid training programme were realistically based on an accurate initial identification of training needs in relation to the criteria of effectiveness adopted by the organisation. In short – were they the right objectives?
evaluation: The assessment of the total value of a training programme, training system or training course in both value- and cost-effective terms. It differs from validation in that it is concerned with the overall benefit of the complete training programme and its implementation and not just the achievement of the laid-down learning objectives of the training course. It includes all the pre-course action, the post-course action and the post-course implementation of the learning by the learner back at work.
Hope these help.
Jeffrey Brooks
Institute of Training and Occupational Learning.
http://www.traininginstitute.co.uk
Happy Sheets and real evaluation
Brian mentions the happy sheet, which is exactly that.
Many delegates, students, call them what you will, will indicate that the trainer was at least in the ‘ok’ sector, because of a fear of confrontation. This is particularly the case if the trainer collects the sheets.
The real value of training can be measured by the increased effectiveness of the trainee, in the area that the training was supposed to address, over a period of time.
Immediate measurement has little value – no one was an expert driver the day after they passed their test. There is a necessary period of experimentation during which new skills are honed.
The impact on performance should be measured after three months as an indicator and six as an absolute. Improvements beyond that will have been triggered more by the individual than the training.
Is the learning valid for the individual and the organisation?
It’s to be hoped that you didn’t waste money on a programme that was not. That’s like buying a painting that you immediately consign to the attic.
Happy Sheets measure …what?
As a couple of previous replies have intimated, Happy Sheets are rather misleading when it comes to measuring the effectiveness of a training course (if we take “effectiveness” to mean things such as coherent transfer of knowledge, improved performance, etc.).
Given that most people imagine that both the trainer and the course should “ideally” receive a rating of 90%+, a study conducted by IBM’s training facility some time back revealed a fascinating – and crucial – anomaly:
When course effectiveness was measured 3, 6 and 9 months later, it turned out that delegates who gave a mark of 61-80% showed significantly more benefit from the training than those who gave marks of 81-100%.
It seems that the really high marks (90%+) do NOT reflect the high quality of the course at all – they are much more likely to be a measure of the trainer’s SOCIAL SKILLS!
(Having said that, of course a rating of 61-80% does not – per se – prove the worth of the course.)