In many professions, compliance-based training is supplemented by on-the-job experience, but the two exist quite separately. Measuring the impact of learning and learner behaviour tends to happen in a silo, after the event and away from the day-to-day work of the individuals being trained. New technology and more creative ways of using it are changing all of this, however.
In this article, I’ll outline a project that is changing the way fire and rescue services in the UK approach their training using innovative new technology.
A major challenge is how to measure cognitive skills like decision-making and interpersonal skills such as leadership. These cannot be directly measured, but they can be indirectly assessed via behaviours.
A challenge for the UK fire and rescue service is to prepare its incident commanders to take charge of emergencies, and to effectively and safely bring them to a conclusion. Commanders gain experience implementing standard operational procedures at real emergencies which, when combined with training, shapes their command expertise.
The fire and rescue service is a victim of its own success, however. Falling numbers of emergencies across the UK, in part due to the effective community safety work of the fire and rescue service, has led to fewer opportunities for incident commanders to gain experience in using their command skills.
This fact has driven the fire and rescue service to explore new ways to prepare their commanders beyond meeting the performance criteria in the service’s National Occupational Standards.
Development of incident command guidance
A major overhaul of the UK fire and rescue service’s in-house guidance for incident command was published in 2015. The revised guidance emphasised the importance of command skills such as leadership, decision-making and situational awareness. These are the non-technical cognitive and interpersonal qualities critical for assertive, effective and safe incident command that complement a commanders’ technical skills.
Challenges in assessing command skills
A major challenge is how to measure cognitive skills like decision-making and interpersonal skills such as leadership. These cannot be directly measured, but they can be indirectly assessed via behaviours. For example, decision-making can be assessed through communications with crew members).
A challenge from a social science perspective is for assessment tools to be valid and reliable: they need to measure what they are supposed to and enable assessors to provide a consistent assessment for the same observation. The use of fire and rescue service subject matter experts to help design and evaluate an assessment tool helps to achieve these benchmarks.
A practical challenge is for the assessment process to be unobtrusive, so as not to interfere with an incident commander’s ongoing performance. A portable, simple, easy to use observation-based assessment tool written in familiar terminology with clear links to performance outcomes would help overcome this challenge.
When combined with a credible assessor, whose role includes monitoring and providing feedback on incident commander performance, such a tool would be more likely to appeal to commanders as valuable and to increase its acceptability and use.
THINCS assessment process
Between 2016 and 2018, a system was developed to meet the command skills assessment challenges identified above. The research was funded by an Economic and Social Research Council PhD studentship and evolved from award-winning research on decision-making conducted by Cardiff University and the National Fire Chiefs Council. The resulting tool was THe INcident Command Skills (THINCS) behavioural marker system.
The system comprises six command skills and with 20 sub skills distributed across them. For each sub skill there are examples of observable behaviours associated with good and poor practice. The system also includes a rating scale and documents to record observations and provide feedback. Such systems are linked to improving safety and are widely used within high reliability organisations to assess the non-technical performance of safety critical personnel (e.g. surgeons).
A trained THINCS rater uses the system to record, review and rate their behavioural observations of a commander. The rater provides feedback based on behavioural observations made at the time they occurred as opposed to relying on memory to recall them after the fact. A preliminary evaluation found that the system was comprehensive and straightforward to use. Also, it could discriminate between different levels of performance with a high degree of inter-rater reliability.
Measurement at real emergencies
In 2018, a project to develop a mobile app for the THINCS system was completed, which met the practical challenges of using the system at real emergencies or live exercises, and in adverse weather conditions. The project was a partnership between a mobile app development company, MyOxygen, and Cardiff University’s School of Psychology. It was supported by the National Fire Chiefs Council and funded by the Economic and Social Research Council.
Ultimately, the finished THINCS app automated many functions of the original paper-based system, but also generated a convenient automated database of incident commander behaviour.
Impact of THINCS behavioural marker system
THINCS fills a gap in the desired competence framework of the UK fire and rescue service by measuring the behaviours of incident commanders. This is an essential part of ensuring that command skills are being appropriately practiced by commanders, to reduce occurrences of human error and improve safety.
THINCS provides a vocabulary to discuss the performance of command skills. It also provides valuable performance data that can become a feature of safety management. Together these benefits can support and shape individual, team and organisational learning.
We envisage that the use of the THINCS behavioural marker system and its associated app will become a central component of incident command training and assessment across the UK, and might also be used by aligned fire and rescue services globally.
More information about this system can be found in our paper on the subject here.
Interested in the topic of measurement? Read Measuring L&D impact: a realistic approach in a VUCA climate.
