Introduction
Recent developments in neurophysiological sensor technologies have enabled the development of wearable and portable devices, leading to new research areas such as neuroergonomics.1 Prior work in this field foreshadowed great promise for decoding the mental states of pilots/operators, including changes in mental workload, vigilance, and drowsiness.2 In particular, studies utilizing brain imaging sensors, such as electroencephalography and functional Near-Infrared Spectroscopy (fNIRS), have demonstrated that certain measurements enable objective assessment of key cognitive states underlying pilot’s performance. The changes observed in neurophysiological signals during stressful or cognitively demanding phases of flight give further insight into pilots’ performance.3 Such findings have opened up the possibility of developing passive Brain-Computer Interface (pBCI) applications that can infer and adaptively respond to the cognitive status of pilots by monitoring their neurophysiological signals in relation to performance indicators logged in a simulation system.4 Such pBCI systems have the potential to enable the development of personalized training simulators for optimized learning and improved man-machine interaction via adaptive interfaces that can help pilots prioritize tasks and manage information complexity.
Figure 1: The placement of the sensors used in the Genius pBCI system. © Copyrighted
There are multiple technical and methodological challenges to monitoring pilots’ physical and cognitive states in real-time conditions, as well as in simulator training situations, via pBCI applications. Firstly, to aid interpretation of the neurophysiological signals and facilitate performance assessment, appropriate task scenarios and task performance models need to be developed. Secondly, the technical infrastructure must synchronize heterogeneous data streams, develop methods for processing and cross-checking signals originating from different body parts at various spatial and temporal scales, and ensure good data quality without sacrificing operators’ comfort. Finally, finding appropriate ways to fuse neurophysiological data with machine learning and artificial intelligence techniques, devising measures for real-time assessment of pilots’ cognitive states, and translating pBCI-enhanced training into the real world are considered higher-level challenges.
In an ongoing research project, we have been integrating several neurophysiological sensors with a high-fidelity flight simulator to develop a pBCI system that optimizes flight simulator training for fighter pilots. In this paper, we provide an overview of related efforts in utilizing neurophysiological measures in aviation and summarize our preliminary findings and observations regarding the technical and organizational challenges involved in the design, development, and deployment of such a pBCI system. We further discuss potential ways of contributing to next-generation training concepts for the Turkish Air Force or other Air Forces.
Figure 2: Overview of the GENIUS pBCI flight simulator training system. © Copyrighted
Pilot Cognitive State Assessment
Human factors such as cognitive workload, situational awareness, stress, divided attention, vigilance, mental fatigue, drowsiness, and incapacitation are generally listed among the fundamental cognitive processes and states that play a decisive role in pilots’ performance.5 Although modern avionics systems assist pilots by reducing the cognitive and physical loads, piloting is still a cognitively demanding task. In particular, piloting tasks significantly burden attentional and working memory resources, especially in mission-critical phases of flight such as take-off, landing, or target engagement, as well as in unexpected events such as emergencies or instrument failures. Such an increase in mental workload may lead to the loss of situational awareness and, eventually, mishaps when tasks consume the available cognitive resources. According to ICAO (2022) safety reports, high-risk categories of incidents (including loss of control in flight, runway incursions/excursions, and mid-air collisions), which are closely associated with cognitive factors, constituted 10.4% of all incidents and 100% of the fatal accidents in 2021.6,7
Flight simulators play a critical role in modern aviation training as they allow pilots to master flight procedures, controls, and avionics interfaces in a realistic but safe environment. In addition, flight simulators provide a safe environment for training on emergency procedures that can be extremely dangerous to attempt during actual flight. Pilots are expected to develop the much-needed skills for mastering interfaces and appropriate task prioritization to handle mentally demanding situations and use their cognitive resources effectively. Simulators also provide a controlled environment where progressive task sequences can be repeated, thus creating opportunities for devising measures to monitor and assess learning.
Although studies on simulator training suggest that the skills acquired in these environments have positive implications on real task performance, innovative approaches are needed to increase effectiveness.8 In the aviation domain, the mental workload is typically studied with behavioural methods. The NASA Task Load Index9 is a popular instrument to develop a subjective measure of pilots’ workload by asking the pilots to rate their experience along specific constructs. Such ratings are often associated with other performance measures, like the level of mental workload elicited by a given task.
Another common subjective method of assessing pilots’ workload is through the deliberate assignment of secondary tasks, such as mental arithmetic exercises or initiating additional air traffic control queries while the pilot is engaged in his primary piloting task. Under such settings, some pilots’ primary task performance may decrease significantly, indicating their lack of cognitive resources to spare for the extra task. This suggests that some pilots must allocate most of their cognitive resources to the main piloting task, with little cognitive reserves for reacting to unexpected events or emergencies. It has been observed that as pilots gain experience they tend to perform better in multitasking scenarios, which is often attributed to their improved cognitive capacities.10 However, the difficulties arising from the objective characterization of cognitive states and capacities solely from behavioural data, and the questionable validity of some of the secondary tasks imposed relative to the aviation domain, have spurred research for alternative methods based on the neurophysiological underpinnings of the related cognitive processes.
Neurophysiological Underpinnings
The need for more objective measures to monitor cognitive states has motivated the use of various neurophysiological sensors. Existing studies have utilized electrocardiogram, electromyogram, Galvanic Skin-Conductance Response (GSR), electrooculogram, eye tracking, electroencephalography, and fNIRS to measure parameters such as heart rate variability, muscle contractions, electrodermal activity, eye movements, brain oscillations, and cerebral oxygenation changes.11 These studies focused predominantly on a single or a small number of sensors to explore neurophysiological correlates. For instance, increased heart rate variability, rate of respiration, GSR amplitude, lateral prefrontal cortex oxygenation, and power in specific electroencephalography bands are associated with increased mental workload. Eye movements also provide important information regarding changes in attentiveness and mental workload levels. Observed findings suggest that an increase in mental workload is associated with a decrease in the rate and duration of eye blinks and an increase in gaze durations. Altogether, these sensors offer different perspectives on pilots’ mental workloads.
Comparing the neurophysiological responses of different pilots to changing task conditions can bring further insights into the level of experienced cognitive load. This can help us better identify why some pilots’ performance dramatically degrades when introducing a secondary task. For instance, oxygen consumption in the prefrontal cortex of the brain, as measured by fNIRS, can be an indicator. Two pilots performing at the same level during the single-task scenario may have significantly different prefrontal oxygenation trends. The pilot consuming more prefrontal oxygen will be more likely to have difficulty coping with the secondary task than the other pilot who can perform the task with less oxygenation. Moreover, the struggling pilot’s eventual disengagement from the task can be observed as a sudden drop in prefrontal oxygenation.12 In addition, after repeated exposure to the tasks through training, systematic decreases in prefrontal oxygenation are also observed along with improved performance.13 Therefore, such measurements have the potential to improve secondary task-based subjective evaluations by providing objective indicators for cognitive workload and learning assessments.
Prior work in neuroergonomics has demonstrated neurophysiological sensors’ potential for measuring pilots’ cognitive states in simulated and live flight settings. There are many challenges to collecting objective data, such as differences in individual ability and experience, and contextual factors that heavily influence the interpretation of mental workload-related measurements.
Unimodal measures such as changes in heart rate, respiration rate, pupillometry, and prefrontal oxygenation can be equally influenced by physical load, muscle fatigue, ambient light conditions, motion, or noise. Some of these challenges can be addressed with offline signal processing techniques. Still, such techniques may not be suitable for real-time application. The sensors also measure at different temporal and spatial resolutions, requiring additional data normalization and fusion procedures to enable real-time cognitive state monitoring. Finally, prior studies have also demonstrated that the neurophysiological responses to different workload levels might change with practice.13 This opens the possibility of utilizing such measures to monitor expertise development. Still, it also complicates identification of thresholds defining mental workload levels.
A Multimodal Approach
The challenges mentioned above highlight the need to integrate neurophysiological sensing modalities to improve the robustness and accuracy of pBCI applications and, ultimately, aid flight simulator training. To enrich flight simulator training with multimodal neurophysiological data, we have conducted a research project titled ‘Cognitive Workload Monitoring for the Personalization of Flight Simulator Training (GENIUS)’, supported by the Presidency of Defence Industries and the Turkish Air Force. The goal is to build a multimodal cognitive state-monitoring application that provides a scientific basis for assessing expertise development during simulation training and developing intelligent, personalized training systems. For this purpose, we developed a system that collects data wirelessly from several wearable sensors in tandem with ambient sensors and streaming flight data from the simulator. The sensors are synchronized and integrated with the flight simulator data.
A total of 42 F-16 pilots were recruited during the data collection stage; they were split into three groups (i.e. beginner, intermediate, and advanced) based on their flight hours. After installing the sensors, the pilots had to perform a full-flight F-16 simulator mission, which included basic flight manoeuvres, air-to-air, air-to-ground strike tasks, and a return-to-base episode under zero visibility and light turbulence conditions. Also, they had to engage an enemy aircraft at the beginning of the session under poor visibility conditions.
Our initial data analysis included a multistage feature extraction process where each sensor’s output is screened and pre-processed for possible artefacts. The raw signals are then converted into higher-level measures such as cortical oxygenation changes, alpha/beta/theta frequency power, eye dispersion, phasic/tonic responses, blink rate, pupil diameter changes, heart rate variability, respiration rate, head movements, and muscle activity on the arms, around the neck, and eyes. Further, the signals were resampled for multimodal analysis in sliding time windows and labelled based on the scenario episodes and events observed in the flight recordings. Finally, machine-learning techniques were employed to detect changes in the mental workload, physical workload, disengagement, and stress levels.
Significant insights were gained from our initial results. We observed that having multiple modalities allows fast crosschecks across sensors and optimizes the required number and type of sensors. By integrating such measures with training tasks and clear performance goals, we aim to develop an intelligent flight simulator that can guide the trainees based on a multimodal assessment of their behavioural and neurophysiological responses and, consequently, to personalize pilots’ training regimen according to the deficiencies identified by the pBCI system. We expect multimodal analysis of mental workload changes to be a vital component of a system of this type due to its relevance in assessing pilots’ performance. We also plan to expand our dataset by covering the development of the same group of pilots over time and by involving pilots from different platforms to better account for expertise and platform dependency.
Way Ahead
Based on our initial findings, we expect the wealth of information on pilots’ cognitive states that can be obtained from neurophysiological sensors to significantly impact fighter pilot training and, potentially, selection practices in the near future. The possibility of integrating such measures into existing Augmented Reality/Virtual Reality (AR/VR) sets can bring personalized and effective training opportunities in a variety of live and simulated contexts. Moreover, recent developments in flexible electronics and advanced sensing materials increase the probability of integrating neurophysiological sensors into the real cockpit, in the near future, with minimal discomfort for the pilot, to enable monitoring health and cognitive states during live flight.
Acknowledgements: The authors are indebted to the GENIUS project team members for their contribution to the project work summarized in this article. This project was funded by the Presidency of Defence Industries (SSB).
