Your research on cognitive states specifically explores the importance of human factors, like mental workload, in the aviation industry. What are some key characteristics you would like to explore with your research?
A. Hamann: In aviation, there is a constant trend towards higher levels of automation. While human operators such as pilots and air traffic controllers still play an important role and will do so in the foreseeable future, their tasks and responsibilities will gradually change from active engagement to a more passive monitoring of systems. At the Institute of Flight Guidance of the German Aerospace Center (DLR), Braunschweig, research is undertaken to understand the challenges the operators will face, and to develop new operational concepts, interfaces and assistance systems to support them. We believe that the physiological assessment of the operators’ current cognitive state will help us to evaluate such new concepts and interfaces, and provide adaptive and intelligent assistance systems with information about the operators’ current needs. We work on defining valid and reliable EEG and fNIRS measures that can be used for such applications, and on understanding and accounting for interactions of different cognitive states. In our current research, we focus on mental workload and mental fatigue, two of the most central concepts of human performance in aviation, and aim to identify physiological measures that allow us to differentiate both states.
What kind of experimental setup do you use to investigate cognitive states in aviation in your participants?
A. Hamann: Depending on the research goal, we use different flight simulators with varying degrees of realism, and utilize expert samples. For this study, however, we explicitly wanted to control the environment, task demands and sample characteristics as much as possible. Therefore, we used a low-fidelity but highly flexible A321 cockpit simulator and recruited a student sample. Our aim was to identify valid physiological measures that differentiate levels of mental workload in a cockpit environment while controlling for confounding effects of mental fatigue. Therefore, we designed a simplified flight task that puts similar cognitive demands on the participants as a real flight would do, but that they could learn easily: The participants monitored and corrected the altitude of their aircraft, and changed its heading in accordance with an adapted n-back task in four difficulty levels. We controlled for mental fatigue by means of task duration and randomization. We recorded the participants’ brain activity with concurrent EEG-fNIRS measurement with a Brain Products LiveAmp-32 with active electrodes, and a NIRx NIRSport2 with 8 sources, 8 detectors and additional short-distance channels.
Why did you decide to combine EEG and fNIRS technology for this study? How did this multimodal approach help you address your research goals?
A. Hamann: Both EEG and fNIRS have advantages and shortcomings. With EEG, you can achieve high temporal precision in your measurements, but movement artefacts and noise from the simulator environment can limit the data quality. There is a large body of research on mental workload assessment with EEG, but tasks and analysis strategies vary across studies. On the other hand, fNIRS measurement has a higher spatial precision than EEG, and is less influenced by movement artefacts. However, systemic artefacts have to be considered and controlled for. Finally, not much fNIRS research has been done in aviation yet, and montage designs and analysis strategies vary considerably. With the combination of both measurements we wanted to make use of the complementary nature of the measurements and cross-validate the results. The real challenge was to design an fNIRS montage around the EEG positions that would optimally cover the areas of interest and still be comparable to other fNIRS-only studies. This, and getting the electrode gel out of the multitude of cables!
Which analyses did you perform on your recorded EEG and fNIRS data?
A. Hamann: We performed spectral analyses on the EEG data. These have the great advantage over evoked potentials that no additional stimulus is needed that could distract the pilot in a real flight. We focused on frontal theta and parietal alpha and beta power because these bands and locations have shown high sensitivity to mental workload in past studies. For the fNIRS data, we used a montage designed to optimally cover dorsolateral prefrontal cortical area which have proven to be sensitive to changes in task difficulty in n-back tasks and driving simulations. We used a GLM-based approach and controlled for systemic and motion artefacts by including data from short-distance channels as an additional predictor. In both datasets, we compared the four induced n-back levels in order to see how many levels of mental workload could be differentiated with which method.
How do you interpret your findings for EEG and fNIRS’ sensitivity to task difficulty?
A. Hamann: We found that increasing mental workload is associated with increasing activation in frontal cortical areas, i.e. increasing frontal theta power and increasing cortical oxygenation. Both methods, EEG and fNIRS, were able to detect changes and differentiate most of the n-back levels. Interestingly, EEG proved to be more sensitive to high demands and could differentiate between the highest n-back levels, but not the lowest. fNIRS, on the other hand, could not differentiate between the highest levels, but indicated changes between the lowest levels where not even performance was impaired. Only with the combination of both data sets could we differentiate all four induced levels of mental workload. Covering the whole range from low to (very) high mental workload will help us to get an overall picture of the pilot’s state, observe changes and be able to intervene before mental workload reaches a level where performance starts to decline. We therefore conclude that EEG and fNIRS are indeed complimentary and mental workload assessment benefits from combining them.
Can you please provide an overview of your future research plans?
A. Hamann: We started out inducing mental workload while controlling for mental fatigue. Adding to this research, we have just completed a follow-up study on mental fatigue in which we used the same setup and tasks, but explicitly controlled for mental workload. With the results gained from both studies under highly controlled conditions, we can draw inferences as to which EEG and/or fNIRS measures are most sensitive to which state and how to differentiate them. These controlled experiments lay the foundation for future research in more applied settings. Next, we will expand our research to expert samples, more realistic flight simulators and scenarios, and non-nominal flight conditions (i.e. when things do not go as planned, like an engine failure). There are still many questions to be answered before adaptive assistance systems based on physiological assessment will find their way into real cockpits, but our results show that EEG and fNIRS are highly promising candidates for these applications.