Translating visual perception from the lab to the real world using mobile EEG and mixed reality displays
by Alexandra Krugliak & Alex Clarke
Department of Psychology, University of Cambridge, UK
This user research article summarizes our publication “Towards real-world neuroscience using mobile EEG and augmented reality“. Krugliak, A., Clarke, A. Sci Rep 12, 2291 (2022). doi.org/10.1038/s41598-022-06296-3.
Using naturalistic paradigms and stimuli is becoming increasingly prominent in cognitive neuroscience, where experimentation beyond the lab is increasingly possible. Studying the human brain in its ‘natural habitat’ provides ecological validity to our theories and promises novel insights into complex cognitive functioning. This relies on emerging technologies to take research out of the lab and into the wild, and combining these techniques offers a toolkit to study neural function in uncontrolled complex environments. We present and validate such an approach, through studying face processing in naturalistic, real-world environments, while importantly retaining the ability to manipulate our visual perceptions. We achieve this by combining mobile EEG (mEEG) with head-mounted cameras and augmented reality (AR). AR allows us to utilise immersive head-mounted displays to present virtual objects on the background of the real world, allowing for full experimental control over what people see and where those items are located. In order to establish the feasibility of our approach, participants completed three EEG face inversion tasks: (1) a computer-based task, (2) a mobile task with photographs of faces on the walls, and (3) a mobile task where virtual faces were presented through the head-mounted augmented reality (AR) device.
The experiment had three distinct tasks, which all participants completed; (1) (2) (3)
Eight healthy participants took part. The experiment was conducted in accordance with the declaration of Helsinki and approved by the Cambridge Psychology Research Ethics Committee. All participants gave informed consent.
Stimuli & Procedure:
Computer-based task: The computer-based task used 100 upright and 100 inverted face images (Figure 1A), presented in a random order. Participants were instructed to press a button on the keyboard when they had seen the face. Each trial began with a fixation cross lasting between 500 and 525 ms, followed by the face image lasting until the button press, before a blank screen lasting 1 second. The experiment was presented using Psychtoolbox version 3 and Matlab R2019b, and triggers recorded via a USB to TTL module (https://www.blackboxtoolkit.com).
mEEG + photos task: Here, photos of 8 upright and 8 inverted faces were attached to the walls at various points along a corridor, and participants were asked to view the faces while walking along the corridor multiple times (Figure 1B). In addition to EEG, participants were fitted with a head-mounted camera attached to a Raspberry Pi Zero. Participants were asked to move along the corridor, and into side rooms, and pause at each face image. When they were paused at, and viewing the image, they pushed a handheld button which sent a 1-bit signal to the LiveAmp trigger input. The condition the image belonged to could then be derived from the head-mounted video. Participants repeatedly viewed the face images resulting in an average of 42 upright (range 24-70) and 39 inverted trials (range 22-72). The variable number of trials across participants reflects the different routes participants chose to take.
mEEG + AR task: To display AR faces, the Microsoft Hololens 2 AR device was placed over the actiCAP slim electrodes (Figure 1C). The clear lens of the Hololens allowed the participant to see the actual environment, while virtual faces were presented anchored to specific locations in the corridor (Figure 1DE). The onboard camera of the Hololens captured first-person video, including the virtual faces. Four different virtual heads were used (2 male, 2 female), with each head appearing in 4 locations, half of which the faces were inverted. Participants paused at each virtual face and pushed a button, and saw each face numerous times resulting in an average of 63 upright (range 28-90) and 53 inverted trials (range 27-87).
Figure 1. Experimental setup. A) Computer-based task. B) Mobile EEG setup and face photos. C) Mobile EEG and AR setup and example virtual face used for AR task. D) 3D spatial map created by the Hololens 2 of the experimental environment, showing locations of upright (yellow) and inverted (orange) virtual faces. E) 2D map of environment showing locations of upright (yellow) and inverted (orange) virtual faces.
EEG was recorded with a LiveAmp 64 mobile system (Brain Products GmbH, Gilching, Germany), using actiCAP slim active Ag/AgCl electrodes referenced to FCz, with a sampling rate of 500 Hz, positioned according to the international 10/20 system. Electrode cables were carefully routed through the cable ties and kept flat to the head to minimise cable sway during recording. EEG signals were amplified using two wireless amplifiers and recorded to onboard memory cards. Data recording was controlled using BrainVision Recorder, and during setup, impedances were reduced to below approximately 5–10 kOhm.
The same EEG analysis pipeline was used for the data from all three tasks, with a slight modification for the mobile tasks (2 and 3; Figure 2). EEG processing used EEGlab (Delorme & Makeig, 2004). Data were band-pass filtered using a onepass zero-phase Blackman-windowed sinc FIR filter with transition width 0.1 Hz and order 27500. Band-pass settings were 0.5 to 40 Hz for task 1, and 1 to 20 Hz for task 2 and 3. The narrower filter range for task 2 and 3 was due to the additional noise suppression this afforded, and the frequency range of interest identified in task 1. Bad channels were detected and subsequently interpolated, and cleaned using the clean_artifacts function. For task 1, the data were epoched between -1 and 2 seconds around the onset of the face images, and for task 2-3 the data were epoched between -2 and +2 seconds centred on the button press. ICA was applied to the epoched data, and analysed using ICLabel (Pion-Tonachini et al., 2019) to identify components related to brain activity which were retained when reconstructing the EEG data. The processed EEG signals were converted to Fieldtrip (Oostenveld et al., 2010), and time-frequency representations were calculated using Morlet wavelets between 4 and 35 Hz. Given prior studies, we focussed on low frequency signals (5–15 Hz) over posterior electrodes. A linear mixed effects model was used to model trial-wise power (EEG_power ~ condition + (1|Subject)). Trials with a value more than 4 standard deviations away from the mean were additionally removed prior to the mixed effects model (task 1: 0.5 % of data; task 2: 0.2 %; task 3: 0.2 %).
Figure 2. EEG preprocessing, core methods across tasks.
Task 1: Computer face inversion task
We first replicated the well-characterised face inversion effect in a standard laboratory setting. Linear mixed effects modelling was used to test for changes in power for upright and inverted faces, showing that EEG low frequency power over posterior electrodes was significantly greater for inverted faces compared to upright faces (mean difference = 0.404, t(1427) = 8.27, p < 0.0001; Figure 3A). Plotting EEG power across frequencies further illustrates that low frequency power differences peaked between 5 and 12 Hz (Figure 3B). Further, we calculated power differences for each electrode, showing greater power for inverted compared to upright faces that were primarily located at posterior central and posterior lateral electrodes (Figure 3C).
Task 2: mEEG + photos
Our next analysis asked whether face inversion effects could be seen in a more naturalistic setting using mobile EEG while participants viewed upright and inverted pictures of faces placed on the walls. Following the analyses used for the computer-based task, we first performed a linear mixed effects analysis of EEG power at posterior electrodes. This revealed a significant effect of face inversion, with greater power for inverted faces compared to upright faces (mean difference = 0.190, t(597) = 2.38, p = 0.017; Figure 3). Across the scalp, inversion effects were greatest over posterior central electrodes and frontal electrodes (Figure 3D). This shows that face inversion effects are detectable using mobile EEG in a natural indoor setting.
Task 3: mEEG + AR
An important issue for studies using mobile EEG in natural settings, is the ability to manipulate the environment for the purposes of the experiment. Here, we combine mobile EEG with a head-mounted AR system which enables us to present virtual objects embedded within the real environment. Upright and inverted virtual heads were placed at various locations along the corridor and in the adjoining rooms, and participants repeatedly viewed the faces and pressed a button when they were fixating on the face. A linear mixed effects model of EEG power averaged over posterior electrodes showed a significant effect of face inversion, with greater power for inverted compared to upright faces (mean difference = 0.170, t(828) = 2.54, p = 0.0112; Figure 3A). Across the scalp, the inversion effect was maximal over posterior central electrodes (Figure 3E), and were partially overlapping with those seen in the computer-based and Mobile+photos tasks, with similar effect sizes in both mobile tasks. Through the combination of mobile EEG and head-mounted AR, this analysis establishes a feasible approach to studying cognitive processes in natural, real environments in which the participant is immersed.
Figure 3. EEG results. A) Face inversion effect sizes for each experimental task. Red x indicates mean inversion effect with individual subjects shown by grey circles. B) Spectrogram showing group mean power between 4 and 35 Hz for upright and inverted conditions, and the difference between them. C-E) Topographies showing mean power difference for inverted-upright faces between 5 and 15 Hz for the computer task (C), mEEG + photos (D) and mEEG + AR (E).
Participant motion and EEG:
Finally, we quantified how participant motion, measured by the accelerometers within the LiveAmp 64 amplifiers, related to EEG signal amplitudes. To do this, the continuous data (EEG channels plus 3 accelerometer channels) were split into 2 second non-overlapping chunks, and the root mean square (RMS) calculated for each channel. The RMS was averaged across the three accelerometer channels, creating one value per 2 second period. These values were then binned into low, medium and high motion RMS groups, before averaging the EEG RMS according to the motion RMS bins. This resulted in an EEG RMS value for each electrode and each of the low, medium and high RMS bins. A linear mixed effects model was used to test if the EEG RMS values related to the levels of participant motion (as defined by the motion RMS bins).
Our results suggest EEG RMS signals appear stable over the 3 motion levels for both mobile tasks (Figure 4AB), with no significant effects of motion level on EEG RMS (Mobile + photos: estimate -0.003, t(1534) = -0.12, p = 0.90; Mobile + AR: estimate 0.022, t(1534) = 0.79, p = 0.43).
Figure 4. EEG RMS during low, medium and high participant motion for A) the Mobile + photos task and B) the Mobile + AR task. Red x indicates mean RMS over electrodes and participants with boxplots showing the distribution of RMS values across electrodes. Topographies show electrode RMS values for each motion RMS bin.
Here we show that cognitively relevant neural signals can be detected in AR and mobile EEG paradigms. Similar to lab-based effects, we showed inverted AR faces elicit greater low frequency power compared to upright AR faces while participants freely moved through an indoor office space. The combination of AR and mobile EEG could offer a new paradigm for cognitive neuroscience, whereby cognition can be studied while participants are immersed in natural environments yet experimenters can retain some control over what items people see and how. One example we’re pursuing is the impact real environments have on object recognition processes, again using mobile EEG and AR (Nicholls et al., 2022). Such research endeavours help pave the way to exploring neurocognitive processes in real-world environments while maintaining experimental control using AR.
Delorme, A., & Makeig, S. (2004). EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. Journal of Neuroscience Methods, 134, 9–21.
Nicholls VI, Alsbury-Nealy B, Krugliak A and Clarke A. Context effects on object recognition in real-world environments: A study protocol [version 2; peer review: 1 approved, 1 approved with reservations]. Wellcome Open Res 2022, 7:165. https://doi.org/10.12688/wellcomeopenres.17856.2
Oostenveld, R., Fries, P., Maris, E., & Schoffelen, J.-M. (2010). FieldTrip: Open Source Software for Advanced Analysis of MEG, EEG, and Invasive Electrophysiological Data. Computational Intelligence and Neuroscience, 2011, e156869. https://doi.org/10.1155/2011/156869
Pion-Tonachini, L., Kreutz-Delgado, K., & Makeig, S. (2019). ICLabel: An automated electroencephalographic independent component classifier, dataset, and website. NeuroImage, 198, 181–197. https://doi.org/10.1016/j.neuroimage.2019.05.026