Translating visual perception from the lab to the real world using mobile EEG and mixed reality displays

This user research article summarizes our publication “Towards real-world neuroscience using mobile EEG and augmented reality“. Krugliak, A., Clarke, A. Sci Rep 12, 2291 (2022). doi.org/10.1038/s41598-022-06296-3.

Using naturalistic paradigms and stimuli is becoming increasingly prominent in cognitive neuroscience, where experimentation beyond the lab is increasingly possible. Studying the human brain in its ‘natural habitat’ provides ecological validity to our theories and promises novel insights into complex cognitive functioning. This relies on emerging technologies to take research out of the lab and into the wild, and combining these techniques offers a toolkit to study neural function in uncontrolled complex environments. We present and validate such an approach, through studying face processing in naturalistic, real-world environments, while importantly retaining the ability to manipulate our visual perceptions. We achieve this by combining mobile EEG (mEEG) with head-mounted cameras and augmented reality (AR). AR allows us to utilise immersive head-mounted displays to present virtual objects on the background of the real world, allowing for full experimental control over what people see and where those items are located. In order to establish the feasibility of our approach, participants completed three EEG face inversion tasks: (1) a computer-based task, (2) a mobile task with photographs of faces on the walls, and (3) a mobile task where virtual faces were presented through the head-mounted augmented reality (AR) device.

The experiment had three distinct tasks, which all participants completed; (1) (2) (3)

Participants:

Eight healthy participants took part. The experiment was conducted in accordance with the declaration of Helsinki and approved by the Cambridge Psychology Research Ethics Committee. All participants gave informed consent.

Stimuli & Procedure:

Computer-based task: The computer-based task used 100 upright and 100 inverted face images (Figure 1A), presented in a random order. Participants were instructed to press a button on the keyboard when they had seen the face. Each trial began with a fixation cross lasting between 500 and 525 ms, followed by the face image lasting until the button press, before a blank screen lasting 1 second. The experiment was presented using Psychtoolbox version 3 and Matlab R2019b, and triggers recorded via a USB to TTL module (https://www.blackboxtoolkit.com).

mEEG + photos task: Here, photos of 8 upright and 8 inverted faces were attached to the walls at various points along a corridor, and participants were asked to view the faces while walking along the corridor multiple times (Figure 1B). In addition to EEG, participants were fitted with a head-mounted camera attached to a Raspberry Pi Zero. Participants were asked to move along the corridor, and into side rooms, and pause at each face image. When they were paused at, and viewing the image, they pushed a handheld button which sent a 1-bit signal to the LiveAmp trigger input. The condition the image belonged to could then be derived from the head-mounted video. Participants repeatedly viewed the face images resulting in an average of 42 upright (range 24-70) and 39 inverted trials (range 22-72). The variable number of trials across participants reflects the different routes participants chose to take.

mEEG + AR task: To display AR faces, the Microsoft Hololens 2 AR device was placed over the actiCAP slim electrodes (Figure 1C). The clear lens of the Hololens allowed the participant to see the actual environment, while virtual faces were presented anchored to specific locations in the corridor (Figure 1DE). The onboard camera of the Hololens captured first-person video, including the virtual faces. Four different virtual heads were used (2 male, 2 female), with each head appearing in 4 locations, half of which the faces were inverted. Participants paused at each virtual face and pushed a button, and saw each face numerous times resulting in an average of 63 upright (range 28-90) and 53 inverted trials (range 27-87).

Figure 1. Experimental setup. A) Computer-based task. B) Mobile EEG setup and face photos. C) Mobile EEG and AR setup and example virtual face used for AR task. D) 3D spatial map created by the Hololens 2 of the experimental environment, showing locations of upright (yellow) and inverted (orange) virtual faces. E) 2D map of environment showing locations of upright (yellow) and inverted (orange) virtual faces.

EEG recording:

EEG was recorded with a LiveAmp 64 mobile system (Brain Products GmbH, Gilching, Germany), using actiCAP slim active Ag/AgCl electrodes referenced to FCz, with a sampling rate of 500 Hz, positioned according to the international 10/20 system. Electrode cables were carefully routed through the cable ties and kept flat to the head to minimise cable sway during recording. EEG signals were amplified using two wireless amplifiers and recorded to onboard memory cards. Data recording was controlled using BrainVision Recorder, and during setup, impedances were reduced to below approximately 5–10 kOhm.

EEG processing:

The same EEG analysis pipeline was used for the data from all three tasks, with a slight modification for the mobile tasks (2 and 3; Figure 2). EEG processing used EEGlab (Delorme & Makeig, 2004). Data were band-pass filtered using a onepass zero-phase Blackman-windowed sinc FIR filter with transition width 0.1 Hz and order 27500. Band-pass settings were 0.5 to 40 Hz for task 1, and 1 to 20 Hz for task 2 and 3. The narrower filter range for task 2 and 3 was due to the additional noise suppression this afforded, and the frequency range of interest identified in task 1. Bad channels were detected and subsequently interpolated, and cleaned using the clean_artifacts function. For task 1, the data were epoched between -1 and 2 seconds around the onset of the face images, and for task 2-3 the data were epoched between -2 and +2 seconds centred on the button press. ICA was applied to the epoched data, and analysed using ICLabel (Pion-Tonachini et al., 2019) to identify components related to brain activity which were retained when reconstructing the EEG data. The processed EEG signals were converted to Fieldtrip (Oostenveld et al., 2010), and time-frequency representations were calculated using Morlet wavelets between 4 and 35 Hz. Given prior studies, we focussed on low frequency signals (5–15 Hz) over posterior electrodes. A linear mixed effects model was used to model trial-wise power (EEG_power ~ condition + (1|Subject)). Trials with a value more than 4 standard deviations away from the mean were additionally removed prior to the mixed effects model (task 1: 0.5 % of data; task 2: 0.2 %; task 3: 0.2 %).

Figure 2. EEG preprocessing, core methods across tasks.