New insights in motor imagery from real-time EEG feedback during concurrent fMRI
by Catharina Zich1, Stefan Debener1 and Maarten De Vos1,2
1Neuropsychology Lab, Department of Psychology, European Medical School, University of Oldenburg, Oldenburg, Germany
2Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, Oxford, UK
This user research summaries our publication “Zich C, Debener S, Kranczioch C, Bleichner G, Gutberlet I, De Vos M (2015). Real-time EEG feedback during simultaneous EEG-fMRI identifies the cortical signature of motor imagery. NeuroImage 114:438-47″.
The mental rehearsal of a motor act, without any overt motor output, is called motor imagery (MI). MI combined with neurofeedback has been suggested as a promising rehabilitation approach for paralyzed individuals, which is a frequent consequence of stroke , . Electroencephalogram (EEG) based MI feedback is particularly promising for therapeutic applications because EEG systems are portable, easy to use, and affordable, hence enable home use and high-intensity practice –. Yet whether EEG feedback indeed targets specific sensorimotor activation patterns cannot unambiguously inferred from EEG alone. One way to address this issue is to combine EEG feedback and functional magnetic resonance imaging (fMRI). Although it is highly desirable to have full spatiotemporal information of activity induced by MI supported by EEG feedback, it is a technical challenge to remove in real time the artifacts in the EEG caused by the simultaneous scanning. First efforts towards implementing online EEG feedback inside the MRI scanner demonstrated the potential of this approach . Here we demonstrate that online correction of gradient artifacts (GA) and ballistocardiogram artifacts (BCG) enables reliable MI EEG feedback inside the MRI scanner. Online artifact correction was implemented with the BrainVision RecView software (Brain Products GmbH, Gilching, Germany). EEG neurofeedback inside the MRI scanner allowed us to identify the brain patters underlying MI, confirm the role of EEG feedback for task-specific activation on these brain areas using a feedback independent modality, and shed light on the causes of brain-computer interface (BCI) illiteracy.
Twenty-four individuals (23.9 ± 2.4 years) with no MI experience participated, two subjects were excluded due to noncompliance with task instructions. Participants conducted the same experiment twice separated by a six-week interval. During the first session concurrent EEG-fMRI was recorded, while in the second session only EEG was acquired in an EEG lab. Each session consisted of four blocks comprising of 40 trials (20 left and 20 right hand). Participants were instructed to execute finger tapping in the first block and to imagine the same movement in the last three blocks. The first two imagery blocks were performed without feedback, whereas in the last one online EEG feedback was provided. Stimulus presentation was controlled with OpenViBE  according to the standard Graz MI protocol with longer inter-trial intervals (4.5 s to 9 s in steps of 1.5 s) to accommodate the fMRI protocol timing.
EEG data were collected from 64 sites (EasyCap, Falk Minow Services, Herrsching-Breitbrunn, Germany) using an MR-compatible BrainAmp MR plus system and the BrainVision Recorder software (Brain Products GmbH, Gilching, Germany). During concurrent EEG-fMRI EEG raw data, sampled at 5 kHz, were saved without artifact correction and after correcting GA and BCG artifacts in real-time as described below. The corrected data were passed on to OpenViBE via a direct network link to provide online feedback. FMRI data acquisition was performed on a 3T Siemens MRI scanner (Siemens AG, Erlangen, Germany). During functional measurements 420 T2*-weighted gradient echo planer imaging volumes (3.1 x 3.1 x 3.0 mm voxels, 0.75 mm gap, TR = 1.5 s, FoV = 200 * 200, Flip Angle = 90°, 27 transversal slices) were obtained within each block. The EEG data obtained outside the scanner were acquired using the same paradigm and EEG setup, except that a lower sampling rate of 500 Hz was used.
To enable real-time EEG feedback inside the MRI scanner, EEG raw data were online corrected for GA and BCG artifacts using BrainVision RecView software (Brain Products GmbH, Gilching, Germany). The online GA correction method employed is based on the average artifact attenuation method proposed by  with the difference that the GA correction template was based only on the already recorded data. After GA artifact correction data were down-sampled to 500 Hz and 35 Hz low-pass filtered. For online BCG correction a search for the most suitable BCG template was performed within the first 30 s of GA-corrected data. This prototypical BCG template was then updated in a moving template matching approach to identify subsequent valid BCG instances. The classifier for online feedback was created after the first two MI blocks. Therefore EEG data from these blocks were 8 to 30 Hz band-pass filtered, non-stereotypical artifacts rejected, the optimal spatial filters derived using a common spatial pattern implementation using EEGLAB 10.2.2.4b . Finally, the optimal linear discrimination analysis (LDA) classifier was estimated in a seven-fold cross-validation procedure for discriminating 1 s power features of right and left hand movement. During the feedback block incoming GA and BCG corrected data were both spatially and temporally filtered in a similar fashion and real time feedback was given to the subject based on evaluating the classifier on the power features extracted from the ongoing EEG activity. To validate the online GA and BCG correction procedures, raw data were also offline corrected using BrainVision Analyzer Professional software package (Brain Products GmbH, Gilching, Germany). The main differences to the online correction procedure was a more advanced BCG detection approach.
Online and offline corrected data were offline re-referenced to common average, 8 to 30 Hz band-pass filtered, stereotypical artifacts were corrected using independent component analysis (ICA) and non-stereotypical artifacts were rejected using EEGLAB 10.2.2.4b. The time course of event-related desynchronization (ERD) patterns were computed following , whereby left and right ERD time courses were merged and only contra- and ipsilateral ERD considered. EEG data recorded outside the MRI scanner were processed in exactly the same way except for the obvious absence of the MRI artifact correction procedure.
MRI data were analyzed using SPM8 (FIL, Wellcome Trust Center for Neuroimage, UCL, London, UK). Data were realigned to the first image of each block, normalized to the Montreal Neurological Institute (MNI) template brain and smoothed with a three-dimensional Gaussian kernel of 8 mm full-width-half-maximum (FWHM). Functional regions of interest (fROIs) were defined based on the group-level activation for movement execution. Similarly to the ERD, only contra- and ipsilateral BOLD activity was considered, whereby contra- and ipsilateral BOLD activity were defined as average beta values of all voxels within the contra- or ipsilateral fROIs of the contrasts Right hand MI > Base and Left hand MI > Base.
Validation of EEG data
Figure 1A demonstrates that a reliable MI induced ERD pattern could be obtained from the online-corrected data as compared to offline-corrected data and data obtained outside the MRI scanner. Analysis of the contralateral ERD in the MI interval using a one-way ANOVA with factor condition (online-corrected, offline-corrected, outside-scanner) revealed a significant main effect (F2,42 = 12.47, p < .001, η² = .37). Follow-up t-tests revealed lower contralateral ERD in the online-corrected compared to the offline-corrected data (t21 = 5.83, p < .001). Moreover, the across-trials standard deviation of the contralateral ERD during the MI interval was significantly higher for online-corrected compared to offline-corrected data (t21 = -3.5, p = .003). Despite this, in all cases strong and significant positive associations between the three conditions could be seen (Figure 1B).
Figure 1: Reliability of EEG MI data recorded inside and outside the MRI scanner.
Although the importance of feedback for BCI performance has been known for many years ,  the effect of feedback based on one modality has hardly been validated with a second, feedback independent modality. We found that both contralateral ERD and contralateral BOLD activity were significantly stronger when feedback was provided compared to when it was absent (t21 = 6.42, p < .001, t21 = -3.99, p = .001), indicating the importance of neurofeedback (Figure 2).
Figure 2: Feedback effect in EEG and fMRI activity.
Furthermore, we investigated the relationship across subjects between ERD and BOLD activity patterns. While a significant association between both modalities was found for the contralateral activity (Pearson r(20) = -.45, p = .03), no significant association could be observed for the lateralization, i.e. the difference between contra- and ipsilateral activity (Pearson r(20) = -.25, p = .27). Further inspection of the EEG-fMRI patterns suggested three different types of response profiles, namely (A) BCI literates, (B) real BCI illiterates and (C) pseudo BCI illiterates. BCI literates are characterized by lateralized EEG activity (Figure 3A), which is absent in BCI illiterates (Figure 3B-C). The fMRI and EEG lateralization was systematically related in the group of BCI literates, but dissociated in the BCI illiteracy group. Based on their fMRI lateralization BCI illiterates can be divided into real BCI illiterates, who show nearly no task-related activity (Fig. 3B), and pseudo BCI illiterates, who show task-related activity through strong bilateral EEG patterns and lateralized fMRI activity (Figure 3C).
Figure 3: MI induced signatures in BCI literates, real BCI illiterates and pseudo BCI illiterates.
We successfully combined online MI EEG feedback with the simultaneous and continuous acquisition of fMRI signals. Positive associations between the EEG data obtained inside and outside the MRI scanner demonstrate a good test-retest reliable signal quality after online and offline correction. However, the contralateral ERD was weaker for online- compared to offline-corrected data, which indicates that there is room for improvement for online artifact correction. Despite this, our results show that it is possible to attenuate GA- and BCG-related artifacts online to a degree that is clearly sufficient for a successful implementation of online EEG feedback. This is reflected in an activity increase in EEG and fMRI when neurofeedback was provided compared to when it was absent. The relationship between EEG and fMRI signals revealed a significant association between contralateral ERD and BOLD activity, in the absence of a clear correspondence of lateralization patterns. The relationship between EEG and fMRI seems to be more complex when the ipsilateral hemisphere is involved .
Our findings suggest that individuals can be clustered into three groups: BCI literates, real BCI illiterates and pseudo BCI illiterates. The topic of BCI illiteracy is extensively discussed and various reasons for BCI illiteracy have been proposed, such as noncompliance to task instructions, particularities in individual neurophysiology or inefficiency of the BCI system, to name a few –. Our results demonstrate the existence of two BCI illiterate groups, whereby particularities in individual neurophysiology seem to underlie pseudo BCI illiteracy. These results inspired us to modify the EEG feedback implementation in further studies in order to better account for the observed individual differences . We are convinced that the validation of EEG neurofeedback signals by means of concurrently acquired fMRI or functional near infrared spectroscopy signals will help to improve the efficiency of brain-computer interface devices and neurofeedback training protocols.
 E. Buch, C. Weber, L. G. Cohen, C. Braun, M. a Dimyan, T. Ard, J. Mellinger, A. Caria, S. Soekadar, A. Fourkas, and N. Birbaumer
Think to move: a neuromagnetic brain-computer interface (BCI) system for chronic stroke.
Stroke, vol. 39, no. 3, pp. 910–7, Mar. 2008.
 A. Caria, C. Weber, D. Brötz, A. Ramos, L. F. Ticini, A. Gharabaghi, C. Braun, and N. Birbaumer
Chronic stroke recovery after combined BCI training and physiotherapy: a case report.
Psychophysiology, vol. 48, no. 4, pp. 578–82, Apr. 2011.
 C. Zich, M. De Vos, C. Kranczioch, and S. Debener
Wireless EEG with individualized channel layout enables efficient motor imagery training.
Clin. Neurophysiol., vol. 126, no. 4, pp. 698–710, 2015.
 S. Debener, F. Minow, R. Emkes, K. Gandras, and M. De Vos
How about taking a low-cost, small, and wireless EEG for a walk?
Psychophysiology, vol. 49, no. 11, pp. 1617–21, Nov. 2012.
 M. De Vos, M. Kroesen, R. Emkes, and S. Debener
P300 speller BCI with a mobile EEG system: comparison to a traditional amplifier.
J. Neural Eng., vol. 11, no. 3, p. 036008, Jun. 2014.
 V. Zotev, R. Phillips, H. Yuan, M. Misaki, and J. Bodurka
Self-regulation of human brain activity using simultaneous real-time fMRI and EEG neurofeedback.
Neuroimage, vol. 85 Pt 3, pp. 985–95, Jan. 2014.
 Y. Renard, M. Congedo, V. Delannoy, and A. Le
OpenViBE : An Open-Source Software Platform to Design , Test , and Use Brain – Computer Interfaces in Real and Virtual.
Presence, vol. 19, no. 1, pp. 35–53, 2010.
 P. J. Allen, O. Josephs, and R. Turner
A method for removing imaging artifact from continuous EEG recorded during functional MRI.
Neuroimage, vol. 12, no. 2, pp. 230–9, Aug. 2000.
 A. Delorme and S. Makeig
EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis.
J. Neurosci. Methods, vol. 134, no. 1, pp. 9–21, Mar. 2004.
 G. Pfurtscheller and F. H. Lopes da Silva
Event-related EEG/MEG synchronization and desynchronization: basic principles.
Clin. Neurophysiol., vol. 110, no. 11, pp. 1842–57, Nov. 1999.
 C. Neuper, A. Schögl, and G. Pfurtscheller
Enhancement of Left-Right Sensorimotor EEG Differences During Feedback-Regulated Motor Imagery.
J. Clin. Neurophysiol. Off. Publ. Am. Electroencephalogr. Soc., vol. 16, no. 4, pp. 373–382, 1999.
 J. R. Wolpaw, N. Birbaumer, W. J. Heetderks, D. J. McFarland, P. H. Peckham, G. Schalk, E. Donchin, L. a Quatrano, C. J. Robinson, and T. M. Vaughan
Brain-computer interface technology: a review of the first international meeting.
IEEE Trans. Rehabil. Eng., vol. 8, no. 2, pp. 164–73, Jun. 2000.
 K. J. Mullinger, S. D. Mayhew, a P. Bagshaw, R. Bowtell, and S. T. Francis
Evidence that the negative BOLD response is neuronal in origin: a simultaneous EEG-BOLD-CBF study in humans.
Neuroimage, vol. 94, pp. 263–74, Jul. 2014.
 B. Blankertz, C. Sannelli, S. Halder, E. M. Hammer, A. Kübler, K.-R. Müller, G. Curio, and T. Dickhaus
Neurophysiological predictor of SMR-based BCI performance.
Neuroimage, vol. 51, no. 4, pp. 1303–9, Jul. 2010.
 C. Vidaurre and B. Blankertz
Towards a cure for BCI illiteracy.
Brain Topogr., vol. 23, no. 2, pp. 194–8, Jun. 2010.
 M. Grosse-Wentrup, B. Schölkopf, and J. Hill
Causal influence of gamma oscillations on the sensorimotor rhythm.
Neuroimage, vol. 56, no. 2, pp. 837–42, May 2011.
 C. Zich, S. Debener, M. De Vos, S. Frerichs, S. Maurer, and C. Kranczioch
Lateralization patterns of covert but not overt movements change with age: An EEG neurofeedback study.