by Jerzy Bodurka1,2
1Laureate Institute for Brain Research, Tulsa, OK, USA
2College of Engineering, The University of Oklahoma, Tulsa, OK, USA
We integrated concurrent real-time fMRI (rtfMRI) and electroencephalography (EEG) data on commercial MRI and EEG equipment. We also report a proof-of-concept experiment using simultaneous multimodal rtfMRI and EEG neurofeedback (rtfMRI-EEG-nf). With this approach participants receive information about their electrophysiological (EEG) and hemodynamic (BOLD fMRI) activity in real-time, and volitionally regulate their own brain activity. This brain neuromodulation technique can enable novel cognitive neuroscience research paradigms and therapeutic approaches for major psychiatric disorders.
This user research summary is primarily based on article  published as Vadim Zotev, Raquel Phillips, Han Yuan, Masaya Misaki, Jerzy Bodurka (2014) Self-regulation of human brain activity using simultaneous real-time fMRI and EEG neurofeedback. NeuroImage 85, 985-995, doi: 10.1016/j.neuroimage.2013.04.126.
Concurrent functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) studies of brain electrophysiological and hemodynamic activity offer multiple advantages [2,3]. However, data streams from both modalities are independent, and multimodal data fusion is typically performed after experiments. Integration of real-time data streams from both fMRI and EEG can make it possible to develop a novel class of experiments and experimental techniques to study brain function. In particular, it allows for a real-time simultaneous fMRI and EEG neurofeedback (rtfMRI-EEG-nf) approach, in which participants can receive information about both their electrophysiological and hemodynamic activity in what is experienced as “real time” and then use this information to volitionally regulate subsequent neural responses. We describe a design in which such multimodal fMRI and EEG data integration was achieved using commercially available MRI and MR-compatible EEG systems. We also describe a proof-of-concept study utilizing multimodal rtfMRI-EEG-nf for simultaneous self-regulation of both BOLD-fMRI activation of the left amygdala (LA) and frontal EEG asymmetry (FEA) in the high-beta (21−30Hz) band during a positive-mood induction task in healthy volunteers. The LA is a critical brain region involved in emotion regulation . FEA has been shown to reflect and influence emotional processing . High-beta FEA abnormality (excessive activity in right prefrontal regions) is particularly relevant in patients with major depression [6,7].
Integration of simultaneous rtfMRI and EEG data
The system was designed to work with a General Electric MR750 3Tesla MRI scanner equipped with 8-element receive-only brain coil and a Brain Products GmbH 128-channel MR-compatible EEG system. The rtfMRI-EEG-nf design (Figure 1) was based on a custom-developed real-time fMRI system , which enabled rtfMRI neurofeedback (rtfMRI-nf) . The design utilizes real-time features of AFNI  and of the BrainVision RecView software (Brain Products, GmbH). Control and communication programs were written in Python (RTeeg, eeg_client, math modules) and Perl (RTmri, RTcontrol, mGUI). All programs run on a Linux workstation with a kernel customized for high-speed inter-process communications with message queues, synchronization with semaphores, and large data exchange via shared memory. The BrainVision RecView software allows for partial removal of MR and cardioballistic (CB) artifacts from EEG data in real time. The system combines rtfMRI and EEG data streams with real-time processing and analysis and includes custom multimodal graphical user interface (mGUI) neurofeedback software where the final data integration for rtfMRI-EEG-nf takes place.
Six healthy subjects (24 ± 9 years) provided IRB-approved written informed consent. Subjects wore an MR-compatible EEG cap (BrainCap MR from EASYCAP GmbH; 32 EEG, 1 ECG electrodes, 10-20 system).
For fMRI, a single-shot gradient echo EPI sequence was used . Concurrent EEG recordings were performed using a 32-channel BrainAmp MR plus amplifier (Brain Products, GmbH). The superb temporal stability of pulse sequence parameters provided by the GE MR750 scanner enabled EEG-fMRI recordings with high measurement resolution (0.1 μV). The EEG system’s clock was synchronized with the 10 MHz MRI scanner’s clock using Brain Products’ SyncBox device. Additionally, the EPI sequence was custom-modified (to achieve 1μs TR accuracy) for accurate correction of MRI artifacts in the EEG data.
The experimental procedure is illustrated in Figure 2. The rtfMRI-nf was based on fMRI activation in an LA target ROI as in . The rtfMRI-nf bar height (a percent change relative to the resting baseline) was updated every 2s (Figure 2a, right red bar). For the EEG-nf, the RecView software was used to perform real-time partial removal of MRI and CB artifacts. The corrected data were exported in real time as data blocks of 8ms duration. The EEG power spectrum was computed every 400ms for a moving data interval (lasting 2.048s). FEA was defined for frontal EEG electrodes F3 and F4 (Figure 2c) as
A = (P (F3) − P (F4)) / (P (F3) + P (F4))
where P is the EEG power in the high-beta band (21 – 30Hz). The difference in A values between the task and the resting baseline was calculated every 0.4s and was used to generate the neurofeedback visual representation, expressed as the height of the EEG-nf bar (red bar on the left in Fig. 2a).
The experimental protocol included seven runs (Figure 2b), and each run (except Rest) consisted of 40s-long blocks of Rest (R), Happy Memories (H), and Count (C) conditions . For each Happy condition, the subject was instructed to feel happy by evoking positive autobiographical memories while also trying to raise the levels of both red bars (EEG left and fMRI right) on the screen toward the fixed levels of the blue target bars. During the Count condition, the subjects were instructed to count backwards from 300 by subtracting a given integer. During the Rest condition, the participants were instructed to relax and breathe regularly while looking at the display screen. No bars were displayed during the Count and Rest conditions. Similarly, no bars were shown for the Happy condition during the Transfer run (TR).
Offline analysis of the fMRI data was performed in AFNI . Analysis of the EEG data, acquired simultaneously with fMRI, was performed in BrainVision Analyzer 2 software . The voxel-wise percent signal change data for Happy vs Rest contrast (for each run) were averaged within the LA ROI and used as a GLM-based measure of fMRI activation. Average EEG power spectra in the high-beta (21−30Hz) band were computed for each experimental condition. A moving window FFT with 2.048s data interval length was applied. The FEA in the high-beta band was then calculated for channels F3 and F4. The average normalizedAn = atanh(A) values for Happy and Rest conditions were then compared for each run. Additionally, EEG-informed fMRI analysis was performed to investigate BOLD fMRI correlates of the EEG-nf (psychophysiological interaction (PPI) analysis  using regressors based on the EEG asymmetry An), as described in .
The results are based on offline EEG data analysis with careful removal of artifacts. The corresponding average LA activation levels, based on offline GLM analysis, are shown in Figure 3b.
Both quantities increased during the neurofeedback runs (PR, R1, R2, R3), and the training effects persisted, to some extent, during the Transfer run.
The group statistical maps for the
interaction term in Figure 4 are thresholded at p < 0.05 (uncorrected).
Significant positive interaction effects are observed in Figure 4 for several regions involved in emotion regulation and visual processing.
We have integrated real-time EEG and fMRI data streams and implemented simultaneous rtfMRI-EEG-nf.
Our proof-of-concept experiment demonstrates that healthy participants can learn to simultaneously upregulate their frontal high-beta EEG asymmetry and left amygdala fMRI activation using rtfMRI-EEG-nf during positive emotion induction based on retrieval of happy autobiographical memories. The EEG-informed fMRI analysis revealed that several brain regions exhibited positive correlation with the high-beta FEA that was significantly stronger during the Happy Memories condition than during the Count condition. Importantly, the positive PPI interaction effect was observed in the left amygdala region, which was the target area for the rtfMRI-nf (Fig. 4a). This result suggests that the EEG-nf and the rtfMRI-nf employed in the present study are mutually compatible, i.e. increases in the high-beta FEA during the Happy condition with rtfMRI-EEG-nf are associated with increases in fMRI activation of the left amygdala. Our results show that EEG-nf based on the frontal high-beta EEG asymmetry can be naturally combined and used simultaneously with the rtfMRI-nf based on the amygdala activation. Conceivably, the rtfMRI-EEG-nf may prove more efficient in training of emotional self-regulation than either the rtfMRI-nf or the EEG-nf applied separately, particularly for the purpose of developing novel treatment for depression [14,15,16].
In our study, EEG-nf was carefully designed to reduce the effects of EEG-fMRI artifacts as much as possible irrespective of the real-time artifact removal procedure. However we estimate that residual artifacts constitute as much as 50% of the average EEG signal power after the real-time EEG signal processing . Clearly, further improvements in procedures for real-time removal of MR, CB, and motion artifacts are essential for successful implementation of EEG-nf with simultaneous fMRI.
Despite the challenges, real-time integration of fMRI and EEG data holds potential for development of novel neuroscience research paradigms and enhanced therapeutic approaches for major psychiatric disorders. The rtfMRI-EEG-nf allows simultaneous regulation of both hemodynamic (BOLD) and electrophysiological (EEG) processes in the human brain. The high temporal resolution of EEG-nf makes it a valuable complement to rtfMRI-nf (with slow hemodynamic response) for training paradigms, in which neuromodulation speed is essential. The rtfMRI-EEG-nf design makes it possible to dynamically modify an experimental protocol and an individual strategy in real time based on both rtfMRI-nf and EEG-nf information. Finally, the availability of both rtfMRI-nf and EEG-nf at any time during an experiment may lead to development of new training paradigms in which the effects of rtfMRI-nf could be approximated using only EEG-nf. This would have profound practical significance, because of more affordable and portable instrumentation for EEG-nf.
 Zotev, V., Phillips, R., Yuan, H., Misaki, M., Bodurka, J., 2014. Self-regulation of human brain activity using simultaneous real-time fMRI and EEG neurofeedback. NeuroImage 85, 985-995.
 Mulert, C., Lemieux, L., Eds., 2010. EEG-fMRI: Physiological Basis, Technique, and Applications. Springer-Verlag, Berlin Heidelberg.
 Zotev, V., Yuan, H., Phillips, R., Bodurka, J., 2012. EEG-assisted retrospective motion correction for fMRI: E-REMCOR. NeuroImage 63, 698-712.
 Sergerie, K., Chochol, C., Armony, J.L., 2008. The role of the amygdala in emotional processing: a quantitative meta-analysis of functional neuroimaging studies. Neurosci. Biobehav. Rev. 32, 811-830.
 Sutton, S.K., Davidson, R.J., 1997. Prefrontal brain asymmetry: a biological substrate of the behavioral approach and inhibition systems. Psychol. Sci. 8, 204-210.
 Pizzagalli, D.A., Nitschke, J.B., Oakes, T.R., Hendrick, A.M., Horras, K.A., Larson, C.L., Abercrombie, H.C., Schaefer, S.M., Koger, J.V., Benca, R.M., Pascual-Marqui, R.D, Davidson, R.J., 2002. Brain electrical tomography in depression: the importance of symptom severity, anxiety, and melancholic features. Biol. Psychiatry 52, 73-85.
 Paquette, V., Beauregard, M., Beaulieu-Prevost, D., 2009. Effect of a psychoneurotherapy on brain electromagnetic tomography in individuals with major depressive disorder. Psychiatry Res. Neuroimaging 174, 231-239.
 Bodurka J., Bandettini P.A., 2008. Real time software for monitoring MRI scanner operation. Proc. 2008 OHBM Annual Meeting, NeuroImage 41 (Supp 1), S85.
 deCharms, R.C., Maeda, F., Glover, G.H., Ludlow, D., Pauly, J.M., Soneji, D., Gabrieli, J.D.E., Mackey, S.C., 2005. Control over brain activation and pain learned by using real-time functional MRI. Proc. Nat. Acad. Sci. USA 102, 18626-18631.
 Cox, R.W., Jesmanowicz, A., Hyde, J.S., 1995. Real-time functional magnetic resonance imaging. Magn. Reson. Med. 33, 230-236.
 Zotev, V., Krueger, F., Phillips, R., Alvarez, R.P., Simmons, W.K., Bellgowan, P., Drevets, W.C., Bodurka, J., 2011. Self-regulation of amygdala activation using real-time fMRI neurofeedback. PLoS ONE 6, e24522 (1-17).
 Cox, R.W., 1996. AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. Comput. Biomed. Res. 29, 162-173.
 Friston, K.J., Buechel, C., Fink, G.R., Morris, J., Rolls, E., Dolan, R.J., 1997. Psychophysiological and modulatory interactions in neuroimaging. NeuroImage 6, 218-229.
 Allen, J.J.B., Harmon-Jones, E., Cavender, J.H., 2001. Manipulation of frontal EEG asymmetry through biofeedback alters self-reported emotional responses and facial EMG. Psychophysiol. 38, 685-693.
 Young, K.D., Zotev, V., Phillips, R., Misaki, M., Yuan, H., Drevets, W.C., Bodurka, J., 2014. Real-time fMRI neurofeedback training of amygdala activity in patients with major depressive disorder. PLoS ONE 9, e88785 (1-13).
 Zotev, V., Yuan, H., Misaki, M., Phillips, R., Young, K.D., Feldner, M.T., Bodurka, J. Correlation between amygdala BOLD activity and frontal EEG asymmetry during real-time fMRI neurofeedback training in patients with depression. Preprint http://arxiv.org/abs/1409.2046
©Brain Products GmbH 2014