Exploring Inner Speech Recognition via Cross-Perception Approach in EEG and fMRI
Source abstract
Abstract
Multimodal brain signal analysis has shown great potential in decoding complex cognitive processes, particularly in the challenging task of inner speech recognition. This paper introduces an innovative I nner Speech Recognition via Cross-Perception (ISRCP) approach that significantly enhances accuracy by fusing electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) data. Our approach comprises three core components: (1) multigranularity encoders that separately process EEG time series, EEG Markov Transition Fields, and fMRI spatial data; (2) a cross-perception expert structure that learns both modality-specific and shared representations; and (3) an attention-based adaptive fusion strategy that dynamically adjusts the contributions of different modalities based on task relevance. Extensive experiments on the Bimodal Dataset on Inner Speech demonstrate that our model outperforms existing methods across accuracy and F1 score.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: EEG and Brain-Computer Interfaces · Blind Source Separation Techniques · Neural dynamics and brain function
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Science and Engineering, University of Liverpool, Liverpool L69 3BX, UK
Department of Financial and Actuarial Mathematics, Xi’an Jiaotong-Liverpool University, Suzhou 215123, China
School of Computer Science and Technology, East China Normal University, Shanghai 200062, China
References (39)
Show source-provided references
1. Inner Speech: Development, Cognitive Functions, Phenomenology, and Neurobiology10.1037/bul0000021 ↗
2. Speech Synthesis from Neural Decoding of Spoken Sentences10.1038/s41586-019-1119-1 ↗
3. Martin, S., Iturrate, I., Millán, J.d.R., Knight, R.T., and Pasley, B.N. (2018). Decoding Inner Speech Using Electrocorticography: Progress and Challenges Toward a Speech Prosthesis. Front. Neurosci., 12.10.3389/fnins.2018.00422 ↗
4. Methods for Simultaneous EEG-fMRI: An Introductory Review10.1523/JNEUROSCI.0447-12.2012 ↗
5. Cooney, C., Folli, R., and Coyle, D. (2019, January 6–9). Optimizing Layers Improves CNN Generalization and Transfer Learning for Imagined Speech Decoding from EEG. Proceedings of the 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC), Bari, Italy.10.1109/SMC.2019.8914246 ↗
6. (2024). Agarwal and Kumar(2024) EEG-based Imagined Words Classification using Hilbert Transform and Deep Networks. Multimed. Tools Appl., 83, 2725–2748.10.1007/s11042-023-15664-8 ↗
7. Porbadnigk, A., Wester, M., Calliess, J., and Schultz, T. (2009, January 14–17). EEG-Based Speech Recognition—Impact of Temporal Effects. Proceedings of the International Conference on Bio-Inspired Systems and Signal Processing—Volume 1: BIOSIGNALS, (BIOSTEC 2009), Porto, Portugal.
8. Inferring imagined speech using EEG signals: A new approach using Riemannian manifold features10.1088/1741-2552/aa8235 ↗
9. Lee, Y.E., Lee, S.H., Kim, S.H., and Lee, S.W. (2023, January 7–14). Towards Voice Reconstruction from EEG during Imagined Speech. Proceedings of the AAAI Conference on Artificial Intelligence, Washington, DC, USA.10.1609/aaai.v37i5.25745 ↗
10. EEG and MEG: Relevance to Neuroscience10.1016/j.neuron.2013.10.017 ↗
11. Gu, J., Buidze, T., Zhao, K., Gläscher, J., and Fu, X. (2024). The neural network of sensory attenuation: A neuroimaging meta-analysis. Psychon. Bull. Rev.10.3758/s13423-024-02532-1 ↗
12. Contrast, Attend and Diffuse to Decode High-Resolution Images from Brain Activities
13. Functional and structural abnormalities of the speech disorders: A multimodal activation likelihood estimation meta-analysis10.1093/cercor/bhae075 ↗
14. Takagi, Y., and Nishimoto, S. (2023, January 17–24). High-Resolution Image Reconstruction with Latent Diffusion Models from Human Brain Activity. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.01389 ↗
15. Gong, P., Jia, Z., Wang, P., Zhou, Y., and Zhang, D. (November, January 29). ASTDF-Net: Attention-Based Spatial-Temporal Dual-Stream Fusion Network for EEG-Based Emotion Recognition. Proceedings of the 31st ACM International Conference on Multimedia (MM’23), Ottawa, ON, Canada.
16. Su, W.C., Dashtestani, H., Miguel, H.O., Condy, E., Buckley, A., Park, S., Perreault, J.B., Nguyen, T., Zeytinoglu, S., and Millerhagen, J. (2023). Simultaneous multimodal fNIRS-EEG recordings reveal new insights in neural activity during motor execution, observation, and imagery. Sci. Rep., 13.10.1038/s41598-023-31609-5 ↗
17. Multimodal audio-visual information fusion using canonical-correlated Graph Neural Network for energy-efficient speech enhancement10.1016/j.inffus.2022.09.006 ↗
18. Mulert, C., and Lemieux, L. (2022). The Added Value of EEG-fMRI in Imaging Neuroscience. EEG—fMRI: Physiological Basis, Technique, and Applications, Springer International Publishing.10.1007/978-3-031-07121-8 ↗
19. Carmichael, D.W., Vulliemoz, S., Murta, T., Chaudhary, U., Perani, S., Rodionov, R., Rosa, M.J., Friston, K.J., and Lemieux, L. (2024). Measurement of the Mapping between Intracranial EEG and fMRI Recordings in the Human Brain. Bioengineering, 11.10.3390/bioengineering11030224 ↗
20. Mental image reconstruction from human brain activity: Neural decoding of mental imagery via deep neural network-based Bayesian estimation10.1016/j.neunet.2023.11.024 ↗
21. Bimodal Electroencephalography-Functional Magnetic Resonance Imaging Dataset for Inner-Speech Recognition10.1038/s41597-023-02286-w ↗
22. Visual Image Reconstruction from Human Brain Activity using a Combination of Multiscale Local Image Decoders10.1016/j.neuron.2008.11.004 ↗
23. Decoding individual differences in STEM learning from functional MRI data10.1038/s41467-019-10053-y ↗
24. Decoding the Contents of Visual Working Memory: Evidence for Process-Based and Content-Based Working Memory Areas?10.1523/JNEUROSCI.4860-12.2013 ↗
25. Herff, C., Krusienski, D.J., and Kubben, P. (2020). The Potential of Stereotactic-EEG for Brain-Computer Interfaces: Current Progress and Future Directions. Front. Neurosci., 14.10.3389/fnins.2020.00123 ↗
26. A Survey on Deep Learning for Multimodal Data Fusion10.1162/neco_a_01273 ↗
27. Review of Machine Learning Techniques for EEG Based Brain Computer Interface10.1007/s11831-021-09684-6 ↗
28. Zadeh, A.B., Liang, P.P., Poria, S., Cambria, E., and Morency, L.P. (2018, January 15–20). Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia.
29. Liu, Z., Shen, Y., Lakshminarasimhan, V.B., Liang, P.P., Zadeh, A., and Morency, L.P. (2018). Efficient low-rank multimodal fusion with modality-specific factors. arXiv.10.18653/v1/P18-1209 ↗
30. Multimodal transformer for unaligned multimodal language sequences10.18653/v1/P19-1656 ↗
31. Yu, W., Xu, H., Yuan, Z., and Wu, J. (2021, January 2–9). Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis. Proceedings of the AAAI Conference on Artificial Intelligence, Virtually.10.1609/aaai.v35i12.17289 ↗
32. Han, W., Chen, H., and Poria, S. (2021, January 7–11). Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online.10.18653/v1/2021.emnlp-main.723 ↗
33. Yuan, Z., Li, W., Xu, H., and Yu, W. (2021, January 20–24). Transformer-based feature reconstruction network for robust multimodal sentiment analysis. Proceedings of the 29th ACM International Conference on Multimedia, Virtual.10.1145/3474085.3475585 ↗
34. Learning to learn better unimodal representations via adaptive multimodal meta-learning10.1109/TAFFC.2022.3178231 ↗
35. Liu, F., Shen, S.Y., Fu, Z.W., Wang, H.Y., Zhou, A.M., and Qi, J.Y. (2022). Lgcct: A light gated and crossed complementation transformer for multimodal speech emotion recognition. Entropy, 24.10.3390/e24071010 ↗
36. Efficient multimodal transformer with dual-level feature restoration for robust multimodal sentiment analysis10.1109/TAFFC.2023.3274829 ↗
37. Fu, Z., Liu, F., Xu, Q., Fu, X., and Qi, J. (2024). LMR-CBT: Learning modality-fused representations with CB-transformer for multimodal emotion recognition from unaligned multimodal sequences. Front. Comput. Sci., 18.10.1007/s11704-023-2444-y ↗
38. A cross modal hierarchical fusion multimodal sentiment analysis method based on multi-task learning10.1016/j.ipm.2024.103675 ↗
39. Co-space Representation Interaction Network for multimodal sentiment analysis10.1016/j.knosys.2023.111149 ↗