Vocabulary Retention Under Multimodal Coupling Strength Index (MCSI): Insights from Eye Tracking
Source abstract
Abstract
This eye-tracking investigation employed a 2 × 2 experimental design to examine multimodal lexical encoding processes. Eighty participants were systematically assigned to four conditions: Group A (text-only), Group B (text + image), Group C (text + sound), and Group D (text + image + sound). The results demonstrated significantly superior recall accuracy in Group D (92.00%) compared with unimodal conditions (Group B: 82.07%; Group C: 76.00%; Group A: 59.60%; p < 0.001), confirming robust audiovisual synergy. The novel Multimodal Coupling Strength Index (MCSI) dynamically quantified crossmodal integration efficacy through eye-tracking metrics (Attentional Synchronization Coefficient, ASC; Saccade Duration–Fixation Duration differential, SD-FD), revealing significantly stronger coupling in audiovisual conditions (C/D: 0.71; B/D: 0.54). Crucially, the established MCSI provides a transferable diagnostic framework for evaluating multimodal integration efficiency in learning environments.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multisensory perception and integration · Visual and Cognitive Learning Processes · Olfactory and Sensory Function Studies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Furnishings and Industrial Design, Nanjing Forestry University, Nanjing 210037, China
Funding
National Nature Science Foundation of China · China Postdoctoral Science Foundation
References (52)
Show source-provided references
1. Memory: A contribution to experimental psychology10.5214/ans.0972.7531.200408 ↗
2. The Effects of Writing Tasks on Vocabulary Learning among Korean EFL Learners and Involvement Load Hypothesis10.15738/kjell.24..202406.554 ↗
3. Integration of auditory and visual information about objects in superior temporal sulcus10.1016/S0896-6273(04)00070-4 ↗
4. Dual coding: A cognitive model for psychoanalytic research10.1177/000306518503300305 ↗
5. Dual coding theory: Retrospect and current status10.1037/h0084295 ↗
6. Effects of tasks and multimedia annotations on vocabulary learning10.1016/j.system.2023.103050 ↗
7. Nine ways to reduce cognitive load in multimedia learning10.1207/S15326985EP3801_6 ↗
8. Working memory in adults who stutter using a visual N-back task10.1016/j.jfludis.2021.105846 ↗
9. Cognitive Architecture and Instructional Design: 20 Years Later10.1007/s10648-019-09465-5 ↗
10. Emotional dissociations in temporal associations: Opposing effects of arousal on memory for details surrounding unpleasant events10.1080/02699931.2023.2270196 ↗
11. Reality monitoring and memory distortion: Effects of negative, arousing content10.3758/BF03193403 ↗
12. Disrupted basolateral amygdala circuits supports negative valence bias in depressive states10.1038/s41398-024-03085-6 ↗
13. The effects of trauma exposure and posttraumatic stress disorder (PTSD) on the emotion-induced memory trade-off
14. Processing of face identity in the affective flanker task: A diffusion model analysis10.1007/s00426-015-0696-3 ↗
15. Clarke, S., Da Costa, S., and Crottaz-Herbette, S. (2024). Dual Representation of the Auditory Space. Brain Sci., 14.10.3390/brainsci14060535 ↗
16. Multisensory perception: Beyond modularity and convergence10.1016/S0960-9822(00)00740-5 ↗
17. Semantically congruent audiovisual integration with modal-based attention accelerates auditory short-term memory retrieval10.3758/s13414-021-02437-4 ↗
18. Memory, navigation and theta rhythm in the hippocampal-entorhinal system10.1038/nn.3304 ↗
19. Creativity and Perception: Unveiling the Role of Cross-Modal Audiovisual Integration10.1002/jocb.668 ↗
20. Esposito, D., Bollini, A., and Gorr, M. (2021, January 1–5). The link between blindness onset and audiospatial processing: Testing audiomotor cues in acoustic virtual reality. Proceedings of the 43rd Annual International Conference of the IEEE-Engineering-in-Medicine-and-Biology-Society (IEEE EMBC), Virtual Event.10.36227/techrxiv.14473047.v1 ↗
21. Domain-general auditory processing determines success in second language pronunciation learning in adulthood: A longitudinal study10.1017/S0142716420000491 ↗
22. Does phonological short-term memory causally determine vocabulary learning? Toward a computational resolution of the debate10.1016/j.jml.2009.08.001 ↗
23. The tug of war between phonological, semantic and shape information in language-mediated visual search10.1016/j.jml.2007.02.001 ↗
24. Effects of reverberation on brainstem representation of speech in musicians and non-musicians10.1016/j.brainres.2010.07.100 ↗
25. Fernandez-Quezada, D., Martinez-Fernandez, D.E., Fuentes, I., Garcia-Estrada, J., and Luquin, S. (2025). The Influence of Noise Exposure on Cognitive Function in Children and Adolescents: A Meta-Analysis. NeuroSci, 6.10.20944/preprints202501.1366.v1 ↗
26. Marsh, J.E., Ljung, R., Nostl, A., Threadgold, E., and Campbell, T.A. (2015). Failing to get the gist of what’s being said: Background noise impairs higher-order cognitive processing. Front. Psychol., 6.10.3389/fpsyg.2015.00548 ↗
27. Neural and behavioral correlates of drawing in an early blind painter: A case study10.1016/j.brainres.2008.07.088 ↗
28. A connectionist model of anticipation in visual worlds
29. Imagery retrieval may explain why recall of negative scenes contains more accurate detail10.3758/s13421-018-0876-7 ↗
30. Drawings of real-world scenes during free recall reveal detailed object and spatial information in memory10.1038/s41467-018-07830-6 ↗
31. Shang, L., Huang, M., Shi, W., Liu, Y., Liu, Y., Steven, W., Sun, B., Xie, X., and Qiao, Y. (2023, January 7–14). Improving Training and Inference of Face Recognition Models via Random Temperature Scaling. Proceedings of the 37th AAAI Conference on Artificial Intelligence, Washington, DC, USA.10.1609/aaai.v37i12.26760 ↗
32. Li, R., Gong, D., Yin, W., Chen, H., Zhu, Y., Wang, K., Chen, X., Sun, J., and Zhang, Y. (2023). Learning to Fuse Monocular and Multi-view Cues for Multi-frame Depth Estimation in Dynamic Scenes. arXiv.10.1109/CVPR52729.2023.02063 ↗
33. Focused attention predicts visual working memory performance in 13-month-old infants: A pupillometric study10.1016/j.dcn.2019.100616 ↗
34. Cilia, F., Aubry, A., Le Driant, B., Bourdin, B., and Vandromme, L. (2019). Visual Exploration of Dynamic or Static Joint Attention Bids in Children with Autism Syndrome Disorder. Front. Psychol., 10.10.3389/fpsyg.2019.02187 ↗
35. Orienting of attention with eye and arrow cues and the effect of overtraining10.1016/j.actpsy.2010.03.008 ↗
36. A Dual-Stream-Modulated Learning Framework for Illuminating and Super-Resolving Ultra-Dark Images10.1109/TNNLS.2024.3409056 ↗
37. A summary statistic representation in peripheral vision explains visual search10.1167/12.4.14 ↗
38. Top-down and bottom-up control of visual selection10.1016/j.actpsy.2010.02.006 ↗
39. Environmental noise degrades hippocampus-related learning and memory10.1073/pnas.2017841117 ↗
40. Is simpler better? Semantic content modulates the emotional prosody perception in Mandarin-speaking children with autism spectrum disorder10.1016/j.jcomdis.2025.106495 ↗
41. Using multimedia for e-learning10.1111/jcal.12197 ↗
42. Lexicons, Contexts, Events, and Images: Commentary on Elman (2009) From the Perspective of Dual Coding Theory10.1111/j.1551-6709.2010.01146.x ↗
43. Cao, J., Li, X., and Zhang, L. (2025). Is Relevancy Everything? A Deep-Learning Approach to Understand the Effect of Image-Text Congruence. Manag. Sci.10.1287/mnsc.2022.01896 ↗
44. A review of research and a meta-analysis of the seductive detail effect10.1016/j.edurev.2012.05.003 ↗
45. Memorability shapes perceived time (and vice versa)10.1038/s41562-024-01863-2 ↗
46. Detecting meaning in RSVP at 13 ms per picture10.3758/s13414-013-0605-z ↗
47. View from the top: Hierarchies and reverse hierarchies in the visual system10.1016/S0896-6273(02)01091-7 ↗
48. Visual saliency and semantic incongruency influence eye movements when inspecting pictures10.1080/17470210500416342 ↗
49. Extensions of the Paivio, Yuille, and Madigan (1968) norms10.3758/BF03195584 ↗
50. Speed and Accuracy of Gaze Gestures10.16910/jemr.3.2.1 ↗
51. Gaze behavior in audiovisual speech perception: The influence of ocular fixations on the McGurk effect10.3758/BF03194582 ↗
52. How Inspecting a Picture Affects Processing of Text in Multimedia Learning10.1002/acp.2922 ↗