Colour Classification Analysis Based on MFCC Acoustic Feature Sets and Machine Learning Algorithms in Sound–Colour Synaesthesia
Source abstract
Abstract
Sound–colour synaesthesia is a rare phenomenon in which auditory stimuli automatically evoke stable, subjectively real colour experiences. This study aimed to investigate whether the colours most frequently reported by a synesthete can be reliably predicted based on objective acoustic parameters of voice signals. The study analysed the responses of a 24-year-old blind woman to different voices, which she consciously associates with distinct coloured silhouettes. A classification analysis based on MFCC acoustic feature sets and machine learning algorithms (SVM, XGBoost) demonstrated that the models could be trained with very high Accuracy—up to 97–100% in binary classification and 89–90% in multi-class classification. These results provide new insights into how specific sound characteristics are linked to imagery arising from the human subconscious.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Multisensory perception and integration · Color perception and design · Olfactory and Sensory Function Studies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Natural Sciences, Vytautas Magnus University, LT-53361 Akademija, Lithuania
Research Institute of Natural and Technological Sciences, Vytautas Magnus University, LT-53361 Akademija, Lithuania
Ultrasound Research Institute, Kaunas University of Technology, K. Baršausko St. 59, LT-51423 Kaunas, Lithuania
Department of System Analysis, Faculty of Informatics, Vytautas Magnus University, Universiteto Str. 10-213, LT-53361 Akademija, Lithuania
Department of Electrical Power Systems, Faculty of Electrical and Electronics Engineering, Kaunas University of Technology, Studentų St. 48, LT-51367 Kaunas, Lithuania
Department of Informatics, Vytautas Magnus University, LT-53361 Akademija, Lithuania
Neuroscience Institute, Lithuanian University of Health Sciences, LT- 50161 Kaunas, Lithuania
Department of Health Psychology, Faculty of Public Health, Lithuanian University of Health Sciences, LT-50161 Kaunas, Lithuania
References (80)
Show source-provided references
1. Color synesthesia. Insight into perception, emotion, and consciousness10.1097/WCO.0000000000000169 ↗
2. Mylopoulos, M.I., and Ro, T. (2013). Synesthesia: A colorful word with a touching sound?. Front. Psychol., 4.10.3389/fpsyg.2013.00763 ↗
3. Sinestesia e percepção musical
4. Lexical-gustatory synaesthesia: Linguistic and conceptual factors10.1016/S0010-0277(03)00122-7 ↗
5. Similarly shaped letters evoke similar colors in grapheme–color synesthesia10.1016/j.neuropsychologia.2011.01.002 ↗
6. The automaticity of vantage point shifts within a synaesthetes’ spatial calendar10.1111/j.1748-6653.2011.02011.x ↗
7. Grapheme-Color Synesthesia and Its Connection to Memory
8. A standardized test battery for the study of synesthesia10.1016/j.jneumeth.2006.07.012 ↗
9. Rothen, N., and Meier, B. (2014). Acquiring synaesthesia: Insights from training studies. Front. Hum. Neurosci., 8.10.3389/fnhum.2014.00109 ↗
10. Enhanced memory ability: Insights from synaesthesia10.1016/j.neubiorev.2012.05.004 ↗
11. Rare variants in axonogenesis genes connect three families with sound-color synesthesia10.1073/pnas.1715492115 ↗
12. A case report of acquired synesthesia and heightened creativity in a musician after traumatic brain injury10.1080/13554794.2023.2208271 ↗
13. Seeing mathematics: Perceptual experience and brain activity in acquired synesthesia10.1080/13554794.2012.701646 ↗
14. Non-optic vision: Beyond synesthesia?10.1016/j.bandc.2016.05.007 ↗
15. The cross-activation theory at 1010.1111/j.1748-6653.2011.02014.x ↗
16. Reduced perceptual narrowing in synesthesia10.1073/pnas.1914668117 ↗
17. Laeng, B., Flaaten, C.B., Walle, K.M., Hochkeppler, A., and Specht, K. (2021). ‘Mickey Mousing’ in the Brain: Motion-Sound Synesthesia and the Subcortical Substrate of Audio-Visual Integration. Front. Hum. Neurosci., 15.10.3389/fnhum.2021.605166 ↗
18. Priming Letters by Colors: Evidence for the Bidirectionality of Grapheme–Color Synesthesia10.1162/jocn.2008.21166 ↗
19. Hupé, J.M., and Dojat, M. (2015). A critical review of the neuroimaging literature on synesthesia. Front. Hum. Neurosci., 9.10.3389/fnhum.2015.00103 ↗
20. Stochastic resonance model of synaesthesia10.1098/rstb.2019.0029 ↗
21. The neuroanatomy of grapheme-color synesthesia10.1111/j.1460-9568.2009.06673.x ↗
22. Eckardt, N., Sinke, C., Bleich, S., Lichtinghagen, R., and Zedler, M. (2024). Investigation of the relationship between neuroplasticity and grapheme-color synesthesia. Front. Neurosci., 18.10.3389/fnins.2024.1434309 ↗
23. Estimating a neutral reference for electroencephalographic recordings: The importance of using a high-density montage and a realistic head model10.1088/1741-2560/12/5/056012 ↗
24. Neuronal Correlates of Colour-Graphemic Synaesthesia: Afmri Study10.1016/S0010-9452(08)70355-1 ↗
25. Multisensory attention and tactile information-processing10.1016/S0166-4328(02)00155-9 ↗
26. Multisensory integration: Perceptual grouping by eye and ear10.1016/S0960-9822(01)00175-0 ↗
27. Sensory Perception: Lessons from Synesthesia using Synesthesia to inform the understanding of Sensory Perception
28. Synaesthesia—A Window Into Perception, Thought and Language
29. Mechanisms of synesthesia: Cognitive and physiological constraints10.1016/S1364-6613(00)01571-0 ↗
30. Reduced audio–visual integration in synaesthetes indicated by the double-flash illusion10.1016/j.brainres.2012.07.011 ↗
31. Neurophysiology of synesthesia10.1007/s11920-007-0018-6 ↗
32. The chemical induction of synaesthesia10.1002/hup.2832 ↗
33. Hearing words and seeing colours: An experimental investigation of a case of synaesthesia10.1068/p160761 ↗
34. Diagnosing and Phenotyping Visual Synaesthesia: A Preliminary Evaluation of the Revised Test of Genuineness (TOG-R)10.1016/S0010-9452(08)70337-X ↗
35. Bartulienė, R., Saudargienė, A., Reinytė, K., Davidavičius, G., Davidavičienė, R., Ašmantas, Š., Raškinis, G., and Šatkauskas, S. (2025). Voice-Evoked Color Prediction Using Deep Neural Networks in Sound–Color Synesthesia. Brain Sci., 15.10.3390/brainsci15050520 ↗
36. Colored hearing synesthesia: An investigation of neural factors10.1212/WNL.39.6.781 ↗
37. Itoh, K., Sakata, H., Kwee, I.L., and Nakada, T. (2017). Musical pitch classes have rainbow hues in pitch class-color synesthesia. Sci. Rep., 7.10.1038/s41598-017-18150-y ↗
38. Goller, A.I., Otten, L.J., and Ward, J. (2025, July 01). Seeing Sounds and Hearing Colors: An Event-Related Potential Study of Auditory-Visual Synesthesia. Available online: http://mitprc.silverchair.com/jocn/article-pdf/21/10/1869/1759787/jocn.2009.21134.pdf.10.1162/jocn.2009.21134 ↗
39. Colored-hearing synesthesia
40. Analysis of EEG records in an epileptic patient using wavelet transform10.1016/S0165-0270(02)00340-0 ↗
41. Implicit associations between individual properties of color and sound10.3758/s13414-018-01639-7 ↗
42. Synesthesia: Strong and weak10.1111/1467-8721.00116 ↗
43. Asci, F., Costantini, G., Di Leo, P., Zampogna, A., Ruoppolo, G., Berardelli, A., Saggio, G., and Suppa, A. (2020). Machine-learning analysis of voice samples recorded through smartphones: The combined effect of ageing and gender. Sensors, 20.10.3390/s20185022 ↗
44. Majkowski, A., and Kołodziej, M. (2025). Emotion Recognition from Speech in a Subject-Independent Approach. Appl. Sci., 15.10.3390/app15136958 ↗
45. Sarbast, H. (2024). Voice Recognition Based on Machine Learning Classification Algorithms: A Review. Indones. J. Comput. Sci., 4414–4431.10.33022/ijcs.v13i3.4110 ↗
46. Gender Recognition by Voice Using Machine Learning
47. Bock, J.R. (2018). A deep learning model of perception in color-letter synesthesia. Big Data Cogn. Comput., 2.10.3390/bdcc2010008 ↗
48. Synaesthesia is linked to a distinctive and heritable cognitive profile10.1016/j.cortex.2020.01.002 ↗
49. Lindborg, P., and Friberg, A.K. (2015). Colour Association with Music Is Mediated by Emotion: Evidence from an Experiment Using a CIE Lab Interface and Interviews. PLoS ONE, 10.10.1371/journal.pone.0144013 ↗
50. Moos, A., Simmons, D., Simner, J., and Smith, R. (2013). Color and texture associations in voice-induced synesthesia. Front. Psychol., 4.10.3389/fpsyg.2013.00568 ↗
51. Synesthetic colors are elicited by sound quality in Japanese synesthetes10.1016/j.concog.2011.05.012 ↗
52. Neurocognitive Mechanisms of Synesthesia10.1016/j.neuron.2005.10.012 ↗
53. Giannakopoulos, T. (2015). pyAudioAnalysis: An Open-Source Python Library for Audio Signal Analysis. PLoS ONE, 10.10.1371/journal.pone.0144610 ↗
54. Trajectory of Fifths Based on Chroma Subbands Extraction—A New Approach to Music Representation, Analysis, and Classification10.1109/TPAMI.2024.3519420 ↗
55. Mel Frequency Cepstral Coefficient and its Applications: A Review10.1109/ACCESS.2022.3223444 ↗
56. On Splitting Training and Validation Set: A Comparative Study of Cross-Validation, Bootstrap and Systematic Sampling for Estimating the Generalization Performance of Supervised Learning10.1007/s41664-018-0068-2 ↗
57. Guo, Y., Graber, A., McBurney, R.N., and Balasubramanian, R. (2010). Sample size and statistical power considerations in high-dimensionality data settings: A comparative study of classification algorithms. BMC Bioinform., 11.10.1186/1471-2105-11-447 ↗
58. Elgeldawi, E., Sayed, A., Galal, A.R., and Zaki, A.M. (2021). Hyperparameter Tuning for Machine Learning Algorithms Used for Arabic Sentiment Analysis. Informatics, 8.10.3390/informatics8040079 ↗
59. Jin, W., Wang, X., and Zhan, Y. (2022). Environmental Sound Classification Algorithm Based on Region Joint Signal Analysis Feature and Boosting Ensemble Learning. Electronics, 11.10.3390/electronics11223743 ↗
60. Worldwide Healthy Adult Voice Baseline Parameters: A Comprehensive Review10.1016/j.jvoice.2020.08.028 ↗
61. Characterization of Healthy and Pathological Voice Through Measures Based on Nonlinear Dynamics10.1109/TASL.2009.2016734 ↗
62. Dimensionality Reduction of a Pathological Voice Quality Assessment System Based on Gaussian Mixture Models and Short-Term Cepstral Parameters10.1109/TBME.2006.871883 ↗
63. A nonlinear operator-based speech feature analysis method with application to vocal fold pathology assessment10.1109/10.661155 ↗
64. Voice Disorder Classification Based on Multitaper Mel Frequency Cepstral Coefficients Features10.1155/2015/956249 ↗
65. Lung disease recognition methods using audio-based analysis with machine learning10.1016/j.heliyon.2024.e26218 ↗
66. Ishimaru, M., Okada, Y., Uchiyama, R., Horiguchi, R., and Toyoshima, I. (2023). A New Regression Model for Depression Severity Prediction Based on Correlation among Audio Features Using a Graph Convolutional Neural Network. Diagnostics, 13.10.3390/diagnostics13040727 ↗
67. Srinivasan, S., Ramadass, P., Mathivanan, S.K., Selvam, K.P., Shivahare, B.D., and Shah, M.A. (2024). Detection of Parkinson disease using multiclass machine learning approach. Sci. Rep., 14.10.1038/s41598-024-64004-9 ↗
68. Ashwini, K., Vincent, P.M.D.R., Srinivasan, K., and Chang, C.Y. (2021). Deep Learning Assisted Neonatal Cry Classification via Support Vector Machine Models. Front. Public Health, 9.10.3389/fpubh.2021.670352 ↗
69. Yanushevskaya, I., Gobl, C., and Chasaide, A.N. (2013). Voice quality in affect cueing: Does loudness matter?. Front. Psychol., 4.10.3389/fpsyg.2013.00335 ↗
70. Helmholtz’s critique of Goethe’s Theory of Color: More than meets the eye10.1016/j.survophthal.2018.10.004 ↗
71. Elliot, A.J. (2015). Color and psychological functioning: A review of theoretical and empirical work. Front. Psychol., 6.10.3389/fpsyg.2015.00368 ↗
72. Some Experimental Observations Concerning the Influence of Colors on the Function of the Organism
73. Fairchild, M. (2013). Color Appearance Models, John Wiley & Sons.10.1002/9781118653128 ↗
74. Extending Color Psychology to the Personality Realm: Interpersonal Hostility Varies by Red Preferences and Perceptual Biases10.1111/jopy.12087 ↗
75. Effects of color–emotion association on facial expression judgments10.1016/j.heliyon.2022.e08804 ↗
76. Universal Patterns in Color-Emotion Associations Are Further Shaped by Linguistic and Geographic Proximity10.1177/0956797620948810 ↗
77. Do we feel colours? A systematic review of 128 years of psychological research linking colours and emotions10.3758/s13423-024-02615-z ↗
78. Study of MFCC and IHC feature extraction methods with probabilistic acoustic models for speaker biometric applications10.1016/j.procs.2018.10.395 ↗
79. A Circumplex Model of Affect10.1037/h0077714 ↗
80. Yin, Y., Shao, Y., Hao, Y., and Lu, X. (2024). Perceived Soundscape Experiences and Human Emotions in Urban Green Spaces: Application of Russell’s Circumplex Model of Affect. Appl. Sci., 14.10.3390/app14135828 ↗