Data Augmentation for Audio-Visual Emotion Recognition with an Efficient Multimodal Conditional GAN
Source abstract
Abstract
Audio-visual emotion recognition is the research of identifying human emotional states by combining the audio modality and the visual modality simultaneously, which plays an important role in intelligent human-machine interactions. With the help of deep learning, previous works have made great progress for audio-visual emotion recognition. However, these deep learning methods often require a large amount of data for training. In reality, data acquisition is difficult and expensive, especially for the multimodal data with different modalities. As a result, the training data may be in the low-data regime, which cannot be effectively used for deep learning. In addition, class imbalance may occur in the emotional data, which can further degrade the performance of audio-visual emotion recognition. To address these problems, we propose an efficient data augmentation framework by designing a multimodal conditional generative adversarial network (GAN) for audio-visual emotion recognition. Specifically, we design generators and discriminators for audio and visual modalities. The category information is used as their shared input to make sure our GAN can generate fake data of different categories. In addition, the high dependence between the audio modality and the visual modality in the generated multimodal data is modeled based on Hirschfeld-Gebelein-Rényi (HGR) maximal correlation. In this way, we relate different modalities in the generated data to approximate the real data. Then, the generated data are used to augment our data manifold. We further apply our approach to deal with the problem of class imbalance. To the best of our knowledge, this is the first work to propose a data augmentation strategy with a multimodal conditional GAN for audio-visual emotion recognition. We conduct a series of experiments on three public multimodal datasets, including eNTERFACE’05, RAVDESS, and CMEW. The results indicate that our multimodal conditional GAN has high effectiveness for data augmentation of audio-visual emotion recognition.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech and Audio Processing · Music and Audio Processing · Image and Signal Denoising Methods
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Tsinghua-Berkeley Shenzhen Institute, Tsinghua University, Shenzhen 518055, China
Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, China
Funding
National Key R&D Program of China · National Natural Science Foundation of China · Shenzhen Science and Technology Program
References (100+)
Show source-provided references
1. Picard, R.W. (2020). Affective Computing, MIT Press.
2. Emotion recognition in human-computer interaction10.1109/79.911197 ↗
3. Canedo, D., and Neves, A.J.R. (2019). Facial Expression Recognition Using Computer Vision: A Systematic Review. Appl. Sci., 9.10.3390/app9214678 ↗
4. Singh, M.I., and Singh, M. (2021). Emotion Recognition: An Evaluation of ERP Features Acquired from Frontal EEG Electrodes. Appl. Sci., 11.10.3390/app11094131 ↗
5. Zhang, H., Huang, H., and Han, H. (2021). A Novel Heterogeneous Parallel Convolution Bi-LSTM for Speech Emotion Recognition. Appl. Sci., 11.10.20944/preprints202108.0433.v1 ↗
6. Luna-Jiménez, C., Cristóbal-Martín, J., Kleinlein, R., Gil-Martín, M., Moya, J.M., and Fernández-Martínez, F. (2021). Guided Spatial Transformers for Facial Expression Recognition. Appl. Sci., 11.10.3390/app11167217 ↗
7. Koromilas, P., and Giannakopoulos, T. (2021). Deep Multimodal Emotion Recognition on Human Speech: A Review. Appl. Sci., 11.10.3390/app11177962 ↗
8. Emotion recognition based on convolutional neural networks and heterogeneous bio-signal data sources10.1016/j.inffus.2021.07.007 ↗
9. Chen, S., and Jin, Q. (2016, January 6–9). Multi-modal conditional attention fusion for dimensional emotion prediction. Proceedings of the 2016 ACM on Multimedia Conference, New York, NY, USA.10.1145/2964284.2967286 ↗
10. Learning Affective Features with a Hybrid Deep Model for Audio-Visual Emotion Recognition10.1109/TCSVT.2017.2719043 ↗
11. Emotional Design Tutoring System Based on Multimodal Affective Computing Techniques10.4018/IJDET.2018010106 ↗
12. Video analytics for customer emotion and satisfaction at contact centers10.1109/THMS.2017.2695613 ↗
13. Karpouzis, K., and Yannakakis, G.N. (2016). Emotion in Games, Springer.10.1007/978-3-319-41316-7 ↗
14. Isbister, K. (2016). How Games Move Us: Emotion by Design, Mit Press.10.7551/mitpress/9267.001.0001 ↗
15. Kim, Y., Lee, H., and Provost, E.M. (2013, January 26–31). Deep learning for robust feature generation in audiovisual emotion recognition. Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Vancouver, BC, Canada.10.1109/ICASSP.2013.6638346 ↗
16. Audio-visual emotion recognition using multi-directional regression and Ridgelet transform10.1007/s12193-015-0207-2 ↗
17. Ma, F., Zhang, W., Li, Y., Huang, S.L., and Zhang, L. (2020). Learning better representations for audio-visual emotion recognition with common information. Appl. Sci., 10.10.3390/app10207239 ↗
18. Peri, R., Parthasarathy, S., Bradshaw, C., and Sundaram, S. (2021, January 6–11). Disentanglement for audio-visual emotion recognition using multitask setup. Proceedings of the ICASSP 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada.10.1109/ICASSP39728.2021.9414705 ↗
19. Data Integration: The Current Status and the Way Forward
20. Data lifecycle challenges in production machine learning: A survey10.1145/3299887.3299891 ↗
21. A survey on data collection for machine learning: A big data-ai integration perspective10.1109/TKDE.2019.2946162 ↗
22. Survey on deep learning with class imbalance10.1186/s40537-019-0192-5 ↗
23. Imbalance problems in object detection: A review10.1109/TPAMI.2020.2981890 ↗
24. Ko, T., Peddinti, V., Povey, D., Seltzer, M.L., and Khudanpur, S. (2017, January 5–9). A study on data augmentation of reverberant speech for robust speech recognition. Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA.10.1109/ICASSP.2017.7953152 ↗
25. Sahu, S., Gupta, R., and Espy-Wilson, C. (2018). On enhancing speech emotion recognition using generative adversarial networks. arXiv.10.21437/Interspeech.2018-1883 ↗
26. Park, D.S., Chan, W., Zhang, Y., Chiu, C.C., Zoph, B., Cubuk, E.D., and Le, Q.V. (2019). Specaugment: A simple data augmentation method for automatic speech recognition. arXiv.10.21437/Interspeech.2019-2680 ↗
27. Chatziagapi, A., Paraskevopoulos, G., Sgouropoulos, D., Pantazopoulos, G., Nikandrou, M., Giannakopoulos, T., Katsamanis, A., Potamianos, A., and Narayanan, S. (2019). Data Augmentation Using GANs for Speech Emotion Recognition, Interspeech.10.21437/Interspeech.2019-2561 ↗
28. Shilandari, A., Marvi, H., and Khosravi, H. (2021, December 04). Speech Emotion Recognition using Data Augmentation Method by Cycle-Generative Adversarial Networks. Available online: https://www.preprints.org/manuscript/202104.0651/v1.10.20944/preprints202104.0651.v1 ↗
29. Madani, A., Moradi, M., Karargyris, A., and Syeda-Mahmood, T. (2018, January 11–13). Chest x-ray generation and data augmentation for cardiovascular abnormality classification. Proceedings of the Medical Imaging 2018: Image Processing. International Society for Optics and Photonics, Houston, TX, USA.10.1117/12.2293971 ↗
30. Deep leaf-bootstrapping generative adversarial network for structural image data augmentation10.1111/mice.12458 ↗
31. Zhu, X., Liu, Y., Li, J., Wan, T., and Qin, Z. (2018). Emotion classification with data augmentation using generative adversarial networks. Pacific-Asia Conference on Knowledge Discovery and Data Mining, Springer.10.1007/978-3-319-93040-4_28 ↗
32. Cassava disease recognition from low-quality images using enhanced data augmentation model and deep learning10.1111/exsy.12746 ↗
33. Abayomi-Alli, O.O., Damaševičius, R., Maskeliūnas, R., and Misra, S. (2021). Few-shot learning with a novel Voronoi tessellation-based image augmentation method for facial palsy detection. Electronics, 10.10.3390/electronics10080978 ↗
34. Malignant skin melanoma detection using image augmentation by oversampling in nonlinear lower-dimensional embedding manifold10.3906/elk-2101-133 ↗
35. Wen, Q., Sun, L., Yang, F., Song, X., Gao, J., Wang, X., and Xu, H. (2020). Time series data augmentation for deep learning: A survey. arXiv.10.24963/ijcai.2021/631 ↗
36. Abayomi-Alli, O.O., Sidekerskienė, T., Damaševičius, R., Siłka, J., and Połap, D. (2020). Empirical Mode Decomposition Based Data Augmentation for Time Series Prediction Using NARX Network. International Conference on Artificial Intelligence and Soft Computing, Springer.10.1007/978-3-030-61401-0_65 ↗
37. Iwana, B.K., and Uchida, S. (2021). An empirical survey of data augmentation for time series classification with neural networks. PLoS ONE, 16.10.1371/journal.pone.0254841 ↗
38. Improving the accuracy of global forecasting models using time series data augmentation10.1016/j.patcog.2021.108148 ↗
39. Kaur, P., Khehra, B.S., and Mavi, E.B.S. (2021, January 9–11). Data Augmentation for Object Detection: A Review. Proceedings of the 2021 IEEE International Midwest Symposium on Circuits and Systems (MWSCAS), Lansing, MI, USA.10.1109/MWSCAS47672.2021.9531849 ↗
40. Generative Adversarial Nets
41. Conditional generative adversarial network-based data augmentation for enhancement of iris recognition accuracy10.1109/ACCESS.2019.2937809 ↗
42. Generative adversarial networks-based data augmentation for brain-computer interface10.1109/TNNLS.2020.3016666 ↗
43. Using a one-dimensional convolutional neural network with a conditional generative adversarial network to classify plant electrical signals10.1016/j.compag.2020.105464 ↗
44. Data augmentation for enhancing EEG-based emotion recognition with deep generative models10.1088/1741-2552/abb580 ↗
45. Antoniou, A., Storkey, A., and Edwards, H. (2017). Data augmentation generative adversarial networks. arXiv.
46. A survey on image data augmentation for deep learning10.1186/s40537-019-0197-0 ↗
47. Ma, F., Huang, S.L., and Zhang, L. (2021, January 5–9). An Efficient Approach for Audio-Visual Emotion Recognition With Missing Labels And Missing Modalities. Proceedings of the 2021 IEEE International Conference on Multimedia and Expo (ICME), Shenzhen, China.10.1109/ICME51207.2021.9428219 ↗
48. Ma, F., Zhang, W., Li, Y., Huang, S.L., and Zhang, L. (2019, January 8–12). An end-to-end learning approach for multimodal emotion recognition: Extracting common and private information. Proceedings of the 2019 IEEE International Conference on Multimedia and Expo (ICME), Shanghai, China.10.1109/ICME.2019.00200 ↗
49. Liang, Y., Ma, F., Li, Y., and Huang, S.L. (2021, January 10–15). Person Recognition with HGR Maximal Correlation on Multimodal Data. Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR), Milan, Italy.10.1109/ICPR48806.2021.9413164 ↗
50. A connection between correlation and contingency10.1017/S0305004100013517 ↗
51. Das statistische Problem der Korrelation als Variations-und Eigenwertproblem und sein Zusammenhang mit der Ausgleichsrechnung10.1002/zamm.19410210604 ↗
52. On measures of dependence10.1007/BF02024507 ↗
53. Martin, O., Kotsia, I., Macq, B., and Pitas, I. (2006, January 3–7). The enterface’05 audio-visual emotion database. Proceedings of the 22nd International Conference on Data Engineering Workshops 2006, Atlanta, GA, USA.10.1109/ICDEW.2006.145 ↗
54. Livingstone, S.R., and Russo, F.A. (2018). The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English. PLoS ONE, 13.10.1371/journal.pone.0196391 ↗
55. Ma, F., Gu, W., Zhang, W., Ni, S., Huang, S.L., and Zhang, L. (2018, January 4–7). Speech Emotion Recognition via Attention-based DNN from Multi-Task Learning. Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems, Shenzhen, China.10.1145/3274783.3275184 ↗
56. Zhang, W., Gu, W., Ma, F., Ni, S., Zhang, L., and Huang, S.L. (2018, January 4–7). Multimodal Emotion Recognition by extracting common and modality-specific information. Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems, Shenzhen, China.10.1145/3274783.3275200 ↗
57. Gu, W., Zhang, Y., Ma, F., Mosalam, K., Zhang, L., and Ni, S. (2018, January 4–7). Real-Time Emotion Detection via E-See. Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems, Shenzhen, China.10.1145/3274783.3275213 ↗
58. Multimodal machine learning: A survey and taxonomy
59. Deep multimodal learning: A survey on recent advances and trends10.1109/MSP.2017.2738401 ↗
60. Multimedia Intelligence: When Multimedia Meets Artificial Intelligence10.1109/TMM.2020.2969791 ↗
61. Deep multimodal representation learning: A survey10.1109/ACCESS.2019.2916887 ↗
62. A survey on deep learning for multimodal data fusion10.1162/neco_a_01273 ↗
63. Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., and Ng, A.Y. (July, January 28). Multimodal deep learning. Proceedings of the 28th International Conference on Machine Learning (ICML-11), Bellevue, WA, USA.
64. Wu, Z., Jiang, Y.G., Wang, J., Pu, J., and Xue, X. (2014, January 3–7). Exploring inter-feature and inter-class relationships with deep neural networks for video classification. Proceedings of the 22nd ACM international Conference on Multimedia, Orlando, FL, USA.10.1145/2647868.2654931 ↗
65. Pan, Y., Mei, T., Yao, T., Li, H., and Rui, Y. (2016, January 27–30). Jointly modeling embedding and translation to bridge video and language. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.497 ↗
66. Xu, R., Xiong, C., Chen, W., and Corso, J. (2015, January 25–30). Jointly modeling deep video and compositional text to bridge vision and language in a unified framework. Proceedings of the AAAI Conference on Artificial Intelligence, Austin, TX, USA.10.1609/aaai.v29i1.9512 ↗
67. Multimodal fusion for multimedia analysis: A survey10.1007/s00530-010-0182-0 ↗
68. A review of affective computing: From unimodal analysis to multimodal fusion10.1016/j.inffus.2017.02.003 ↗
69. Generative adversarial networks: An overview10.1109/MSP.2017.2765202 ↗
70. A survey on generative adversarial networks: Variants, applications, and training10.1145/3463475 ↗
71. Generative adversarial networks: A survey on applications and challenges10.1007/s13735-020-00196-w ↗
72. Zhu, J.Y., Park, T., Isola, P., and Efros, A.A. (2017, January 22–29). Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.244 ↗
73. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., and Wang, Z. (2017, January 21–26). Photo-realistic single image super-resolution using a generative adversarial network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.19 ↗
74. Karras, T., Laine, S., and Aila, T. (2019, January 16–20). A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00453 ↗
75. Vaccari, I., Orani, V., Paglialonga, A., Cambiaso, E., and Mongelli, M. (2021). A Generative Adversarial Network (GAN) Technique for Internet of Medical Things Data. Sensors, 21.10.3390/s21113726 ↗
76. Improved Classification of White Blood Cells with the Generative Adversarial Network and Deep Convolutional Neural Network10.1155/2020/6490479 ↗
77. Debie, E., Moustafa, N., and Whitty, M.T. (2020, January 19–24). A privacy-preserving generative adversarial network method for securing eeg brain signals. Proceedings of the 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, UK.10.1109/IJCNN48605.2020.9206683 ↗
78. Data augmentation using improved cDCGAN for plant vigor rating10.1016/j.compag.2020.105603 ↗
79. Improving Image-Based Plant Disease Classification with Generative Adversarial Network Under Limited Training Set10.3389/fpls.2020.583438 ↗
80. Investigation on the data augmentation using machine learning algorithms in structural health monitoring information
81. Latif, S., Asim, M., Rana, R., Khalifa, S., Jurdak, R., and Schuller, B.W. (2020, January 25–29). Augmenting generative adversarial networks for speech emotion recognition. Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, Shanghai, China.10.21437/Interspeech.2020-3194 ↗
82. Lai, Y.H., and Lai, S.H. (2018, January 15–19). Emotion-preserving representation learning via generative adversarial network for multi-view facial expression recognition. Proceedings of the 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), Xi’an, China.10.1109/FG.2018.00046 ↗
83. Yi, W., Sun, Y., and He, S. (2018, January 1–4). Data augmentation using conditional GANs for facial emotion recognition. Proceedings of the 2018 Progress in Electromagnetics Research Symposium (PIERS-Toyama), Toyama, Japan.10.23919/PIERS.2018.8598226 ↗
84. Luo, Y., and Lu, B.L. (2018, January 17–21). EEG data augmentation for emotion recognition using a conditional Wasserstein GAN. Proceedings of the 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Honolulu, HI, USA.10.1109/EMBC.2018.8512865 ↗
85. C-GCN: Correlation based graph convolutional network for audio-video emotion recognition10.1109/TMM.2020.3032037 ↗
86. Effective data generation for imbalanced learning using conditional generative adversarial networks10.1016/j.eswa.2017.09.030 ↗
87. MFC-GAN: Class-imbalanced dataset classification using multiple fake class generative adversarial network10.1016/j.neucom.2019.06.043 ↗
88. Machinery fault diagnosis with imbalanced data using deep generative adversarial networks10.1016/j.measurement.2019.107377 ↗
89. Balanced semisupervised generative adversarial network for damage assessment from low-data imbalanced-class regime10.1111/mice.12741 ↗
90. Data Augmentation for Sentiment Analysis Using Sentence Compression-Based SeqGAN With Data Screening10.1109/ACCESS.2021.3094023 ↗
91. Ma, F., Xu, X., Huang, S.L., and Zhang, L. (2021). Maximum Likelihood Estimation for Multimodal Learning with Missing Modality. arXiv.
92. Joint face detection and alignment using multitask cascaded convolutional networks10.1109/LSP.2016.2603342 ↗
93. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference On Computer Vision And Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
94. Khorrami, P., Le Paine, T., Brady, K., Dagli, C., and Huang, T.S. (2016, January 25–28). How deep neural networks can improve emotion recognition on video data. Proceedings of the 2016 IEEE International Conference On Image Processing (ICIP), Phoenix, AZ, USA.10.1109/ICIP.2016.7532431 ↗
95. Zhang, C., Cui, Y., Han, Z., Zhou, J.T., Fu, H., and Hu, Q. (2020). Deep partial multi-view learning. IEEE Trans. Pattern Anal. Mach. Intell.10.1109/TPAMI.2020.3037734 ↗
96. Audio-Visual Deep Neural Network for Robust Person Verification10.1109/TASLP.2021.3057230 ↗
97. Xi, C., Lu, G., and Yan, J. (2020, January 17–19). Multimodal sentiment analysis based on multi-head attention mechanism. Proceedings of the 4th International Conference on Machine Learning and Soft Computing, Haiphong City, Viet Nam.10.1145/3380688.3380693 ↗
98. Feature Fusion for Multimodal Emotion Recognition Based on Deep Canonical Correlation Analysis10.1109/LSP.2021.3112314 ↗
99. Multimodal Sentiment Analysis Using Multi-tensor Fusion Network with Cross-modal Modeling
100. Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.