Continuous Emotion Recognition with Spatiotemporal Convolutional Neural Networks
Source abstract
Abstract
Facial expressions are one of the most powerful ways to depict specific patterns in human behavior and describe the human emotional state. However, despite the impressive advances of affective computing over the last decade, automatic video-based systems for facial expression recognition still cannot correctly handle variations in facial expression among individuals as well as cross-cultural and demographic aspects. Nevertheless, recognizing facial expressions is a difficult task, even for humans. This paper investigates the suitability of state-of-the-art deep learning architectures based on convolutional neural networks (CNNs) to deal with long video sequences captured in the wild for continuous emotion recognition. For such an aim, several 2D CNN models that were designed to model spatial information are extended to allow spatiotemporal representation learning from videos, considering a complex and multi-dimensional emotion space, where continuous values of valence and arousal must be predicted. We have developed and evaluated convolutional recurrent neural networks, combining 2D CNNs and long short term-memory units and inflated 3D CNN models, which are built by inflating the weights of a pre-trained 2D CNN model during fine-tuning, using application-specific videos. Experimental results on the challenging SEWA-DB dataset have shown that these architectures can effectively be fine-tuned to encode spatiotemporal information from successive raw pixel images and achieve state-of-the-art results on such a dataset.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Emotion and Mood Recognition · Face recognition and analysis · Face and Expression Recognition
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
École de Technologie Supérieure, Université du Québec, 1100, Rue Notre-Dame Ouest, Montreal, QC H3C 1K3, Canada
References (76)
Show source-provided references
1. Li, S., and Deng, W. (2018). Deep facial expression recognition: A survey. arXiv.
2. Constants across cultures in the face and emotion10.1037/h0030377 ↗
3. Strong evidence for universals in facial expressions: A reply to russell’s mistaken critique10.1037/0033-2909.115.2.268 ↗
4. Emotional Expressions Reconsidered: Challenges to Inferring Emotion From Human Facial Movements10.1177/1529100619832930 ↗
5. AI weighs in on debate about universal facial expressions10.1038/d41586-020-03509-5 ↗
6. Facial expressions of emotion are not culturally universal10.1073/pnas.1200155109 ↗
7. Four-quadrant investigation of job-related affects and behaviours10.1080/1359432X.2012.744449 ↗
8. 2D Principal Component Analysis for Face and Facial-Expression Recognition10.1109/MCSE.2010.149 ↗
9. Zavaschi, T.H.H., Koerich, A.L., and Oliveira, L.E.S. (2011, January 22–27). Facial expression recognition using ensemble of classifiers. Proceedings of the 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Prague, Czech Republic.10.1109/ICASSP.2011.5946775 ↗
10. Facial expression recognition based on local binary patterns: A comprehensive study10.1016/j.imavis.2008.08.005 ↗
11. Fusion of feature sets and classifiers for facial expression recognition10.1016/j.eswa.2012.07.074 ↗
12. Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions10.1109/TPAMI.2007.1110 ↗
13. LBP with Six Intersection Points: Reducing Redundant Information in LBP-TOP for Micro-expression Recognition
14. Cossetin, M.J., Nievola, J.C., and Koerich, A.L. (2016, January 24–29). Facial Expression Recognition Using a Pairwise Feature Selection and Classification Approach. Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN), Vancouver, BC, Canada.10.1109/IJCNN.2016.7727879 ↗
15. Challenges in Representation Learning: A Report on Three Machine Learning Contests
16. Susskind, J.M., Anderson, A.K., and Hinton, G.E. (2010). The Toronto Face Dataset, U. Toronto. Technical Report TR 2010-001.
17. Dhall, A., Goecke, R., Lucey, S., and Gedeon, T. (2011, January 6–13). Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark. Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCV Workshops), Barcelona, Spain.10.1109/ICCVW.2011.6130508 ↗
18. Local Learning With Deep and Handcrafted Features for Facial Expression Recognition10.1109/ACCESS.2019.2917266 ↗
19. Kim, B.K., Lee, H., Roh, J., and Lee, S.Y. (2015, January 9–13). Hierarchical Committee of Deep CNNs with Exponentially-Weighted Decision Fusion for Static Facial Expression Recognition. Proceedings of the International Conference on Multimodal Interaction, ICMI ’15, Seattle, WA, USA.10.1145/2818346.2830590 ↗
20. Liu, X., Kumar, B.V.K.V., You, J., and Jia, P. (2017, January 21–26). Adaptive Deep Metric Learning for Identity-Aware Facial Expression Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, USA.10.1109/CVPRW.2017.79 ↗
21. Zhang, Z., Luo, P., Loy, C.C., and Tang, X. (2015, January 7–13). Learning Social Relation Traits from Face Images. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.414 ↗
22. Guo, Y., Tao, D., Yu, J., Xiong, H., and Li, Y. (2016, January 11–15). Deep Neural Networks with Relativity Learning for facial expression recognition. Proceedings of the IEEE International Conference on Multimedia Expo Workshops (ICMEW), Seattle, WA, USA.
23. Kim, B., Dong, S., Roh, J., Kim, G., and Lee, S. (July, January 26). Fusing Aligned and Non-aligned Face Information for Automatic Affect Recognition in the Wild: A Deep Learning Approach. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Las Vegas, NV, USA.
24. Pramerdorfer, C., and Kampel, M. (2016). Facial Expression Recognition using Convolutional Neural Networks: State of the Art. arXiv.
25. Fayolle, S.L., and Droit-Volet, S. (2014). Time Perception and Dynamics of Facial Expressions of Emotions. PLoS ONE, 9.10.1371/journal.pone.0097944 ↗
26. Bargal, S.A., Barsoum, E., Ferrer, C.C., and Zhang, C. (2016, January 12–16). Emotion Recognition in the Wild from Videos Using Images. Proceedings of the International Conference on Multimodal Interaction, ICMI ’16, Tokyo, Japan.10.1145/2993148.2997627 ↗
27. Ding, W., Xu, M., Huang, D.Y., Lin, W., Dong, M., Yu, X., and Li, H. (2016, January 12–16). Audio and Face Video Emotion Recognition in the Wild Using Deep Neural Networks and Small Datasets. Proceedings of the 18th ACM International Conference on Multimodal Interaction, ICMI ’16, Tokyo, Japan.10.1145/2993148.2997637 ↗
28. Ayral, T., Pedersoli, M., Bacon, S., and Granger, E. (2021, January 5–9). Temporal Stochastic Softmax for 3D CNNs: An Application in Facial Expression Recognition. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Waikola, HI, USA.10.1109/WACV48630.2021.00307 ↗
29. Carneiro de Melo, W., Granger, E., and Hadid, A. (2020). A Deep Multiscale Spatiotemporal Network for Assessing Depression from Facial Dynamics. IEEE Trans. Affect. Comput.
30. Tran, D., Bourdev, L.D., Fergus, R., Torresani, L., and Paluri, M. (2015, January 7–13). Learning Spatiotemporal Features with 3D Convolutional Networks. Proceedings of the IEEE International Conference on Computer Vision, (ICCV), Santiago, Chile.10.1109/ICCV.2015.510 ↗
31. Abbasnejad, I., Sridharan, S., Nguyen, D., Denman, S., Fookes, C., and Lucey, S. (2017, January 22–29). Using Synthetic Data to Improve Facial Expression Analysis with 3D Convolutional Networks. Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCVW), Venice, Italy.10.1109/ICCVW.2017.189 ↗
32. Fan, Y., Lu, X., Li, D., and Liu, Y. (2016, January 12–16). Video-Based Emotion Recognition Using CNN-RNN and C3D Hybrid Networks. Proceedings of the International Conference on Multimodal Interaction, ICMI ’16, Tokyo, Japan.10.1145/2993148.2997632 ↗
33. Nguyen, D., Nguyen, K., Sridharan, S., Ghasemi, A., Dean, D., and Fookes, C. (2017, January 24–31). Deep Spatio-Temporal Features for Multimodal Emotion Recognition. Proceedings of the 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), Santa Rosa, CA, USA.10.1109/WACV.2017.140 ↗
34. Liu, K., Liu, W., Gan, C., Tan, M., and Ma, H. (2018, January 2–7). T-C3D: Temporal Convolutional 3D Network for Real-Time Action Recognition. Proceedings of the 32nd AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.12333 ↗
35. Developing crossmodal expression recognition based on a deep neural model10.1177/1059712316664017 ↗
36. Learning deep facial expression features from image and optical flow sequences using 3D CNN10.1007/s00371-018-1477-y ↗
37. Ouyang, X., Kawaai, S., Goh, E., Shen, S., Ding, W., Ming, H., and Huang, D.Y. (2017, January 13–17). Audio-visual emotion recognition using deep transfer learning and multiple temporal models. Proceedings of the 19th ACM International Conference on Multimodal Interaction, Glasgow, UK.10.1145/3136755.3143012 ↗
38. 3D Convolutional Neural Networks for Human Action Recognition10.1109/TPAMI.2012.59 ↗
39. Vielzeuf, V., Pateux, S., and Jurie, F. (2017, January 13–17). Temporal multimodal fusion for video emotion classification in the wild. Proceedings of the 19th ACM International Conference on Multimodal Interaction (ICMI), Glasgow, UK.10.1145/3136755.3143011 ↗
40. Campos, V., Salvador, A., Giro-iNieto, X., and Jou, B. (2015, January 30). Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction. Proceedings of the 1st International Workshop on Affect and Sentiment in Multimedia, Brisbane, Australia.10.1145/2813524.2813530 ↗
41. Xu, C., Cetintas, S., Lee, K.C., and Li, L.J. (2014). Visual Sentiment Prediction with Deep Convolutional Neural Networks. arXiv.
42. Parkhi, O.M., Vedaldi, A., and Zisserman, A. (2015, January 7–10). Deep Face Recognition. Proceedings of the British Machine Vision Conference (BMVC), Swansea, UK.10.5244/C.29.41 ↗
43. Simonyan, K., and Zisserman, A. (2015, January 7–9). Very Deep Convolutional Networks for Large-Scale Image Recognition. Proceedings of the 3rd International Conference on Learning Representations, (ICLR), San Diego, CA, USA.
44. Dhall, A., Kaur, A., Goecke, R., and Gedeon, T. (2018, January 16–20). EmotiW 2018: Audio-Video, Student Engagement and Group-Level Affect Prediction. Proceedings of the International Conference on Multimodal Interaction (ICMI), Boulder, CO, USA.10.1145/3242969.3264993 ↗
45. Ringeval, F., Schuller, B., Valstar, M., Jaiswal, S., Marchi, E., Lalanne, D., Cowie, R., and Pantic, M. (2015, January 26). AV+EC 2015: The First Affect Recognition Challenge Bridging Across Audio, Video, and Physiological Data. Proceedings of the 5th International Workshop on Audio/Visual Emotion Challenge, AVEC ’15, Brisbane, Australia.10.1145/2808196.2811642 ↗
46. Wan, L., Liu, N., Huo, H., and Fang, T. (2017, January 2–4). Face Recognition with Convolutional Neural Networks and subspace learning. Proceedings of the 2nd International Conference on Image, Vision and Computing (ICIVC), Chengdu, China.
47. Knyazev, B., Shvetsov, R., Efremova, N., and Kuharenko, A. (2017). Convolutional neural networks pretrained on large face recognition datasets for emotion classification from video. arXiv.10.1109/FG.2018.00109 ↗
48. Ding, H., Zhou, S.K., and Chellappa, R. (June, January 30). FaceNet2ExpNet: Regularizing a Deep Face Recognition Net for Expression Recognition. Proceedings of the 12th IEEE International Conference on Automatic Face & Gesture Recognition, Washington, DC, USA.
49. Carreira, J., and Zisserman, A. (2017, January 21–26). Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR, Honolulu, HI, USA.10.1109/CVPR.2017.502 ↗
50. Diba, A., Fayyaz, M., Sharma, V., Karami, A.H., Arzani, M.M., Yousefzadeh, R., and Gool, L.V. (2017). Temporal 3D ConvNets: New Architecture and Transfer Learning for Video Classification. arXiv.
51. Praveen, G., Granger, E., and Cardinal, P. (2020). Deep DA for Ordinal Regression of Pain Intensity Estimation Using Weakly-Labeled Videos. arXiv.
52. Praveen, G., Granger, E., and Cardinal, P. (2020, January 18–22). Deep Weakly-Supervised Domain Adaptation for Pain Localization in Videos. Proceedings of the 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), Buenos Aires, Argentina.
53. Cardinal, P., Dehak, N., Koerich, A.L., Alam, J., and Boucher, P. (2015, January 26–30). ETS System for AV+EC 2015 Challenge. Proceedings of the ACM Multimedia Conference, Brisbane, Australia.10.1145/2808196.2811639 ↗
54. Simonyan, K., and Zisserman, A. (2014, January 8–13). Two-Stream Convolutional Networks for Action Recognition in Videos. Proceedings of the Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems, Montreal, QC, Canada.
55. Building Emotional Machines: Recognizing Image Emotions through Deep Neural Networks10.1109/TMM.2018.2827782 ↗
56. Mou, W., Celiktutan, O., and Gunes, H. (2015, January 4–8). Group-level arousal and valence recognition in static images: Face, body and context. Proceedings of the 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Ljubljana, Slovenia.
57. Ben Henia, W.M., and Lachiri, Z. (2017, January 8–10). Emotion classification in arousal-valence dimension using discrete affective keywords tagging. Proceedings of the International Conference on Engineering MIS (ICEMIS), Monastir, Tunisia.10.1109/ICEMIS.2017.8272991 ↗
58. Kollias, D., and Zafeiriou, S. (2018). A Multi-component CNN-RNN Approach for Dimensional Emotion Recognition in-the-wild. arXiv.
59. Deep Affect Prediction in-the-Wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond10.1007/s11263-019-01158-4 ↗
60. Kossaifi, J., Schuller, B.W., Star, K., Hajiyev, E., Pantic, M., Walecki, R., Panagakis, Y., Shen, J., Schmitt, M., and Ringeval, F. (2019). SEWA DB: A Rich Database for Audio-Visual Emotion and Sentiment Research in the Wild. IEEE Trans. Pattern Anal. Mach. Intell.
61. Cao, Q., Shen, L., Xie, W., Parkhi, O.M., and Zisserman, A. (2018, January 15–19). VGGFace2: A Dataset for Recognising Faces across Pose and Age. Proceedings of the 13th IEEE International Conference on Automatic Face & Gesture Recognition, Xi’an, China.10.1109/FG.2018.00020 ↗
62. Jyoti, S., Sharma, G., and Dhall, A. (2019, January 14–18). Expression Empowered ResiDen Network for Facial Action Unit Detection. Proceedings of the 14th IEEE International Conference on Automatic Face & Gesture Recognition, Lille, France.10.1109/FG.2019.8756580 ↗
63. ImageNet Classification with Deep Convolutional Neural Networks10.1145/3065386 ↗
64. Video-Based Emotion Recognition in the Wild using Deep Transfer Learning and Score Fusion10.1016/j.imavis.2017.01.012 ↗
65. Li, S., Deng, W., and Du, J. (2017, January 21–26). Reliable Crowdsourcing and Deep Locality-Preserving Learning for Expression Recognition in the Wild. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.277 ↗
66. Reliable Crowdsourcing and Deep Locality-Preserving Learning for Unconstrained Facial Expression Recognition10.1109/TIP.2018.2868382 ↗
67. Tannugi, D.L., Britto, A.S., and Koerich, A.L. (2019, January 6–9). Memory Integrity of CNNs for Cross-Dataset Facial Expression Recognition. Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (SMC), Bari, Italy.10.1109/SMC.2019.8914172 ↗
68. Li, S., and Deng, W. (2020). A Deeper Look at Facial Expression Dataset Bias. IEEE Trans. Affect. Comput.
69. de Matos, J., Britto, A.S., Oliveira, L.E.S., and Koerich, A.L. (2019, January 14–19). Double Transfer Learning for Breast Cancer Histopathologic Image Classification. Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN), Budapest, Hungary.10.1109/IJCNN.2019.8852092 ↗
70. Joint face detection and alignment using multitask cascaded convolutional networks10.1109/LSP.2016.2603342 ↗
71. Yu, F., and Koltun, V. (2016, January 2–4). Multi-Scale Context Aggregation by Dilated Convolutions. Proceedings of the 4th International Conference on Learning Representations (ICLR), San Juan, Puerto Rico.
72. Ortega, J.D.S., Cardinal, P., and Koerich, A.L. (2019, January 6–9). Emotion Recognition Using Fusion of Audio and Video Features. Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (SMC), Bari, Italy.10.1109/SMC.2019.8914655 ↗
73. Occlusion Aware Facial Expression Recognition Using CNN With Attention Mechanism10.1109/TIP.2018.2886767 ↗
74. Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition10.1109/TIP.2019.2956143 ↗
75. Chen, H., Deng, Y., Cheng, S., Wang, Y., Jiang, D., and Sahli, H. (2019, January 21). Efficient Spatial Temporal Convolutional Features for Audiovisual Continuous Affect Recognition. Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop, AVEC ’19, Nice, France.10.1145/3347320.3357690 ↗
76. Zhao, J., Li, R., Liang, J., Chen, S., and Jin, Q. (2019, January 21). Adversarial Domain Adaption for Multi-Cultural Dimensional Emotion Recognition in Dyadic Interactions. Proceedings of the 9th International on Audio/Visual Emotion Challenge and Workshop, AVEC ’19, Nice, France.10.1145/3347320.3357692 ↗