On-Road Driver Emotion Recognition Using Facial Expression
Source abstract
Abstract
With the development of intelligent automotive human-machine systems, driver emotion detection and recognition has become an emerging research topic. Facial expression-based emotion recognition approaches have achieved outstanding results on laboratory-controlled data. However, these studies cannot represent the environment of real driving situations. In order to address this, this paper proposes a facial expression-based on-road driver emotion recognition network called FERDERnet. This method divides the on-road driver facial expression recognition task into three modules: a face detection module that detects the driver’s face, an augmentation-based resampling module that performs data augmentation and resampling, and an emotion recognition module that adopts a deep convolutional neural network pre-trained on FER and CK+ datasets and then fine-tuned as a backbone for driver emotion recognition. This method adopts five different backbone networks as well as an ensemble method. Furthermore, to evaluate the proposed method, this paper collected an on-road driver facial expression dataset, which contains various road scenarios and the corresponding driver’s facial expression during the driving task. Experiments were performed on the on-road driver facial expression dataset that this paper collected. Based on efficiency and accuracy, the proposed FERDERnet with Xception backbone was effective in identifying on-road driver facial expressions and obtained superior performance compared to the baseline networks and some state-of-the-art networks.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Emotion and Mood Recognition · EEG and Brain-Computer Interfaces · Sleep and Work-Related Fatigue
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Mechanical and Vehicle Engineering, Chongqing University, Chongqing 400044, China
Funding
National Key Research and Development Program of China
References (82)
Show source-provided references
1. World Health Organization (2018). Global Status Report on Road Safety 2018: Summary, World Health Organization. Technical Report.
2. Influence of traffic congestion on driver behavior in post-congestion driving10.1016/j.aap.2020.105508 ↗
3. Braun, M., Chadowitz, R., and Alt, F. (2019, January 2–6). User Experience of Driver State Visualizations: A Look at Demographics and Personalities. Proceedings of the IFIP Conference on Human-Computer Interaction, Paphos, Cyprus.10.1007/978-3-030-29390-1_9 ↗
4. A survey of affective brain computer interfaces: Principles, state-of-the-art, and challenges10.1080/2326263X.2014.912881 ↗
5. Emotions recognition using EEG signals: A survey10.1109/TAFFC.2017.2714671 ↗
6. Nisa’Minhad, K., Ali, S.H.M., Khai, J.O.S., and Ahmad, S.A. (2016, January 14–16). Human emotion classifications for automotive driver using skin conductance response signal. Proceedings of the 2016 International Conference on Advances in Electrical, Electronic and Systems Engineering (ICAEES), Putrajaya, Malaysia.10.1109/ICAEES.2016.7888072 ↗
7. Eesee, A.K. (2019, January 13–14). The suitability of the Galvanic Skin Response (GSR) as a measure of emotions and the possibility of using the scapula as an alternative recording site of GSR. Proceedings of the 2019 2nd International Conference on Electrical, Communication, Computer, Power and Control Engineering (ICECCPCE), Mosul, Iraq.10.1109/ICECCPCE46549.2019.203752 ↗
8. Shen, Z., Cheng, J., Hu, X., and Dong, Q. (2019, January 22–25). Emotion Recognition Based on Multi-View Body Gestures. Proceedings of the 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan.10.1109/ICIP.2019.8803460 ↗
9. Emotion recognition of affective speech based on multiple classifiers using acoustic-prosodic information and semantic labels
10. Speech emotion recognition using Fourier parameters10.1109/TAFFC.2015.2392101 ↗
11. Recognizing action units for facial expression analysis10.1109/34.908962 ↗
12. Li, S., and Deng, W. (2020). Deep facial expression recognition: A survey. IEEE Trans. Affect. Comput.
13. Towards affect-integrated driving behaviour research10.1080/1463922X.2015.1067934 ↗
14. An exploratory study of the relationship between road rage and crash experience in a representative sample of US drivers10.1016/S0001-4575(01)00021-5 ↗
15. Measuring drivers’ frustration in a driving simulator10.1177/154193121005401937 ↗
16. The role of personality traits and driving experience in self-reported risky driving behaviors and accident risk among Chinese drivers10.1016/j.aap.2016.12.009 ↗
17. Driving risk classification methodology for intelligent drive in real traffic event10.1142/S0218001419500149 ↗
18. Gabor feature based classification using the enhanced fisher linear discriminant model for face recognition10.1109/TIP.2002.999679 ↗
19. Facial expression recognition from line-based caricatures10.1109/TSMCA.2003.817057 ↗
20. Distinctive image features from scale-invariant keypoints10.1023/B:VISI.0000029664.99615.94 ↗
21. Dalal, N., and Triggs, B. (2005, January 20–25). Histograms of oriented gradients for human detection. Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), San Diego, CA, USA.
22. A comparative study of human thermal face recognition based on Haar wavelet transform and local binary pattern10.1155/2012/261089 ↗
23. Automated thermal face recognition based on minutiae extraction
24. Seal, A., Ganguly, S., Bhattacharjee, D., Nasipuri, M., and Basu, D.K. (2013). Thermal human face recognition based on haar wavelet transform and series matching technique. Multimedia Processing, Communication and Computing Applications, Springer.10.1007/978-81-322-1143-3_13 ↗
25. Seal, A., Bhattacharjee, D., Nasipuri, M., Gonzalo-Martin, C., and Menasalvas, E. (2014). Histogram of bunched intensity values based thermal face recognition. Rough Sets and Intelligent Systems Paradigms, Springer.10.1007/978-3-319-08729-0_38 ↗
26. A Dynamic-Bayesian Network framework for modeling and evaluating learning from observation10.1016/j.eswa.2014.02.049 ↗
27. Trujillo, L., Olague, G., Hammoud, R., and Hernandez, B. (2005, January 21–23). Automatic feature localization in thermal images for facial expression recognition. Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)-Workshops, San Diego, CA, USA.
28. Littlewort, G., Bartlett, M.S., Fasel, I., Susskind, J., and Movellan, J. (July, January 27). Dynamics of facial expression extracted automatically from video. Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop, Washington, DC, USA.
29. Facial expression recognition in image sequences using geometric deformation features and support vector machines10.1109/TIP.2006.884954 ↗
30. 3D facial expression recognition using SIFT descriptors of automatically detected keypoints10.1007/s00371-011-0611-x ↗
31. Lemaire, P., Ardabilian, M., Chen, L., and Daoudi, M. (2013, January 22–26). Fully automatic 3D facial expression recognition using differential mean curvature maps and histograms of oriented gradients. Proceedings of the 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Shanghai, China.10.1109/FG.2013.6553821 ↗
32. Liu, P., Han, S., Meng, Z., and Tong, Y. (2014, January 23–28). Facial expression recognition via a boosted deep belief network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.233 ↗
33. Yu, Z., and Zhang, C. (2015, January 9–13). Image based static facial expression recognition with multiple deep network learning. Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, Seattle, WA, USA.10.1145/2818346.2830595 ↗
34. Zhao, X., Liang, X., Liu, L., Li, T., Han, Y., Vasconcelos, N., and Yan, S. (2016, January 11–14). Peak-piloted deep network for facial expression recognition. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46475-6_27 ↗
35. Deep neural network architecture: Application for facial expression recognition10.1109/TLA.2020.9099774 ↗
36. Facial Expression Recognition Using Local Gravitational Force Descriptor-Based Deep Convolution Neural Networks
37. Luan, P., Huynh, V., and Tuan Anh, T. (2020, January 10–15). Facial Expression Recognition using Residual Masking Network. Proceedings of the IEEE 25th International Conference on Pattern Recognition, Milan, Italy.
38. Minaee, S., Minaei, M., and Abdolrashidi, A. (2021). Deep-emotion: Facial expression recognition using attentional convolutional network. Sensors, 21.10.3390/s21093046 ↗
39. Shi, J., and Zhu, S. (2021). Learning to amend facial expression representation via de-albino and affinity. arXiv.10.23919/CCC55666.2022.9901738 ↗
40. Schmidhuber, J. (1995). On Learning How to Learn Learning Strategies, Fakultat fur Informatik. Technical Report FKI-198-94.
41. Thrun, S. (1996). Is learning the n-th thing any easier than learning the first?. Advances in Neural Information Processing Systems, MIT Press.
42. Multitask learning10.1023/A:1007379606734 ↗
43. A survey on transfer learning10.1109/TKDE.2009.191 ↗
44. Ng, H.W., Nguyen, V.D., Vonikakis, V., and Winkler, S. (2015, January 9–13). Deep learning for emotion recognition on small datasets using transfer learning. Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, Seattle, WA, USA.10.1145/2818346.2830593 ↗
45. Video-based emotion recognition in the wild using deep transfer learning and score fusion10.1016/j.imavis.2017.01.012 ↗
46. Orozco, D., Lee, C., Arabadzhi, Y., and Gupta, D. (2018). Transfer Learning for Facial Expression Recognition, Florida State Univ.
47. Ravi, A. (2018). Pre-Trained Convolutional Neural Network Features for Facial Expression Recognition. arXiv.
48. Khaireddin, Y., and Chen, Z. (2021). Facial Emotion Recognition: State of the Art Performance on FER2013. arXiv.
49. Drivers’ visual scanning behavior at signalized and unsignalized intersections: A naturalistic driving study in China10.1016/j.jsr.2019.09.012 ↗
50. A deep learning based image enhancement approach for autonomous driving at night10.1016/j.knosys.2020.106617 ↗
51. Estimation of driving style in naturalistic highway traffic using maneuver transition probabilities10.1016/j.trc.2016.11.011 ↗
52. Automatic segmentation and understanding on driving behavioral signals using unsupervised Bayesian methods10.1016/j.ymssp.2020.107589 ↗
53. Risk assessment based collision avoidance decision-making for autonomous vehicles in multi-scenarios10.1016/j.trc.2020.102820 ↗
54. Li, W., Zhang, B., Wang, P., Sun, C., Zeng, G., Tang, Q., Guo, G., and Cao, D. (2021). Visual-Attribute-Based Emotion Regulation of Angry Driving Behaviours. IEEE Intell. Transp. Syst. Mag.10.1109/MITS.2021.3050890 ↗
55. Li, W., Zeng, G., Zhang, J., Xu, Y., Xing, Y., Zhou, R., Guo, G., Shen, Y., Cao, D., and Fei-Yue, W. (2021). CogEmoNet: A Cognitive-Feature-Augmented Driver Emotion Recognition Model for Smart Cockpit. IEEE Trans. Comput. Soc. Syst., 1–12.
56. Gao, H., Yüce, A., and Thiran, J.P. (2014, January 27–30). Detecting emotional stress from facial expressions for driving safety. Proceedings of the 2014 IEEE International Conference on Image Processing (ICIP), Paris, France.10.1109/ICIP.2014.7026203 ↗
57. Li, W., Cui, Y., Ma, Y., Chen, X., Li, G., Zeng, G., Guo, G., and Cao, D. (2021). A Spontaneous Driver Emotion Facial Expression (DEFE) Dataset for Intelligent Vehicles: Emotions Triggered by Video-audio Clips in Driving Scenarios. IEEE Trans. Affect. Comput.
58. Woźniak, M., Siłka, J., and Wieczorek, M. (2021, January 29). Deep learning based crowd counting model for drone assisted systems. Proceedings of the 4th ACM MobiCom Workshop on Drone Assisted Wireless Communications for 5G and Beyond, Virtual Event.10.1145/3477090.3481054 ↗
59. Wieczorek, M., Sika, J., Wozniak, M., Garg, S., and Hassan, M. (2021). Lightweight CNN model for human face detection in risk situations. IEEE Trans. Ind. Inform.10.1109/TII.2021.3129629 ↗
60. Liu, X., Chen, S., Song, L., Woźniak, M., and Liu, S. (2021). Self-attention negative feedback network for real-time image super-resolution. J. King Saud Univ. Comput. Inf. Sci.10.1016/j.jksuci.2021.07.014 ↗
61. Neural network-based face detection10.1109/34.655647 ↗
62. Rowley, H.A., Baluja, S., and Kanade, T. (1998, January 25). Rotation invariant neural network-based face detection. Proceedings of the 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No. 98CB36231), Santa Barbara, CA, USA.
63. Li, S.Z., Zhu, L., Zhang, Z., Blake, A., Zhang, H., and Shum, H. (2002, January 28–31). Statistical learning of multi-view face detection. Proceedings of the European Conference on Computer Vision, Copenhagen, Denmark.10.1007/3-540-47979-1_5 ↗
64. Li, H., Lin, Z., Shen, X., Brandt, J., and Hua, G. (2015, January 7–12). A convolutional neural network cascade for face detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7299170 ↗
65. Huang, L., Yang, Y., Deng, Y., and Yu, Y. (2015). Densebox: Unifying landmark localization with end to end object detection. arXiv.
66. Faceness-net: Face detection through deep facial part responses10.1109/TPAMI.2017.2738644 ↗
67. Joint face detection and alignment using multitask cascaded convolutional networks10.1109/LSP.2016.2603342 ↗
68. The driver monitor system: A means of assessing driver performance
69. Learning from imbalanced data10.1109/TKDE.2008.239 ↗
70. Survey on deep learning with class imbalance10.1186/s40537-019-0192-5 ↗
71. Zhou, B., Cui, Q., Wei, X.S., and Chen, Z.M. (2020, January 14–19). BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00974 ↗
72. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7–12). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗
73. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
74. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016, January 27–30). Rethinking the inception architecture for computer vision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.308 ↗
75. Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A. (2017, January 4–9). Inception-v4, inception-resnet and the impact of residual connections on learning. Proceedings of the AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.10.1609/aaai.v31i1.11231 ↗
76. Chollet, F. (2017, January 21–26). Xception: Deep learning with depthwise separable convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.195 ↗
77. Barsoum, E., Zhang, C., Ferrer, C.C., and Zhang, Z. (2016, January 12–16). Training deep networks for facial expression recognition with crowd-sourced label distribution. Proceedings of the 18th ACM International Conference on Multimodal Interaction, Tokyo, Japan.10.1145/2993148.2993165 ↗
78. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., and Matthews, I. (2010, January 13–18). The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, San Francisco, CA, USA.10.1109/CVPRW.2010.5543262 ↗
79. Ma, Z., Mahmoud, M., Robinson, P., Dias, E., and Skrypchuk, L. (2017, January 3–6). Automatic detection of a driver’s complex mental states. Proceedings of the International Conference on Computational Science and Its Applications, Trieste, Italy.10.1007/978-3-319-62398-6_48 ↗
80. Yan, Y., Lu, K., Xue, J., Gao, P., and Lyu, J. (2019, January 8–12). Feafa: A well-annotated dataset for facial expression analysis and 3d facial animation. Proceedings of the 2019 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), Shanghai, China.10.1109/ICMEW.2019.0-104 ↗
81. Inthanon, P., and Mungsing, S. (2020, January 24–27). Detection of drowsiness from facial images in real-time video media using nvidia Jetson Nano. Proceedings of the 2020 17th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON), Phuket, Thailand.10.1109/ECTI-CON49241.2020.9158235 ↗
82. Xun, D.T.W., Lim, Y.L., and Srigrarom, S. (2021, January 20–22). Drone detection using YOLOv3 with transfer learning on NVIDIA Jetson TX2. Proceedings of the 2021 Second International Symposium on Instrumentation, Control, Artificial Intelligence, and Robotics (ICA-SYMP), Bangkok, Thailand.