Driver Behavior Analysis via Two-Stream Deep Convolutional Neural Network
Source abstract
Abstract
According to the World Health Organization global status report on road safety, traffic accidents are the eighth leading cause of death in the world, and nearly one-fifth of the traffic accidents were cause by driver distractions. Inspired by the famous two-stream convolutional neural network (CNN) model, we propose a driver behavior analysis system using one spatial stream ConvNet to extract the spatial features and one temporal stream ConvNet to capture the driver’s motion information. Instead of using three-dimensional (3D) ConvNet, which would suffer from large parameters and the lack of a pre-trained model, two-dimensional (2D) ConvNet is used to construct the spatial and temporal ConvNet streams, and they were pre-trained by the large-scale ImageNet. In addition, in order to integrate different modalities, the feature-level fusion methodology was applied, and a fusion network was designed to integrate the spatial and temporal features for further classification. Moreover, a self-compiled dataset of 10 actions in the vehicle was established. According to the experimental results, the proposed system can increase the accuracy rate by nearly 30% compared to the two-stream CNN model with a score-level fusion.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human Pose and Action Recognition · Autonomous Vehicle Technology and Safety · Video Surveillance and Tracking Methods
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science and Information Engineering, National Kaohsiung University of Science and Technology, Kaohsiung city 8078, Taiwan
Funding
ministry of science and technology
References (54)
Show source-provided references
1. World Health Organization (WHO) (2019, February 13). Global Status Report, Available online: https://www.who.int/violence_injury_prevention/road_safety_status/2018/en/.
2. Driver Distraction Identification with an Ensemble of Convolutional Neural Networks10.1155/2019/4125865 ↗
3. Driving safety10.1518/155723405783703037 ↗
4. (2019, February 13). Traffic Safety Facts, Available online: https://www.nhtsa.gov/risky-driving/distracted-driving.
5. (2019, February 13). Distracted Driving, Available online: https://www.cdc.gov/motorvehiclesafety/distracted_driving/.
6. National Highway Traffic Safety Administration (2019, February 13). Traffic Safety Facts, Available online: https://www.nhtsa.gov/risky-driving/distracted-driving.
7. Zhang, X., Zheng, N., Wang, F., and He, Y. (2011, January 10–12). Visual recognition of driver hand-held cell phone use based on hidden CRF. Proceedings of the 2011 IEEE International Conference on Vehicular Electronics and Safety, Beijing, China.10.1109/ICVES.2011.5983823 ↗
8. Berri, R., Silva, A.G., Parpinelli, R.S., Girardi, E., and Arthur, R. (2014, January 5–8). A Pattern Recognition System for Detecting Use of Mobile Phones While Driving. Proceedings of the International Conference on Computer Vision Theory and Applications, Lisbon, Portugal.
9. Craye, C., and Karray, F. (2015). Driver distraction detection and recognition using RGB-D sensor. arXiv.
10. Seshadri, K., Juefei-Xu, F., Pal, D.K., Savvides, M., and Thor, C.P. (2015, January 7–12). Driver cell phone usage detection on Strategic Highway Research Program (SHRP2) face view videos. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Boston, MA, USA.10.1109/CVPRW.2015.7301397 ↗
11. Das, N., Ohn-Bar, E., and Trivedi, M.M. (2015, January 15–18). On Performance Evaluation of Driver Hand Detection Algorithms: Challenges, Dataset, and Metrics. Proceedings of the 2015 IEEE 18th International Conference on Intelligent Transportation Systems, Gran Canaria, Spain.10.1109/ITSC.2015.473 ↗
12. Pdf ImageNet classification with deep convolutional neural networks10.1145/3065386 ↗
13. Hoang, T., Do, T.-T., Le Tan, D.-K., and Cheung, N.-M. (2017, January 25–28). Selective Deep Convolutional Features for Image Retrieval. Proceedings of the 2017 ACM on Web Science Conference—WebSci ’17, Troy, NY, USA.10.1145/3123266.3123417 ↗
14. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
15. Le, T.H.N., Zheng, Y., Zhu, C., Luu, K., and Savvides, M. (July, January 26). Multiple Scale Faster-RCNN Approach to Driver’s Cell-Phone Usage and Hands on Steering Wheel Detection. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Las Vegas, NV, USA.10.1109/CVPRW.2016.13 ↗
16. Faster R-CNN: Towards real-time object detection with region proposal networks10.1109/TPAMI.2016.2577031 ↗
17. Yuen, K., Martin, S., and Trivedi, M.M. (2016, January 1–4). Looking at faces in a vehicle: A deep CNN based approach and evaluation. Proceedings of the 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC), Rio de Janeiro, Brazil.10.1109/ITSC.2016.7795622 ↗
18. Streiffer, C., Raghavendra, R., Benson, T., and Srivatsa, M. (2017, January 11–15). Darnet: A deep learning solution for distracted driving detection. Proceedings of the ACM/IFIP/USENIX Middleware Conference, Las Vegas, NV, USA.10.1145/3154448.3154452 ↗
19. Majdi, M.S., Ram, S., Gill, J.T., and Rodriguez, J.J. (2018, January 8–10). Drive-Net: Convolutional Network for Driver Distraction Detection. Proceedings of the 2018 IEEE Southwest Symposium on Image Analysis and Interpretation (SSIAI), Las Vegas, NV, USA.10.1109/SSIAI.2018.8470309 ↗
20. Real-time detection of distracted driving based on deep learning10.1049/iet-its.2018.5172 ↗
21. Simonyan, K., and Zisserman, A. (2014, January 14–16). Very deep convolutional networks for large-scale image recognition. Proceedings of the International Conference on Learning Representations, Banff, AB, Canada.
22. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7–12). Going deeper with convolutions. Proceedings of the Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗
23. Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A.A. (2016, January 12–17). Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning. Proceedings of the AAAI, Phoenix, AZ, USA.10.1609/aaai.v31i1.11231 ↗
24. Abouelnaga, Y., Eraqi, H.M., and Moustafa, M.N. (2018, January 8). Real-time distracted driver posture classification. Proceedings of the Workshop on Machine Learning for Intelligent Transportation Systems, Montréal, QC, Canada.
25. Baheti, B., Gajre, S., and Talbar, S. (2018, January 18–22). Detection of Distracted Driver Using Convolutional Neural Network. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Salt Lake City, UT, USA.10.1109/CVPRW.2018.00150 ↗
26. Masood, S., Rai, A., Aggarwal, A., Doja, M., and Ahmad, M. (2018). Detecting distraction of drivers using Convolutional Neural Network. Pattern Recognit. Lett.
27. Simonyan, K., and Zisserman, A. (2014, January 8–13). Two-stream convolutional networks for action recognition in videos. Proceedings of the NIPS, Montreal, QC, Canada.
28. Zach, C., Pock, T., and Bischof, H. (2007). A Duality Based Approach for Realtime TV-L 1 Optical Flow. Lecture Notes in Computer Science, Springer.10.1007/978-3-540-74936-3_22 ↗
29. Handcrafted vs. non-handcrafted features for computer vision classification10.1016/j.patcog.2017.05.025 ↗
30. Laptev, I., and Lindeberg, T. (2003, January 14–17). Velocity adaptation of space-time interest points. Proceedings of the Ninth IEEE International Conference on Computer Vision, Nice, France.10.1109/ICCV.2003.1238378 ↗
31. Dollár, P., Rabaud, V., Cottrell, G., and Belongie, S. (2006, January 15–16). Behavior Recognition via Sparse Spatio-Temporal Features. Proceedings of the 2005 IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance, Beijing, China.
32. Sadanand, S., and Corso, J.J. (2012, January 16–21). Action bank: A high-level representation of activity in video. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.10.1109/CVPR.2012.6247806 ↗
33. Wang, H., and Schmid, C. (2013, January 1–8). Action Recognition with Improved Trajectories. Proceedings of the 2013 IEEE International Conference on Computer Vision, Sydney, Australia.10.1109/ICCV.2013.441 ↗
34. Kong, Y., and Fu, Y. (2018). Human Action Recognition and Prediction: A Survey. arXiv.
35. Martin, S., Ohn-Bar, E., Tawari, A., Trivedi, M.M., and Martin, S. (2014, January 8–11). Understanding head and hand activities and coordination in naturalistic driving videos. Proceedings of the 2014 IEEE Intelligent Vehicles Symposium, Dearborn, MI, USA.10.1109/IVS.2014.6856610 ↗
36. Driver hand activity analysis in naturalistic driving studies: Challenges, algorithms, and experimental studies10.1117/1.JEI.22.4.041119 ↗
37. Recognition of driving postures by contourlet transform and random forests10.1049/iet-its.2011.0116 ↗
38. (2019, February 13). StateFarm’s Distracted Driver Detection Dataset. Available online: https://www.kaggle.com/c/state-farm-distracted-driver-detection.
39. Carreira, J., and Zisserman, A. (2017, January 21–26). Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.502 ↗
40. Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., Fei-Fei, L., Shetty, S., and Leung, T. (2014, January 24–27). Large-Scale Video Classification with Convolutional Neural Networks. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Washington, DC, USA.10.1109/CVPR.2014.223 ↗
41. Donahue, J., Hendricks, L.A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., and Saenko, K. (2015, January 7–12). Long-term recurrent convolutional networks for visual recognition and description. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/CVPR.2015.7298878 ↗
42. Ng, J.Y.-H., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., and Toderici, G. (2015, January 7–12). Beyond short snippets: Deep networks for video classification. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.
43. 3D Convolutional Neural Networks for Human Action Recognition10.1109/TPAMI.2012.59 ↗
44. Tran, D., Bourdev, L., Fergus, R., Torresani, L., and Paluri, M. (2015, January 7–13). Learning Spatiotemporal Features with 3D Convolutional Networks. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.510 ↗
45. A Bi-LSTM mention hypergraph model with encoding schema for mention extraction10.1016/j.engappai.2019.06.005 ↗
46. BILU-NEMH: A BILU neural-encoded mention hypergraph for mention extraction10.1016/j.ins.2019.04.059 ↗
47. Driving posture recognition by convolutional neural networks10.1049/iet-cvi.2015.0175 ↗
48. Driver behavior recognition using recurrent neural network in multiple depth cameras environment10.2352/ISSN.2470-1173.2019.15.AVM-056 ↗
49. Gebert, P., Roitberg, A., Haurilet, M., and Stiefelhagen, R. (2019, January 9–12). End-to-end Prediction of Driver Intention using 3D Convolutional Neural Networks. Proceedings of the 2019 IEEE Intelligent Vehicles Symposium (IV), Paris, France.10.1109/IVS.2019.8814249 ↗
50. Xu, L., and Fujimura, K. (2014, January 17–19). Real-Time Driver Activity Recognition with Random Forests. Proceedings of the 6th international conference on Multimodal interfaces—ICMI ’04, Seattle, WA, USA.10.1145/2667317.2667333 ↗
51. Martin, M., Popp, J., Anneken, M., Voit, M., and Stiefelhagen, R. (2018, January 26–30). Body Pose and Context Information for Driver Secondary Task Detection. Proceedings of the 2018 IEEE Intelligent Vehicles Symposium (IV), Changshu, China.10.1109/IVS.2018.8500523 ↗
52. Martin, M., Roiterg, A., Haurilet, M., Horne, M., Reib, S., Voit, M., and Stiefelhagen, R. (2019, January 23–25). Drive & Act: A multi-modal dataset for fine-grained driver behavior recognition in autonomous vehicles. Proceedings of the International Conference on Computer Vision, Thessaloniki, Greece.10.1109/ICCV.2019.00289 ↗
53. Soomro, K., Zamir, A.R., and Shah, M. (2012). UCF101: A Dataset of 101 Human Actions Classes From Videos in the Wild. arXiv.
54. Soleymani, S., Dabouei, A., Kazemi, H., Dawson, J., and Nasrabadi, N.M. (2018, January 20–24). Multi-Level Feature Abstraction from Convolutional Neural Networks for Multimodal Biometric Identification. Proceedings of the 2018 24th International Conference on Pattern Recognition (ICPR), Beijing, China.10.1109/ICPR.2018.8545061 ↗