Real-Time Hand Gesture Spotting and Recognition Using RGB-D Camera and 3D Convolutional Neural Network
Source abstract
Abstract
Using hand gestures is a natural method of interaction between humans and computers. We use gestures to express meaning and thoughts in our everyday conversations. Gesture-based interfaces are used in many applications in a variety of fields, such as smartphones, televisions (TVs), video gaming, and so on. With advancements in technology, hand gesture recognition is becoming an increasingly promising and attractive technique in human–computer interaction. In this paper, we propose a novel method for fingertip detection and hand gesture recognition in real-time using an RGB-D camera and a 3D convolution neural network (3DCNN). This system can accurately and robustly extract fingertip locations and recognize gestures in real-time. We demonstrate the accurateness and robustness of the interface by evaluating hand gesture recognition across a variety of gestures. In addition, we develop a tool to manipulate computer programs to show the possibility of using hand gesture recognition. The experimental results showed that our system has a high level of accuracy of hand gesture recognition. This is thus considered to be a good approach to a gesture-based interface for human–computer interaction by hand in the future.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Hand Gesture Recognition Systems · Human Pose and Action Recognition · Gait Recognition and Analysis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Electronics and Computer Engineering, Chonnam National University, 77 Yongbong-ro, Gwangju 500-757, Korea
Funding
Basic Science Research Program · National Research Foundation of Korea
References (53)
Show source-provided references
1. Starner, T., and Pentland, A. (1997). Real-time american sign language recognition from video using hidden markov models. Motion-Based Recognition, Springer.10.1007/978-94-015-8935-2_10 ↗
2. Malima, A.K., Özgür, E., and Çetin, M. (2006, January 17–19). A Fast Algorithm for Vision-Based Hand Gesture Recognition for Robot Control. Proceedings of the 2006 IEEE 14th Signal Processing and Communications Applications, Antalya, Turkey.
3. Tsai, T.-H., Huang, C.-C., and Zhang, K.-L. (2015, January 6–8). Embedded Virtual Mouse System by Using Hand Gesture Recognition. Proceedings of the 2015 IEEE International Conference on Consumer Electronics-Taiwan (ICCE-TW), Taipei, Taiwan.10.1109/ICCE-TW.2015.7216939 ↗
4. Mouse Operations using Finger Tracking
5. Hand-Mouse Interface Using Virtual Monitor Concept for Natural Interaction10.1109/ACCESS.2017.2768405 ↗
6. Virtual Mouse Using Hand Gesture
7. Le, P.D., and Nguyen, V.H. (2014). Remote mouse control using fingertip tracking technique. AETA 2013: Recent Advances in Electrical Engineering and Related Sciences, Springer.10.1007/978-3-642-41968-3_47 ↗
8. Reza, M.N., Hossain, M.S., and Ahmad, M. (2015, January 21–23). Real Time Mouse Cursor Control Based on Bare Finger Movement Using Webcam to Improve HCI. Proceedings of the 2015 International Conference on Electrical Engineering and Information Communication Technology (ICEEICT), Dhaka, Bangladesh.10.1109/ICEEICT.2015.7307391 ↗
9. Gallo, L., Placitelli, A.P., and Ciampi, M. (2011, January 27–30). Controller-Free Exploration of Medical Image Data: Experiencing the Kinect. Proceedings of the 2011 24th International Symposium on Computer-Based Medical Systems (CBMS), Bristol, UK.10.1109/CBMS.2011.5999138 ↗
10. Dong, G., Yan, Y., and Xie, M. (2018, January 18–21). Vision-Based Hand Gesture Recognition for Human-Vehicle Interaction. Proceedings of the International Conference on Control, Automation and Computer Vision, Singapore.
11. Zhang, X., Chen, X., Wang, W., Yang, J., Lantz, V., and Wang, K. (2009, January 25). Hand Gesture Recognition and Virtual Game Control Based on 3D Accelerometer and EMG Sensors. Proceedings of the 14th International Conference on Intelligent User Interfaces, Sanibel Island, FL, USA.10.1145/1502650.1502708 ↗
12. Deng, L. (2014). A tutorial survey of architectures, algorithms, and applications for deep learning. APSIPA Trans. Signal Inf. Process., 3.10.1017/atsip.2013.9 ↗
13. Wang, Y., and Neff, M. (2013, January 26–27). Data-driven Glove Calibration for Hand Motion Capture. Proceedings of the 12th ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Anaheim, CA, USA.10.1145/2485895.2485901 ↗
14. Hand Gesture Recognition using Microcontroller Flex Sensor
15. Parvini, F., McLeod, D., Shahabi, C., Navai, B., Zali, B., and Ghandeharizadeh, S. (2009, January 19–24). An Approach to Glove-Based Gesture Recognition. Proceedings of the International Conference on Human-Computer Interaction, San Diego, CA, USA.10.1007/978-3-642-02577-8_26 ↗
16. Allevard, T., Benoit, E., and Foulloy, L. (2003, January 22–28). Fuzzy Glove for Gesture Recognition. Proceedings of the XVII IMEKO World Congress, Dubrovnik, Croatia.
17. Multi-Point Gesture Recognition Using LED Gloves for Interactive HCI
18. Ganzeboom, M. (2018, December 12). How Hand Gestures Are Recognized Using a Dataglove. Available online: https://pdfs.semanticscholar.org/bd6b/40dca3813367272c917e6d28a45a2f053004.pdf?_ga=2.35948259.294260165.1579427347-803309327.1579427347.
19. Hand gesture recognition application for physically disabled people
20. Real time hand gesture recognition system for dynamic applications10.5121/iju.2012.3103 ↗
21. Murugeswari, M., and Veluchamy, S. (2014, January 8–10). Hand Gesture Recognition System for Real-Time Application. Proceedings of the 2014 International Conference on Advanced Communication Control and Computing Technologies (ICACCCT), Ramanathapuram, India.10.1109/ICACCCT.2014.7019293 ↗
22. Hand gesture recognition for human computer interaction10.1016/j.procs.2017.09.092 ↗
23. Chen, Z., Kim, J.-T., Liang, J., Zhang, J., and Yuan, Y.-B. (2014). Real-time hand gesture recognition using finger segmentation. Sci. World J., 2014.10.1155/2014/267872 ↗
24. Xu, P. (2017). A Real-time Hand Gesture Recognition and Human-Computer Interaction System. arXiv.
25. Neto, P., Pereira, D., Pires, J.N., and Moreira, A.P. (2013, January 6–10). Real-Time and Continuous Hand Gesture Spotting: An Approach Based on Artificial Neural Networks. Proceedings of the 2013 IEEE International Conference on Robotics and Automation (ICRA), Karlsruhe, Germany.10.1109/ICRA.2013.6630573 ↗
26. Banerjee, A., Ghosh, A., Bharadwaj, K., and Saikia, H. (2014). Mouse control using a web camera based on colour detection. arXiv.10.14445/22312803/IJCTT-V9P104 ↗
27. Ge, L., Ren, Z., Li, Y., Xue, Z., Wang, Y., Cai, J., and Yuan, J. (2019, January 15–20). 3D Hand Shape and Pose Estimation from a Single RGB Image. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.01109 ↗
28. Li, Y., Xue, Z., Wang, Y., Ge, L., Ren, Z., and Rodriguez, J. (2019, December 25). End-to-End 3D Hand Pose Estimation from Stereo Cameras. Available online: https://bmvc2019.org/wp-content/uploads/papers/0219-paper.pdf.
29. Fossati, A., Gall, J., Grabner, H., Ren, X., and Konolige, K. (2012). Consumer Depth Cameras for Computer Vision: Research Topics and Applications, Springer Science & Business Media.10.1007/978-1-4471-4640-7 ↗
30. Sharp, T., Keskin, C., Robertson, D., Taylor, J., Shotton, J., Kim, D., Rhemann, C., Leichter, I., Vinnikov, A., and Wei, Y. (2015, January 18–23). Accurate, Robust, and Flexible Real-Time Hand Tracking. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, Seoul, Korea.10.1145/2702123.2702179 ↗
31. Khamis, S., Taylor, J., Shotton, J., Keskin, C., Izadi, S., and Fitzgibbon, A. (2015, January 7–12). Learning an Efficient Model of Hand Shape Variation from Depth Images. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298869 ↗
32. Oikonomidis, I., Kyriazis, N., and Argyros, A.A. (September, January 29). Efficient Model-Based 3D Tracking of Hand Articulations Using Kinect. Proceedings of the BmVC, Dundee, UK.
33. K-Cosine Corner Detection
34. Bakar, M.Z.A., Samad, R., Pebrianti, D., Mustafa, M., and Abdullah, N.R.H. (2015, January 25–27). Finger Application Using K-Curvature Method and Kinect Sensor in Real-Time. Proceedings of the 2015 International Symposium on Technology Management and Emerging Technologies (ISTMET), Langkawai Island, Malaysia.
35. Nguyen, H.D., Kim, Y.C., Kim, S.H., and Na, I.S. (2017, January 29–31). A Method for Fingertips Detection Using RGB-D Image and Convolution Neural Network. Proceedings of the 2017 13th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD), Guilin, China.10.1109/FSKD.2017.8393373 ↗
36. Robust rgb-d hand tracking using deep learning priors
37. Molchanov, P., Gupta, S., Kim, K., and Kautz, J. (2015, January 7–12). Hand Gesture Recognition with 3D Convolutional Neural Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Boston, MA, USA.10.1109/CVPRW.2015.7301342 ↗
38. Hoang, N.N., Lee, G.-S., Kim, S.-H., and Yang, H.-J. (2018, January 28–30). A Real-Time Multimodal Hand Gesture Recognition via 3D Convolutional Neural Network and Key Frame Extraction. Proceedings of the 2018 International Conference on Machine Learning and Machine Intelligence, Hanoi, Vietnam.10.1145/3278312.3278314 ↗
39. (2018, December 10). The 20BN-JESTER Dataset. Available online: https://20bn.com/datasets/jester/v1.
40. Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3–6). Imagenet Classification with Deep Convolutional Neural Networks. Proceedings of the Advances in Neural Information Processing Systems, Lake Tahoe, NV, USA.
41. Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv.
42. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7–12). Going Deeper with Convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗
43. He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
44. Baccouche, M., Mamalet, F., Wolf, C., Garcia, C., and Baskurt, A. (2011, January 16). Sequential Deep Learning for Human Action Recognition. Proceedings of the International Workshop on Human Behavior Understanding, Amsterdam, The Netherlands.10.1007/978-3-642-25446-8_4 ↗
45. Schuldt, C., Laptev, I., and Caputo, B. (2004, January 26). Recognizing Human Actions: A Local SVM Approach. Proceedings of the 17th International Conference on Pattern Recognition, Cambridge, UK.10.1109/ICPR.2004.1334462 ↗
46. Tran, D., Bourdev, L., Fergus, R., Torresani, L., and Paluri, M. (2015, January 7–13). Learning Spatiotemporal Features with 3D Convolutional Networks. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.510 ↗
47. Shou, Z., Wang, D., and Chang, S.-F. (2016, January 27–30). Temporal Action Localization in Untrimmed Videos via Multi-Stage Cnns. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.119 ↗
48. Molchanov, P., Yang, X., Gupta, S., Kim, K., Tyree, S., and Kautz, J. (2016, January 27–30). Online Detection and Classification of Dynamic Hand Gestures with Recurrent 3D Convolutional Neural Network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.456 ↗
49. Pan, Y., Mei, T., Yao, T., Li, H., and Rui, Y. (2016, January 27–30). Jointly Modeling Embedding and Translation to Bridge Video and Language. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.497 ↗
50. Dietterich, T.G. (2000, January 21–23). Ensemble Methods in Machine Learning. Proceedings of the International Workshop on Multiple Classifier Systems, Cagliari, Italy.10.1007/3-540-45014-9_1 ↗
51. Gonzalez, R.W.R. (2008). Digital Image Processing, Prentice Hall. [3rd ed.].
52. Contour line tracing algorithm for digital topographic maps
53. Cassels, J.W.S. (1957). An Introduction to Diophantine Approximation, Cambridge University Press.