Automatic Lip-Reading System Based on Deep Convolutional Neural Network and Attention-Based Long Short-Term Memory
Source abstract
Abstract
With the improvement of computer performance, virtual reality (VR) as a new way of visual operation and interaction method gives the automatic lip-reading technology based on visual features broad development prospects. In an immersive VR environment, the user’s state can be successfully captured through lip movements, thereby analyzing the user’s real-time thinking. Due to complex image processing, hard-to-train classifiers and long-term recognition processes, the traditional lip-reading recognition system is difficult to meet the requirements of practical applications. In this paper, the convolutional neural network (CNN) used to image feature extraction is combined with a recurrent neural network (RNN) based on attention mechanism for automatic lip-reading recognition. Our proposed method for automatic lip-reading recognition can be divided into three steps. Firstly, we extract keyframes from our own established independent database (English pronunciation of numbers from zero to nine by three males and three females). Then, we use the Visual Geometry Group (VGG) network to extract the lip image features. It is found that the image feature extraction results are fault-tolerant and effective. Finally, we compare two lip-reading models: (1) a fusion model with an attention mechanism and (2) a fusion model of two networks. The results show that the accuracy of the proposed model is 88.2% in the test dataset and 84.9% for the contrastive model. Therefore, our proposed method is superior to the traditional lip-reading recognition methods and the general neural networks.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech and Audio Processing · Face recognition and analysis · Facial Nerve Paralysis Treatment and Research
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information Science and Technology, North China University of Technology, Beijing 100144, China
Funding
National Natural Science Foundation of China · Beijing Municipal Natural Science Foundation · Science and Technology Development Program of Beijing Municipal Education Commission · Great Wall Scholar Reserved Talent Program of North China University of Technology
References (25)
Show source-provided references
1. Multimodal human–computer interaction: A survey10.1016/j.cviu.2006.10.019 ↗
2. Immersive virtual environment technology as a basic research tool in psychology10.3758/BF03200735 ↗
3. Hassanat, A.B. (arXiv, 2014). Visual passwords using automatic lip reading, arXiv.
4. Thanda, A., and Venkatesan, S.M. (arXiv, 2017). Multi-task learning of deep neural networks for audio visual automatic speech recognition, arXiv.
5. Multiple cameras audio visual speech recognition using active appearance model visual features in car environment10.1007/s10772-016-9332-x ↗
6. Scanlon, P., and Reilly, R. (2001, January 3–5). Feature analysis for automatic speechreading. Proceedings of the 2001 IEEE Fourth Workshop on Multimedia Signal Processing (Cat. No. 01TH8564), Cannes, France.
7. Matthews, I., Potamianos, G., Neti, C., and Luettin, J. (2001, January 22–25). A comparison of model and transform-based visual features for audio-visual LVCSR. Proceedings of the IEEE International Conference on Multimedia and Expo, ICME 2001, Tokyo, Japan.10.1109/ICME.2001.1237849 ↗
8. Aleksic, P.S., and Katsaggelos, A.K. (2004, January 17–21). Comparison of low-and high-level visual features for audio-visual continuous automatic speech recognition. Proceedings of the 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing, Montreal, QC, Canada.
9. Chitu, A.G., Driel, K., and Rothkrantz, L.J. (2010, January 6–10). Automatic lip reading in the Dutch language using active appearance models on high speed recordings. Proceedings of the International Conference on Text, Speech and Dialogue, Brno, Czech Republic.10.1007/978-3-642-15760-8_33 ↗
10. Luettin, J., Thacker, N.A., and Beet, S.W. (1996, January 9). Visual speech recognition using active shape models and hidden Markov models. Proceedings of the 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings, Atlanta, GA, USA.
11. Shaikh, A.A., Kumar, D.K., Yau, W.C., Azemin, M.C., and Gubbi, J. (2010, January 16–18). Lip reading using optical flow and support vector machines. Proceedings of the 2010 3rd International Congress on Image and Signal Processing, Yantai, China.10.1109/CISP.2010.5646264 ↗
12. Lip reading of hearing impaired persons using HMM10.1016/j.eswa.2010.09.119 ↗
13. Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. (2015, January 7–12). Show and tell: A neural image caption generator. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298935 ↗
14. Gers, F.A., Schmidhuber, J., and Cummins, F. (1999, January 7–10). Learning to forget: Continual prediction with LSTM. Proceedings of the 9th International Conference on Artificial Neural Networks: ICANN’99, Edinburgh, UK.10.1049/cp:19991218 ↗
15. Wang, Y., Huang, M., and Zhao, L. (2016, January 1–5). Attention-based LSTM for aspect-level sentiment classification. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.10.18653/v1/D16-1058 ↗
16. Simonyan, K., and Zisserman, A. (arXiv, 2014). Very deep convolutional networks for large-scale image recognition, arXiv.
17. Extraction of visual features for lipreading10.1109/34.982900 ↗
18. Fan, X., Zhang, F., Wang, H., and Lu, X. (2012, January 23–25). The system of face detection based on OpenCV. Proceedings of the 2012 24th Chinese Control and Decision Conference (CCDC), Taiyuan, China.
19. Dlib-ml: A machine learning toolkit
20. Martins, A., and Astudillo, R. (2016, January 19–24). From softmax to sparsemax: A sparse model of attention and multi-label classification. Proceedings of the International Conference on Machine Learning, New York, NY, USA.
21. Long short-term memory10.1162/neco.1997.9.8.1735 ↗
22. Graves, A. (2012). Long short-term memory. Supervised Sequence Labelling with Recurrent Neural Networks, Springer.10.1007/978-3-642-24797-2 ↗
23. Describing multimedia content using attention-based encoder-decoder networks10.1109/TMM.2015.2477044 ↗
24. Zhang, Y., Pezeshki, M., Brakel, P., Zhang, S., Bengio, C.L.Y., and Courville, A. (arXiv, 2017). Towards end-to-end speech recognition with deep convolutional neural networks, arXiv.10.21437/Interspeech.2016-1446 ↗
25. Graves, A. (arXiv, 2013). Generating sequences with recurrent neural networks, arXiv.