Diverse Pose Lip-Reading Framework
Source abstract
Abstract
Lip-reading is a technique to understand speech by observing a speaker’s lips movement. It has numerous applications; for example, it is helpful for hearing impaired persons and understanding the speech in noisy environments. Most of the previous works of lips-reading focused on frontal and near frontal face lip-reading and some of them targeted multiple poses in high quality videos. However, their results are not satisfactory on low quality videos containing multiple poses. In this research work, a lip-reading framework is proposed for improving the recognition rate in low quality videos. In this work, a Multiple Pose (MP) dataset of low quality videos containing multiple extreme poses is built. The proposed framework decomposes the input video into frames and enhances them by applying the Contrast Limited Adaptive Histogram Equalization (CLAHE) method. Next, faces are detected from enhanced frames and frontalized the multiple poses using the face frontalization Generative Adversarial Network (FF-GAN). After face frontalization, the mouth region is extracted. The extracted mouth region in the whole video and its respective sentences are then provided to the ResNet during the training process. The proposed framework achieved a sentence prediction accuracy of 90% on a testing dataset containing 100 silent low-quality videos with multiple poses that are better as compared to state-of-the-art methods.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech and Audio Processing · Face recognition and analysis · Infant Health and Development
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science & Information Technology, Hazara University Mansehra, Mansehra 21120, Pakistan
Department of Computer Science & IT, Neelum Campus, The University of Azad Jammu and Kashmir, Athmuqam 13230, Pakistan
Department of Computer Science & IT, King Abdullah Campus, The University of Azad Jammu and Kashmir, Muzaffarabad 13100, Pakistan
Department of Telecommunication, Hazara University Mansehra, Mansehra 21120, Pakistan
Department of Computer Science, National Textile University, Faisalabad 37610, Pakistan
Information and Computing Technology (ICT) Division, College of Science and Engineering (CSE), Hamad Bin Khalifa University (HBKU), Doha 34110, Qatar
References (32)
Show source-provided references
1. Hearing lips and seeing voices10.1038/264746a0 ↗
2. Scanlon, P., and Reilly, R. (2001, January 3–5). Feature analysis for automatic speechreading. Proceedings of the 2001 IEEE Fourth Workshop on Multimedia Signal Processing (Cat. No. 01TH8564), Cannes, France.
3. Matthews, I., Potamianos, G., Neti, C., and Luettin, J. (2001, January 22–25). A comparison of model and transform-based visual features for audio-visual LVCSR. Proceedings of the IEEE International Conference on Multimedia and Expo, 2001 (ICME 2001), Tokyo, Japan.10.1109/ICME.2001.1237849 ↗
4. Aleksic, P.S., and Katsaggelos, A.K. (2004, January 17–21). Comparison of low-and high-level visual features for audio-visual continuous automatic speech recognition. Proceedings of the 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing, Montreal, QC, Canada.
5. Shaikh, A.A., Kumar, D.K., Yau, W.C., Azemin, M.C., and Gubbi, J. (2010, January 16–18). Lip reading using optical flow and support vector machines. Proceedings of the 2010 3rd International Congress on Image and Signal Processing, Yantai, China.10.1109/CISP.2010.5646264 ↗
6. Lip reading of hearing impaired persons using HMM10.1016/j.eswa.2010.09.119 ↗
7. Bear, H.L., Harvey, R.W., and Lan, Y. (2017). Finding phonemes: Improving machine lip-reading. arXiv.10.1109/ICASSP.2016.7472029 ↗
8. Learning to lip read words by watching videos10.1016/j.cviu.2018.02.001 ↗
9. Experimenting with lipreading for large vocabulary continuous speech recognition10.1007/s12193-018-0266-2 ↗
10. Thangthai, K., Bear, H.L., and Harvey, R. (2018). Comparing phonemes and visemes with DNN-based lipreading. arXiv.
11. Bear, H.L. (2016). Decoding Visemes: Improving Machine Lip-Reading. [Ph.D. Thesis, University of East Anglia].10.1109/ICASSP.2016.7472029 ↗
12. Bear, H.L., Cox, S.J., and Harvey, R.W. (2017). Speaker-independent machine lip-reading with speaker-dependent viseme classifiers. arXiv.
13. Chung, J.S., Senior, A., Vinyals, O., and Zisserman, A. (2017, January 21–26). Lip reading sentences in the wild. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.367 ↗
14. Chung, J.S., and Zisserman, A. (2016, January 20–24). Lip reading in the wild. Proceedings of the Asian Conference on Computer Vision, Taipei, Taiwan.
15. Visual speech recognition
16. Almajai, I., Cox, S., Harvey, R., and Lan, Y. (2016, January 20–25). Improved speaker independent lip reading using speaker adaptive training and deep neural networks. Proceedings of the 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Shanghai, China.10.1109/ICASSP.2016.7472172 ↗
17. Phoneme-to-viseme mappings: The good, the bad, and the ugly10.1016/j.specom.2017.07.001 ↗
18. Assael, Y.M., Shillingford, B., Whiteson, S., and De Freitas, N. (2016). Lipnet: End-to-end sentence-level lipreading. arXiv.
19. Koller, O., Ney, H., and Bowden, R. (2015, January 7–13). Deep learning of mouth shapes for sign language. Proceedings of the IEEE International Conference on Computer Vision Workshops, Santiago, Chile.10.1109/ICCVW.2015.69 ↗
20. Lu, Y., and Li, H. (2019). Automatic Lip-Reading System Based on Deep Convolutional Neural Network and Attention-Based Long Short-Term Memory. Appl. Sci., 9.10.3390/app9081599 ↗
21. Lan, Y., Theobald, B.J., and Harvey, R. (2012, January 9–13). View independent computer lip-reading. Proceedings of the 2012 IEEE International Conference on Multimedia and Expo, Melbourne, VIC, Australia.10.1109/ICME.2012.192 ↗
22. Lucey, P.J., Potamianos, G., and Sridharan, S. (2007, January 27–31). A unified approach to multi-pose audio-visual ASR. Proceedings of the INTERSPEECH 2007, 8th Annual Conference of the International Speech Communication Association, Antwerp, Belgium.10.21437/Interspeech.2007-281 ↗
23. The impact of the Lombard effect on audio and visual speech recognition systems10.1016/j.specom.2018.04.006 ↗
24. Audio-visual speech recognition using deep learning10.1007/s10489-014-0629-7 ↗
25. Trustworthy Method for Person Identification in IIoT Environments by Means of Facial Dynamics10.1109/TII.2020.2977774 ↗
26. Chung, J.S., and Zisserman, A. (2017, January 4–7). Lip reading in profile. Proceedings of the British Machine Vision Conference (BMVC), London, UK.
27. Yadav, G., Maheshwari, S., and Agarwal, A. (2014, January 24–27). Contrast limited adaptive histogram equalization based enhancement for real time video system. Proceedings of the 2014 International Conference on Advances in Computing, Communications and Informatics (ICACCI), Delhi, India.10.1109/ICACCI.2014.6968381 ↗
28. An adaptive histogram equalization algorithm on the image gray level mapping10.1016/j.phpro.2012.03.132 ↗
29. Kazemi, V., and Sullivan, J. (2014, January 23–28). One millisecond face alignment with an ensemble of regression trees. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.241 ↗
30. Yin, X., Yu, X., Sohn, K., Liu, X., and Chandraker, M. (2017, January 22–29). Towards large-pose face frontalization in the wild. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.430 ↗
31. Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. (2007, January 4–9). Automatic differentiation in PyTorch. Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.
32. An audio-visual corpus for speech perception and automatic speech recognition10.1121/1.2229005 ↗