Deep Learning-Based Approach for Arabic Visual Speech Recognition
Source abstract
Abstract
Lip-reading technologies are rapidly progressing following the breakthrough of deep learning. It plays a vital role in its many applications, such as: human-machine communication practices or security applications. In this paper, we propose to develop an effective lip-reading recognition model for Arabic visual speech recognition by implementing deep learning algorithms. The Arabic visual datasets that have been collected contains 2400 records of Arabic digits and 960 records of Arabic phrases from 24 native speakers. The primary purpose is to provide a high-performance model in terms of enhancing the preprocessing phase. Firstly, we extract keyframes from our dataset. Secondly, we produce a Concatenated Frame Images (CFIs) that represent the utterance sequence in one single image. Finally, the VGG-19 is employed for visual features extraction in our proposed model. We have examined different keyframes: 10, 15, and 20 for comparing two types of approaches in the proposed model: (1) the VGG-19 base model and (2) VGG-19 base model with batch normalization. The results show that the second approach achieves greater accuracy: 94% for digit recognition, 97% for phrase recognition, and 93% for digits and phrases recognition in the test dataset. Therefore, our proposed model is superior to models based on CFIs input.
Source: openalex_inverted_index · Quality: baseline
Research context
OpenAlex topics: Speech and Audio Processing · Indoor and Outdoor Localization Technologies · Blind Source Separation Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (45)
Show source-provided references
1. A review on methods and classifiers in lip reading
2. Result based analysis of various lip tracking systems
3. Lip reading techniques: A survey
4. A survey of visual lip reading and lip-password verification
5. A survey-based study on lip segmentation techniques for lip reading applications
6. Word spotting in silent lip videos
7. Artificial intelligence: A survey on lip-reading techniques
8. Lip reading using committee networks with two different types of concatenated frame images10.1109/ACCESS.2019.2927166 ↗
9. An Arabic figures recognition model based on automatic learning of lip movement10.21608/auej.2017.19304 ↗
10. A silent password recognition framework based on lip analysis10.1109/ACCESS.2020.2982359 ↗
11. An Arabic visual dataset for visual speech recognition10.1016/j.procs.2019.12.122 ↗
12. P. C. Rabaneda, “Lip reading visual passwords for user authentication,” B.A. Thesis, Federico Santa María Technical University, Spain, 2018.
13. An audio-visual corpus for speech perception and automatic speech recognition10.1121/1.2229005 ↗
14. Ouluvs2: A multi-view audiovisual database for non-rigid mouth motion analysis
15. Speaker independent audio-visual continuous speech recognition
16. Robust AAM-based audio-visual speech recognition against face direction changes
17. Visual speech recognition using active shape models and hidden Markov models
18. Movellan · Advances in Neural Information Processing Systems · 1995 · 7 · 851
19. Automatic lipreading with principal component analysis
20. A segment-based audio-visual speech recognizer: Data collection, development, and initial experiments
21. Lip reading using optical flow and support vector machines
22. Geometrical-based lip-reading using template probabilistic multi-dimension dynamic time warping10.1016/j.jvcir.2015.04.013 ↗
23. CUAVE: A new audio-visual database for multimodal human-computer interface research
24. Visual speech recognition using optical flow and hidden Markov model10.1007/s11277-018-5930-z ↗
25. Arabic lip-reading system: A combination of Hypercolumn neural
26. Automatic lip-reading system based on deep convolutional neural network and attention-based long short-term memory10.3390/app9081599 ↗
27. Understanding pictograph with facial features: end-to-end sentence-level lip reading of Chinese10.1609/aaai.v33i01.33019211 ↗
28. A lip reading model using CNN with batch normalization
29. A new visual speech recognition approach for RGB-D cameras
30. Automatic lip reading system based on a fusion lightweight neural network with Raspberry Pi10.3390/app9245432 ↗
31. M. Faisal and S. Manzoor, “Deep learning for lip reading using audiovisual information for Urdu language,” arXiv preprint arXiv: 1802.0552, 2018.
32. End-to-end visual speech recognition with LSTMs
33. A. Gutierrez and Z. Robert, “Lip reading word classification,” Technical Report, Stanford University, CS231n project report, 2017.
34. Y. M. Assael, B. Shillingford, S. Whiteson and D. N. Freitas, “LipNet: End-to-end sentence-level lipreading,” arXiv preprint arXiv: 1611.01599, 2016.
35. T. Stafylakis and G. Tzimiropoulos, “Combining residual networks with LSTMs for lipreading,” arXiv preprint arXiv: 1703.04105, 2017.10.21437/Interspeech.2017-85 ↗
36. Out of time: Automated lip sync in the wild
37. A. Garg, J. Noyola and S. Bagadia, “Lip reading using CNN and LSTM,” Technical report, Stanford University, CS231n project report, 2016.
38. Lip reading with Hahn convolutional neural networks10.1016/j.imavis.2019.04.010 ↗
39. A bidirectional interactive system of sign language and visual speech based on portable devices
40. A lip reading model with DenseNet and E3D-LSTM
41. Concatenated frame image-based CNN for visual speech recognition
42. The sum-absolute-difference motion estimation accelerator
43. Imagenet: A large-scale hierarchical image database
44. Batch normalization: Accelerating deep network training by reducing internal covariate shift
45. Age estimation using specific domain transfer learning