Spatiotemporal Feature Enhancement for Lip-Reading: A Survey
Source abstract
Abstract
Lip-reading, a crucial technique to recognize human lip movement patterns for semantic output, has gained increasing attention due to its broad applications in public safety, healthcare, the military, and entertainment. Spatiotemporal feature enhancement techniques have played a significant role in advancing lip-reading research in deep learning. This paper presents a comprehensive review of the latest advancements in methods for lip-reading by exploring key properties of diversity enhancement techniques, involving spatial features, spatiotemporal convolution, attention mechanisms, pulse features, audio-visual features, and so on. Furthermore, according to different network structures, the six spatiotemporal feature enhancement method for lip-reading is offered. And each spatiotemporal feature enhancement method was divided into different subclasses based on the differences in the architecture structure, feature attributes, and application types. Ultimately, this is followed by an in-depth discussion of state-of-the-art spatiotemporal feature enhancement methods, accompanied by an analysis of the challenges and limitations faced, and a discussion of future research directions. From different views, this comprehensive review reveals the limitations and intrinsic disparities among these techniques in different categories for scholars to embark on innovative paths in the advancement of lip-reading.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech and Audio Processing · Face recognition and analysis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Computing, City University of Hong Kong, Hong Kong SAR 999077, China
School of Computer Science and Information Engineering, Hefei University of Technology, Hefei 230009, China
Funding
The National Natural Science Foundation of China · Anhui Province Key R&D Program
References (100+)
Show source-provided references
1. Chen, H., Li, W., Cheng, Z., Liang, X., and Zhang, Q. (2023, January 26–29). TCS-LipNet: Temporal & Channel & Spatial Attention-Based Lip Reading Network. Proceedings of the International Conference on Artificial Neural Networks, Heraklion, Crete, Greece.10.1007/978-3-031-44201-8_34 ↗
2. Ma, P., Wang, Y., Petridis, S., Shen, J., and Pantic, M. (2022, January 23–27). Training Strategies for Improved Lip-Reading. Proceedings of the 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore.10.1109/ICASSP43922.2022.9746706 ↗
3. Analyzing lower half facial gestures for lip reading applications: Survey on vision techniques10.1016/j.cviu.2023.103738 ↗
4. Deep Learning-Based Automated Lip-Reading: A Survey10.1109/ACCESS.2021.3107946 ↗
5. Review on research progress of machine lip reading10.1007/s00371-022-02511-4 ↗
6. Review on Automatic Lip Reading Techniques10.1142/S0218001418560074 ↗
7. Deep Learning for Visual Speech Analysis: A Survey10.1109/TPAMI.2024.3376710 ↗
8. End-to-End Speech Recognition: A Survey10.1109/TASLP.2023.3328283 ↗
9. Santos, C., Cunha, A., and Coelho, P. (2023, January 29–30). A Review on Deep Learning-Based Automatic Lipreading. Proceedings of the International Conference on Wireless Mobile Communication and Healthcare, Vila Real, Portugal.10.1007/978-3-031-32029-3_17 ↗
10. Oghbaie, M., Sabaghi, A., Hashemifard, K., and Akbari, M. (2021). Advances and Challenges in Deep Lip Reading. arXiv.
11. Wang, C. (2019). Multi-grained spatio-temporal modeling for lip-reading. arXiv.
12. Zhang, X., Cheng, F., and Shilin, W. (November, January 27). Spatio-Temporal Fusion Based Convolutional Sequence Learning for Lip Reading. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.10.1109/ICCV.2019.00080 ↗
13. Intelligent event-based lip reading word classification with spiking neural networks using spatio-temporal attention features and triplet loss10.1016/j.ins.2024.120660 ↗
14. Dampfhoffer, M., and Mesquida, T. (2024, January 17–18). Neuromorphic Lip-reading with signed spiking gated recurrent units. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPRW63382.2024.00219 ↗
15. Visual Lip Reading using 3D-DCT and 3D-DWT and LSDA
16. Morade, S.S., and Patnaik, S. (2014, January 21–22). Lip reading using DWT and LSDA. Proceedings of the 2014 IEEE International Advance Computing Conference (IACC), Gurgaon, India.10.1109/IAdCC.2014.6779463 ↗
17. Potamianos, G., Graf, H.P., and Cosatto, E. (1998, January 7). An image transform approach for HMM based automatic lipreading. Proceedings of the 1998 International Conference on Image Processing, Chicago, IL, USA.
18. Yu, K., Jiang, X., and Bunke, H. (1997, January 10–12). Lipreading using Fourier transform over time. Proceedings of the Computer Analysis of Images and Patterns, Kiel, Germany.10.1007/3-540-63460-6_152 ↗
19. Lan, Y., Harvey, R., and Theobald, B.-J. (2012, January 25–30). Insights into machine lip reading. Proceedings of the 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Kyoto, Japan.10.1109/ICASSP.2012.6288999 ↗
20. Boosting a multi-linear classifier with application to visual lip reading10.1016/j.eswa.2010.07.078 ↗
21. Development of Novel Lip-Reading Recognition Algorithm10.1109/ACCESS.2017.2649838 ↗
22. Automatic Lip reading by optical flow analysis10.1002/scj.4690220607 ↗
23. Multi-Modal Speech Recognition Using Optical-Flow Analysis for Lip Images10.1023/B:VLSI.0000015091.47302.07 ↗
24. Ma, X., Yan, L., and Zhong, Q. (2016, January 27–29). Lip Feature Extraction Based on Improved Jumping-Snake Model. Proceedings of the 2016 35th Chinese Control Conference (CCC), Chengdu, China.10.1109/ChiCC.2016.7554449 ↗
25. Automatic Lip Contour Extraction Using Both Pixel-Based and Parametric Models
26. Chen, J., Tiddeman, B., and Zhao, G. (2008, January 1–3). Real-Time Lip Contour Extraction and Tracking Using an Improved Active Contour Model. Proceedings of the 4th International Symposium (ISVC), Las Vegas, NV, USA.10.1007/978-3-540-89646-3_23 ↗
27. Active Appearance Models10.1109/34.927467 ↗
28. Active Shape Models-Their Training and Application10.1006/cviu.1995.1004 ↗
29. Haque, S., Togneri, R., Bennamoun, M., and Sui, C. (2013, January 15–17). A lip extraction algorithm using region-based ACM with automatic contour initialization. Proceedings of the 2013 IEEE Workshop on Applications of Computer Vision (WACV), Clearwater Beach, FL, USA.
30. Petajan, E.D. (1984). Automatic Lipreading to Enhance Speech Recognition (Speech Reading). [Ph.D. Thesis, Electrical Engineering, University of Illinois at Urbana].
31. A tutorial on hidden Markov models and selected applications in speech recognition10.1109/5.18626 ↗
32. Extraction of visual features for lipreading10.1109/34.982900 ↗
33. Geometrical-based lip-reading using template probabilistic multi-dimension dynamic time warping10.1016/j.jvcir.2015.04.013 ↗
34. Bregler, C., Hild, H., Manke, S., and Waibe, A. (1993, January 27–30). Improving connected letter recognition by lipreading. Proceedings of the 1993 IEEE International Conference on Acoustics, Speech, and Signal Processing, Minneapolis, MN, USA.10.1109/ICASSP.1993.319179 ↗
35. Comparison of classifiers for lip reading with CUAVE and TULIPS database10.1016/j.ijleo.2015.08.192 ↗
36. Pei, Y., Kim, T.-K., and Zha, H. (2013, January 1–8). Unsupervised Random Forest Manifold Alignment for Lipreading. Proceedings of the 2013 IEEE International Conference on Computer Vision, Sydney, NSW, Australia.10.1109/ICCV.2013.23 ↗
37. Enhancing Lip Reading: A Deep Learning Approach with CNN and RNN Integration10.52783/jes.1367 ↗
38. A lip reading method based on 3D convolutional vision transformer10.1109/ACCESS.2022.3193231 ↗
39. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
40. Huang, G., Liu, Z., Maaten LV, D., and Weinberger, K.Q. (2017, January 21–26). Densely Connected Convolutional Networks. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.243 ↗
41. Tan, M., and Le, Q.V. (2019, January 9–15). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA.
42. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv.
43. Zhang, X., Zhou, X., Lin, M., and Sun, J. (2018, January 18–23). ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00716 ↗
44. Schmidt, R.M. (2019). Recurrent Neural Networks (RNNs): A gentle Introduction and Overview. arXiv.
45. Long Short-Term memory10.1162/neco.1997.9.8.1735 ↗
46. Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. (2014). Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. arXiv.
47. Graves, A., Fernández, S., Gomez, F., and Schmidhuber, J. (2006, January 25–29). Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks. Proceedings of the 23rd International Conference on Machine Learning (ICML), Pittsburgh, PA, USA.10.1145/1143844.1143891 ↗
48. Lin, J., Gan, C., and Han, S. (November, January 27). Tsm: Temporal shift module for efficient video understanding. Proceedings of the IEEE/CVF international conference on computer vision, Seoul, Republic of Korea.
49. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention Is All You Need. arXiv.
50. Ni, R., Jiang, H., Zhou, L., and Lu, Y. (2024, January 27–30). Lip Recognition Based on Bi-GRU with Multi-Head Self-Attention. Proceedings of the International Conference on Artificial Intelligence Applications and Innovations, Corfu, Greece.10.1007/978-3-031-63219-8_8 ↗
51. Lip reading of words with lip segmentation and deep learning10.1007/s11042-022-13321-0 ↗
52. Turkish lip-reading using Bi-LSTM and deep learning models
53. Ma, P., Wang, Y., Shen, J., Petridis, S., and Pantic, M. (2021, January 3–8). Lip-reading with Densely Connected Temporal Convolutional Networks. Proceedings of the 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA.10.1109/WACV48630.2021.00290 ↗
54. End-to-End Sentence-Level Multi-View Lipreading Architecture with Spatial Attention Module Integrated Multiple CNNs and Cascaded Local Self-Attention-CTC10.3390/s22093597 ↗
55. Liu, Q., Xing, D., Tang, H., Ma, D., and Pan, G. (2021, January 19–26). Event-based action recognition using motion information and spiking neural networks. Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, Montreal-themed Virtual Reality, Montreal, QC, Canada.10.24963/ijcai.2021/240 ↗
56. Spiking neural network methodology for modelling, classification and understanding of EEG spatio-temporal data measuring cognitive processes10.1016/j.ins.2014.06.028 ↗
57. Luo, M., Yang, S., Chen, X., Liu, Z., and Shan, S. (2020). Synchronous bidirectional learning for multilingual lip reading. arXiv.
58. Zhang, X., Zhang, C., Sui, J., Sheng, C., Deng, W., and Liu, L. (2022, January 18–22). Boosting lip reading with a multi-view fusion network. Proceedings of the 2022 IEEE International Conference on Multimedia and Expo (ICME), Taipei, Taiwan.10.1109/ICME52920.2022.9859810 ↗
59. Exploring complementarity of global and local information for effective lip reading10.1117/1.JEI.32.2.023001 ↗
60. Collaborative Viseme Subword and End-to-end Modeling for Word-level Lip Reading10.1109/TMM.2024.3390148 ↗
61. Chen, H., Du, J., Hu, Y., Dai, L.-R., Lee, C.-H., and Yin, B.-C. (2020). Lip-reading with hierarchical pyramidal convolution and self-attention. arXiv.10.21437/Interspeech.2021-723 ↗
62. Stafylakis, T., and Tzimiropoulos, G. (2017). Combining residual networks with LSTMs for lipreading. arXiv.10.21437/Interspeech.2017-85 ↗
63. Mudaliar, N.K., Hegde, K., Ramesh, A., and Patil, V. (2020, January 10–12). Visual speech recognition: A deep learning approach. Proceedings of the 2020 5th International Conference on Communication and Electronics Systems (ICCES), Coimbatore, India.10.1109/ICCES48766.2020.9137926 ↗
64. Lip reading sentences using deep learning with only visual cues10.1109/ACCESS.2020.3040906 ↗
65. Wu, Z., Chen, W., Xu, J., and Wang, Y. (2021, January 10–13). Lip Reading Based on 3D Face Modeling and Spatial Transformation Learning. Proceedings of the 2021 7th International Conference on Computer and Communications (ICCC), Chengdu, China.10.1109/ICCC54389.2021.9674610 ↗
66. Jiang, J., Zhao, Z., Yang, Y., and Tian, W. (July, January 30). GSLip: A Global Lip-Reading Framework with Solid Dilated Convolutions. Proceedings of the 2024 International Joint Conference on Neural Networks (IJCNN), Yokohama, Japan.10.1109/IJCNN60899.2024.10651423 ↗
67. Arakane, T., and Saitoh, T. (2023). Efficient DNN Model for Word Lip-Reading. Algorithms, 16.10.3390/a16060269 ↗
68. Xiao, J., Yang, S., Zhang, Y., Shan, S., and Chen, X. (2020, January 16–20). Deformation flow based two-stream network for lip reading. Proceedings of the 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), Buenos Aires, Argentina.10.1109/FG47880.2020.00132 ↗
69. Developing phoneme-based lip-reading sentences system for silent speech recognition10.1049/cit2.12131 ↗
70. Zeng, Q., Du, J., and Wang, Z. (2021, January 14–17). HMM-based Lip Reading with Stingy Residual 3D Convolution. Proceedings of the 2021 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), Tokyo, Japan.
71. Sun, B., Xie, D., Luo, D., and Yin, X. (November, January 31). A Lipreading Model Based on Fine-Grained Global Synergy of Lip Movement. Proceedings of the 2022 IEEE 34th International Conference on Tools with Artificial Intelligence (ICTAI), Macao, China.10.1109/ICTAI56018.2022.00130 ↗
72. Dual-flow Spatio-temporal Separation Network for Lip Reading10.1088/1742-6596/2400/1/012028 ↗
73. Xu, K., Li, D., Cassimatis, N., and Wang, X. (2018, January 15–19). LCANet: End-to-end lipreading with cascaded attention-CTC. Proceedings of the 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), Xi’an, China.10.1109/FG.2018.00088 ↗
74. An optimal 3D convolutional neural network based lipreading method10.1049/ipr2.12337 ↗
75. Bi, C., Zhang, D., Yang, L., and Chen, P. (2019, January 2–4). An lipreading Modle with DenseNet and E3D-LSTM. Proceedings of the 2019 6th International Conference on Systems and Informatics (ICSAI), Shanghai, China.10.1109/ICSAI48974.2019.9010432 ↗
76. Jeon, S., and Kim, M.S. (2022). End-to-end lip-reading open cloud-based speech architecture. Sensors, 22.10.3390/s22082938 ↗
77. Wen, J., and Lu, Y. (2019). Automatic lip reading system based on a fusion lightweight neural network with Raspberry Pi. Appl. Sci., 9.10.3390/app9245432 ↗
78. LipSyncNet: A Novel Deep Learning Approach for Visual Speech Recognition in Audio-Challenged Situations10.1109/ACCESS.2024.3436931 ↗
79. Zhang, G., and Lu, Y. (2023). Research on a Lip Reading Algorithm Based on Efficient-GhostNet. Electronics, 12.10.3390/electronics12051151 ↗
80. Fu, Y., Lu, Y., and Ni, R. (2023). Chinese lip-reading research based on ShuffleNet and CBAM. Appl. Sci., 13.10.3390/app13021106 ↗
81. AI-based visual speech recognition towards realistic avatars and lip-reading applications in the metaverse10.1016/j.asoc.2024.111906 ↗
82. Sumanth, S., Jyosthana, K., Reddy, J.K., and Geetha, G. (2022, January 19–20). Computer Vision Lip Reading(CV). Proceedings of the 2022 International Conference on Advancements in Smart, Secure and Intelligent Computing (ASSIC), Bhubaneswar, India.10.1109/ASSIC55218.2022.10088386 ↗
83. Koumparoulis, A., and Potamianos, G. (2022, January 23–27). Accurate and resource-efficient lipreading with efficientnetv2 and transformers. Proceedings of the 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore.10.1109/ICASSP43922.2022.9747729 ↗
84. HLR-net: A hybrid lip-reading model based on deep convolutional neural networks
85. Tian, W., Zhang, H., Peng, C., and Zhao, Z.-Q. (2022, January 23–27). Lipreading model based on whole-part collaborative learning. Proceedings of the 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore.10.1109/ICASSP43922.2022.9747052 ↗
86. New Feature Extraction Approaches Based on Spatial Points for Visual-Only Lip-Reading10.18280/ts.390229 ↗
87. Peng, C., Li, J., Chai, J., Zhao, Z., Zhang, H., and Tian, W. (2022, January 6–9). Lip Reading Using Deformable 3D Convolution and Channel-Temporal Attention. Proceedings of the International Conference on Artificial Neural Networks, Bristol, UK.10.1007/978-3-031-15937-4_59 ↗
88. Tsourounis, D., Kastaniotis, D., and Fotopoulos, S. (2021). Lip reading by alternating between spatiotemporal and spatial convolutions. J. Imaging, 7.10.3390/jimaging7050091 ↗
89. Jeon, S., Elsharkawy, A., and Kim, M.S. (2022). Lipreading Architecture Based on Multiple Convolutional Neural Networks for Sentence-Level Visual Speech Recognition. Sensors, 22.10.3390/s22010072 ↗
90. Huang, J., Teng, L., Xiao, Y., Zhu, A., and Liu, X. (2023, January 20–23). Lip Reading Using Temporal Adaptive Module. Proceedings of the International Conference on Neural Information Processing, Changsha, China.10.1007/978-981-99-8141-0_26 ↗
91. MALip: Modal Amplification Lipreading based on reconstructed audio features10.1016/j.image.2023.117002 ↗
92. Hao, M., Mamut, M., Yadikar, N., Aysa, A., and Ubul, K. (2021, January 6–11). How to use time information effectively? Combining with time shift module for lipreading. Proceedings of the 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada.10.1109/ICASSP39728.2021.9414659 ↗
93. Li, H., Mamut, M., Yadikar, N., Zhu, Y., and Ubul, K. (2021, January 10–12). Channel Enhanced Temporal-Shift Module for Efficient Lipreading. Proceedings of the Chinese Conference on Biometric Recognition, Shanghai, China.10.1007/978-3-030-86608-2_52 ↗
94. Wiriyathammabhum, P. (2020, January 23–27). SpotFast Networks with Memory Augmented Lateral Transformers for Lipreading. Proceedings of the International Conference on Neural Information Processing, Bangkok, Thailand.10.1007/978-3-030-63820-7_63 ↗
95. Assael, Y.M., Shillingford, B., Whiteson, S., and Freitas, N.D. (2016). Lipnet: End-to-end sentence-level lipreading. arXiv.
96. 3D convolutional neural networks for human action recognition10.1109/TPAMI.2012.59 ↗
97. Zhang, P., Wang, D., Lu, H., Wang, H., and Ruan, X. (2017, January 22–29). Amulet: Aggregating multi-level convolutional features for salient object detection. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.31 ↗
98. Using lip reading recognition to predict daily Mandarin conversation10.1109/ACCESS.2022.3175867 ↗
99. A Chinese Lip-Reading System Based on Convolutional Block Attention Module10.1155/2021/6250879 ↗
100. Pan, X., Chen, P., Gong, Y., Zhou, H., Wang, X., and Lin, Z. (2022). Leveraging unimodal self-supervised learning for multimodal audio-visual speech recognition. arXiv.10.18653/v1/2022.acl-long.308 ↗