Multi-Layer Progressive Face Alignment by Integrating Global Match and Local Refinement
Source abstract
Abstract
Robust and accurate face alignment remains a challenging task, especially when local noises, illumination variations and partial occlusions exist in images. The existing local search and global match methods often misalign due to local optima without global constraints or limited local representation of global appearance. To solve these problems, we propose a new multi-layer progressive face alignment method that combines global matches for a whole face with local refinement for a given region, where the errors caused by local optima are restricted by globally-matched appearance, and the local misalignments in the global method are avoided by supplementing the representation of local details. Our method consists of the following processes: Firstly, an input image is encoded as a multi-mode Local Binary Pattern (LBP) image to regress the face shape parameters. Secondly, the local multi-mode histogram of oriented gradient (HOG) features is applied to update each landmark position. Thirdly, the above two alignment shapes are weighted as the final result. The contributions of this paper are as follows: (1) Shape initialization by applying an affine transformation to the mean shape. (2) Face representation by integrating multi-mode information in a whole face or a face region. (3) Face alignment by combining handcrafted features with convolutional neural networks (CNN). Extensive experiments on public datasets show that our method demonstrates improved performance in real environments in comparison to some state-of-the-art methods which apply single scale features or single CNN networks. Applying our method to the challenging HELEN dataset, the samples with fewer than 8 mean errors reach 81.1%.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Face recognition and analysis · Face and Expression Recognition · Video Surveillance and Tracking Methods
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Electronic Information and Electrical Engineering, Dalian University of Technology, Dalian 116024, China
School of Information Science and Technology, Dalian Maritime University, Dalian 116026, China
Funding
Natural Science Foundation of China · the Password Theory Project of the 13th Five-Year Plan National Cryptography Development Fund
References (61)
Show source-provided references
1. Tzimiropoulos, G., and Pantic, M. (2013, January 1–8). Optimization Problems for Fast AAM Fitting in-the-Wild. Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia.10.1109/ICCV.2013.79 ↗
2. Navarathna, R., Sridharan, S., and Lucey, S. (2011, January 6–13). Fourier Active Appearance Models. Proceedings of the IEEE International Conference on Computer Vision, Barcelona, Spain.10.1109/ICCV.2011.6126461 ↗
3. Sun, Y., Wang, X., and Tang, X. (2013, January 25–27). Deep Convolutional Network Cascade for Facial Point Detection. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Portland, OR, USA.10.1109/CVPR.2013.446 ↗
4. Trigeorgis, G., Snape, P., and Nicolaou, M.A. (2016, January 27–30). Mnemonic Descent Method: A Recurrent Process Applied for End-to-End Face Alignment. Proceedings of the IEEE International Conference on computer vision and pattern recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.453 ↗
5. Zhang, J., Shan, S., and Kan, M. (2014, January 6–12). Coarse-to-Fine Auto-Encoder Networks (CFAN) for Real-Time Face Alignment. Proceedings of the European Conference on Computer Vision, Zurich, Switzerland.10.1007/978-3-319-10605-2_1 ↗
6. Sun, P., Min, J.K., and Xiong, G. (2015, January 8–10). Globally Tuned Cascade Pose Regression via Back Propagation with Application in 2D Face Pose Estimation and Heart Segmentation in 3D CT Images. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.
7. Discriminative Deep Face Shape Model for Facial Point Detection10.1007/s11263-014-0775-8 ↗
8. Integrated Deep Model for Face Detection and Landmark Localization From “In The Wild” Images10.1109/ACCESS.2018.2882227 ↗
9. HyperFace: A Deep Multi-Task Learning Framework for Face Detection, Landmark Localization, Pose Estimation, and Gender Recognition
10. Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks10.1109/LSP.2016.2603342 ↗
11. Zhu, S., Li, C., and Loy, C.C. (2016, January 27–30). Unconstrained Face Alignment via Cascaded Compositional Learning. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.371 ↗
12. Kumar, A., and Chellappa, R. (2018, January 18–22). Disentangling 3D Pose in a Dendritic CNN for Unconstrained 2D Face Alignment. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00052 ↗
13. Feng, Y., Wu, F., and Shao, X. (2018, January 8–14). Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network. Proceedings of the European Conference on Computer Vision, Munich, Germany.10.1007/978-3-030-01264-9_33 ↗
14. Pose-Invariant Face Alignment via CNN-Based Dense 3D Model Fitting10.1007/s11263-017-1012-z ↗
15. Fan, L., Tao, X., and Tong, S. (2018, January 17–20). Kernel PCA and Nonlinear ASM. Proceedings of the IEEE International Conference on Control Systems, GangWon, Korea.
16. Zhao, X., Kim, T., and Luo, W. (2014, January 23–28). Unified Face Analysis by Iterative Multi-output Random Forests. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.228 ↗
17. Asthana, A., Zafeiriou, S., and Cheng, S. (2013, January 25–27). Robust Discriminative Response Map Fitting with Constrained Local Models. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Portland, OR, USA.10.1109/CVPR.2013.442 ↗
18. Zadeh, A., Baltrusaitis, T., and Morency, L. (2016, January 27–30). Deep Constrained Local Models for Facial Landmark Detection. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPRW.2017.256 ↗
19. Deformable Model Fitting by Regularized Landmark Mean-Shift10.1007/s11263-010-0380-4 ↗
20. Tzimiropoulos, G., and Pantic, M. (2014, January 23–28). Gauss-Newton Deformable Part Models for Face Alignment In-the-Wild. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.239 ↗
21. Lee, D., Park, H., and Yoo, C.D. (2015, January 8–10). Face alignment using cascade Gaussian process regression trees. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7299048 ↗
22. Zhu, S., Li, C., and Loy, C.C. (2015, January 8–10). Face alignment by coarse-to-fine shape searching. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.
23. Xiong, X., and LaTorre, F.D. (2015, January 8–10). Global supervised descent method. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298882 ↗
24. Xiong, X., and LaTorre, F.D. (2013, January 25–27). Supervised Descent Method and Its Applications to Face Alignment. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Portland, OR, USA.10.1109/CVPR.2013.75 ↗
25. Dollar, P., Welinder, P., and Perona, P. (2010, January 13–18). Cascaded pose regression. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, San Francisco, CA, USA.10.1109/CVPR.2010.5540094 ↗
26. Face Alignment by Explicit Shape Regression10.1007/s11263-013-0667-3 ↗
27. Ren, S., Cao, X., and Wei, Y. (2014, January 23–28). Face Alignment at 3000 FPS via Regressing Local Binary Features. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.218 ↗
28. Kazemi, V., and Sullivan, J. (2014, January 23–28). One Millisecond Face Alignment with an Ensemble of Regression Trees. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.241 ↗
29. Burgosartizzu, X.P., Perona, P., and Dollar, P. (2013, January 1–8). Robust Face Landmark Estimation under Occlusion. Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia.10.1109/ICCV.2013.191 ↗
30. Tzimiropoulos, G. (2015, January 8–10). Project-Out Cascaded Regression with an application to face alignment. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298989 ↗
31. Alqunaieer, F.S., and Alkanhal, M.I. (2016, January 24–26). Learning from Partially Occluded Faces. Proceedings of the IEEE International Conference on Pattern Recognition Applications and Methods, Rome, Italy.10.5220/0005665605340539 ↗
32. Wu, Y., Wang, Z., and Ji, Q. (2013, January 25–27). Facial Feature Tracking Under Varying Facial Expressions and Face Poses Based on Restricted Boltzmann Machines. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Portland, OR, USA.10.1109/CVPR.2013.443 ↗
33. Baltrusaitis, T., Robinson, P., and Morency, L. (2012, January 16–21). 3D Constrained Local Model for rigid and non-rigid facial tracking. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.10.1109/CVPR.2012.6247980 ↗
34. Yu, X., Huang, J., and Zhang, S. (2013, January 1–8). Pose-Free Facial Landmark Fitting via Optimized Part Mixtures and Cascaded Deformable Shape Model. Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia.10.1109/ICCV.2013.244 ↗
35. Ghiasi, G., and Fowlkes, C.C. (2014, January 23–28). Occlusion Coherence: Localizing Occluded Faces with a Hierarchical Deformable Part Model. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.306 ↗
36. Alabortimedina, J., and Zafeiriou, S. (2015, January 8–10). Unifying holistic and Parts-Based Deformable Model fitting. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298991 ↗
37. Zhou, E., Fan, H., and Cao, Z. (2013, January 1–8). Extensive Facial Landmark Localization with Coarse-to-Fine Convolutional Network Cascade. Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia.10.1109/ICCVW.2013.58 ↗
38. Approaching human level facial landmark localization by deep learning10.1016/j.imavis.2015.11.004 ↗
39. Face alignment under occlusion based on local and global feature regression10.1007/s11042-016-3470-7 ↗
40. Xu, X., and Kakadiaris, I.A. (June, January 30). Joint Head Pose Estimation and Face Alignment Framework Using Global and Local CNN Features. Proceedings of the IEEE International Conference on Automatic Face Gesture Recognition, Washington, DC, USA.
41. Robust face alignment and tracking by combining local search and global fitting10.1016/j.imavis.2016.04.012 ↗
42. Face alignment using a deep neural network with local feature learning and recurrent regression
43. Huang, Z., Zhou, E., and Cao, Z. (2015, January 8–10). Coarse-to-fine Face Alignment with Multi-Scale Local Patch Regression. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.
44. Duffner, S., and Garcia, C. (2005, January 7–8). A hierarchical approach for precise facial feature detection. Proceedings of the Compression et Représentation des Signaux Audiovisuels, Rennes, France.
45. Lv, J., Shao, X., and Xing, J. (2017, January 21–26). A Deep Regression Architecture with Two-Stage Re-initialization for High Performance Facial Landmark Detection. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.393 ↗
46. Liang, P., Zadeh, A., and Morency, L. (2018, January 16–20). Multimodal Local-Global Ranking Fusion for Emotion Recognition. Proceedings of the IEEE International Conference on Multimodal Interfaces, Boulder, CO, USA.10.1145/3242969.3243019 ↗
47. End-to-End Multimodal Emotion Recognition Using Deep Neural Networks10.1109/JSTSP.2017.2764438 ↗
48. Liu, W., Zheng, W., and Lu, B. (2016, January 16–21). Emotion Recognition Using Multimodal Deep Learning. Proceedings of the IEEE International Conference on Neural Information Processing, Kyoto, Japan.10.1007/978-3-319-46672-9_58 ↗
49. Felipe, B., Luis, M., and Pedro, T. (October, January 28). A Novel Multimodal Emotion Recognition Approach for Affective Human Robot. Proceedings of the IEEE International Conference on Intelligent Robots and Systems, Hamburg, Germany.
50. Ranganathan, H., Chakraborty, S., and Panchanathan, S. (2016, January 7–10). Multimodal emotion recognition using deep learning architectures. Proceedings of the IEEE Winter Conference on Applications of Computer Vision, Lake Placid, NY, USA.10.1109/WACV.2016.7477679 ↗
51. Yan, J., Lei, Z., and Li, S. (2013, January 1–8). Learn to combine multiple hypotheses for accurate face alignment. Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia.10.1109/ICCVW.2013.126 ↗
52. Springenberg, J., Dosovitskiy, A., and Brox, T. (2014, January 14–16). Striving for Simplicity: The All Convolutional Net. Proceedings of the IEEE International Conference on Learning Representations, Banff, AB, Canada.
53. Howard, G., Zhu, M., and Chen, B. (2017, January 21–26). MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.
54. Tzimiropoulos, G., Zafeiriou, S., and Pantic, M. (2011, January 6–13). Robust and efficient parametric face alignment. Proceedings of the IEEE International Conference on Computer Vision, Barcelona, Spain.10.1109/ICCV.2011.6126452 ↗
55. Multi-Scale Colour Completed Local Binary Patterns for Scene and Event Sport Image Categorisation
56. Ioffe, S., and Szegedy, C. (2015, January 6–11). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. Proceedings of the IEEE International Conference on Machine Learning, Lille, France.
57. He, K., Girshick, B., and Dollar, P. (2018, January 18–22). Rethinking ImageNet Pre-training. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/ICCV.2019.00502 ↗
58. Xin, W., Gongde, G., and Hui, W. (2015, January 24–27). A Multiscale Method for HOG-Based Face Recognition. Proceedings of the IEEE International Conference on Intelligent Robotics and Applications, Portsmouth, UK.
59. HOG-assisted deep feature learning for pedestrian gender recognition10.1016/j.jfranklin.2017.09.003 ↗
60. Liu, L., Jiani, H., and Shuo, Z. (2014, January 1–5). Extended Supervised Descent Method for Robust Face Alignment. Proceedings of the IEEE Asian Conference on Computer Vision, Singapore.
61. Eyes closeness detection from still images with multi-scale histograms of principal oriented gradients10.1016/j.patcog.2014.03.024 ↗