Animal Pose Estimation Based on 3D Priors
Source abstract
Abstract
Animal pose estimation is very useful in analyzing animal behavior, monitoring animal health and moving trajectories, etc. However, occlusions, complex backgrounds, and unconstrained illumination conditions in wild-animal images often lead to large errors in pose estimation, i.e., the detected key points have large deviations from their true positions in 2D images. In this paper, we propose a method to improve animal pose estimation accuracy by exploiting 3D prior constraints. Firstly, we learn the 3D animal pose dictionary, in which each atom provides prior knowledge about 3D animal poses. Secondly, given the initially estimated 2D animal pose in the image, we represent its latent 3D pose with the learned dictionary. Finally, the representation coefficients are optimized to minimize the difference between the initially estimated 2D pose and the 2D projection of the latent 3D pose. Furthermore, we construct 2D and 3D animal pose datasets, which are used to evaluate the algorithm’s performance and learn the 3D pose dictionary, respectively. Our experimental results demonstrate that the proposed method makes good use of the 3D pose knowledge and can effectively improve 2D animal pose estimation.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human Pose and Action Recognition · Advanced Vision and Imaging · Video Surveillance and Tracking Methods
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
National Key Laboratory of Fundamental Science on Synthetic Vision, Sichuan University, Chengdu 610065, China
College of Information Science and Engineering, Guilin University of Technology, Guilin 541006, China
College of Computer Science, Sichuan University, Chengdu 610065, China
Funding
National Natural Science Foundation of China · Science and Technology Department of Tibet · 2021 Guidance Special Provincial Supporting-2021 Pilot Program · CAAI-Huawei MindSpore Open Found
References (58)
Show source-provided references
1. DeepLabCut: Markerless Pose Estimation of User-Defined Body Parts with Deep Learning10.1038/s41593-018-0209-y ↗
2. DeepPoseKit, A Software Toolkit for Fast and Robust Animal Pose Estimation Using Deep Learning10.7554/eLife.47994 ↗
3. Deep Learning Tools for the Measurement of Animal Behavior in Neuroscience10.1016/j.conb.2019.10.008 ↗
4. A Primer on Motion Capture with Deep Learning: Principles, Pitfalls, and Perspectives10.1016/j.neuron.2020.09.017 ↗
5. Biggs, B., Roddick, T., Fitzgibbon, A., and Cipolla, R. (2018, January 2–6). Creatures Great and SMAL: Recovering the Shape and Motion of Animals From Video. Proceedings of the Asian Conference on Computer Vision, Perth, Australia.10.1007/978-3-030-20873-8_1 ↗
6. Zuffi, S., Kanazawa, A., and Black, M.J. (2018, January 18–22). Lions and Tigers and Bears: Capturing Non-Rigid, 3D, Articulated Shape From Images. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00416 ↗
7. Zuffi, S., Kanazawa, A., Berger-Wolf, T., and Black, M.J. (November, January 27). Three-D Safari: Learning to Estimate Zebra Pose, Shape, and Texture From Images “In the Wild”. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Republic of Korea.
8. Shih, L.Y., Chen, B.Y., and Wu, J.L. (2009, January 13–16). Video-Based Motion Capturing for Skeleton-Based 3D Models. Proceedings of the Pacific-Rim Symposium on Image and Video Technology, Tokyo, Japan.10.1007/978-3-540-92957-4_65 ↗
9. A Novel Template-Based Automatic Rigging Algorithm for Articulated-Character Animation10.1002/cav.1429 ↗
10. Quantifying Behavior to Understand the Brain10.1038/s41593-020-00734-z ↗
11. Seok, S., Wang, A., Chuah, M.Y., Otten, D., Lang, J., and Kim, S. (2013, January 6–10). Design Principles for Highly Efficient Quadrupeds and Implementation on the MIT Cheetah Robot. Proceedings of the IEEE International Conference on Robotics and Automation, Karlsruhe, Germany.10.1109/ICRA.2013.6631038 ↗
12. Zhao, D., Song, S., Su, J., Jiang, Z., and Zhang, J. (2020, January 13–16). Learning Bionic Motions by Imitating Animals. Proceedings of the IEEE International Conference on Mechatronics and Automation, Beijing, China.10.1109/ICMA49215.2020.9233839 ↗
13. Human3. 6m: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments10.1109/TPAMI.2013.248 ↗
14. Zuffi, S., Kanazawa, A., Jacobs, D.W., and Black, M.J. (2017, January 21–26). 3D Menagerie: Modeling the 3D Shape and Pose of Animals. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.586 ↗
15. Mu, J., Qiu, W., Hager, G.D., and Yuille, A.L. (2020, January 14–19). Learning From Synthetic Animals. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Virtual.10.1109/CVPR42600.2020.01240 ↗
16. Li, C., and Lee, G.H. (2021, January 19–25). From Synthetic to Real: Unsupervised Domain Adaptation for Animal Pose Estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Virtual.10.1109/CVPR46437.2021.00153 ↗
17. Cao, J., Tang, H., Fang, H.S., Shen, X., Lu, C., and Tai, Y.W. (November, January 27). Cross-Domain Adaptation for Animal Pose Estimation. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Republic of Korea.
18. Chen, W., Wang, H., Li, Y., Su, H., Wang, Z., Tu, C., Lischinski, D., Cohen-Or, D., and Chen, B. In Proceedings of the Fourth International Conference on 3D Vision, Stanford, CA, USA, 25–28 October 2016.
19. Varol, G., Romero, J., Martin, X., Mahmood, N., Black, M.J., Laptev, I., and Schmid, C. (2017, January 21–26). Learning From Synthetic Humans. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.492 ↗
20. A Quantum-clustering Optimization Method for COVID-19 CT Scan Image Segmentation10.1016/j.eswa.2021.115637 ↗
21. A Comprehensive Survey of Image Segmentation: Clustering Methods, Performance Parameters, and Benchmark Datasets10.1007/s11042-021-10594-9 ↗
22. Ambiguous D-means Fusion Clustering Algorithm Based on Ambiguous Set Theory: Special Application in Clustering of CT Scan Images of COVID-1910.1016/j.knosys.2021.107432 ↗
23. Dai, X., Li, S., Zhao, Q., and Yang, H. (2022, January 21–24). Animal Pose Refinement in 2D Images with 3D Constraints. Proceedings of the 2022-33rd British Machine Vision Conference, London, UK.
24. Li, S., Li, J., Tang, H., Qian, R., and Lin, W. (2020, January 12–16). ATRW: A Benchmark for Amur Tiger Re-Identification in the Wild. Proceedings of the ACM International Conference on Multimedia, Seattle, WA USA.10.1145/3394171.3413569 ↗
25. Sun, K., Xiao, B., Liu, D., and Wang, J. (2019, January 16–20). Deep High-Resolution Representation Learning for Human Pose Estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00584 ↗
26. Martinez, J., Hossain, R., Romero, J., and Little, J.J. (2017, January 22–29). A Simple Yet Effective Baseline for 3D Human Pose Estimation. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.288 ↗
27. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is All You Need. Adv. Neural Inf. Process. Syst., 30.
28. Zhao, W., Wang, W., and Tian, Y. (2022, January 19–23). GraFormer: Graph-Oriented Transformer for 3D Pose Estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01979 ↗
29. Li, W., Liu, H., Tang, H., Wang, P., and Van, G.L. (2022, January 19–23). Mhformer: Multi-Hypothesis Transformer for 3D Human Pose Estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01280 ↗
30. Wandt, B., and Rosenhahn, B. (2019, January 16–20). Repnet: Weakly Supervised Training of an Adversarial Reprojection Network for 3D Human Pose Estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00797 ↗
31. Li, C., and Lee, G.H. (2020, January 7–10). Weakly Supervised Generative Network for Multiple 3D Human Pose Hypotheses. Proceedings of the 2020—31st British Machine Vision Conference, Virtual Event, UK.
32. Usman, B., Tagliasacchi, A., Saenko, K., and Sud, A. (2022, January 19–23). MetaPose: Fast 3D Pose from Multiple Views without 3D Supervision. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00664 ↗
33. Wandt, B., Rudolph, M., Zell, P., Rhodin, H., and Rosenhahn, B. (2021, January 19–25). Canonpose: Self-supervised Monocular 3D Human Pose Estimation in the Wild. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.01309 ↗
34. Drover, D., MV, R., Chen, C.H., Agrawal, A., Tyagi, A., and Phuoc, H.C. Can 3D Pose be Learned from 2D Projections Alone? In Proceedings of the European Conference on Computer Vision Workshops, Munich, Germany, 8–14 September 2018.10.1007/978-3-030-11018-5_7 ↗
35. Chen, C.H., Tyagi, A., Agrawal, A., Drover, D., Mv, R., Stojanov, S., and Rehg, J.M. (2019, January 16–20). Unsupervised 3D Pose Estimation with Geometric Self-supervision. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00586 ↗
36. Yu, Z., Ni, B., Xu, J., Wang, J., Zhao, C., and Zhang, W. (2021, January 10–17). Towards Alleviating the Modeling Ambiguity of Unsupervised Monocular 3D Human Pose Estimation. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada.10.1109/ICCV48922.2021.00853 ↗
37. Wandt, B., Little, J.J., and Rhodin, H. (2022, January 19–23). ElePose: Unsupervised 3D Human Pose Estimation by Predicting Camera Elevation and Learning Normalizing Flows on 2D Poses. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00652 ↗
38. Sparse Representation for 3D Shape Estimation: A Convex Relaxation Approach10.1109/TPAMI.2016.2605097 ↗
39. Wang, C., Qiu, H., Yuille, A.L., and Zeng, W. (February, January 27). Learning Basis Representation to Refine 3D Human Pose Estimations. Proceedings of the AAAI Conference on Artificial intelligence, Honolulu, HI, USA.
40. Ramakrishna, V., Kanade, T., and Sheikh, Y. (2012, January 7–13). Reconstructing 3D Human Pose from 2D Image Landmarks. Proceedings of the European Conference on Computer Vision, Florence, Italy.10.1007/978-3-642-33765-9_41 ↗
41. Wang, C., Wang, Y., Lin, Z., Yuille, A.L., and Gao, W. (2014, January 23–28). Robust Estimation of 3D Human Poses from A Single Image. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.303 ↗
42. Akhter, I., and Black, M.J. (2015, January 7–12). Pose-Conditioned Joint Angle Limits for 3D Human Pose Reconstruction. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298751 ↗
43. Newell, A., Yang, K., and Deng, J. (2016, January 11–14). Stacked Hourglass Networks for Human Pose Estimation. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46484-8_29 ↗
44. Wei, S.E., Ramakrishna, V., Kanade, T., and Sheikh, Y. (2016, January 27–30). Convolutional Pose Machines. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.511 ↗
45. Xiao, B., Wu, H., and Wei, Y. (2018, January 8–14). Simple Baselines for Human Pose Estimation and Tracking. Proceedings of the European Conference on Computer Vision, Munich, Germany.10.1007/978-3-030-01231-1_29 ↗
46. Fast Animal Pose Estimation Using Deep Neural Networks10.1038/s41592-018-0234-5 ↗
47. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
48. Insafutdinov, E., Pishchulin, L., Andres, B., Andriluka, M., and Schiele, B. (2016, January 11–14). DeeperCut: A Deeper, Stronger, and Faster Multi-person Pose Estimation Model. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46466-4_3 ↗
49. SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation10.1109/TPAMI.2016.2644615 ↗
50. Jégou, S., Drozdzal, M., Vazquez, D., Romero, A., and Bengio, Y. (2017, January 21–26). The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA.10.1109/CVPRW.2017.156 ↗
51. Learning 3D Deformation of Animals From 2d Images10.1111/cgf.12838 ↗
52. SMPL: A Skinned Multi-Person Linear Model10.1145/2816795.2818013 ↗
53. Online Learning for Matrix Factorization and Sparse Coding
54. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C.L. (2014, January 6–12). Microsoft COCO: Common Objects in Context. Proceedings of the European Conference on Computer Vision, Zurich, Switzerland.10.1007/978-3-319-10602-1_48 ↗
55. Del Pero, L., Ricco, S., Sukthankar, R., and Ferrari, V. (2015, January 7–12). Articulated Motion Discovery Using Pairs of Trajectories. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298827 ↗
56. Yu, X., Zhou, F., and Chandraker, M. (2016, January 11–14). Deep Deformation Network for Object Landmark Localization. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46454-1_4 ↗
57. Mei, J., Chen, X., Wang, C., Yuille, A., Lan, X., and Zeng, W. (2019, January 16–19). Learning to Refine 3D Human Pose Sequences. Proceedings of the International Conference on 3D Vision, Québec City, QC, Canada.10.1109/3DV.2019.00047 ↗
58. (2022, July 28). OpenMMLab. Available online: https://github.com/open-mmlab/mmpose.