DeepProfile: Accurate Under-the-Clothes Body Profile Estimation
Source abstract
Abstract
Accurate human body profiles have many potential applications. Image-based human body profile estimation can be regarded as a fine-grained semantic segmentation problem, which is typically used to locate objects and boundaries in images. However, existing image segmentation methods, such as human parsing, require significant amounts of annotation and their datasets consider clothes as part of the human body profile. Therefore, the results they generate are not accurate when the human subject is dressed in loose-fitting clothing. In this paper, we created and labeled an under-the-clothes human body contour keypoint dataset; we utilized a convolutional neural network (CNN) to extract the contour keypoints, then combined them with a body profile database to generate under-the-clothes profiles. In order to improve the precision of keypoint detection, we propose a short-skip multi-scale dense (SMSD) block in the CNN to keep the details of the image and increase the information flow among different layers. Extensive experiments were conducted to show the effectiveness of our method. We demonstrate that our method achieved better results—especially when the person was dressed in loose-fitting clothes—than and competitive quantitative performance compared to state-of-the-art methods, while requiring less annotation effort. We also extended our method to the applications of 3D human model reconstruction and body size measurement.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: 3D Shape Modeling and Analysis · Human Pose and Action Recognition · Advanced Neural Network Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou 310023, China
Shenzhen TOZI Technology Co., Ltd., Shenzhen 518052, China
Funding
Zhejiang Provincial Natural Science Foundation of China
References (46)
Show source-provided references
1. Ruan, T., Liu, T., Huang, Z., Wei, Y., Wei, S., and Zhao, Y. (2019, January 17). Devil in the details: Towards accurate single and multiple human parsing. Proceedings of the AAAI Conference on Artificial Intelligence, Virtual.10.1609/aaai.v33i01.33014814 ↗
2. Look into person: Joint body parsing & pose estimation network and a new benchmark10.1109/TPAMI.2018.2820063 ↗
3. Sun, K., Zhao, Y., Jiang, B., Cheng, T., Xiao, B., Liu, D., Mu, Y., Wang, X., Liu, W., and Wang, J. (2019). High-resolution representations for labeling pixels and regions. arXiv.
4. Huang, G., Chen, D., Li, T., Wu, F., van der Maaten, L., and Weinberger, K.Q. (2017). Multi-scale dense networks for resource efficient image classification. arXiv.
5. Predicting realistic and precise human body models under clothing based on orthogonal-view photos10.1016/j.promfg.2015.07.884 ↗
6. Long, J., Shelhamer, E., and Darrell, T. (2015, January 7–12). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298965 ↗
7. Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., and Adam, H. (2018, January 8–14). Encoder-decoder with atrous separable convolution for semantic image segmentation. Proceedings of the European Conference on Computer Vision, Munich, Germany.10.1007/978-3-030-01234-2_49 ↗
8. He, K., Gkioxari, G., Dollár, P., and Girshick, R. (2017, January 22–29). Mask r-cnn. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.322 ↗
9. Yang, M., Yu, K., Zhang, C., Li, Z., and Yang, K. (2018, January 18–23). Denseaspp for semantic segmentation in street scenes. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00388 ↗
10. Takikawa, T., Acuna, D., Jampani, V., and Fidler, S. (November, January 27). Gated-scnn: Gated shape cnns for semantic segmentation. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea.
11. Huang, Z., Wang, X., Wei, Y., Huang, L., Shi, H., Liu, W., and Huang, T.S. (November, January 27). CCNet: Criss-cross attention for semantic segmentation. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea.
12. Boudiaf, M., Kervadec, H., Masud, Z.I., Piantanida, P., Ayed, I.B., and Dolz, J. (2021, January 20–25). Few-Shot Segmentation Without Meta-Learning: A Good Transductive Inference Is All You Need?. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.01376 ↗
13. Hou, Q., Zhang, L., Cheng, M.-M., and Feng, J. (2020, January 13–19). Strip Pooling: Rethinking Spatial Pooling for Scene Parsing. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00406 ↗
14. Yamaguchi, K., Kiapour, M.H., Ortiz, L.E., and Berg, T.L. (2012, January 16–21). Parsing clothing in fashion photographs. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.10.1109/CVPR.2012.6248101 ↗
15. Dong, J., Chen, Q., Shen, X., Yang, J., and Yan, S. (2014, January 23–28). Towards Unified Human Parsing and Pose Estimation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.113 ↗
16. Liu, S., Liang, X., Liu, L., Shen, X., Yang, J., Xu, C., Lin, L., Cao, X., and Yan, S. (2015, January 7–12). Matching-CNN meets KNN: Quasi-parametric human parsing. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298748 ↗
17. Tripathi, S., Collins, M., Brown, M., and Belongie, S. (2017). Pose2instance: Harnessing keypoints for person instance segmentation. arXiv.
18. Papandreou, G., Zhu, T., Chen, L.C., Gidaris, S., Tompson, J., and Murphy, K. (2018, January 8–14). PersonLab: Person Pose Estimation and Instance Segmentation with a Bottom-Up, Part-Based, Geometric Embedding Model. Proceedings of the European Conference on Computer Vision, Munich, Germany.10.1007/978-3-030-01264-9_17 ↗
19. Zhang, S.H., Li, R., Dong, X., Rosin, P., Cai, Z., Han, X., Yang, D., Huang, H., and Hu, S.M. (2019, January 15–20). Pose2Seg: Detection Free Human Instance Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00098 ↗
20. Kanazawa, A., Black, M.J., Jacobs, D.W., and Malik, J. (2018, January 18–22). End-to-end recovery of human shape and pose. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00744 ↗
21. SMPL: A skinned multi-person linear model10.1145/2816795.2818013 ↗
22. Güler, R.A., Neverova, N., and Kokkinos, I. (2018, January 18–23). DensePose: Dense Human Pose Estimation in the Wild. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00762 ↗
23. Omran, M., Lassner, C., Pons-Moll, G., Gehler, P., and Schiele, B. (2018, January 5–8). Neural body fitting: Unifying deep learning and model based human pose and shape estimation. Proceedings of the 2018 International Conference on 3D Vision, Verona, Italy.10.1109/3DV.2018.00062 ↗
24. Pavlakos, G., Choutas, V., Ghorbani, N., Bolkart, T., Osman, A.A., Tzionas, D., and Black, M.J. (2019, January 15–20). Expressive body capture: 3d hands, face, and body from a single image. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.01123 ↗
25. Bogo, F., Kanazawa, A., Lassner, C., Gehler, P., Romero, J., and Black, M.J. (2016, January 11–14). Keep It SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46454-1_34 ↗
26. Bălan, A.O., and Black, M.J. (2016, January 12–18). The Naked Truth: Estimating Body Shape Under Clothing. Proceedings of the European Conference on Computer Vision, Marseille, France.
27. Estimation of human body shape and posture under clothing
28. Learning to Estimate the Body Shape Under Clothing From a Single 3-D Scan10.1109/TII.2020.3016591 ↗
29. Body talk: Crowdshaping realistic 3D avatars with words10.1145/2897824.2925981 ↗
30. Estimating 3D human shape under clothing from a single RGB image
31. Xiao, B., Wu, H., and Wei, Y. (2018, January 8–14). Simple baselines for human pose estimation and tracking. Proceedings of the European Conference on Computer Vision, Munich, Germany.10.1007/978-3-030-01231-1_29 ↗
32. Tompson, J., Goroshin, R., Jain, A., LeCun, Y., and Bregler, C. (2015, January 7–12). Efficient object localization using Convolutional Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298664 ↗
33. Newell, A., Yang, K., and Deng, J. (2016, January 11–14). Stacked hourglass networks for human pose estimation. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46484-8_29 ↗
34. Nie, X., Feng, J., Zhang, J., and Yan, S. (November, January 27). Single-stage multi-person pose machines. Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea.
35. Papandreou, G., Zhu, T., Kanazawa, N., Toshev, A., Tompson, J., Bregler, C., and Murphy, K. (2017, January 21–26). Towards accurate multi-person pose estimation in the wild. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.395 ↗
36. Chen, Y., Wang, Z., Peng, Y., Zhang, Z., Yu, G., and Sun, J. (2018, January 18–23). Cascaded pyramid network for multi-person pose estimation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00742 ↗
37. Li, W., Wang, Z., Yin, B., Peng, Q., Du, Y., Xiao, T., Yu, G., Lu, H., Wei, Y., and Sun, J. (2019). Rethinking on multi-stage networks for human pose estimation. arXiv.
38. Sun, K., Xiao, B., Liu, D., and Wang, J. (2019, January 15–20). Deep high-resolution representation learning for human pose estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00584 ↗
39. Zhang, F., Zhu, X., Dai, H., Ye, M., and Zhu, C. (2020, January 13–19). Distribution-Aware Coordinate Representation for Human Pose Estimation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00712 ↗
40. Geng, Z., Sun, K., Xiao, B., Zhang, Z., and Wang, J. (2021, January 20–25). Bottom-Up Human Pose Estimation via Disentangled Keypoint Regression. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.01444 ↗
41. Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B., and Belongie, S. (2017, January 21–26). Feature Pyramid Networks for Object Detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.106 ↗
42. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C.L. (2014, January 6–12). Microsoft coco: Common objects in context. Proceedings of the European Conference on Computer Vision, Zurich, Switzerland.10.1007/978-3-319-10602-1_48 ↗
43. Res2net: A new multi-scale backbone architecture10.1109/TPAMI.2019.2938758 ↗
44. Conditional random fields meet deep neural networks for semantic segmentation: Combining probabilistic graphical models with deep learning forstructured prediction10.1109/MSP.2017.2762355 ↗
45. Verma, D., Gulati, K., Goel, V., and Shah, R.R. (2020, January 12–16). Fashionist: Personalising outfit recommendation for cold-start scenarios. Proceedings of the 28th ACM International Conference on Multimedia, Seattle, WA, USA.10.1145/3394171.3414446 ↗
46. Sagar, D., Garg, J., Kansal, P., Bhalla, S., Shah, R.R., and Yu, Y. (2020, January 24–26). Pai-bpr: Personalized outfit recommendation scheme with attribute-wise interpretability. Proceedings of the 2020 IEEE Sixth International Conference on Multimedia Big Data, New Delhi, India.10.1109/BigMM50055.2020.00039 ↗