Voxel-Based Multi-Person Multi-View 3D Pose Estimation in Operating Room
Source abstract
Abstract
The localization and pose estimation of clinicians in the operating room is a critical component for building intelligent perception systems, playing a vital role in enhancing surgical standardization and safety. Multi-view, multi-person 3D pose estimation is a highly challenging task—especially in the operating room, where the presence of sterile clothing, occlusion from surgical instruments, and limited data availability due to privacy concerns exacerbate the difficulty. While voxel-based 3D pose estimation methods have shown promising results in general scenarios, their performance is significantly challenged in surgical environments with limited camera views and severe occlusions. To address these issues, this paper proposes a fine-grained voxel feature reconstruction method enhanced with depth information, effectively mitigating projection errors caused by reduced viewpoints. Additionally, an attention mechanism is integrated into the encoder–decoder architecture to improve the network’s capacity for global information modeling and enhance the accuracy of keypoint regression. Experiments conducted in real-world operating room scenarios, using the Multi-View Operating Room (MVOR) dataset, demonstrate that the proposed method maintains high accuracy even under limited camera views and outperforms existing state-of-the-art multi-view 3D pose estimation approaches. This work provides a novel and efficient solution for human pose estimation (HPE) in complex medical environments.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human Pose and Action Recognition · Advanced Vision and Imaging · Surgical Simulation and Training
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China
Funding
National Natural Science Foundation of China
References (38)
Show source-provided references
1. OR black box and surgical control tower: Recording and streaming data and analytics to improve surgical care10.1016/j.jviscsurg.2021.01.004 ↗
2. Özsoy, E., Pellegrini, C., Czempiel, T., Tristram, F., Yuan, K., Bani-Harouni, D., Eck, U., Busam, B., Keicher, M., and Navab, N. (2025, January 11–15). Mm-or: A large multimodal operating room dataset for semantic understanding of high-intensity surgical environments. Proceedings of the Computer Vision and Pattern Recognition Conference, Nashville, TN, USA.10.1109/CVPR52734.2025.01805 ↗
3. Radio Frequency Identification (RFID) in health care: Where are we? A scoping review10.1007/s12553-022-00696-1 ↗
4. Qiu, H., Wang, C., Wang, J., Wang, N., and Zeng, W. (2019, January 27–28). Cross view fusion for 3D human pose estimation. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.10.1109/ICCV.2019.00444 ↗
5. Iskakov, K., Burkov, E., Lempitsky, V., and Malkov, Y. (2019, January 27–28). Learnable triangulation of human pose. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.10.1109/ICCV.2019.00781 ↗
6. Ye, H., Zhu, W., Wang, C., Wu, R., and Wang, Y. (2022, January 23–27). Faster voxelpose: Real-time 3D human pose estimation by orthographic projection. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-20068-7_9 ↗
7. Zhuang, Z., and Zhou, Y. (2024, January 5–8). FasterVoxelPose+: Fast and accurate voxel-based 3D human pose estimation by depth-wise projection decay. Proceedings of the Asian Conference on Machine Learning, Hanoi, Vietnam.
8. Srivastav, V., Issenhuth, T., Kadkhodamohammadi, A., de Mathelin, M., Gangi, A., and Padoy, N. (2018). MVOR: A multi-view RGB-D operating room dataset for 2D and 3D human pose estimation. arXiv.
9. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2020). An image is worth 16×16 words: Transformers for image recognition at scale. arXiv.
10. 3D human pose estimation from multi-view thermal vision sensors10.1016/j.inffus.2023.102154 ↗
11. Real-time multi-camera 3D human pose estimation at the edge for industrial applications10.1016/j.eswa.2024.124089 ↗
12. GAN-BodyPose: Real-time 3D human body pose data key point detection and quality assessment assisted by generative adversarial network10.1016/j.imavis.2024.105144 ↗
13. Srivastav, V., Chen, K., and Padoy, N. (2024, January 17–21). Selfpose3d: Self-supervised multi-person multi-view 3D pose estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.00242 ↗
14. Li, Y.J., Xu, Y., Khirodkar, R., Park, J., and Kitani, K. (2024). Multi-person 3D pose estimation from multi-view uncalibrated depth cameras. arXiv.
15. Skeleton Cluster Tracking for robust multi-view multi-person 3D human pose estimation10.1016/j.cviu.2024.104059 ↗
16. Amara, K., Guerroudji, M.A., Zenati, N., Kerdjidj, O., Atalla, S., Mansoor, W., and Ramzan, N. (2024, January 21–22). Augmented Reality localisation using 6 DoF phantom head Pose Estimation-based generalisable Deep Learning model. Proceedings of the 2024 8th International Conference on Image and Signal Processing and their Applications (ISPA), Biskra, Algeria.10.1109/ISPA59904.2024.10536799 ↗
17. Xu, M., Shu, Q., Huang, Z., Chen, G., and Poslad, S. (2025). ARLO: Augmented Reality Localization Optimization for Real-Time Pose Estimation and Human–Computer Interaction. Electronics, 14.10.3390/electronics14071478 ↗
18. In-bed human pose estimation using multi-source information fusion for health monitoring in real-world scenarios10.1016/j.inffus.2023.102209 ↗
19. Image-free single-pixel keypoint detection for privacy preserving human pose estimation10.1364/OL.514213 ↗
20. Salehin, S., Kearney, S., and Gurbuz, Z. (2024, January 27–30). Kinematic Cycle Consistency Using Wearables and RF Data for Improved Human Skeleton Estimation. Proceedings of the 2024 58th Asilomar Conference on Signals, Systems, and Computers, Pacific Grove, CA, USA.10.1109/IEEECONF60004.2024.10942986 ↗
21. Joo, H., Liu, H., Tan, L., Gui, L., Nabbe, B., Matthews, I., Kanade, T., Nobuhara, S., and Sheikh, Y. (2015, January 7–13). Panoptic studio: A massively multiview system for social motion capture. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.381 ↗
22. Belagiannis, V., Amin, S., Andriluka, M., Schiele, B., Navab, N., and Ilic, S. (2014, January 23–28). 3D pictorial structures for multiple human pose estimation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.216 ↗
23. Guo, Y., Li, Z., Li, Z., Du, X., Quan, S., and Xu, Y. (2022, January 3–8). PoP-Net: Pose Over Parts Network for Multi-Person 3D Pose Estimation From a Depth Image. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA.10.1109/WACV51458.2022.00397 ↗
24. Jiang, T., Billingham, J., Müksch, S., Zarate, J., Evans, N., Oswald, M., Pollefeys, M., Hilliges, O., Kaufmann, M., and Song, J. (2024). WorldPose: A World Cup Dataset for Global 3D Human Pose Estimation. European Conference on Computer Vision, Springer Nature.10.1007/978-3-031-72655-2_20 ↗
25. Tu, H., Wang, C., and Zeng, W. (2020, January 23–28). Voxelpose: Towards multi-camera 3D human pose estimation in wild environment. Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK. Proceedings, Part I 16.10.1007/978-3-030-58452-8_12 ↗
26. Zheng, C., Zhu, S., Mendieta, M., Yang, T., Chen, C., and Ding, Z. (2021, January 11–17). 3D human pose estimation with spatial and temporal transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.01145 ↗
27. Direct multi-view multi-person 3D pose estimation
28. SVTformer: Spatial-View-Temporal Transformer for Multi-View 3D Human Pose Estimation
29. Chen, L., Ai, H., Chen, R., Zhuang, Z., and Liu, S. (2020, January 13–19). Cross-view tracking for multi-human 3D pose estimation at over 100 fps. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00334 ↗
30. Surgical data science–from concepts toward clinical translation10.1016/j.media.2021.102306 ↗
31. Holistic or domain modeling: A semantic scene graph approach
32. Parsing human skeletons in an operating room10.1007/s00138-016-0792-4 ↗
33. Özsoy, E., Örnek, E.P., Eck, U., Czempiel, T., Tombari, F., and Navab, N. (2022, January 18–22). 4d-or: Semantic scene graphs for or domain modeling. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Singapore.10.1007/978-3-031-16449-1_45 ↗
34. Fusing information from multiple 2D depth cameras for 3D human pose estimation in the operating room10.1007/s11548-019-02044-7 ↗
35. A generalizable approach for multi-view 3D human pose regression10.1007/s00138-020-01120-2 ↗
36. Unsupervised domain adaptation for clinician pose estimation and instance segmentation in the operating room10.1016/j.media.2022.102525 ↗
37. 3D human pose estimation in multi-view operating room videos using differentiable camera projections10.1080/21681163.2022.2155580 ↗
38. Sun, K., Xiao, B., Liu, D., and Wang, J. (2019, January 15–20). Deep high-resolution representation learning for human pose estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00584 ↗