MonoMR: Synthesizing Pseudo-2.5D Mixed Reality Content from Monocular Videos
Source abstract
Abstract
MonoMR is a system that synthesizes pseudo-2.5D content from monocular videos for mixed reality (MR) head-mounted displays (HMDs). Unlike conventional systems that require multiple cameras, the MonoMR system can be used by casual end-users to generate MR content from a single camera only. In order to synthesize the content, the system detects people in the video sequence via a deep neural network, and then the detected person’s pseudo-3D position is estimated by our proposed novel algorithm through a homography matrix. Finally, the person’s texture is extracted using a background subtraction algorithm and is placed on an estimated 3D position. The synthesized content can be played in MR HMD, and users can freely change their viewpoint and the content’s position. In order to evaluate the efficiency and interactive potential of MonoMR, we conducted performance evaluations and a user study with 12 participants. Moreover, we demonstrated the feasibility and usability of the MonoMR system to generate pseudo-2.5D content using three example application scenarios.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Vision and Imaging · Visual Attention and Saliency Detection · Face recognition and analysis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science, School of Computing, Tokyo Institute of Technology, Tokyo 152-8550, Japan
Funding
Japan Science and Technology Agency
References (51)
Show source-provided references
1. High-quality Streamable Free-viewpoint Video10.1145/2766945 ↗
2. Orts-Escolano, S., Rhemann, C., Fanello, S., Chang, W., Kowdle, A., Degtyarev, Y., Kim, D., Davidson, P.L., Khamis, S., and Dou, M. (2016, January 16–19). Holoportation: Virtual 3D Teleportation in Real-time. Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST ’16), Tokyo, Japan.10.1145/2984511.2984517 ↗
3. (2021, August 16). Canon Announces Development of the Free Viewpoint Video System Virtual Camera System That Creates an Immersive Viewing Experience. Available online: https://global.canon/en/news/2017/20170921.html.
4. (2021, July 24). Intel® True View—See More Game Than Ever. Available online: https://www.intel.com/content/www/us/en/sports/technology/true-view.html.
5. Goorts, P., Maesen, S., Dumont, M., Rogmans, S., and Bekaert, P. (2014, January 5–8). Free viewpoint video for soccer using histogram-based validity maps in plane sweeping. Proceedings of the 2014 International Conference on Computer Vision Theory and Applications (VISAPP), Lisbon, Portugal.
6. Free-Viewpoint Video of Human Actors Using Multiple Handheld Kinects10.1109/TCYB.2013.2272321 ↗
7. Pavlakos, G., Zhu, L., Zhou, X., and Daniilidis, K. (2018, January 18–23). Learning to Estimate 3D Human Pose and Shape From a Single Color Image. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00055 ↗
8. Unstructured Video-based Rendering: Interactive Exploration of Casually Captured Videos10.1145/1778765.1778824 ↗
9. Chen, J., Paris, S., Wang, J., Matusik, W., Cohen, M., and Durand, F. (2011, January 8–10). The video mesh: A data structure for image-based three-dimensional video editing. Proceedings of the 2011 IEEE International Conference on Computational Photography (ICCP), Pittsburgh, PA, USA.
10. Fleet, D., Pajdla, T., Schiele, B., and Tuytelaars, T. (2014). Video Pop-up: Monocular 3D Reconstruction of Dynamic Scenes. Computer Vision–ECCV 2014, Springer International Publishing.10.1007/978-3-319-10602-1 ↗
11. Langlotz, T., Zingerle, M., Grasset, R., Kaufmann, H., and Reitmayr, G. (2012, January 26–30). AR Record & Replay: Situated Compositing of Video Content in Mobile Augmented Reality. Proceedings of the 24th Australian Computer-Human Interaction Conference (OzCHI ’12), Melbourne, Australia.10.1145/2414536.2414588 ↗
12. Mohr, P., Mandl, D., Tatzgern, M., Veas, E., Schmalstieg, D., and Kalkofen, D. (2017, January 6–11). Retargeting Video Tutorials Showing Tools With Surface Contact to Augmented Reality. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17), Denver, CO, USA.10.1145/3025453.3025688 ↗
13. Virtualized reality: Constructing virtual worlds from real scenes10.1109/93.580394 ↗
14. Recording of multiple videos in a large-scale space for large-scale virtualized reality
15. Koyama, T., Kitahara, I., and Ohta, Y. (2003, January 10). Live mixed-reality 3D video in soccer stadium. Proceedings of the Second IEEE and ACM International Symposium on Mixed and Augmented Reality, Tokyo, Japan.10.1145/965400.965463 ↗
16. Kameda, Y., Koyama, T., Mukaigawa, Y., Yoshikawa, F., and Ohta, Y. (2004, January 27–30). Free viewpoint browsing of live soccer games. Proceedings of the 2004 IEEE International Conference on Multimedia and Expo (ICME) (IEEE Cat. No.04TH8763), Taipei, Taiwan.
17. A Free-Viewpoint Video System for Visualization of Sport Scenes10.5594/J11445 ↗
18. Grau, O., Thomas, G.A., Hilton, A., Kilner, J., and Starck, J. (2007, January 7–9). A Robust Free-Viewpoint Video System for Sport Scenes. Proceedings of the 2007 3DTV Conference, Kos, Greece.10.1109/3DTV.2007.4379384 ↗
19. Bogomjakov, A., Gotsman, C., and Magnor, M. (2006, January 22–24). Free-viewpoint video from depth cameras. Proceedings of the International Workshop on Vision, Modeling and Visualization (VMV), Aachen, Germany.
20. Kuster, C., Popa, T., Zach, C., Gotsman, C., and Gross, M.H. (2011, January 4–6). FreeCam: A Hybrid Camera System for Interactive Free-Viewpoint Video. Proceedings of the International Workshop on Vision, Modeling and Visualization (VMV), Berlin, Germany.
21. Matsumoto, K., Song, C., de Sorbier, F., and Saito, H. (2013, January 10–12). Free viewpoint video synthesis using multi-view depth and color cameras. Proceedings of the IVMSP 2013, Seoul, Korea.10.1109/IVMSPW.2013.6611922 ↗
22. Horry, Y., Anjyo, K.I., and Arai, K. (1997, January 3–8). Tour into the Picture: Using a Spidery Mesh Interface to Make Animation from a Single Image. Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques ( SIGGRAPH ’97), Los Angeles, CA, USA.10.1145/258734.258854 ↗
23. Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., and Weinberger, K.Q. (2014). Depth Map Prediction from a Single Image using a Multi-Scale Deep Network. Advances in Neural Information Processing Systems 27, Curran Associates, Inc.
24. Godard, C., Aodha, O.M., and Brostow, G.J. (2017, January 21–26). Unsupervised Monocular Depth Estimation with Left-Right Consistency. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.699 ↗
25. Rematas, K., Kemelmacher-Shlizerman, I., Curless, B., and Seitz, S. (2018, January 18–23). Soccer on Your Tabletop. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00498 ↗
26. Watanabe, T., Kitahara, I., Kameda, Y., and Ohta, Y. (2010, January 7–9). 3D Free-viewpoint video capturing interface by using bimanual operation. Proceedings of the 2010 3DTV-Conference: The True Vision-Capture, Transmission and Display of 3D Video, Tampere, Finland.10.1109/3DTV.2010.5506506 ↗
27. Kashiwakuma, J., Kitahara, I., Kameda, Y., and Ohta, Y. (2013, January 24–26). A Virtual Camera Controlling Method Using Multi-touch Gestures for Capturing Free-viewpoint Video. Proceedings of the 11th European Conference on Interactive TV and Video (EuroITV ’13), Como, Italy.10.1145/2465958.2465961 ↗
28. Inamoto, N., and Saito, H. (2004, January 3–5). Free Viewpoint Video Synthesis and Presentation of Sporting Events for Mixed Reality Entertainment. Proceedings of the 2004 ACM SIGCHI International Conference on Advances in Computer Entertainment Technology (ACE ’04), Singapore.10.1145/1067343.1067348 ↗
29. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks10.1109/TPAMI.2016.2577031 ↗
30. Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 27–30). You Only Look Once: Unified, Real-Time Object Detection. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.91 ↗
31. Pishchulin, L., Insafutdinov, E., Tang, S., Andres, B., Andriluka, M., Gehler, P., and Schiele, B. (2016, January 27–30). DeepCut: Joint Subset Partition and Labeling for Multi Person Pose Estimation. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.533 ↗
32. Cao, Z., Simon, T., Wei, S., and Sheikh, Y. (2017, January 21–26). Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.143 ↗
33. He, K., Gkioxari, G., Dollár, P., and Girshick, R. (2017, January 22–29). Mask R-CNN. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.322 ↗
34. SMPL: A Skinned Multi-Person Linear Model
35. Bogo, F., Kanazawa, A., Lassner, C., Gehler, P., Romero, J., and Black, M.J. (2016). Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image. Computer Vision–ECCV 2016, Springer International Publishing. Lecture Notes in Computer Science.10.1007/978-3-319-46454-1_34 ↗
36. The Hungarian Method for the assignment problem10.1002/nav.3800020109 ↗
37. Hartley, R., and Zisserman, A. (2003). Multiple View Geometry in Computer Vision, Cambridge University Press. [2nd ed.].10.1017/CBO9780511811685 ↗
38. Graph Cuts and Efficient N-D Image Segmentation10.1007/s11263-006-7934-5 ↗
39. Efficient Adaptive Density Estimation Per Image Pixel for the Task of Background Subtraction10.1016/j.patrec.2005.11.005 ↗
40. Akenine-Moller, T., Haines, E., and Hoffman, N. (2008). Real-Time Rendering, A. K. Peters, Ltd.. [3rd ed.].10.1201/b10644 ↗
41. D’Orazio, T., Leo, M., Mosca, N., Spagnolo, P., and Mazzeo, P.L. (2009, January 2–4). A Semi-automatic System for Ground Truth Generation of Soccer Video Sequences. Proceedings of the 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS ’09), Genova, Italy.10.1109/AVSS.2009.69 ↗
42. Daniel, W. (1990). Applied Nonparametric Statistics, PWS-KENT.
43. Individual comparisons by ranking methods10.2307/3001968 ↗
44. Kotz, S., and Johnson, N.L. (1992). Statistical Methods for Research Workers. Breakthroughs in Statistics: Methodology and Distribution, Springer.10.1007/978-1-4612-4380-9 ↗
45. Du, R., Bista, S., and Varshney, A. (2016, January 22–24). Video Fields: Fusing Multiple Surveillance Videos into a Dynamic Virtual Environment. Proceedings of the 21st International Conference on Web3D Technology (Web3D ’16), Anaheim, CA, USA.10.1145/2945292.2945299 ↗
46. Hattori, K., Hattori, H., Ono, Y., Nishino, K., Itoh, M., Boddeti, V., and Kanade, T. (2014). Image Dataset for Researches about Surveillance Camera-CMUSRD (Surveillance Research Dataset), Technical Report; Carnegie Mellon University.
47. Yao, Q., Sankoh, H., Nonaka, K., and Naito, S. (2016, January 21–23). Automatic camera self-calibration for immersive navigation of free viewpoint sports video. Proceedings of the 2016 IEEE 18th International Workshop on Multimedia Signal Processing (MMSP), Montreal, QC, Canada.10.1109/MMSP.2016.7813399 ↗
48. Shade, J., Gortler, S., He, L.W., and Szeliski, R. (1998, January 19–24). Layered Depth Images. Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’98), Orlando, FL, USA.10.1145/280814.280882 ↗
49. Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., and Weinberger, K.Q. (2014). Generative Adversarial Nets. Advances in Neural Information Processing Systems 27, Curran Associates, Inc.
50. Choi, H., Moon, G., and Lee, K.M. (2020, January 23–28). Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose. Proceedings of the European Conference on Computer Vision (ECCV), Glasgow, UK.10.1007/978-3-030-58571-6_45 ↗
51. Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences, Lawrence Erlbaum Associates.