MVL-Loc: Leveraging Vision-Language Model for Generalizable Multi-Scene Camera Relocalization
Source abstract
Abstract
Camera relocalization, a cornerstone capability of modern computer vision, accurately determines a camera’s position and orientation from images and is essential for applications in augmented reality, mixed reality, autonomous driving, delivery drones, and robotic navigation. Unlike traditional deep learning-based methods regress camera pose from images in a single scene which lack generalization and robustness in diverse environments. We propose MVL-Loc, a novel end-to-end multi-scene six degrees of freedom camera relocalization framework. MVL-Loc leverages pretrained world knowledge from vision-language models and incorporates multimodal data to generalize across both indoor and outdoor settings. Furthermore, natural language is employed as a directive tool to guide the multi-scene learning process, facilitating semantic understanding of complex scenes and capturing spatial relationships among objects. Extensive experiments on the 7Scenes and Cambridge Landmarks datasets demonstrate MVL-Loc’s robustness and state-of-the-art performance in real-world multi-scene camera relocalization, with improved accuracy in both positional and orientational estimates.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Robotics and Sensor-Based Localization · Advanced Image and Video Retrieval Techniques · Advanced Vision and Imaging
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Automation Science and Engineering, South China University of Technology, Guangzhou 510641, China
References (29)
Show source-provided references
1. Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography10.1145/358669.358692 ↗
2. Kendall, A., Grimes, M., and Cipolla, R. (2015, January 7–12). Posenet: A convolutional network for real-time 6-dof camera relocalization. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA.10.1109/ICCV.2015.336 ↗
3. Kendall, A., and Cipolla, R. (2017, January 21–26). Geometric loss functions for camera pose regression with deep learning. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.694 ↗
4. Clark, R., Wang, S., Markham, A., Trigoni, N., and Wen, H. (2017, January 21–26). Vidloc: A deep spatio-temporal model for 6-dof video-clip relocalization. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.284 ↗
5. Wang, B., Chen, C., Lu, C.X., Zhao, P., Trigoni, N., and Markham, A. (2020, January 7–12). Atloc: Attention guided camera localization. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i06.6608 ↗
6. Xiao, Z., Chen, C., Yang, S., and Wei, W. (2024, January 13–17). EffLoc: Lightweight Vision Transformer for Efficient 6-DOF Camera Relocalization. Proceedings of the 2024 IEEE International Conference on Robotics and Automation (ICRA), Yokohama, Japan.10.1109/ICRA57147.2024.10611622 ↗
7. Zhou, G., Hong, Y., and Wu, Q. (2023). NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models. arXiv.10.1609/aaai.v38i7.28597 ↗
8. Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., and Chen, X. (2023). RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. arXiv.
9. Blanton, H., Greenwell, C., Workman, S., and Jacobs, N. (2020, January 14–19). Extending Absolute Pose Regression to Multiple Scenes. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Seattle, WA, USA.10.1109/CVPRW50498.2020.00027 ↗
10. Coarse-to-Fine Multi-Scene Pose Regression with Transformers10.1109/TPAMI.2023.3310929 ↗
11. Chen, D.M., Baatz, G., Köser, K., Tsai, S.S., Vedantham, R., Pylvänäinen, T., Roimela, K., Chen, X., Bach, J., and Pollefeys, M. (2011, January 20–25). City-scale landmark identification on mobile devices. Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Colorado Springs, CO, USA.10.1109/CVPR.2011.5995610 ↗
12. Shavit, Y., Ferens, R., and Keller, Y. (2021, January 10–17). Learning Multi-Scene Absolute Pose Regression with Transformers. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.10.1109/ICCV48922.2021.00273 ↗
13. Brahmbhatt, S., Gu, J., Kim, K., Hays, J., and Kautz, J. (2018, January 18–23). Geometry-aware learning of maps for camera localization. Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.10.1109/CVPR.2018.00277 ↗
14. FusionLoc: Camera-2D LiDAR Fusion Using Multi-Head Self-Attention for End-to-End Serving Robot Relocalization10.1109/ACCESS.2023.3297202 ↗
15. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning transferable visual models from natural language supervision. Proceedings of the International Conference on Machine Learning, Virtual.
16. Li, J., Tan, H., and Bansal, M. (2022, January 18–24). Envedit: Environment Editing for Vision-and-Language Navigation. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA.10.1109/CVPR52688.2022.01497 ↗
17. Mirjalili, R., Krawez, M., and Burgard, W. (2023, January 1–5). FM-Loc: Using Foundation Models for Improved Vision-Based Localization. Proceedings of the 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Detroit, MI, USA.10.1109/IROS55552.2023.10342439 ↗
18. Matsuzaki, S., Sugino, T., Tanaka, K., Sha, Z., Nakaoka, S., Yoshizawa, S., and Shintani, K. (2024, January 13–17). CLIP-Loc: Multi-modal Landmark Association for Global Localization in Object-based Maps. Proceedings of the 2024 IEEE International Conference on Robotics and Automation (ICRA), Yokohama, Japan.10.1109/ICRA57147.2024.10611393 ↗
19. Manvi, R., Khanna, S., Mai, G., Burke, M., Lobell, D.B., and Ermon, S. (2024, January 7–11). GeoLLM: Extracting Geospatial Knowledge from Large Language Models. Proceedings of the Twelfth International Conference on Learning Representations, Vienna, Austria.
20. Li, L., Ye, Y., Jiang, B., and Zeng, W. (2024, January 21–27). GeoReasoner: Geo-localization with Reasoning in Street Views using a Large Vision-Language Model. Proceedings of the International Conference on Machine Learning (ICML), Vienna, Austria.
21. Sun, X., Albin, D., Mauceri, C., Woods, D., and Heckman, C. (2025). Spatial-LLaVA: Enhancing Large Language Models with Spatial Referring Expressions for Visual Understanding. arXiv.
22. Chen, H., Müller, T., and Park, Y. (2025, January 11–15). UniSceneVL: A Unified Vision-Language Model for 3D Scene Understanding and Spatial Correspondence Learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.
23. Shen, F., Ye, H., Zhang, J., Wang, C., Han, X., and Yang, W. (2023). Advancing pose-guided image synthesis with progressive conditional diffusion models. arXiv.
24. Shen, F., Ye, H., Liu, S., Zhang, J., Wang, C., Han, X., and Wei, Y. (March, January 25). Boosting consistency in story visualization with rich-contextual conditional diffusion models. Proceedings of the AAAI Conference on Artificial Intelligence, Philadelphia, PA, USA.
25. Criminisi, A., Shotton, J., Glocker, B., Izadi, S., and Fitzgibbon, A. (2013, January 1). 7-Scenes RGB-D Dataset. Proceedings of the Microsoft Research Cambridge Vision and Graphics Group Dataset Release, Cambridge, UK. Available online: https://www.microsoft.com/en-us/research/project/rgb-d-dataset-7-scenes/.
26. Kendall, A., and Cipolla, R. (2016, January 16–21). Modelling uncertainty in deep learning for camera relocalization. Proceedings of the 2016 IEEE International Conference on Robotics and Automation (ICRA), Stockholm, Sweden.10.1109/ICRA.2016.7487679 ↗
27. Walch, F., Hazirbas, C., Leal-Taixe, L., Sattler, T., Hilsenbeck, S., and Cremers, D. (2017, January 21–26). Image-based localization using lstms for structured feature correlation. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/ICCV.2017.75 ↗
28. Shavit, Y., and Ferens, R. (2021, January 10–15). Do We Really Need Scene-specific Pose Encoders?. Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR), Milan, Italy.10.1109/ICPR48806.2021.9412225 ↗
29. Melekhov, I., Ylioinas, J., Kannala, J., and Rahtu, E. (2017, January 22–29). Image-Based Localization Using Hourglass Networks. Proceedings of the 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), Venice, Italy.10.1109/ICCVW.2017.107 ↗