journal article·2020-05-05Open access

Autonomous Navigation Framework for Intelligent Robots Based on a Semantic Environment Modeling

Sung-Hyeon Joo, Sumaira Manzoor, Yuri Goncalves Rocha, Sang-Hyeon Bae, Kwang-Hee Lee, Tae-Yong Kuc, Minsung Kim

Applied Sciences· Volume 10· Issue 9

Abstract

Humans have an innate ability of environment modeling, perception, and planning while simultaneously performing tasks. However, it is still a challenging problem in the study of robotic cognition. We address this issue by proposing a neuro-inspired cognitive navigation framework, which is composed of three major components: semantic modeling framework (SMF), semantic information processing (SIP) module, and semantic autonomous navigation (SAN) module to enable the robot to perform cognitive tasks. The SMF creates an environment database using Triplet Ontological Semantic Model (TOSM) and builds semantic models of the environment. The environment maps from these semantic models are generated in an on-demand database and downloaded in SIP and SAN modules when required to by the robot. The SIP module contains active environment perception components for recognition and localization. It also feeds relevant perception information to behavior planner for safely performing the task. The SAN module uses a behavior planner that is connected with a knowledge base and behavior database for querying during action planning and execution. The main contributions of our work are the development of the TOSM, integration of SMF, SIP, and SAN modules in one single framework, and interaction between these components based on the findings of cognitive science. We deploy our cognitive navigation framework on a mobile robot platform, considering implicit and explicit constraints for autonomous robot navigation in a real-world environment. The robotic experiments demonstrate the validity of our proposed framework.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Robotics and Automated Systems · AI-based Problem Solving and Planning · Robotic Path Planning Algorithms

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Department of Electrical and Computer Engineering, College of Information and Communication Engineering, Sungkyunkwan University, Suwon 16419, Korea

Robot R&D Group, Korea Institute of Industrial Technology (KITECH), Ansan 15588, Korea

Department of Electronic and Electrical Engineering, Dongguk University-Seoul Campus, Seoul 04620, Korea

Funding

Ministry of Trade, Industry and Energy

References (100+)

Show source-provided references
1. Visual place recognition: A survey10.1109/TRO.2015.2496823 ↗
2. Robot planning in the real world: Research challenges and opportunities
3. Pendleton, S.D., Andersen, H., Du, X., Shen, X., Meghjani, M., Eng, Y.H., Rus, D., and Ang, M.H. (2017). Perception, planning, control, and coordination for autonomous vehicles. Machines, 5.10.3390/machines5010006 ↗
4. Ontology based knowledge representation technique, domain modeling languages and planners for robotic path planning: A survey10.1016/j.icte.2018.04.008 ↗
5. Towards semantic maps for mobile robots10.1016/j.robot.2008.08.001 ↗
6. Exploiting semantic knowledge for robot object recognition10.1016/j.knosys.2015.05.032 ↗
7. Galindo, C., Fernández-Madrigal, J.A., González, J., and Saffiotti, A. (2007, January 10). Using semantic information for improving efficiency of robot task planning. Proceedings of the ICRA Workshop: Semantic Information in Robotics, Rome, Italy.
8. Abdi, L., and Meddeb, A. (2017, January 3–7). Semantic recognition: Unified framework for joint object detection and semantic segmentation. Proceedings of the Symposium on Applied Computing, Marrakech, Morocco.10.1145/3019612.3019636 ↗
9. Generic decoding of seen and imagined objects using hierarchical visual features10.1038/ncomms15037 ↗
10. PDDL2. 1: An extension to PDDL for expressing temporal planning domains10.1613/jair.1129 ↗
11. Cashmore, M., Fox, M., Long, D., Magazzeni, D., Ridder, B., Carrera, A., Palomeras, N., Hurtos, N., and Carreras, M. (2015, January 7–11). Rosplan: Planning in the robot operating system. Proceedings of the Twenty-Fifth International Conference on Automated Planning and Scheduling, Jerusalem, Israel.10.1609/icaps.v25i1.13699 ↗
12. Ontology-based unified robot knowledge for service robots in indoor environments10.1109/TSMCA.2010.2076404 ↗
13. RoboEarth-A world wide web for robots
14. Joo, S.H., Manzoor, S., Rocha, Y.G., Lee, H.U., and Kuc, T.Y. (arXiv, 2019). A realtime autonomous robot navigation framework for human like high-level interaction and task planning in global dynamic environment, arXiv.
15. Schlenoff, C., Prestes, E., Madhavan, R., Goncalves, P., Li, H., Balakirsky, S., Kramer, T., and Miguelanez, E. (2012, January 7–12). An IEEE standard ontology for robotics and automation. Proceedings of the 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, Vilamoura, Portugal.10.1109/IROS.2012.6385518 ↗
16. Suh, I.H., Lim, G.H., Hwang, W., Suh, H., Choi, J.H., and Park, Y.T. (November, January 29). Ontology-based multi-layered robot knowledge framework (OMRKF) for robot intelligence. Proceedings of the 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems, San Diego, CA, USA.
17. Bratko, I. (2011). Prolog Programming for Artificial Intelligence, Pearson Education.
18. Representation and exchange of knowledge about actions, objects, and environments in the roboearth framework10.1109/TASE.2013.2244883 ↗
19. KnowRob: A knowledge processing infrastructure for cognition-enabled robots10.1177/0278364913481635 ↗
20. The graph SLAM algorithm with applications to large-scale mapping of urban structures10.1177/0278364906065387 ↗
21. Improved techniques for grid mapping with rao-blackwellized particle filters10.1109/TRO.2006.889486 ↗
22. Murphy, K., and Russell, S. (2001). Rao-Blackwellised particle filtering for dynamic Bayesian networks. Sequential Monte Carlo Methods in Practice, Springer.10.1007/978-1-4757-3437-9_24 ↗
23. Zhang, J., and Singh, S. (2014, January 12–16). LOAM: Lidar Odometry and Mapping in Real-time. Proceedings of the Robotics: Science and Systems Conference, Berkeley, CA, USA.10.15607/RSS.2014.X.007 ↗
24. Klein, G., and Murray, D. (2007, January 13–16). Parallel tracking and mapping for small AR workspaces. Proceedings of the 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality, Nara, Japan.10.1109/ISMAR.2007.4538852 ↗
25. ORB-SLAM: A versatile and accurate monocular SLAM system10.1109/TRO.2015.2463671 ↗
26. Pronobis, A., and Jensfelt, P. (2012, January 14–18). Large-scale semantic mapping and reasoning with heterogeneous modalities. Proceedings of the 2012 IEEE International Conference on Robotics and Automation, Saint Paul, MN, USA.10.1109/ICRA.2012.6224637 ↗
27. Semantic mapping for mobile robotics tasks: A survey10.1016/j.robot.2014.12.006 ↗
28. McCormac, J., Handa, A., Davison, A., and Leutenegger, S. (June, January 29). Semanticfusion: Dense 3d semantic mapping with convolutional neural networks. Proceedings of the 2017 IEEE International Conference on Robotics and automation (ICRA), Singapore, Singapore.10.1109/ICRA.2017.7989538 ↗
29. Yang, S., Huang, Y., and Scherer, S. (2017, January 24–28). Semantic 3D occupancy mapping through efficient high order CRFs. Proceedings of the 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, Canada.10.1109/IROS.2017.8202212 ↗
30. Nakajima, Y., Tateno, K., Tombari, F., and Saito, H. (2018, January 1–5). Fast and accurate semantic mapping through geometric-based incremental segmentation. Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.10.1109/IROS.2018.8593993 ↗
31. Dynamic-SLAM: Semantic monocular visual localization and mapping based on deep learning in dynamic environment10.1016/j.robot.2019.03.012 ↗
32. RoboEarth semantic mapping: A cloud enabled knowledge-based approach10.1109/TASE.2014.2377791 ↗
33. Tenorth, M., Perzylo, A.C., Lafrenz, R., and Beetz, M. (2012, January 14–18). The roboearth language: Representing and exchanging knowledge about actions, objects, and environments. Proceedings of the 2012 IEEE International Conference on Robotics and Automation, Saint Paul, MN, USA.10.1109/ICRA.2012.6224812 ↗
34. Johnston, B., Yang, F., Mendoza, R., Chen, X., and Williams, M.A. (2008). Ontology based object categorization for robots. International Conference on Practical Aspects of Knowledge Management, Springer.10.1007/978-3-540-89447-6_21 ↗
35. Tenorth, M., Kunze, L., Jain, D., and Beetz, M. (2010, January 6–8). Knowrob-map-knowledge-linked semantic object maps. Proceedings of the 2010 10th IEEE-RAS International Conference on Humanoid Robots, Nashville, TN, USA.10.1109/ICHR.2010.5686350 ↗
36. Relational model for robotic semantic navigation in indoor environments10.1007/s10846-017-0469-x ↗
37. A Survey of Deep Learning-Based Object Detection10.1109/ACCESS.2019.2939201 ↗
38. Zou, Z., Shi, Z., Guo, Y., and Ye, J. (arXiv, 2019). Object detection in 20 years: A survey, arXiv.
39. Object recognition in mental representations: Directions for exploring diagnostic features through visual mental imagery10.3389/fpsyg.2017.00833 ↗
40. Luo, Y., Gao, Y., Liu, L., and Huang, X. (2012, January 6–8). A novel object recognition system for cognitive robot. Proceedings of the 2012 IEEE International Conference on Information and Automation, Shenyang, China.10.1109/ICInfA.2012.6246763 ↗
41. Deep neural networks as scientific models10.1016/j.tics.2019.01.009 ↗
42. Redmon, J., and Farhadi, A. (arXiv, 2018). Yolov3: An incremental improvement, arXiv.
43. Weiss, I. (arXiv, 2019). A Dual-hierarchy Semantic Graph for Robust Object Recognition, arXiv.
44. Zhang, H., Xu, T., Elhoseiny, M., Huang, X., Zhang, S., Elgammal, A., and Metaxas, D. (2016, January 27–30). Spda-cnn: Unifying semantic part detection and abstraction for fine-grained recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.129 ↗
45. Wang, J., Xie, C., Zhang, Z., Zhu, J., Xie, L., and Yuille, A. (arXiv, 2017). Detecting semantic parts on partially occluded objects, arXiv.10.5244/C.31.73 ↗
46. Choi, J.H., Park, Y.T., Lim, G.H., and Lee, S. (2007). Ontology-Based Semantic Context Modeling for Object Recognition of Intelligent Mobile Robots. Recent Progress in Robotics: Viable Robotic Service to Human, Springer.
47. Ontology based complex object recognition10.1016/j.imavis.2005.07.027 ↗
48. A knowledge-based image retrieval system integrating semantic and visual features10.1016/j.procs.2016.08.188 ↗
49. Lee, S., Naguib, A.M., and Islam, N.U. (2018, January 1–5). 3D deep object recognition and semantic understanding for visually-guided robotic service. Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.10.1109/IROS.2018.8593985 ↗
50. Chen, Y., Gan, W., Zhang, L., Liu, C., and Wang, X. (2017, January 11–12). A Survey on Visual Place Recognition for Mobile Robots Localization. Proceedings of the 2017 14th Web Information Systems and Applications Conference (WISA), Liuzhou, China.10.1109/WISA.2017.7 ↗
51. Schönberger, J.L., Pollefeys, M., Geiger, A., and Sattler, T. (2018, January 18–23). Semantic visual localization. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00721 ↗
52. Garg, S., Suenderhauf, N., and Milford, M. (arXiv, 2018). Lost? appearance-invariant place recognition for opposite viewpoints using visual semantics, arXiv.10.15607/RSS.2018.XIV.022 ↗
53. Chen, Z., Jacobson, A., Sünderhauf, N., Upcroft, B., Liu, L., Shen, C., Reid, I., and Milford, M. (June, January 29). Deep learning features at scale for visual place recognition. Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore, Singapore.10.1109/ICRA.2017.7989366 ↗
54. Gomez-Ojeda, R., Lopez-Antequera, M., Petkov, N., and Gonzalez-Jimenez, J. (arXiv, 2015). Training a convolutional neural network for appearance-invariant place recognition, arXiv.
55. Mitsuhashi, M., and Kuroda, Y. (2011, January 3–7). Mobile robot localization using place recognition in outdoor environments with similar scenes. Proceedings of the 2011 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM)E, Budapest, Hungary.10.1109/AIM.2011.6027041 ↗
56. Learning place-dependant features for long-term vision-based localisation10.1007/s10514-015-9463-y ↗
57. Zhu, J., Li, Q., Cao, R., Sun, K., Liu, T., Garibaldi, J.M., Li, Q., Liu, B., and Qiu, G. (2019). Indoor topological localization using a visual landmark sequence. Remote. Sens., 11.10.3390/rs11010073 ↗
58. Design and recognition of artificial landmarks for reliable indoor self-localization of mobile robots10.1177/1729881417693489 ↗
59. Simmons, R., Goodwin, R., Haigh, K.Z., Koenig, S., and O’Sullivan, J. (1997, January 5–8). A layered architecture for office delivery robots. Proceedings of the first international conference on Autonomous Agents, Marina del Rey, CA, USA.10.1145/267658.267723 ↗
60. Choset, H.M., Hutchinson, S., Lynch, K.M., Kantor, G., Burgard, W., Kavraki, L.E., and Thrun, S. (2005). Principles of Robot Motion: Theory, Algorithms, and Implementation, MIT Press.
61. Buehler, M., Iagnemma, K., and Singh, S. (2009). The DARPA Urban Challenge: Autonomous Vehicles in City Traffic, Springer.10.1007/978-3-642-03991-1 ↗
62. Heuristic approaches in robot path planning: A survey10.1016/j.robot.2016.08.001 ↗
63. Unified framework for path-planning and task-planning for autonomous robots10.1016/j.robot.2016.04.010 ↗
64. STRIPS: A new approach to the application of theorem proving to problem solving10.1016/0004-3702(71)90010-5 ↗
65. Nilsson, N.J. (1984). Shakey the Robot, SRI International. Technical Report.
66. Estivill-Castro, V., and Ferrer-Mestres, J. (2013, January 25–29). Path-finding in dynamic environments with PDDL-planners. Proceedings of the 2013 16th International Conference on Advanced Robotics, ICAR 2013, Montevideo, Uruguay.10.1109/ICAR.2013.6766456 ↗
67. Dvorak, F., Bit-Monnot, A., Ingrand, F., and Ghallab, M. (2014, January 21–26). A Flexible ANML Actor and Planner in Robotics. Proceedings of the Planning and Robotics (PlanRob) Workshop (ICAPS), Portsmouth, NH, USA.
68. Smith, D.E., Frank, J., and Cushing, W. (2008, January 14–18). The ANML language. Proceedings of the ICAPS-08 Workshop on Knowledge Engineering for Planning and Scheduling (KEPS), Sydney, Australia.
69. Boutilier, C., and Dearden, R. (August, January 31). Using abstractions for decision-theoretic planning with time constraints. Proceedings of the AAAI, Seattle, WA, USA.
70. Dean, T.L., Kaelbling, L.P., Kirman, J., and Nicholson, A.E. (1993, January 11–15). Planning With Deadlines in Stochastic Domains. Proceedings of the AAAI, Washington, DC, USA.
71. Robot task planning using semantic maps10.1016/j.robot.2008.08.007 ↗
72. Beetz, M., Beßler, D., Haidu, A., Pomarlan, M., Bozcuoğlu, A.K., and Bartels, G. (2018, January 21–25). Know rob 2.0—A 2nd generation knowledge processing framework for cognition-enabled robotic agents. Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia.10.1109/ICRA.2018.8460964 ↗
73. Beßler, D., Pomarlan, M., and Beetz, M. (2018, January 10–15). Owl-enabled assembly planning for robotic agents. Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS, Stockholm, Sweden.
74. Lau, M., and Kuffner, J.J. (2005, January 29–31). Behavior planning for character animation. Proceedings of the 2005 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Los Angeles, CA, USA.10.1145/1073368.1073408 ↗
75. Koo, T., Hoffmann, F., Shim, H., Sinopoli, B., and Sastry, S. (1998, January 21–23). Hybrid control of an autonomous helicopter. Proceedings of the IFAC Workshop on Motion Control, Grenoble, France.10.1016/S1474-6670(17)40039-5 ↗
76. Experimental results of robotic excavation using fuzzy behavior control10.1016/0967-0661(95)00220-0 ↗
77. Wei, J., Snider, J.M., Gu, T., Dolan, J.M., and Litkouhi, B. (2014, January 8–11). A behavioral planning framework for autonomous driving. Proceedings of the 2014 IEEE Intelligent Vehicles Symposium Proceedings, Dearborn, MI, USA.10.1109/IVS.2014.6856582 ↗
78. Arney, T. (2007, January 4–6). An efficient solution to autonomous path planning by approximate cell decomposition. Proceedings of the 2007 Third International Conference on Information and Automation for Sustainability, Melbourne, Australia.10.1109/ICIAFS.2007.4544785 ↗
79. Lingelbach, F. (May, January 26). Path planning using probabilistic cell decomposition. Proceedings of the IEEE International Conference on Robotics and Automation, 2004. Proceedings ICRA’04, New Orleans, LA, USA.10.1109/ROBOT.2004.1307193 ↗
80. Šeda, M. (2007, January 16–19). Roadmap methods vs. cell decomposition in robot motion planning. Proceedings of the 6th WSEAS International Conference on Signal Processing, Robotics and Automation, Corfu Island, Greece.
81. Potential field method to navigate several mobile robots10.1007/s10489-006-0110-3 ↗
82. Kang, Y.H., Lee, M.C., Kim, C.Y., Yoon, S.M., and Noh, C.B. (2011, January 7–10). A study of cluster robots line formatted navigation using potential field method. Proceedings of the 2011 IEEE International Conference on Mechatronics and Automation, Beijing, China.10.1109/ICMA.2011.5986370 ↗
83. A selective retraction-based RRT planner for various environments10.1109/TRO.2014.2309836 ↗
84. Kavraki, L., and Latombe, J.C. (1994, January 8–13). Randomized preprocessing of configuration for fast path planning. Proceedings of the 1994 IEEE International Conference on Robotics and Automation, San Diego, CA, USA.
85. LaValle, S.M. (1998). Rapidly-Exploring Random trees: A New Tool for Path Planning, Department of Computer Science, Iowa State University. Research Report 98-11.
86. Sampling-based algorithms for optimal motion planning10.1177/0278364911406761 ↗
87. Sampling-based robot motion planning: A review10.1109/ACCESS.2014.2302442 ↗
88. Human-level control through deep reinforcement learning10.1038/nature14236 ↗
89. Shah, P., Fiser, M., Faust, A., Kew, J.C., and Hakkani-Tur, D. (arXiv, 2018). Follownet: Robot navigation by following natural language directions with deep reinforcement learning, arXiv.
90. Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (arXiv, 2015). Continuous control with deep reinforcement learning, arXiv.
91. Tai, L., Paolo, G., and Liu, M. (2017, January 24–28). Virtual-to-real deep reinforcement learning: Continuous control of mobile robots for mapless navigation. Proceedings of the 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, Canada.10.1109/IROS.2017.8202134 ↗
92. Faust, A., Oslund, K., Ramirez, O., Francis, A., Tapia, L., Fiser, M., and Davidson, J. (2018, January 21–25). PRM-RL: Long-range robotic navigation tasks by combining reinforcement learning and sampling-based planning. Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia.10.1109/ICRA.2018.8461096 ↗
93. Cognitive neuroscience of human memory10.1146/annurev.psych.49.1.87 ↗
94. Bechhofer, S., Van Harmelen, F., Hendler, J., Horrocks, I., McGuinness, D.L., Patel-Schneider, P.F., and Stein, L.A. (2004). OWL web ontology language reference. W3C Recomm., 10.02.
95. Rocha, Y.G., Joo, S.H., Kim, E.J., and Kuc, T.Y. (2019, January 15–18). Automatic Generation of a Simulated Robot from an Ontology-Based Semantic Description. Proceedings of the 2019 19th International Conference on Control, Automation and Systems (ICCAS), Jeju, Korea.10.23919/ICCAS47443.2019.8971767 ↗
96. Lin, T.Y., Goyal, P., Girshick, R., He, K., and Dollár, P. (2017, January 22–29). Focal loss for dense object detection. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.324 ↗
97. Coles, A.J., Coles, A.I., Fox, M., and Long, D. (2010, January 12–16). Forward-chaining partial-order planning. Proceedings of the Twentieth International Conference on Automated Planning and Scheduling, Toronto, ON, Canada.
98. A note on two problems in connexion with graphs10.1007/BF01386390 ↗
99. Kahneman, D., and Tversky, A. (1981). The Simulation Heuristic, Department of Psychology, Stanford University. Technical Report.10.1017/CBO9780511809477.015 ↗
100. Mental simulation for autonomous learning and planning based on triplet ontological semantic model