PDDL Planning with Natural Language-Based Scene Understanding for UAV-UGV Cooperation
Source abstract
Abstract
Natural-language-based scene understanding can enable heterogeneous robots to cooperate efficiently in large and unconstructed environments. However, studies on symbolic planning rarely consider the semantic knowledge acquisition problem associated with the surrounding environments. Further, recent developments in deep learning methods show outstanding performance for semantic scene understanding using natural language. In this paper, a cooperation framework that connects deep learning techniques and a symbolic planner for heterogeneous robots is proposed. The framework is largely composed of the scene understanding engine, planning agent, and knowledge engine. We employ neural networks for natural-language-based scene understanding to share environmental information among robots. We then generate a sequence of actions for each robot using a planning domain definition language planner. JENA-TDB is used for knowledge acquisition storage. The proposed method is validated using simulation results obtained from one unmanned aerial and three ground vehicles.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Robotic Path Planning Algorithms · Multimodal Machine Learning Applications · Robotics and Sensor-Based Localization
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Automation and Systems Research Institute, Department of Electrical and Computer Engineering, Seoul National University, 1 Gwanak-ro, Gwanak-gu, Seoul 08826, Korea
Funding
Agency for Defense Development
References (43)
Show source-provided references
1. Multi-task cooperative control in a heterogeneous ground-air robot team10.1016/j.ifacol.2015.06.463 ↗
2. Flexible Production Systems: Automated Generation of Operations Plans based on ISA-95 and PDDL10.1109/LRA.2019.2929991 ↗
3. Toward Affordance Detection and Ranking on Novel Objects for Real-world Robotic Manipulation10.1109/LRA.2019.2930364 ↗
4. Miranda, D.S.S., de Souza, L.E., and Bastos, G.S. (2018, January 6–10). A ROSPlan-Based Multi-Robot Navigation System. Proceedings of the 2018 Workshop on Robotics in Education, Joao Pessoa, Brazil.
5. Zhang, S., Jiang, Y., Sharon, G., and Stone, P. (2017, January 8–12). Multirobot symbolic planning under temporal uncertainty. Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, Sao Paulo, Brazil.
6. Communication-Efficient Planning and Mapping for Multi-Robot Exploration in Large Environments10.1109/LRA.2019.2897368 ↗
7. Bowman, S.L., Atanasov, N., Daniilidis, K., and Pappas, G.J. (June, January 29). Probabilistic data association for semantic slam. Proceedings of the International Conference on Robotics and Automation, Singapore.
8. Recent trends in deep learning based natural language processing10.1109/MCI.2018.2840738 ↗
9. Aneja, J., Deshpande, A., and Schwing, A.G. (2018, January 18–22). Convolutional image captioning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00583 ↗
10. Semantic SLAM Based on Object Detection and Improved Octomap10.1109/ACCESS.2018.2873617 ↗
11. Karpathy, A., and Fei-Fei, L. (2015, January 7–12). Deep visual-semantic alignments for generating image descriptions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298932 ↗
12. Yao, T., Pan, Y., Li, Y., and Mei, T. (2018, January 8–14). Exploring visual relationship for image captioning. Proceedings of the European Conference on Computer Vision, Munich, Germany.10.1007/978-3-030-01264-9_42 ↗
13. A framework for learning semantic maps from grounded natural language descriptions10.1177/0278364914537359 ↗
14. Johnson, J., Krishna, R., Stark, M., Li, L.J., Shamma, D., Bernstein, M., and Li, F.-F. (2015, January 7–12). Image retrieval using scene graphs. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298990 ↗
15. Ma, L., Lu, Z., and Li, H. (2016, January 12–17). Learning to answer questions from image using convolutional neural network. Proceedings of the AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.10.1609/aaai.v30i1.10442 ↗
16. Garnelo, M., Arulkumaran, K., and Shanahan, M. (2016). Towards deep symbolic reinforcement learning. arXiv.
17. Asai, M., and Fukunaga, A. (2018, January 2–7). Classical planning in deep latent space: Bridging the subsymbolic-symbolic boundary. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.12077 ↗
18. Mao, J., Gan, C., Kohli, P., Tenenbaum, J.B., and Wu, J. (2019). The Neuro-Symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. arXiv.
19. Cashmore, M., Fox, M., Long, D., Magazzeni, D., Ridder, B., Carrera, A., Palomeras, N., Hurtos, N., and Carreras, M. (2015, January 7–11). Rosplan: Planning in the robot operating system. Proceedings of the Twenty-Fifth International Conference on Automated Planning and Scheduling, Jerusalem, Israel.10.1609/icaps.v25i1.13699 ↗
20. Gautam, A., and Mohan, S. (2012, January 6–9). A review of research in multi-robot systems. Proceedings of the IEEE 7th International Conference on Industrial and Information Systems, Chennai, India.10.1109/ICIInfS.2012.6304778 ↗
21. Coordinating heterogeneous teams of robots using temporal symbolic planning10.1007/s10514-012-9320-1 ↗
22. An integrated decision-making framework of a heterogeneous aerial robotic swarm for cooperative tasks with minimum requirements
23. Kingry, N., Liu, Y.C., Martinez, M., Simon, B., Bang, Y., and Dai, R. (2017, January 24–28). Mission planning for a multi-robot team with a solar-powered charging station. Proceedings of the International Conference on Intelligent Robots and Systems, Vancouver, BC, Canada.10.1109/IROS.2017.8206413 ↗
24. Reis, J.C., Lima, P.U., and Garcia, J. (2013). Efficient distributed communications for multi-robot systems. Robot Soccer World Cup, Springer.
25. Jiang, J., and Lu, Z. (2018). Learning attentional communication for multi-agent cooperation. Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems, Montreal, QC, Canada, 2–8 December 2018, Neural Information Processing Systems.
26. Foerster, J., Assael, I.A., de Freitas, N., and Whiteson, S. (2016). Learning to communicate with deep multi-agent reinforcement learning. Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems, Barcelona, Spain, 5–10 December 2016, Neural Information Processing Systems.
27. Semantic SLAM for an AUV using object recognition from point clouds10.1016/j.ifacol.2018.09.497 ↗
28. Li, L., Liu, Z., Özgïner, Ü., Lian, J., Zhou, Y., and Zhao, Y. (2018, January 26–30). Dense 3D Semantic SLAM of traffic environment based on stereo vision. Proceedings of the 2018 IEEE Intelligent Vehicles Symposium, Changshu, China.10.1109/IVS.2018.8500714 ↗
29. SEMANTIC-RTAB-MAP (SRM): A semantic SLAM system with CNNs on depth images10.3934/mfc.2019003 ↗
30. Lu, J., Xiong, C., Parikh, D., and Socher, R. (2017, January 21–26). Knowing when to look: Adaptive attention via a visual sentinel for image captioning. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.345 ↗
31. Lu, J., Yang, J., Batra, D., and Parikh, D. (2016). Hierarchical question-image co-attention for visual question answering. Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems, Barcelona, Spain, 5–10 December 2016, Neural Information Processing Systems.
32. Dai, B., Zhang, Y., and Lin, D. (2017, January 21–26). Detecting visual relationships with deep relational networks. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.352 ↗
33. Srivastava, S., Fang, E., Riano, L., Chitnis, R., Russell, S., and Abbeel, P. (June, January 31). Combined task and motion planning through an extensible planner-independent interface layer. Proceedings of the IEEE International Conference on Robotics and Automation, Hong Kong, China.
34. Dornhege, C., Hertle, A., and Nebel, B. (2013, January 6–10). Lazy evaluation and subsumption caching for search-based integrated task and motion planning. Proceedings of the International Conference on Robotics and Automation Workshop on AI-Based Robotics, Karlsruhe, Germany.
35. Manso, L.J., Bustos, P., Alami, R., Milliez, G., and Núnez, P. (2015, January 5). Planning human-robot interaction tasks using graph models. Proceedings of the International Workshop on Recognition and Action for Scene Understanding, Valletta, Malta.
36. Zhou, Y., and Tuzel, O. (2018, January 18–22). Voxelnet: End-to-end learning for point cloud based 3d object detection. Proceedings of the IEEE international conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00472 ↗
37. Multi-scale deep reinforcement learning for real-time 3D-landmark detection in CT scans10.1109/TPAMI.2017.2782687 ↗
38. Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., and Xiao, J. (2015, January 8–10). 3D shapenets: A deep representation for volumetric shapes. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.
39. Zhang, Q., and Sornette, D. (2017). Learning like humans with Deep Symbolic Networks. arXiv.
40. Liao, Q., and Poggio, T. (2017). Object-Oriented Deep Learning, Center for Brains, Minds and Machines.
41. Scene understanding using natural language description based on 3D semantic graph map10.1007/s11370-018-0257-x ↗
42. Xu, D., Zhu, Y., Choy, C.B., and Li, F.-F. (2017, January 22–29). Scene graph generation by iterative message passing. Proceedings of the IEEE international conference on Computer Vision and Pattern Recognition, Venice, Italy.10.1109/CVPR.2017.330 ↗
43. Simonyan, K., and Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv.