Staircase Detection, Characterization and Approach Pipeline for Search and Rescue Robots
Source abstract
Abstract
Currently, most rescue robots are mainly teleoperated and integrate some level of autonomy to reduce the operator’s workload, allowing them to focus on the primary mission tasks. One of the main causes of mission failure are human errors and increasing the robot’s autonomy can increase the probability of success. For this reason, in this work, a stair detection and characterization pipeline is presented. The pipeline is tested on a differential drive robot using the ROS middleware, YOLOv4-tiny and a region growing based clustering algorithm. The pipeline’s staircase detector was implemented using the Neural Compute Engines (NCEs) of the OpenCV AI Kit with Depth (OAK-D) RGB-D camera, which allowed the implementation using the robot’s computer without a GPU and, thus, could be implemented in similar robots to increase autonomy. Furthermore, by using this pipeline we were able to implement a Fuzzy controller that allows the robot to align itself, autonomously, with the staircase. Our work can be used in different robots running the ROS middleware and can increase autonomy, allowing the operator to focus on the primary mission tasks. Furthermore, due to the design of the pipeline, it can be used with different types of RGB-D cameras, including those that generate noisy point clouds from low disparity depth images.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Robotics and Sensor-Based Localization · Advanced Neural Network Applications · Robotic Path Planning Algorithms
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Graduate Studies Division, Technological University of the Mixteca, Km. 2.5 Carretera a Acatlima, Huajuapan de León 69000, Oaxaca, Mexico
Department of Mechanical Engineering, Faculty of Science and Technology, Tokyo University of Science, 2641 Yamazaki, Noda, Chiba 278-8510, Japan
References (35)
Show source-provided references
1. Murphy, R.R. (2014). Disaster Robotics, MIT Press.10.7551/mitpress/9407.001.0001 ↗
2. Tadokoro, S. (2019). Disaster Robotics: Results from the ImPACT Tough Robotics Challenge, Springer.10.1007/978-3-030-05321-5 ↗
3. Cong, Y., Li, X., Liu, J., and Tang, Y. (2008, January 6–8). A stairway detection algorithm based on vision for ugv stair climbing. Proceedings of the 2008 IEEE International Conference on Networking, Sensing and Control, Sanya, China.10.1109/ICNSC.2008.4525517 ↗
4. Hernández, D.C., and Jo, K.H. (2011, January 9–11). Stairway tracking based on automatic target selection using directional filters. Proceedings of the 2011 17th Korea-Japan Joint Workshop on Frontiers of Computer Vision (FCV), Ulsan, Korea.
5. Huang, X., and Tang, Z. (2018, January 25–27). Staircase Detection Algorithm Based on Projection-Histogram. Proceedings of the 2018 2nd IEEE Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC), Xi’an, China.10.1109/IMCEC.2018.8469186 ↗
6. Lee, H.W., Wang, C., and Lu, B.Y. (2019, January 20–22). Study on the Computer Vision of the Biped Robot for Stable Walking on the Stairs. Proceedings of the 2019 IEEE International Conference on Consumer Electronics-Taiwan (ICCE-TW), Yilan, Taiwan.10.1109/ICCE-TW46550.2019.8991808 ↗
7. Zhong, C., Zhuang, Y., and Wang, W. (2011, January 14–16). Stairway detection using Gabor filter and FFPG. Proceedings of the 2011 International Conference of Soft Computing and Pattern Recognition (SoCPaR), Dalian, China.10.1109/SoCPaR.2011.6089334 ↗
8. Kajabad, E.N., Begen, P., Nizomutdinov, B., and Ivanov, S. (2021, January 27–29). YOLOv4 for Urban Object Detection: Case of Electronic Inventory in St. Petersburg. In Proceedings of the 2021 28th Conference of Open Innovations Association (FRUCT), Moscow, Russia.10.23919/FRUCT50888.2021.9347622 ↗
9. Patil, U., Gujarathi, A., Kulkarni, A., Jain, A., Malke, L., Tekade, R., Paigwar, K., and Chaturvedi, P. (2019, January 25–27). Deep learning based stair detection and statistical image filtering for autonomous stair climbing. Proceedings of the 2019 Third IEEE International Conference on Robotic Computing (IRC), Naples, Italy.10.1109/IRC.2019.00031 ↗
10. Ilyas, M., Lakshmanan, A.K., Le, A.V., and Mohan, R.E. (2018). Staircase recognition and localization using convolution neural network (cnn) for cleaning robot application. Preprints.10.20944/preprints201812.0296.v1 ↗
11. Automatic estimation of the position and orientation of stairs to be reached and climbed by a disaster response robot by analyzing 2D image and 3D point cloud
12. Choi, Y.J., Rahim, T., Ramatryana, I.N.A., and Shin, S.Y. (February, January 31). Improved CNN-Based Path Planning for Stairs Climbing in Autonomous UAV with LiDAR Sensor. Proceedings of the 2021 International Conference on Electronics, Information, and Communication (ICEIC), Jeju, Korea.10.1109/ICEIC51217.2021.9369805 ↗
13. Deep Learning-Based Stair Segmentation and Behavioral Cloning for Autonomous Stair Climbing10.1142/S1793351X1940021X ↗
14. Ciobanu, A., Morar, A., Moldoveanu, F., Petrescu, L., Ferche, O., and Moldoveanu, A. (2017, January 29–31). Real-time indoor staircase detection on mobile devices. Proceedings of the 2017 21st International Conference on Control Systems and Computer Science (CSCS), Bucharest, Romania.10.1109/CSCS.2017.46 ↗
15. Sharma, B., and Syed, I.A. (2019, January 3–4). Where to begin climbing? Computing start-of-stair position for robotic platforms. Proceedings of the 2019 11th International Conference on Computational Intelligence and Communication Networks (CICN), Honolulu, HI, USA.10.1109/CICN.2019.8902461 ↗
16. Souto, L.A., Castro, A., Gonçalves, L.M.G., and Nascimento, T.P. (2017). Stairs and doors recognition as natural landmarks based on clouds of 3D edge-points from RGB-D sensors for mobile robot localization. Sensors, 17.10.3390/s17081824 ↗
17. Woo, S., Shin, J., Lee, Y.H., Lee, Y.H., Lee, H., Kang, H., Choi, H.R., and Moon, H. (2019, January 24–27). Stair-mapping with point-cloud data and stair-modeling for quadruped robot. Proceedings of the 2019 16th International Conference on Ubiquitous Robots (UR), Jeju, Korea.10.1109/URAI.2019.8768786 ↗
18. Robust stairway-detection and localization method for mobile robots using a graph-based model and competing initializations10.1177/0278364918798039 ↗
19. The open images dataset v410.1007/s11263-020-01316-z ↗
20. MCIndoor20000: A fully-labeled image dataset to advance indoor objects detection10.1016/j.dib.2017.12.047 ↗
21. Google (2020, December 10). Google Images. Available online: https://www.google.com/imghp?hl=en.
22. Tzutalin (2021, January 07). LabelImg. Git code., Available online: https://github.com/tzutalin/labelImg.
23. Bochkovskiy, A., Wang, C.Y., and Liao, H.Y.M. (2020). Yolov4: Optimal speed and accuracy of object detection. arXiv.
24. Wang, C.Y., Liao, H.Y.M., Wu, Y.H., Chen, P.Y., Hsieh, J.W., and Yeh, I.H. (2020, January 14–19). CSPNet: A new backbone that can enhance learning capability of CNN. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Seattle, WA, USA.10.1109/CVPRW50498.2020.00203 ↗
25. Spatial pyramid pooling in deep convolutional networks for visual recognition10.1109/TPAMI.2015.2389824 ↗
26. Liu, S., Qi, L., Qin, H., Shi, J., and Jia, J. (2018, January 18–23). Path aggregation network for instance segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00913 ↗
27. Redmon, J., and Farhadi, A. (2018). Yolov3: An incremental improvement. arXiv.
28. Segmentation of point clouds using smoothness constraint
29. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography10.1145/358669.358692 ↗
30. Grossman, S.I. (2012). Álgebra Lineal, McGraw Hill/Interamericana Editores, S.A. de C.V.. [7th ed.].
31. Luxonis (2021, August 01). Depthai-Ros. Git Code., Available online: https://github.com/luxonis/depthai-ros.
32. ROS (2021, July 17). Depth_img_proc. Package Summary., Available online: http://wiki.ros.org/depth_image_proc.
33. Rusu, R.B., and Cousins, S. (2011, January 9–13). 3d is here: Point cloud library (pcl). Proceedings of the 2011 IEEE International Conference on Robotics and Automation, Shanghai, China.10.1109/ICRA.2011.5980567 ↗
34. Gazebo (2021, September 01). Robot Simulation Made Easy. Available online: http://gazebosim.org/.
35. AlexeyAB (2021, August 15). Darknet. Git Code. Available online: https://github.com/AlexeyAB/darknet.