3D Position Estimation of Objects for Inventory Management Automation Using Drones
Source abstract
Abstract
With the recent development of drone technology, drones are being used in various fields. Drones have the advantage of being equipped with various devices to move freely and perform various tasks. In the field of inventory management, many studies have been conducted into management automation based on the drone. Drones scan a marker, such as a quick response code (QR code), attached to the shelves to obtain location information of the shelves on which the inventory is loaded. At the same time, drones perform inventory management by scanning the marker attached to the inventory to obtain inventory information. However, unlike indoor warehouses, where grids or shelves are well-defined, a storage yard is not fixed in the location where the inventory is stored. It is difficult to recognize the loading position from the marker for a storage yard without shelves and grids. Furthermore, the loading position of the inventory is not fixed. For the automation of inventory management of warehouses where shelves and grids are undefined, this paper proposes a framework that estimates the inventory 3D position in the video frame based on a deep learning model. The proposed framework uses the image segmentation model to detect and decode the marker in the video frame to estimate the 3D position of a drone and inventory. In addition, the estimated inventory 3D position is corrected using the continuity of the video frame. Experiment results on the video dataset verified that the proposed framework improved the 3D position estimation performance of the inventory. Consequently, efficient inventory management based on drones can be performed through the proposed framework for the 3D position estimation of inventory in all types of warehouses.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Robotics and Sensor-Based Localization · Smart Parking Systems Research · UAV Applications and Optimization
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Industrial and Systems Engineering, Dongguk University, Seoul 04620, Republic of Korea
Funding
Korea Institute for Advancement of Technology
References (62)
Show source-provided references
1. Yoon, B., Kim, H., Youn, G., and Rhee, J. (2021, January 25–27). 3D position estimation of drone and object based on QR code segmentation model for inventory management automation. Proceedings of the 2021 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), New York, NY, USA.10.1109/SSRR53300.2021.9597865 ↗
2. Saggiani, G., Persiani, F., Ceruti, A., Tortora, P., Troiani, E., Giuletti, F., Amici, S., Buongiorno, M., Distefano, G., and Bentini, G. (2023, September 26). A UAV System for Observing Volcanoes and Natural Hazards. American Geophysical Union, Fall Meeting 2007, Abstract ID. GC11B-05, 2007. Available online: https://ui.adsabs.harvard.edu/abs/2007AGUFMGC11B..05S/abstract.
3. Efficient routing for precedence-constrained package delivery for heterogeneous vehicles10.1109/TASE.2019.2914113 ↗
4. Unmanned aerial vehicles (UAVs): A survey on civil applications and key research challenges10.1109/ACCESS.2019.2909530 ↗
5. Rhiat, A., Chalal, L., and Saadane, A. (2021, January 28–29). A Smart Warehouse Using Robots and Drone to Optimize Inventory Management. Proceedings of the Future Technologies Conference, Vancouver, BC, Canada.10.1007/978-3-030-89906-6_32 ↗
6. Radácsi, L., Gubán, M., Szabó, L., and Udvaros, J. (2022). A Path Planning Model for Stock Inventory Using a Drone. Mathematics, 10.10.3390/math10162899 ↗
7. Gubán, M., and Udvaros, J. (2022). A Path Planning Model with a Genetic Algorithm for Stock Inventory Using a Swarm of Drones. Drones, 6.10.3390/drones6110364 ↗
8. Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras10.1109/TRO.2017.2705103 ↗
9. Gao, X., Wang, R., Demmel, N., and Cremers, D. (2018, January 1–5). LDSO: Direct sparse odometry with loop closure. Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.10.1109/IROS.2018.8593376 ↗
10. Szeliski, R. (2010). Computer Vision: Algorithms and Applications, Springer Science & Business Media.10.1007/978-1-84882-935-0 ↗
11. Places: A 10 million Image Database for Scene Recognition10.1109/TPAMI.2017.2723009 ↗
12. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20–25). Imagenet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
13. Cho, H., Kim, D., Park, J., Roh, K., and Hwang, W. (2018, January 26–30). 2D barcode detection using images for drone-assisted inventory management. Proceedings of the 2018 15th International Conference on Ubiquitous Robots (UR), Honolulu, HI, USA.10.1109/URAI.2018.8441834 ↗
14. Ramaswamy, A., Gubbi, J., Raj, R., and Purushothaman, B. (2018, January 7–10). Frame stitching in indoor environment using drone captured images. Proceedings of the 2018 25th IEEE International Conference on Image Processing (ICIP), Athens, Greece.10.1109/ICIP.2018.8451671 ↗
15. Anand, A., Agrawal, S., Agrawal, S., Chandra, A., and Deshmukh, K. (2019). Grid-based localization stack for inspection drones towards automation of large scale warehouse systems. arXiv.
16. Almalki, F.A. (2020, January 12–15). Utilizing Drone for Food Quality and Safety Detection using Wireless Sensors. Proceedings of the 2020 IEEE 3rd International Conference on Information Communication and Signal Processing (ICICSP), Shanghai, China.10.1109/ICICSP50920.2020.9232046 ↗
17. ReLoc 2.0: UHF-RFID Relative Localization for Drone-Based Inventory Management
18. Cristiani, D., Bottonelli, F., Trotta, A., and Di Felice, M. (September, January 31). Inventory Management through Mini-Drones: Architecture and Proof-of-Concept Implementation. Proceedings of the 2020 IEEE 21st International Symposium on “A World of Wireless, Mobile and Multimedia Networks” (WoWMoM), Cork, Ireland.10.1109/WoWMoM49955.2020.00060 ↗
19. Manjrekar, A., Jha, D., Jagtap, P., and Yadav, V. (2021, January 27–29). Warehouse Inventory Management with Cycle Counting Using Drones. Proceedings of the 4th International Conference on Advances in Science and Technology (ICAST2021), Bahir Dar, Ethiopia.10.2139/ssrn.3869512 ↗
20. Martinez-Martin, E., Ferrer, E., Vasilev, I., and Del Pobil, A.P. (2021). The UJI Aerial Librarian Robot: A Quadcopter for Visual Library Inventory and Book Localisation. Sensors, 21.10.3390/s21041079 ↗
21. Blanger, L., and Hirata, N.S. (2019, January 22–25). An evaluation of deep learning techniques for QR code detection. Proceedings of the 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan.10.1109/ICIP.2019.8803075 ↗
22. Yuan, B., Li, Y., Jiang, F., Xu, X., Zhao, J., Zhang, D., Guo, J., Wang, Y., and Zhang, S. (2019, January 26–29). Fast QR code detection based on BING and AdaBoost-SVM. Proceedings of the 2019 IEEE 20th International Conference on High Performance Switching and Routing (HPSR), Xi’an, China.10.1109/HPSR.2019.8808000 ↗
23. Cheng, M.M., Zhang, Z., Lin, W.Y., and Torr, P. (2014, January 23–28). BING: Binarized normed gradients for objectness estimation at 300fps. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.414 ↗
24. Yuan, B., Li, Y., Jiang, F., Xu, X., Guo, Y., Zhao, J., Zhang, D., Guo, J., and Shen, X. (2019). MU R-CNN: A two-dimensional code instance segmentation network based on deep learning. Future Internet, 11.10.3390/fi11090197 ↗
25. He, K., Gkioxari, G., Dollár, P., and Girshick, R. (2017, January 22–29). Mask r-cnn. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.322 ↗
26. Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5–9). U-net: Convolutional networks for biomedical image segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany.10.1007/978-3-319-24574-4_28 ↗
27. Li, K., Meng, F., Huang, Z., and Wang, Q. (2019). A correction algorithm of QR code on cylindrical surface. J. Phys. Conf. Ser., 1237.10.1088/1742-6596/1237/2/022006 ↗
28. Peng, L., Wen, L., Qiang, L., Min, D., Yue, D., and Yiying, N. (2020). Advances in 3D Image and Graphics Representation, Analysis, Computing and Information Technology, Springer.
29. Choi, J.H., and Choi, B.J. (2017, January 17–20). Design of self-localization based autonomous driving platform for an electric wheelchair. Proceedings of the 2017 11th Asian Control Conference (ASCC), Gold Coast, QLD, Australia.10.1109/ASCC.2017.8287214 ↗
30. Li, Z., and Huang, J. (2018, January 23–26). Study on the use of QR codes as landmarks for indoor positioning: Preliminary results. Proceedings of the 2018 IEEE/ION Position, Location and Navigation Symposium (PLANS), Monterey, CA, USA.10.1109/PLANS.2018.8373516 ↗
31. Lee, D.H., Lee, S.S., Kang, H.H., and Ahn, C.K. (2018, January 15–17). Camera Position Estimation for UAVs Using SolvePnP with Kalman Filter. Proceedings of the 2018 1st IEEE International Conference on Hot Information-Centric Networking (HotICN), Shenzhen, China.10.1109/HOTICN.2018.8606037 ↗
32. Taketani, R., and Kobayashi, H. (2019, January 14–17). A Proposal for Improving Estimation Accuracy of Localization Using QR codes and Image Sensors. Proceedings of the IECON 2019—45th Annual Conference of the IEEE Industrial Electronics Society, Lisbon, Portugal.10.1109/IECON.2019.8927589 ↗
33. Pan, G., Liang, A., Liu, J., Liu, M., and Wang, E.X. (2020, January 20–22). 3-D Positioning System Based QR Code and Monocular Vision. Proceedings of the 2020 5th International Conference on Robotics and Automation Engineering (ICRAE), Singapore.10.1109/ICRAE50850.2020.9310908 ↗
34. Kim, J.I., Gang, H.S., Pyun, J.Y., and Kwon, G.R. (2021). Implementation of QR Code Recognition Technology Using Smartphone Camera for Indoor Positioning. Energies, 14.10.3390/en14102759 ↗
35. Kang, T.W., Choi, Y.S., and Jung, J.W. (2022, January 9–12). Estimation of Relative Position of Drone using Fixed Size QR Code. Proceedings of the 2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Prague, Czech Republic.10.1109/SMC53654.2022.9945142 ↗
36. Long, J., Shelhamer, E., and Darrell, T. (2015, January 7–12). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298965 ↗
37. Image segmentation using deep learning: A survey
38. Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., and Yuille, A.L. (2014). Semantic image segmentation with deep convolutional nets and fully connected crfs. arXiv.
39. Liu, Z., Li, X., Luo, P., Loy, C.C., and Tang, X. (2015, January 7–13). Semantic image segmentation via deep parsing network. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.162 ↗
40. Zhou, Z., Siddiquee, M.M.R., Tajbakhsh, N., and Liang, J. (2018). Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, Springer.
41. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs10.1109/TPAMI.2017.2699184 ↗
42. Chen, L.C., Papandreou, G., Schroff, F., and Adam, H. (2017). Rethinking atrous convolution for semantic image segmentation. arXiv.
43. Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., and Adam, H. (2018, January 8–14). Encoder-decoder with atrous separable convolution for semantic image segmentation. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01234-2_49 ↗
44. Zhao, H., Shi, J., Qi, X., Wang, X., and Jia, J. (2017, January 21–26). Pyramid scene parsing network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.660 ↗
45. Khan, S.D., Alarabi, L., and Basalamah, S. (2021). Deep hybrid network for land cover semantic segmentation in high-spatial resolution satellite images. Information, 12.10.3390/info12060230 ↗
46. Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K.Q. (2017, January 21–26). Densely connected convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.243 ↗
47. DSMSA-Net: Deep Spatial and Multi-scale Attention Network for Road Extraction in High Spatial Resolution Satellite Images10.1007/s13369-022-07082-z ↗
48. An encoder–decoder deep learning framework for building footprints extraction from aerial imagery10.1007/s13369-022-06768-8 ↗
49. Algorithms for the reduction of the number of points required to represent a digitized line or its caricature
50. A simple, robust and fast method for the perspective-n-point problem10.1016/j.patrec.2018.02.028 ↗
51. Complete solution classification for the perspective-three-point problem10.1109/TPAMI.2003.1217599 ↗
52. Hesch, J.A., and Roumeliotis, S.I. (2011, January 6–13). A direct least-squares (DLS) method for PnP. Proceedings of the 2011 International Conference on Computer Vision, Barcelona, Spain.10.1109/ICCV.2011.6126266 ↗
53. Epnp: An accurate o (n) solution to the pnp problem10.1007/s11263-008-0152-6 ↗
54. Ester, M., Kriegel, H.P., Sander, J., and Xu, X. (1996, January 2–4). A density-based algorithm for discovering clusters in large spatial databases with noise. Proceedings of the Second International Conference on Knowledge Discovery and Data Mining (KDD-96), Portland, OR, USA.
55. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
56. Chollet, F. (2017, January 21–26). Xception: Deep learning with depthwise separable convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.195 ↗
57. Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.
58. Milletari, F., Navab, N., and Ahmadi, S.A. (2016, January 25–28). V-net: Fully convolutional neural networks for volumetric medical image segmentation. Proceedings of the 2016 Fourth International Conference on 3D Vision (3DV), Stanford, CA, USA.10.1109/3DV.2016.79 ↗
59. Kim, D., Ka, W., Ahn, P., Joo, D., Chun, S., and Kim, J. (2022). Global-local path networks for monocular depth estimation with vertical cutdepth. arXiv.
60. A hybrid CNN for image denoising
61. Deep generative inpainting with comparative sample augmentation
62. Scale adaptive fitness evaluation-based particle swarm optimisation for hyperparameter and architecture optimisation in neural networks and deep learning10.1049/cit2.12106 ↗