A Review on Sound Source Localization in Robotics: Focusing on Deep Learning Methods
Source abstract
Abstract
Sound source localization (SSL) adds a spatial dimension to auditory perception, allowing a system to pinpoint the origin of speech, machinery noise, warning tones, or other acoustic events, capabilities that facilitate robot navigation, human–machine dialogue, and condition monitoring. While existing surveys provide valuable historical context, they typically address general audio applications and do not fully account for robotic constraints or the latest advancements in deep learning. This review addresses these gaps by offering a robotics-focused synthesis, emphasizing recent progress in deep learning methodologies. We start by reviewing classical methods such as time difference of arrival (TDOA), beamforming, steered-response power (SRP), and subspace analysis. Subsequently, we delve into modern machine learning (ML) and deep learning (DL) approaches, discussing traditional ML and neural networks (NNs), convolutional neural networks (CNNs), convolutional recurrent neural networks (CRNNs), and emerging attention-based architectures. The data and training strategy that are the two cornerstones of DL-based SSL are explored. Studies are further categorized by robot types and application domains to facilitate researchers in identifying relevant work for their specific contexts. Finally, we highlight the current challenges in SSL works in general, regarding environmental robustness, sound source multiplicity, and specific implementation constraints in robotics, as well as data and learning strategies in DL-based SSL. Also, we sketch promising directions to offer an actionable roadmap toward robust, adaptable, efficient, and explainable DL-based SSL for next-generation robots.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech and Audio Processing · Vehicle Noise and Vibration Control · Music and Audio Processing
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Management, Economics and Industrial Engineering, Politecnico di Milano, Via Lambruschini 24/b, 20156 Milan, Italy
Department of Information and Communications Engineering, Aalto University, 02150 Espoo, Finland
Department of Materials and Mechanical Engineering, University of Turku, 20014 Turku, Finland
Department of Electrical and Computer Engineering, University of Victoria, Victoria, BC V8P 5C2, Canada
References (100+)
Show source-provided references
1. Localization of sound sources in robotics: A review10.1016/j.robot.2017.07.011 ↗
2. Jo, H.M., Kim, T.W., and Kwak, K.C. (2025). Sound Source Localization Using Deep Learning for Human–Robot Interaction Under Intelligent Robot Environments. Electronics, 14.10.3390/electronics14051043 ↗
3. Design and implementation of the voice command recognition and the sound source localization system for human–robot interaction10.1017/S0263574720001496 ↗
4. Jalayer, R., Jalayer, M., Orsenigo, C., and Vercellis, C. (2023, January 18–22). A Conceptual Framework for Localization of Active Sound Sources in Manufacturing Environment Based on Artificial Intelligence. Proceedings of the 33rd International Conference on Flexible Automation and Intelligent Manufacturing (FAIM 2023), Porto, Portugal.10.1007/978-3-031-38241-3_78 ↗
5. An Overview of Sound Source Localization based Condition Monitoring Robots10.1016/j.isatra.2024.12.046 ↗
6. Lv, D., Feng, G., Zhen, D., Liang, X., Sun, G., and Gu, F. (2024, January 31). Motor Bearing Fault Source Localization Based on Sound and Robot Movement Characteristics. Proceedings of the 2024 International Conference on Sensing, Measurement & Data Analytics in the era of Artificial Intelligence (ICSMD), Huangshan, China.10.1109/ICSMD64214.2024.10920535 ↗
7. Marques, I., Sousa, J., Sá, B., Costa, D., Sousa, P., Pereira, S., Santos, A., Lima, C., Hammerschmidt, N., and Pinto, S. (2022). Microphone array for speaker localization and identification in shared autonomous vehicles. Electronics, 11.10.20944/preprints202201.0399.v1 ↗
8. Yamada, T., Itoyama, K., Nishida, K., and Nakadai, K. (2020, January 12–15). Sound source tracking by drones with microphone arrays. Proceedings of the 2020 IEEE/SICE International Symposium on System Integration (SII), Honolulu, HI, USA.10.1109/SII46433.2020.9026185 ↗
9. Yamamoto, T., Hoshiba, K., Yen, B., and Nakadai, K. (2024, January 3–6). Implementation of a Robot Operation System-based network for sound source localization using multiple drones. Proceedings of the 2024 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), Macau, China.10.1109/APSIPAASC63619.2025.10849321 ↗
10. Sound localization sensors for search and rescue biobots10.1109/JSEN.2015.2477443 ↗
11. Zhang, B., Masahide, K., and Lim, H. (2019, January 7–9). Sound source localization and interaction based human searching robot under disaster environment. Proceedings of the 2019 SICE International Symposium on Control Systems (SICE ISCS), Kumamoto, Japan.10.23919/SICEISCS.2019.8758766 ↗
12. Mae, N., Mitsui, Y., Makino, S., Kitamura, D., Ono, N., Yamada, T., and Saruwatari, H. (2017, January 12–15). Sound source localization using binaural difference for hose-shaped rescue robot. Proceedings of the 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), Kuala Lumpur, Malaysia.10.1109/APSIPA.2017.8282292 ↗
13. Park, J.H., and Sim, K.B. (2008, January 14–17). A design of mobile robot based on Network Camera and sound source localization for intelligent surveillance system. Proceedings of the 2008 International Conference on Control, Automation and Systems, Seoul, Republic of Korea.
14. Research on sound source localization and real-time facial expression recognition for security robot
15. A review of indoor localization techniques and wireless technologies10.1007/s11277-021-08209-5 ↗
16. Vision-based robotic person following under light variations and difficult walking maneuvers10.1002/rob.20348 ↗
17. Hall, D., Talbot, B., Bista, S.R., Zhang, H., Smith, R., Dayoub, F., and Sünderhauf, N. (2020). The robotic vision scene understanding challenge. arXiv.
18. Real-time lidar-based localization of mobile ground robot10.1016/j.procs.2021.04.164 ↗
19. Yu, Z. (2024, January 26–28). A WiFi Indoor Localization System Based on Robot Data Acquisition and Deep Learning Model. Proceedings of the 2024 6th International Conference on Internet of Things, Automation and Artificial Intelligence (IoTAAI), Guangzhou, China.10.1109/IoTAAI62601.2024.10692494 ↗
20. Indoor positioning system: A review
21. Performance comparison of three types of sensor matrices for indoor multi-robot localization
22. Flynn, A.M., Brooks, R.A., Wells, W.M., and Barrett, D.S. (1989). Squirt: The Prototypical Mobile Robot for Autonomous Graduate Students, Massachusetts Institute of Technology.10.21236/ADA212337 ↗
23. A survey of sound source localization with deep learning methods10.1121/10.0011809 ↗
24. Liaquat, M.U., Munawar, H.S., Rahman, A., Qadir, Z., Kouzani, A.Z., and Mahmud, M.P. (2021). Localization of sound sources: A systematic review. Energies, 14.10.3390/en14133910 ↗
25. A review on sound source localization systems10.1007/s11831-022-09747-2 ↗
26. Nonverbal sound in human–robot interaction: A systematic review10.1145/3583743 ↗
27. Jekateryńczuk, G., and Piotrowski, Z. (2023). A survey of sound source localization and detection methods and their applications. Sensors, 24.10.3390/s24010068 ↗
28. A Review on Recent Advances in Sound Source Localization Techniques, Challenges, and Applications10.1016/j.snr.2025.100313 ↗
29. He, W. (2021). Deep Learning Approaches for Auditory Perception in Robotics. [Ph.D. Thesis, EPFL].
30. Youssef, K., Argentieri, S., and Zarader, J.L. (2013, January 3–7). A learning-based approach to robust binaural sound localization. Proceedings of the 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, Tokyo, Japan.10.1109/IROS.2013.6696771 ↗
31. Nakamura, K., Gomez, R., and Nakadai, K. (2013, January 3–7). Real-time super-resolution three-dimensional sound source localization for robots. Proceedings of the 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, Tokyo, Japan.10.1109/IROS.2013.6696921 ↗
32. Ohata, T., Nakamura, K., Mizumoto, T., Taiki, T., and Nakadai, K. (2014, January 14–18). Improvement in outdoor sound source detection using a quadrotor-embedded microphone array. Proceedings of the 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, Chicago, IL, USA.10.1109/IROS.2014.6942813 ↗
33. Grondin, F., and Michaud, F. (October, January 28). Time difference of arrival estimation based on binary frequency mask for sound source localization on mobile robots. Proceedings of the 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Hamburg, Germany.10.1109/IROS.2015.7354253 ↗
34. Nakamura, K., Sinapayen, L., and Nakadai, K. (October, January 28). Interactive sound source localization using robot audition for tablet devices. Proceedings of the 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Hamburg, Germany.10.1109/IROS.2015.7354251 ↗
35. Li, X., Girin, L., Badeig, F., and Horaud, R. (2016, January 9–14). Reverberant sound localization with a robot head based on direct-path relative transfer function. Proceedings of the 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, Republic of Korea.10.1109/IROS.2016.7759437 ↗
36. Nakadai, K., Kumon, M., Okuno, H.G., Hoshiba, K., Wakabayashi, M., Washizaki, K., Ishiki, T., Gabriel, D., Bando, Y., and Morito, T. (2017, January 24–28). Development of microphone-array-embedded UAV for search and rescue task. Proceedings of the 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, Canada.10.1109/IROS.2017.8206494 ↗
37. Strauss, M., Mordel, P., Miguet, V., and Deleforge, A. (2018, January 1–5). DREGON: Dataset and methods for UAV-embedded sound source localization. Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain.10.1109/IROS.2018.8593581 ↗
38. Wang, L., Sanchez-Matilla, R., and Cavallaro, A. (2019, January 3–8). Audio-visual sensing from a quadcopter: Dataset and baselines for source localization and sound enhancement. Proceedings of the 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Macau, China.10.1109/IROS40897.2019.8968183 ↗
39. Michaud, S., Faucher, S., Grondin, F., Lauzon, J.S., Labbé, M., Létourneau, D., Ferland, F., and Michaud, F. (2020, January 25–29). 3D localization of a sound source using mobile microphone arrays referenced by SLAM. Proceedings of the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA.10.1109/IROS45743.2020.9341098 ↗
40. Sewtz, M., Bodenmüller, T., and Triebel, R. (2020, January 25–29). Robust MUSIC-based sound source localization in reverberant and echoic environments. Proceedings of the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA.10.1109/IROS45743.2020.9340826 ↗
41. Tourbabin, V., Barfuss, H., Rafaely, B., and Kellermann, W. (2015, January 19–24). Enhanced robot audition by dynamic acoustic sensing in moving humanoids. Proceedings of the 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), South Brisbane, Australia.10.1109/ICASSP.2015.7179048 ↗
42. Takeda, R., and Komatani, K. (2016, January 20–25). Sound source localization based on deep neural networks with directional activate function exploiting phase information. Proceedings of the 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Shanghai, China.10.1109/ICASSP.2016.7471706 ↗
43. Takeda, R., and Komatani, K. (2017, January 5–9). Unsupervised adaptation of deep neural networks for sound source localization using entropy minimization. Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA.10.1109/ICASSP.2017.7952550 ↗
44. Ferguson, E.L., Williams, S.B., and Jin, C.T. (2018, January 15–20). Sound source localization in a multipath environment using convolutional neural networks. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8462024 ↗
45. Grondin, F., and Michaud, F. (2016, January 16–21). Noise mask for TDOA sound source localization of speech on mobile robots in noisy environments. Proceedings of the 2016 IEEE International Conference on Robotics and Automation (ICRA), Stockholm, Sweden.10.1109/ICRA.2016.7487652 ↗
46. He, W., Motlicek, P., and Odobez, J.M. (2018, January 21–25). Deep neural networks for multiple speaker detection and localization. Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia.10.1109/ICRA.2018.8461267 ↗
47. An, I., Son, M., Manocha, D., and Yoon, S.E. (2018, January 21–25). Reflection-aware sound source localization. Proceedings of the 2018 IEEE international conference on robotics and automation (ICRA), Brisbane, Australia.10.1109/ICRA.2018.8461268 ↗
48. Automatic patrol and inspection method for machinery diagnosis robot—Sound signal-based fuzzy search approach10.1109/JSEN.2020.2978396 ↗
49. An embedded multichannel sound acquisition system for drone audition10.1109/JSEN.2023.3273330 ↗
50. Lightweight and optimized sound source localization and tracking methods for open and closed microphone array configurations10.1016/j.robot.2019.01.002 ↗
51. Go, Y.J., and Choi, J.S. (2021). An acoustic source localization method using a drone-mounted phased microphone array. Drones, 5.10.3390/drones5030075 ↗
52. Yamada, T., Itoyama, K., Nishida, K., and Nakadai, K. (2023). Placement planning for sound source tracking in active drone audition. Drones, 7.10.3390/drones7070405 ↗
53. Skoczylas, A., Stefaniak, P., Anufriiev, S., and Jachnik, B. (2021). Belt conveyors rollers diagnostics based on acoustic signal collected using autonomous legged inspection robot. Appl. Sci., 11.10.3390/app11052299 ↗
54. Shi, Z., Zhang, L., and Wang, D. (2023). Audio–visual sound source localization and tracking based on mobile robot for the cocktail party problem. Appl. Sci., 13.10.3390/app13106056 ↗
55. Advanced binaural sound localization in 3-D for humanoid robots10.1109/TIM.2014.2308051 ↗
56. Multiple sound source localization exploiting robot motion and approaching control
57. Theoretical framework for the optimization of microphone array configuration for humanoid robot audition10.1109/TASLP.2014.2351133 ↗
58. Drone audition: Sound source localization using on-board microphones10.1109/TASLP.2022.3140550 ↗
59. Gonzalez-Billandon, J., Belgiovine, G., Tata, M., Sciutti, A., Sandini, G., and Rea, F. (2021, January 23–26). Self-supervised learning framework for speaker localisation with a humanoid robot. Proceedings of the 2021 IEEE International Conference on Development and Learning (ICDL), Beijing, China.10.1109/ICDL49984.2021.9515566 ↗
60. Gamboa-Montero, J.J., Basiri, M., Castillo, J.C., Marques-Villarroya, S., and Salichs, M.A. (2022, January 12–15). Real-Time Acoustic Touch Localization in Human-Robot Interaction based on Steered Response Power. Proceedings of the 2022 IEEE International Conference on Development and Learning (ICDL), London, UK.10.1109/ICDL53763.2022.9962225 ↗
61. Sound source localization using deep learning models10.20965/jrm.2017.p0037 ↗
62. Chen, L., Chen, G., Huang, L., Choy, Y.S., and Sun, W. (2022). Multiple sound source localization, separation, and reconstruction by microphone array: A dnn-based approach. Appl. Sci., 12.10.3390/app12073428 ↗
63. Multiple CRNN for SELD
64. Exploiting temporal context in CNN based multisource DOA estimation10.1109/TASLP.2021.3067113 ↗
65. Li, X., Liu, H., and Yang, X. (2011, January 25–30). Sound source localization for mobile robot based on time difference feature and space grid matching. Proceedings of the 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems, San Francisco, CA, USA.10.1109/IROS.2011.6094667 ↗
66. A neural network-based smart antenna for multiple source tracking10.1109/8.855496 ↗
67. Real-time sound source localization in robots using fly Ormia ochracea inspired MEMS directional microphone
68. Athanasopoulos, G., Dekens, T., Brouckxon, H., and Verhelst, W. (2012, January 2–5). The effect of speech denoising algorithms on sound source localization for humanoid robots. Proceedings of the 2012 11th International Conference on Information Science, Signal Processing and their Applications (ISSPA), Montreal, QC, Canada.10.1109/ISSPA.2012.6310569 ↗
69. Deep learning based multi-source localization with source splitting and its effectiveness in multi-talker speech recognition10.1016/j.csl.2022.101360 ↗
70. Deep learning-based speech specific source localization by using binaural and monaural microphone arrays in hearing aids10.1109/TASLP.2023.3268734 ↗
71. Wu, S., Zheng, Y., Ye, K., Cao, H., Zhang, X., and Sun, H. (2024). Sound source localization for unmanned aerial vehicles in low signal-to-noise ratio environments. Remote Sens., 16.10.3390/rs16111847 ↗
72. ConvLSTM-based Sound Source Localization in a manufacturing workplace10.1016/j.cie.2024.110213 ↗
73. Sound source localization10.1016/j.anorl.2018.04.009 ↗
74. Hirvonen, T. (2015, January 7–10). Classification of spatial audio location and content using convolutional neural networks. Proceedings of the Audio Engineering Society Convention 138, Warsaw, Poland.
75. Xiao, X., Zhao, S., Zhong, X., Jones, D.L., Chng, E.S., and Li, H. (2015, January 19–24). A learning-based approach to direction of arrival estimation in noisy and reverberant environments. Proceedings of the 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), South Brisbane, Australia.10.1109/ICASSP.2015.7178484 ↗
76. Geng, Y., Jung, J., and Seol, D. (2008, January 1–8). Sound-source localization system based on neural network for mobile robots. Proceedings of the 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), Hong Kong, China.
77. Liu, G., Yuan, S., Wu, J., and Zhang, R. (December, January 30). A sound source localization method based on microphone array for mobile robot. Proceedings of the 2018 Chinese Automation Congress (CAC), Xi’an, China.10.1109/CAC.2018.8623702 ↗
78. Deep learning-based method for multiple sound source localization with high resolution and accuracy10.1016/j.ymssp.2021.107959 ↗
79. Acosta, O., Hermida, L., Herrera, M., Montenegro, C., Gaona, E., Bejarano, M., Gordillo, K., Pavón, I., and Asensio, C. (2023). Remote Binaural System (RBS) for Noise Acoustic Monitoring. J. Sens. Actuator Netw., 12.10.3390/jsan12040063 ↗
80. Acoustic space learning for sound-source separation and localization on binaural manifolds10.1142/S0129065714400036 ↗
81. Realtime active sound source localization for unmanned ground robots using a self-rotational bi-microphone array10.1007/s10846-018-0908-3 ↗
82. Baxendale, M.D., Nibouche, M., Secco, E.L., Pipe, A.G., and Pearson, M.J. (2019, January 9–12). Feed-forward selection of cerebellar models for calibration of robot sound source localization. Proceedings of the Conference on Biomimetic and Biohybrid Systems, Nara, Japan.10.1007/978-3-030-24741-6_1 ↗
83. Moving sound source localization and tracking for an autonomous robot equipped with a self-rotating bi-microphone array10.1121/10.0020583 ↗
84. Algorithms for acoustic localization based on microphone array in service robotics10.1016/S0921-8890(02)00325-1 ↗
85. Nguyen, Q.V., Colas, F., Vincent, E., and Charpillet, F. (2017, January 1–3). Long-term robot motion planning for active sound source localization with Monte Carlo tree search. Proceedings of the 2017 Hands-free Speech Communications and Microphone Arrays (HSCMA), San Francisco, CA, USA.10.1109/HSCMA.2017.7895562 ↗
86. Tamai, Y., Kagami, S., Amemiya, Y., Sasaki, Y., Mizoguchi, H., and Takano, T. (2004, January 24–27). Circular microphone array for robot’s audition. Proceedings of the SENSORS, 2004 IEEE, Vienna, Austria.
87. Speech enhancement and recognition using circular microphone array for service robots10.1109/IROS.2003.1249700 ↗
88. Sasaki, Y., Kabasawa, M., Thompson, S., Kagami, S., and Oro, K. (2012, January 7–12). Spherical microphone array for spatial sound localization for a mobile robot. Proceedings of the 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, Vilamoura-Algarve, Portugal.10.1109/IROS.2012.6385877 ↗
89. RNN for solving time-variant generalized Sylvester equation with applications to robots and acoustic source localization10.1109/TII.2020.2964817 ↗
90. Bando, Y., Mizumoto, T., Itoyama, K., Nakadai, K., and Okuno, H.G. (2013, January 3–7). Posture estimation of hose-shaped robot using microphone array localization. Proceedings of the 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, Tokyo, Japan.10.1109/IROS.2013.6696847 ↗
91. Kim, U.H. (2013). Improvement of Sound Source Localization for a Binaural Robot of Spherical Head with Pinnae. [Ph.D. Thesis, Kyoto University].
92. Kumon, M., and Noda, Y. (2011, January 25–30). Active soft pinnae for robots. Proceedings of the 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems, San Francisco, CA, USA.10.1109/IROS.2011.6048186 ↗
93. Murray, J.C., and Erwin, H.R. (2012, January 16). A neural network classifier for notch filter classification of sound-source elevation in a mobile robot. Proceedings of the The 2011 International Joint Conference on Neural Networks, San Jose, CA, USA.10.1109/IJCNN.2011.6033298 ↗
94. Grounded auditory development by a developmental robot10.1109/IJCNN.2001.939507 ↗
95. High-accuracy TDOA-based localization without time synchronization10.1109/TPDS.2012.248 ↗
96. The generalized correlation method for estimation of time delay10.1109/TASSP.1976.1162830 ↗
97. Performance analysis of GCC-PHAT-based sound source localization for intelligent robots
98. Wang, J., Qian, X., Pan, Z., Zhang, M., and Li, H. (June, January 30). GCC-PHAT with speech-oriented attention for robotic sound source localization. Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi’an, China.10.1109/ICRA48506.2021.9561885 ↗
99. TDOA estimation for multiple sound sources in noisy and reverberant environments using broadband independent component analysis10.1109/TASL.2010.2092765 ↗
100. TDOA-based source localization with distance-dependent noises10.1109/TWC.2014.2351798 ↗