YOLO-Based Transfer Learning for Sound Event Detection Using Visual Object Detection Techniques
Source abstract
Abstract
Traditional Sound Event Detection (SED) approaches are based on either specialized models or these models in combination with general audio embedding extractors. In this article, we propose to reframe SED as an object detection task in the time–frequency plane and introduce a direct adaptation of modern YOLO detectors to audio. To our knowledge, this is among the first works to employ YOLOv8 and YOLOv11 not merely as feature extractors but as end-to-end models that localize and classify sound events on mel-spectrograms. Methodologically, our approach (i) generates mel-spectrograms on the fly from raw audio to streamline the pipeline and enable transfer learning from vision models; (ii) applies curriculum learning that exposes the detector to progressively more complex mixtures, improving robustness to overlaps; and (iii) augments training with synthetic audio constructed under DCASE 2023 guidelines to enrich rare classes and challenging scenarios. Comprehensive experiments compare our YOLO-based framework against strong CRNN and Conformer baselines. In our experiments on the DCASE-style setting, the method achieves competitive detection accuracy relative to CRNN and Conformer baselines, with gains in some overlapping/noisy conditions and shortcomings for several short-duration classes. These results suggest that adapting modern object detectors to audio can be effective in this setting, while broader generalization and encoder-augmented comparisons remain open.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Music and Audio Processing · Speech and Audio Processing · Anomaly Detection Techniques and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
AUDIAS—Audio, Data Intelligence and Speech Research Group, Universidad Autónoma de Madrid, Calle Francisco Tomás y Valiente, 11, Ciudad Universitaria de Canto Blanco, 28049 Madrid, Spain
References (61)
Show source-provided references
1. Abeßer, J. (2020). A review of deep learning based methods for acoustic scene classification. Appl. Sci., 10.10.3390/app10062020 ↗
2. Hershey, S., Chaudhuri, S., Ellis, D.P.W., Gemmeke, J.F., Jansen, A., Moore, R.C., Plakal, M., Platt, D., Saurous, R.A., and Seybold, B. (2017, January 5–9). CNN architectures for large-scale audio classification. Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA.10.1109/ICASSP.2017.7952132 ↗
3. Enhancing conformer-based sound event detection using frequency dynamic convolutions and beats audio embeddings10.1109/TASLP.2024.3444490 ↗
4. Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 27–30). You only look once: Unified, real-time object detection. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.91 ↗
5. Pham, P., Li, J.B., Szurley, J., and Das, S. (2018, January 15–20). Eventness: Object detection on spectrograms for temporal localization of audio events. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8462062 ↗
6. Jocher, G., Qiu, J., and Chaurasia, A. (2023, January 01). Ultralytics YOLO. Available online: https://github.com/ultralytics/ultralytics.
7. A survey on transfer learning10.1109/TKDE.2009.191 ↗
8. Gemmeke, J.F., Ellis, D.P.W., Freedman, D., Jansen, A., Lawrence, W., Moore, R.C., Plakal, M., and Ritter, M. (2017, January 5–9). AudioSet: An ontology and human-labeled dataset for audio events. Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA.10.1109/ICASSP.2017.7952261 ↗
9. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., and Bernstein, M.S. (2014). Imagenet large scale visual recognition challenge. arXiv.10.1007/s11263-015-0816-y ↗
10. Abeßer, J., Mimilakis, S.I., Gräfe, R., and Lukashevich, H.M. (2017, January 16). Acoustic scene classification by combining autoencoder-based dimensionality reduction and convolutional neural networks. Proceedings of the Detection and Classification of Acoustic Scenes and Events (DCASE) 2017, Munich, Germany.
11. Deep convolutional neural networks and data augmen- tation for environmental sound classification10.1109/LSP.2017.2657381 ↗
12. Xu, J.-X., Lin, T.-C., Yu, T.-C., Tai, T.-C., and Chang, P.-C. (2018, January 10–12). Acoustic scene classification using reduced MobileNet architecture. Proceedings of the 2018 IEEE International Symposium on Multimedia (ISM), Taichung, Taiwan.10.1109/ISM.2018.00038 ↗
13. Zhang, H., Cissé, M., Dauphin, Y.N., and Lopez-Paz, D. (2017). Mixup: Beyond empirical risk minimization. arXiv.10.1007/978-1-4899-7687-1_79 ↗
14. Park, D., Chan, W., Zhang, Y., Chiu, C.-C., Zoph, B., Cubuk, E., and Le, Q. (2019, January 15–19). SpecAugment: A simple data augmentation method for automatic speech recognition. Proceedings of the Interspeech 2019, Graz, Austria.10.21437/Interspeech.2019-2680 ↗
15. Zhong, Z., Zheng, L., Kang, G., Li, S., and Yang, Y. (2017). Random erasing data augmentation. arXiv.
16. Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014, January 8–13). Generative adversarial nets. Proceedings of the Advances in Neural Information Processing Systems 2014, Montreal, QC, Canada.
17. Detection and classification of acoustic scenes and events: Out- come of the dcase 2016 challenge10.1109/TASLP.2017.2778423 ↗
18. Mesaros, A., Heittola, T., and Virtanen, T. (2016). Metrics for polyphonic sound event detection. Appl. Sci., 6.10.3390/app6060162 ↗
19. Mesaros, A., Serizel, R., Heittola, T., Virtanen, T., and Plumbley, M.D. (2025, January 6–11). A decade of DCASE: Achievements, practices, evaluations and future challenges. Proceedings of the ICASSP 2025—IEEE International Conference on Acoustics, Speech and Signal Processing, Hyderabad, India.10.1109/ICASSP49660.2025.10887673 ↗
20. Embedded streaming deep neural networks accelerator with applications10.1109/TNNLS.2016.2545298 ↗
21. Feature extraction with deep neural networks by a generalized discriminant analysis10.1109/TNNLS.2012.2183645 ↗
22. Girshick, R.B., Donahue, J., Darrell, T., and Malik, J. (2014, January 23–28). Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA.10.1109/CVPR.2014.81 ↗
23. Deep learning10.1038/nature14539 ↗
24. Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012). ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, Curran Associates, Inc.
25. Weakly supervised object recognition with convolutional neural networks
26. Oquab, M., Bottou, L., Laptev, I., and Sivic, J. (2014, January 23–28). Learning and transferring mid-level image representations using convolutional neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.222 ↗
27. Probit analysis: A statistical treatment of the sigmoid response curve10.1126/science.116.3011.286-b ↗
28. Kavukcuoglu, K., Ranzato, M., Fergus, R., and LeCun, Y. (2009, January 20–25). Learning invariant features through topographic filter maps. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Miami, FL, USA.10.1109/CVPRW.2009.5206545 ↗
29. Spatial pyramid pooling in deep convolu- tional networks for visual recognition10.1109/TPAMI.2015.2389824 ↗
30. Girshick, R.B. (2015, January 7–13). Fast R-CNN. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.169 ↗
31. Faster r-cnn: Towards real-time object detection with region proposal networks10.1109/TPAMI.2016.2577031 ↗
32. Dai, J., Li, Y., He, K., and Sun, J. (2016). R-FCN: Object detection via region-based fully convolutional networks. Advances in Neural Information Processing Systems, Curran Associates, Inc.
33. Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B., and Belongie, S. (2017, January 21–26). Feature pyramid networks for object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017, Honolulu, HI, USA.10.1109/CVPR.2017.106 ↗
34. He, K., Gkioxari, G., Dollár, P., and Girshick, R. (2017, January 22–29). Mask R-CNN. Proceedings of the IEEE International Conference on Computer Vision (ICCV) 2017, Venice, Italy.10.1109/ICCV.2017.322 ↗
35. Erhan, D., Szegedy, C., Toshev, A., and Anguelov, D. (2014, January 23–28). Scalable object detection using deep neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2014, Columbus, OH, USA.10.1109/CVPR.2014.276 ↗
36. Yoo, D., Park, S., Lee, J.-Y., Paek, A.S., and Kweon, I.S. (2015, January 7–13). AttentionNet: Aggregating weak directions for accurate object detection. Proceedings of the IEEE International Conference on Computer Vision (ICCV) 2015, Santiago, Chile.10.1109/ICCV.2015.305 ↗
37. Najibi, M., Rastegari, M., and Davis, L.S. (2016, January 27–30). G-CNN: An iterative grid-based object detector. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2016, Las Vegas, NV, USA.10.1109/CVPR.2016.260 ↗
38. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., and Berg, A.C. (2016, January 11–14). SSD: Single Shot MultiBox Detector. Proceedings of the Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands. Proceedings, Part I.10.1007/978-3-319-46448-0_2 ↗
39. Redmon, J., and Farhadi, A. (2017, January 21–26). YOLO9000: Better, faster, stronger. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2017, Honolulu, HI, USA.10.1109/CVPR.2017.690 ↗
40. Fu, C.-Y., Liu, W., Ranga, A., Tyagi, A., and Berg, A.C. (2017). Dssd: Deconvolutional single shot detector. arXiv.
41. Shen, Z., Liu, Z., Li, J., Jiang, Y.-G., Chen, Y., and Xue, X. (2017, January 22–29). DSOD: Learning deeply supervised object detectors from scratch. Proceedings of the IEEE International Conference on Computer Vision (ICCV) 2017, Venice, Italy.10.1109/ICCV.2017.212 ↗
42. Kornblith, S., Shlens, J., and Le, Q.V. (2019, January 15–20). Do better ImageNet models transfer better?. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2019, Long Beach, CA, USA.10.1109/CVPR.2019.00277 ↗
43. Zhou, H., Bai, X., and Du, J. (2018, January 26–29). An investigation of transfer learning mechanism for acoustic scene classification. Proceedings of the 11th International Symposium on Chinese Spoken Language Processing (ISCSLP) 2018, Taipei, Taiwan.10.1109/ISCSLP.2018.8706712 ↗
44. Ye, M., Zhong, H., Song, X., Huang, S., and Cheng, G. (2019, January 15–17). Acoustic scene classification using deep convolutional neural network via transfer learning. Proceedings of the International Conference on Asian Language Processing (IALP) 2019, Shanghai, China.10.1109/IALP48816.2019.9037692 ↗
45. Tan, P.S., Lim, K.M., Lee, C.P., and Tan, C.H. (2020, January 22–24). Acoustic event detection with MobileNet and 1D convolutional neural network. Proceedings of the 2020 IEEE 2nd International Conference on Artificial Intelligence in Engineering and Technology (IICAIET), Kota Kinabalu, Malaysia.10.1109/IICAIET49801.2020.9257865 ↗
46. Ren, Z., Pitt, V., Qian, K., Yang, Z., Zhang, Z., and Schuller, B. (2017, January 16). Deep sequential image features on acoustic scene classification. Proceedings of the Detection and Classification of Acoustic Scenes and Events (DCASE) 2017, Munich, Germany.
47. Koutini, K., Eghbalzadeh, H., Dorfer, M., and Widmer, G. (2019, January 2–6). The receptive field as a regularizer in deep convolutional neural networks for acoustic scene classification. Proceedings of the 27th European Signal Processing Conference (EUSIPCO) 2019, A Coruña, Spain.10.23919/EUSIPCO.2019.8902732 ↗
48. Urban acoustic classification based on deep feature transfer learning10.1016/j.jfranklin.2019.10.014 ↗
49. Müller, R., Ritz, F., Illium, S., and Linnhoff-Popien, C. (2020, January 22–24). Acoustic Anomaly Detection for Machine Sounds Based on Image Transfer Learning. Proceedings of the International Conference on Agents and Artificial Intelligence, Valletta, Malta. Available online: https://api.semanticscholar.org/CorpusID:219401687.
50. Chen, S., Wang, C., Wu, Y., Chen, Z., Wu, S., Liu, S., Wang, J., Li, J., and Yu, F. (December, January 28). BEATs: Audio pre-training with acoustic tokenizers. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) 2022, New Orleans, LA, USA.
51. Koutini, K., Schmid, F., and Widmer, G. (2022, January 23–27). Efficient training of audio transformers with Patchout. Proceedings of the 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore.
52. Gong, Y., Chung, Y.-A., and Glass, J. (September, January 30). AST: Audio spectrogram transformer. Proceedings of the 22nd Annual Conference of the International Speech Communication Association (INTERSPEECH) 2021, Brno, Czech Republic.10.21437/Interspeech.2021-698 ↗
53. Baevski, A., Hsu, W.N., Xu, Q., Babu, A., Gu, J., and Auli, M. (2022). Data2vec: A general framework for self-supervised learning in speech, vision and language. arXiv.
54. Venkatesh, S., Moffat, D., and Miranda, E.R. (2022). You only hear once: A yolo-like algo- rithm for audio segmentation and sound event detection. Appl. Sci., 12.10.3390/app12073293 ↗
55. Salamon, J., MacConnell, D., Cartwright, M., Li, P., and Bello, J.P. (2017, January 15–18). Scaper: A library for soundscape synthesis and augmentation. Proceedings of the 2017 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), New Paltz, NY, USA.10.1109/WASPAA.2017.8170052 ↗
56. Tarvainen, A., and Valpola, H. (2017, January 4–9). Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Proceedings of the Advances in Neural Information Processing Systems 2017, Long Beach, CA, USA.
57. Nam, H., Kim, S.-H., Ko, B.-Y., and Park, Y.-H. (2022, January 18–22). Frequency Dynamic Convolution: Frequency-adaptive pattern recognition for sound event detection. Proceedings of the Interspeech 2022, Incheon, Republic of Korea.10.21437/Interspeech.2022-10127 ↗
58. Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., and Wu, Y. (2020). Conformer: Convolution-augmented Transformer for Speech Recognition. arXiv.10.21437/Interspeech.2020-3015 ↗
59. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems 2017, Long Beach, CA, USA.
60. Barahona, S., Benito-Gorrón, D., Segovia, S., Ramos, D., and Toledano, D.T. (2023, January 4–6). Multi-resolution conformer for sound event detection: Analysis and optimization. Proceedings of the 8th Detection and Classification of Acoustic Scenes and Events Workshop (DCASE 2023), Tampere, Finland.
61. BEATs: Audio pre-training with acoustic tokenizers