A Method Based on Dual Cross-Modal Attention and Parameter Sharing for Polyphonic Sound Event Localization and Detection
Source abstract
Abstract
Sound event localization and detection (SELD) is a joint task that unifies sound event detection (SED) and direction-of-arrival estimation (DOAE). The task has become such a popular topic that it was introduced into the challenge on Detection and Classification of Acoustic Scenes and Events (DCASE) Task3 in 2019. In this paper, we propose a method based on dual cross-modal attention (DCMA) and parameter sharing to simultaneously detect and localize sound events. In particular, the DCMA-based decoder commonly used for multiple predictions efficiently learns the associations between SED and DOAE features by exchanging SED and DOAE information in the process of attention, in addition to the encoder with parameter sharing. Furthermore, acoustic features that have not been usually used in the SELD task are additionally adopted to improve the performance, and data augmentation techniques of the mixup to simulate polyphonic events and channel rotation for spatial augmentation are conducted for this task. Experimental results demonstrate that our efficient model using one common decoder block based on the DCMA to predict multiple events in the track-wise output format is effective for the SELD task with up to three overlapping events.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Music and Audio Processing · Speech and Audio Processing · Music Technology and Sound Studies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Electronic Engineering, Sogang University, Seoul 04107, Korea
Autonomous IoT Research Center, Korea Electronics Technology Institute, Seongnam 13509, Korea
Funding
Institute of Information and communications Technology Planning and Evaluation (IITP)
References (42)
Show source-provided references
1. Sound event localization and detection of overlapping sources using convolutional recurrent neural networks10.1109/JSTSP.2018.2885636 ↗
2. Nguyen, T.N.T., Watcharasupat, K.N., Nguyen, N.K., Jones, D.L., and Gan, W.S. (2021). SALSA: Spatial cue-augmented log-spectrogram features for polyphonic sound event localization and detection. arXiv.10.1109/TASLP.2022.3173054 ↗
3. Politis, A., Adavanne, S., Krause, D., Deleforge, A., Srivastava, P., and Virtanen, T. (2021). A dataset of dynamic reverberant sound scenes with directional interferers for sound event localization and detection. arXiv.
4. Lee, S.H., Hwang, J.W., Seo, S.B., and Park, H.M. (2022, May 08). Sound Event Localization and Detection Using Cross-Modal Attention and Parameter Sharing for DCASE2021 Challenge. Available online: https://dcase.community/challenge2021/task-sound-event-localization-and-detection-results.
5. Valenzise, G., Gerosa, L., Tagliasacchi, M., Antonacci, F., and Sarti, A. (2007, January 5–7). Scream and gunshot detection and localization for audio-surveillance systems. Proceedings of the IEEE Conference on Advanced Video and Signal Based Surveillance, London, UK.10.1109/AVSS.2007.4425280 ↗
6. Audio surveillance of roads: A system for detecting anomalous sounds10.1109/TITS.2015.2470216 ↗
7. Valin, J.M., Michaud, F., Hadjou, B., and Rouat, J. (May, January 26). Localization of simultaneous moving sound sources for mobile robot using a frequency-domain steered beamformer approach. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), New Orleans, LA, USA.
8. Overview and evaluation of sound event localization and detection in DCASE 201910.1109/TASLP.2020.3047233 ↗
9. Trowitzsch, I., Taghia, J., Kashef, Y., and Obermayer, K. (2019). The NIGENS general sound events database. arXiv.
10. Politis, A., Adavanne, S., and Virtanen, T. (2020). A dataset of reverberant spatial sound scenes with moving sources for sound event localization and detection. arXiv.
11. Cao, Y., Iqbal, T., Kong, Q., Galindo, M., Wang, W., and Plumbley, M.D. (2022, May 08). Two-Stage Sound Event Localization and Detection Using Intensity Vector and Generalized Cross-Correlation. Available online: https://www.researchgate.net/publication/337335170_TWO-STAGE_SOUND_EVENT_LOCALIZATION_AND_DETECTION_USING_INTENSITY_VECTOR_AND_GENERALIZED_CROSS-CORRELATION.
12. Cao, Y., Iqbal, T., Kong, Q., Zhong, Y., Wang, W., and Plumbley, M.D. (2020). Event-Independent Network for Polyphonic Sound Event Localization and Detection. arXiv.10.1109/ICASSP39728.2021.9413473 ↗
13. Cao, Y., Iqbal, T., Kong, Q., An, F., Wang, W., and Plumbley, M.D. (2021, January 6–11). An improved event-independent network for polyphonic sound event localization and detection. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada.10.1109/ICASSP39728.2021.9413473 ↗
14. Shimada, K., Takahashi, N., Takahashi, S., and Mitsufuji, Y. (2020). Sound Event Localization and Detection Using Activity-Coupled Cartesian DOA Vector and RD3Net. arXiv.
15. Shimada, K., Takahashi, N., Koyama, Y., Takahashi, S., Tsunoo, E., Takahashi, M., and Mitsufuji, Y. (2021). Ensemble of ACCDOA-and EINV2-Based Systems with D3Nets and Impulse Response Simulation for Sound Event Localization and Detection. arXiv.
16. Adavanne, S., Politis, A., and Virtanen, T. (2019). A Multi-Room Reverberant Dataset for Sound Event Localization and Detection. arXiv.10.33682/1xwd-5v76 ↗
17. Nguyen, T.N.T., Watcharasupat, K., Nguyen, N.K., Jones, D.L., and Gan, W.S. (2021). DCASE 2021 Task 3: Spectrotemporally-Aligned Features for Polyphonic Sound Event Localization and Detection. arXiv.
18. Park, S., Suh, S., and Jeong, Y. (2022, May 08). Sound Event Localization and Detection with Various Loss Functions. Available online: https://dcase.community/documents/challenge2020/technical_reports/DCASE2020_Park_89.pdf.
19. Park, S., Jeong, Y., and Lee, T. (2022, May 08). Self-Attention Mechanism for Sound Event Localization and Detection. Available online: http://dcase.community/documents/challenge2021/technical_reports/DCASE2021_Park_106_t3.pdf.
20. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017, January 4–9). Attention is All you Need. Proceedings of the Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems 2017, Long Beach, CA, USA.
21. Lee, Y.H., Jang, D.W., Kim, J.B., Park, R.H., and Park, H.M. (2020). Audio–visual speech recognition based on dual cross-modality attentions with the transformer model. Appl. Sci., 10.10.3390/app10207263 ↗
22. Khare, A., Parthasarathy, S., and Sundaram, S. (2021, January 19–22). Self-supervised learning with cross-modal transformers for emotion recognition. Proceedings of the IEEE Spoken Language Technology Workshop (SLT), Shenzhen, China.10.1109/SLT48900.2021.9383618 ↗
23. Zhang, H., Cisse, M., Dauphin, Y.N., and Lopez-Paz, D. (2017). mixup: Beyond empirical risk minimization. arXiv.10.1007/978-1-4899-7687-1_79 ↗
24. Ronchini, F., Arteaga, D., and Pérez-López, A. (2020). Sound event localization and detection based on CRNN using rectangular filters and channel rotation data augmentation. arXiv.
25. Klapuri, A., and Davy, M. (2007). Signal Processing Methods for Music Transcription, Springer Science & Business Media.10.1007/0-387-32845-9 ↗
26. Jiang, D.N., Lu, L., Zhang, H.J., Tao, J.H., and Cai, L.H. (2002, January 26–29). Music type classification by spectral contrast feature. Proceedings of the IEEE International Conference on Multimedia and Expo, Lausanne, Switzerland.
27. Generalization of spectral flatness measure for non-Gaussian linear processes10.1109/LSP.2004.831663 ↗
28. (2022, May 08). Feature Extraction. Available online: https://librosa.org/doc/main/feature.html#spectral-features.
29. Perceptual effects of spectral modifications on musical timbres10.1121/1.381843 ↗
30. Müller, M., and Ewert, S. (2011, January 24–28). Chroma Toolbox: MATLAB implementations for extracting variants of chroma-based audio features. Proceedings of the International Conference on Music Information Retrieval (ISMIR), Miami, FL, USA.
31. Müller, M., Kurth, F., and Clausen, M. (2005, January 11–15). Audio matching via chroma-based statistical features. Proceedings of the International Conference on Music Information Retrieval (ISMIR), London, UK.
32. Mazzon, L., Yasuda, M., Koizumi, Y., and Harada, N. (2019, January 25–26). Sound event localization and detection using FOA domain spatial augmentation. Proceedings of the Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE), New York, NY, USA.
33. Convolutional neural network: A review of models, methodologies and applications to object detection10.1007/s13748-019-00203-0 ↗
34. Ruder, S. (2017). An overview of multi-task learning in deep neural networks. arXiv.
35. Standley, T., Zamir, A., Chen, D., Guibas, L., Malik, J., and Savarese, S. (2020, January 12–18). Which tasks should be learned together in multi-task learning?. Proceedings of the International Conference on Machine Learning, Virtual.
36. McFee, B., Raffel, C., Liang, D., Ellis, D.P., McVicar, M., Battenberg, E., and Nieto, O. (2015, January 6–12). librosa: Audio and music signal analysis in python. Proceedings of the 14th Python in Science Conference, Austin, TX, USA.10.25080/Majora-7b98e3ed-003 ↗
37. Kingma, D.P., and Ba, J. (2015, January 7–9). Adam: A method for stochastic optimization. Proceedings of the International Conference on Learning Representations, San Diego, CA, USA.
38. (2022, May 08). StepLR. Available online: https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.StepLR.html.
39. Sound event detection in the DCASE 2017 challenge10.1109/TASLP.2019.2907016 ↗
40. Shimada, K., Koyama, Y., Takahashi, N., Takahashi, S., and Mitsufuji, Y. (2021, January 6–11). ACCDOA: Activity-coupled Cartesian direction of arrival representation for sound event localization and detection. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada.10.1109/ICASSP39728.2021.9413609 ↗
41. Emmanuel, P., Parrish, N., and Horton, M. (2022, May 08). Multi-Scale Network for Sound Event Localization and Detection. Available online: http://dcase.community/documents/challenge2021/technical_reports/DCASE2021_Parrish_95_t3.pdf.
42. Zhang, Y., Wang, S., Li, Z., Guo, K., Chen, S., and Pang, Y. (2022, May 08). Data Augmentation and Class-Based Ensembled CNN-Conformer Networks for Sound Event Localization and Detection. Available online: http://dcase.community/documents/challenge2021/technical_reports/DCASE2021_Zhang_67_t3.pdf.