Metrics for Polyphonic Sound Event Detection
Source abstract
Abstract
This paper presents and discusses various metrics proposed for evaluation of polyphonic sound event detection systems used in realistic situations where there are typically multiple sound sources active simultaneously. The system output in this case contains overlapping events, marked as multiple sounds detected as being active at the same time. The polyphonic system output requires a suitable procedure for evaluation against a reference. Metrics from neighboring fields such as speech recognition and speaker diarization can be used, but they need to be partially redefined to deal with the overlapping events. We present a review of the most common metrics in the field and the way they are adapted and interpreted in the polyphonic case. We discuss segment-based and event-based definitions of each metric and explain the consequences of instance-based and class-based averaging using a case study. In parallel, we provide a toolbox containing implementations of presented metrics.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Music and Audio Processing · Speech and Audio Processing · Music Technology and Sound Studies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Signal Processing, Tampere University of Technology, P.O. Box 553, Tampere FI-33101, Finland
Funding
European Research Council
References (43)
Show source-provided references
1. Clavel, C., Ehrette, T., and Richard, G. (2005, January 6). Events Detection for an Audio-Based Surveillance System. Proceedings of the IEEE International Conference on Multimedia and Expo, Amsterdam, The Netherlands.
2. Härmä, A., McKinney, M.F., and Skowronek, J. (2005, January 6). Automatic surveillance of the acoustic activity in our living environment. Proceedings of the IEEE International Conference on Multimedia and Expo (ICME), Amsterdam, The Netherlands.
3. Reliable detection of audio events in highly noisy environments10.1016/j.patrec.2015.06.026 ↗
4. Peng, Y.T., Lin, C.Y., Sun, M.T., and Tsai, K.C. (July, January 28). Healthcare audio event classification using Hidden Markov Models and Hierarchical Hidden Markov Models. Proceedings of the IEEE International Conference on Multimedia and Expo, New York, NY, USA.
5. Acoustic Monitoring and Localization for Social Care10.5626/JCSE.2012.6.1.40 ↗
6. Guyot, P., Pinquier, J., Valero, X., and Alias, F. (2013, January 15–19). Two-step detection of water sound events for the diagnostic and monitoring of dementia. Proceedings of the IEEE International Conference on Multimedia and Expo (ICME), San Jose, CA, USA.10.1109/ICME.2013.6607558 ↗
7. Stowell, D., and Clayton, D. (2015, January 18–21). Acoustic Event Detection for Multiple Overlapping Similar Sources. Proceedings of the IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), New Paltz, NY, USA.10.1109/WASPAA.2015.7336885 ↗
8. A flexible framework for key audio effects detection and auditory context inference10.1109/TSA.2005.857575 ↗
9. Audio Keywords Generation for Sports Video Analysis10.1145/1352012.1352015 ↗
10. Bugalho, M., Portelo, J., Trancoso, I., Pellegrini, T., and Abad, A. (2009, January 6–10). Detecting audio events for semantic video search. Proceedings of the 10th Annual Conference of the International Speech Communication Association, Brighton, UK.10.21437/Interspeech.2009-335 ↗
11. Environmental Sound Recognition With Time-Frequency Audio Features10.1109/TASL.2009.2017438 ↗
12. Sound Event Recognition With Probabilistic Distance SVMs10.1109/TASL.2010.2093519 ↗
13. Dennis, J., Tran, H.D., and Li, H. (2015, January 19–24). Combining robust spike coding with spiking neural networks for sound event classification. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brisbane, QLD, Australia.10.1109/ICASSP.2015.7177955 ↗
14. Zhang, H., McLoughlin, I., and Song, Y. (2015, January 19–24). Robust Sound Event Recognition Using Convolutional Neural Networks. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brisbane, QLD, Australia.10.1109/ICASSP.2015.7178031 ↗
15. Real-world Acoustic Event Detection10.1016/j.patrec.2010.02.005 ↗
16. Heittola, T., Mesaros, A., Virtanen, T., and Gabbouj, M. (2013, January 26–31). Supervised Model Training for Overlapping Sound Events based on Unsupervised Source Separation. Proceedings of the 38th International Conference on Acoustics Speech, and Signal Processing (ICASSP 2013), Vancouver, BC, Canada.10.1109/ICASSP.2013.6639360 ↗
17. Mesaros, A., Dikmen, O., Heittola, T., and Virtanen, T. (2015, January 19–24). Sound event detection in real life recordings using coupled matrix factorization of spectral representations and class activity annotations. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brisbane, QLD, Australia.10.1109/ICASSP.2015.7177950 ↗
18. Espi, M., Fujimoto, M., Kinoshita, K., and Nakatani, T. (2015). Exploiting spectro-temporal locality in deep learning based acoustic event detection. EURASIP J. Audio Speech Music Process., 2015.10.21437/Interspeech.2015-451 ↗
19. Grootel, M., Andringa, T., and Krijnders, J. (2009, January 23–26). DARES-G1: Database of Annotated Real-world Everyday Sounds. Proceedings of the NAG/DAGA Meeting, Rotterdam, The Netherlands.
20. Detection and Classification of Acoustic Scenes and Events10.1109/TMM.2015.2428998 ↗
21. A Discriminative Model for Polyphonic Piano Transcription10.1155/2007/48317 ↗
22. Detection and Classification of Acoustic Scenes and Events 2016, IEEE AASP Challenge. Available online: http://www.cs.tut.fi/sgn/arg/dcase2016/.
23. Mesaros, A., Heittola, T., Eronen, A., and Virtanen, T. (2010, January 23–27). Acoustic Event Detection in Real-life Recordings. Proceedings of the 18th European Signal Processing Conference (EUSIPCO 2010), Aalborg, Denmark.
24. Martinez, E., Celma, O., Sordo, M., Jong, B.D., and Serra, X. (2009, January 23–25). Extending the folksonomies of freesound.org using contentbased audio analysis. Proceedings of the Sound and Music Computing Conference, Porto, Portugal.
25. Foster, P., Sigtia, S., Krstulovic, S., and Barker, J. (2015, January 18–21). CHiME-Home: A Dataset for Sound Source Recognition in a Domestic Environment. Proceedings of the Worshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), New Paltz, NY, USA.10.1109/WASPAA.2015.7336899 ↗
26. Heittola, T., Mesaros, A., Eronen, A., and Virtanen, T. (2013). Context-Dependent Sound Event Detection. EURASIP J. Audio Speech Music Process., 2013.10.1186/1687-4722-2013-1 ↗
27. Cakir, E., Heittola, T., Huttunen, H., and Virtanen, T. (August, January 31). Polyphonic Sound Event Detection Using Multi Label Deep Neural Networks. Proceedings of the International Joint Conference on Neural Networks 2015 (IJCNN 2015), Montreal, QC, Canada.10.1109/IJCNN.2015.7280624 ↗
28. Dikmen, O., and Mesaros, A. (2013, January 20–23). Sound event detection using non-negative dictionaries learned from annotated overlapping events. Proceedings of the IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), New Paltz, NY, USA.10.1109/WASPAA.2013.6701861 ↗
29. Waibel, A.H., and Stiefelhagen, R. (2009). Computers in the Human Interaction Loop, Springer.10.1007/978-1-84882-054-8 ↗
30. Music Information Retrieval Evaluation eXchange (MIREX 2016): Multiple Fundamental Frequency Estimation & Tracking. Available online: http://www.music-ir.org/mirex/wiki/2016:Multiple_Fundamental_Frequency_Estimation_&_Tracking.
31. Giannoulis, D., Benetos, E., Stowell, D., Rossignol, M., Lagrange, M., and Plumbley, M. (2013, January 20–23). Detection and classification of acoustic scenes and events: An IEEE AASP challenge. Proceedings of the IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), New Paltz, NY, USA.10.1109/WASPAA.2013.6701819 ↗
32. Mesaros, A., Heittola, T., and Virtanen, T. TUT Sound Events 2016. Available online: https://zenodo.org/record/45759.
33. Machine Learning in Automated Text Categorization10.1145/505282.505283 ↗
34. On the Stratification of Multi-label Data
35. Apples-to-apples in Cross-validation Studies: Pitfalls in Classifier Performance Measurement10.1145/1882471.1882479 ↗
36. Rijsbergen, C.J.V. (1979). Information Retrieval, Butterworth-Heinemann. [2nd ed.].
37. Dixon, S. (2000, January 10–12). On the computer recognition of solo piano music. Proceedings of the Australasian Computer Music Conference, Brisbane, Australia.
38. Ryynanen, M.P., and Klapuri, A. (2005, January 16–19). Polyphonic music transcription using note event modeling. Proceedings of the IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, New Platz, NY, USA.
39. Sheh, A., and Ellis, D. (2003, January 27–30). Chord Segmentation and Recognition using EM-Trained Hidden Markov Models. Proceedings of the 4th International Conference on Music Information Retrieval ISMIR, Baltimore, MD, USA.
40. Simultaneous Estimation of Chords and Musical Context From Audio10.1109/TASL.2009.2032947 ↗
41. Heittola, T., and Mesaros, A. sed_eval - Evaluation toolbox for Sound Event Detection. Available online: https://github.com/TUT-ARG/sed_eval.
42. Japkowicz, N., and Shah, M. (2011). Evaluating Learning Algorithms, Cambridge University Press.10.1017/CBO9780511921803 ↗
43. Diment, A., Heittola, T., and Virtanen, T. (2013). Sound Event Detection for Office Live and Office Synthetic AASP Challenge, Tampere University of Technology. Technical Report.