Joint Detection and Classification of Singing Voice Melody Using Convolutional Recurrent Neural Networks
Source abstract
Abstract
Singing melody extraction essentially involves two tasks: one is detecting the activity of a singing voice in polyphonic music, and the other is estimating the pitch of a singing voice in the detected voiced segments. In this paper, we present a joint detection and classification (JDC) network that conducts the singing voice detection and the pitch estimation simultaneously. The JDC network is composed of the main network that predicts the pitch contours of the singing melody and an auxiliary network that facilitates the detection of the singing voice. The main network is built with a convolutional recurrent neural network with residual connections and predicts pitch labels that cover the vocal range with a high resolution, as well as non-voice status. The auxiliary network is trained to detect the singing voice using multi-level features shared from the main network. The two optimization processes are tied with a joint melody loss function. We evaluate the proposed model on multiple melody extraction and vocal detection datasets, including cross-dataset evaluation. The experiments demonstrate how the auxiliary network and the joint melody loss function improve the melody extraction performance. Furthermore, the results show that our method outperforms state-of-the-art algorithms on the datasets.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Music and Audio Processing · Speech and Audio Processing · Music Technology and Sound Studies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Music and Audio Computing Lab, Graduate School of Culture Technology, KAIST, 291 Daehak-ro, Yuseong-gu, Daejeon 34141, Korea
Funding
National Research Foundation of Korea
References (46)
Show source-provided references
1. Molina, E., Tardón, L.J., Barbancho-Perez, I., and Barbancho-Perez, A.M. (2014, January 27–31). The importance of F0 tracking in query-by-singing-humming. Proceedings of the International Society for Music Information Retrieval (ISMIR), Taipei, Taiwan.
2. Audio cover song identification and similarity: Background, approaches, evaluation, and beyond
3. Salamon, J., Rocha, B., and Gómez, E. (2012, January 25–30). Musical genre classification using melody features extracted from polyphonic music signals. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Kyoto, Japan.10.1109/ICASSP.2012.6287822 ↗
4. Kako, T., Ohishi, Y., Kameoka, H., Kashino, K., and Takeda, K. (2009, January 26–30). Automatic identification for singing style based on sung melodic contour characterized in phase plane. Proceedings of the International Society for Music Information Retrieval (ISMIR), Kobe, Japan.
5. Schlüter, J., and Lehner, B. (2018, January 23–27). Zero-Mean Convolutions for Level-Invariant Singing Voice Detection. Proceedings of the 19th International Society for Music Information Retrieval Conference (ISMIR 2018), Paris, France.
6. Dressler, K. (2011, January 24–28). An Auditory Streaming Approach for Melody Extraction from Polyphonic Music. Proceedings of the International Society for Music Information Retrieval (ISMIR), Miami, FL, USA.
7. Melody extraction from polyphonic music signals: Approaches, applications, and challenges10.1109/MSP.2013.2271648 ↗
8. Bosch, J.J., Bittner, R.M., Salamon, J., and Gómez, E. (2016, January 7–11). A comparison of melody extraction methods based on source-filter modelling. Proceedings of the International Society for Music Information Retrieval (ISMIR), New York, NY, USA.
9. Predominant melody extraction from vocal polyphonic music signal by time-domain adaptive filtering-based method10.1007/s00034-017-0696-1 ↗
10. Melody extraction from polyphonic music using particle filter and dynamic programming10.1109/TASLP.2018.2834722 ↗
11. Source/filter model for unsupervised main melody extraction from polyphonic audio signals10.1109/TASL.2010.2041114 ↗
12. Tachibana, H., Ono, T., Ono, N., and Sagayama, S. (2010, January 14–19). Melody line estimation in homophonic music audio signals based on temporal-variability of melodic source. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Dallas, TX, USA.10.1109/ICASSP.2010.5495764 ↗
13. Singing voice separation and vocal F0 estimation based on mutual combination of robust principal component analysis and subharmonic summation10.1109/TASLP.2016.2577879 ↗
14. Melody transcription from music audio: Approaches and evaluation10.1109/TASL.2006.889797 ↗
15. Kum, S., Oh, C., and Nam, J. (2016, January 7–11). Melody extraction on vocal segments using multi-column deep neural networks. Proceedings of the International Society for Music Information Retrieval (ISMIR), New York, NY, USA.
16. Rigaud, F., and Radenen, M. (2016, January 7–11). Singing voice melody transcription using deep neural networks. Proceedings of the International Society for Music Information Retrieval (ISMIR), New York, NY, USA.
17. Bittner, R.M., McFee, B., Salamon, J., Li, P., and Bello, J.P. (2017, January 23–27). Deep salience representations for f0 estimation in polyphonic music. Proceedings of the International Society for Music Information Retrieval (ISMIR), Suzhou, China.
18. Su, L. (2018, January 15–20). Vocal melody extraction using patch-based CNN. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8462420 ↗
19. Park, H., and Yoo, C.D. (2017, January 5–9). Melody extraction and detection through LSTM-RNN with harmonic sum loss. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA.10.1109/ICASSP.2017.7952660 ↗
20. Basaran, D., Essid, S., and Peeters, G. (2018, January 23–27). Main melody extraction with source-filter NMF and CRNN. Proceedings of the International Society for Music Information Retrieval (ISMIR), Paris, France.
21. Lu, W.T., and Su, L. (2018, January 23–27). Vocal melody extraction with semantic segmentation and audio-symbolic domain transfer learning. Proceedings of the International Society for Music Information Retrieval (ISMIR), Paris, France.
22. Hsieh, T.H., Su, L., and Yang, Y.H. (arXiv, 2018). A Streamlined Encoder/Decoder Architecture for Melody Extraction, arXiv.10.1109/ICASSP.2019.8682389 ↗
23. Chou, H., Chen, M.T., and Chi, T.S. (2018, January 15–20). A hybrid neural network based on the duplex model of pitch perception for singing melody extraction. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8461483 ↗
24. Zhang, X., and LeCun, Y. (2017, January 4–10). Universum prescription: Regularization using unlabeled data. Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.10.1609/aaai.v31i1.10768 ↗
25. Ruder, S. (arXiv, 2017). An overview of multi-task learning in deep neural networks, arXiv.
26. Joint optimization of masks and deep recurrent neural networks for monaural source separation10.1109/TASLP.2015.2468583 ↗
27. Stoller, D., Ewert, S., and Dixon, S. (2018, January 2–5). Jointly detecting and separating singing voice: A multi-task approach. Proceedings of the International Conference on Latent Variable Analysis and Signal Separation, Guildford, UK.10.1007/978-3-319-93764-9_31 ↗
28. Kong, Q., Xu, Y., and Plumbley, M.D. (September, January 28). Joint detection and classification convolutional neural network on weakly labelled bird audio detection. Proceedings of the Signal Processing Conference (EUSIPCO), Kos Island, Greece.
29. Ioffe, S., and Szegedy, C. (arXiv, 2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift, arXiv.
30. Maas, A.L., Hannun, A.Y., and Ng, A.Y. (2013, January 16–21). Rectifier nonlinearities improve neural network acoustic models. Proceedings of the International Conference on Machine Learning (ICML), Atlanta, GA, USA.
31. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 8–16). Identity mappings in deep residual networks. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46493-0_38 ↗
32. Veit, A., Wilber, M.J., and Belongie, S. (2016, January 5–10). Residual networks behave like ensembles of relatively shallow networks. Proceedings of the Advances in Neural Information Processing Systems 29 (NIPS 2016), Barcelona, Spain.
33. Simonyan, K., and Zisserman, A. (arXiv, 2014). Very deep convolutional networks for large-scale image recognition, arXiv.
34. Kim, J.W., Salamon, J., Li, P., and Bello, J.P. (2018, January 15–20). CREPE: A Convolutional representation for pitch estimation. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8461329 ↗
35. Goto, M., Hashiguchi, H., Nishimura, T., and Oka, R. (2002, January 13–17). RWC music database: Popular, classical and jazz music databases. Proceedings of the International Society for Music Information Retrieval (ISMIR), Paris, France.
36. Bittner, R.M., Salamon, J., Tierney, M., Mauch, M., Cannam, C., and Bello, J.P. (2014, January 27–31). MedleyDB: A multitrack dataset for annotation-intensive MIR research. Proceedings of the International Society for Music Information Retrieval (ISMIR), Taipei, Taiwan.
37. Chan, T.S., Yeh, T.C., Fan, Z.C., Chen, H.W., Su, L., Yang, Y.H., and Jang, R. (2015, January 19–24). Vocal activity informed singing voice separation with the iKala dataset. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), South Brisbane, Queensland, Australia.10.1109/ICASSP.2015.7178063 ↗
38. Schlüter, J., and Grill, T. (2015, January 26–30). Exploring data augmentation for improved singing voice detection with neural networks. Proceedings of the International Society for Music Information Retrieval (ISMIR), Malaga, Spain.
39. Laroche, J. (2002). Time and pitch scale modification of audio signals. Applications of Digital Signal Processing to Audio and Acoustics, Springer.10.1007/0-306-47042-X_7 ↗
40. Ramona, M., Richard, G., and David, B. (April, January 30). Vocal detection in music with support vector machines. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Las Vegas, ND, USA.
41. Raffel, C., McFee, B., Humphrey, E.J., Salamon, J., Nieto, O., Liang, D., Ellis, D.P., and Raffel, C.C. (2014, January 27–31). mir_eval: A transparent implementation of common MIR metrics. Proceedings of the International Society for Music Information Retrieval (ISMIR), Taipei, Taiwan.
42. He, K., Zhang, X., Ren, S., and Sun, J. (2015, January 7–13). Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.123 ↗
43. Chollet, F. (2019, March 29). Keras. Available online: https://github.com/fchollet/keras.
44. Lehner, B., Widmer, G., and Bock, S. (September, January 31). A low-latency, real-time-capable singing voice detection method with LSTM recurrent neural networks. Proceedings of the Signal Processing Conference (EUSIPCO), Nice, France.
45. Leglaive, S., Hennequin, R., and Badeau, R. (2015, January 19–24). Singing voice detection with deep recurrent neural networks. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), South Brisbane, Queensland, Australia.10.1109/ICASSP.2015.7177944 ↗
46. Lee, K., Choi, K., and Nam, J. (2018, January 23–27). Revisiting singing voice detection: A quantitative review and the future outlook. Proceedings of the International Society for Music Information Retrieval (ISMIR), Paris, France.