SampleCNN: End-to-End Deep Convolutional Neural Networks Using Very Small Filters for Music Classification
Source abstract
Abstract
Convolutional Neural Networks (CNN) have been applied to diverse machine learning tasks for different modalities of raw data in an end-to-end fashion. In the audio domain, a raw waveform-based approach has been explored to directly learn hierarchical characteristics of audio. However, the majority of previous studies have limited their model capacity by taking a frame-level structure similar to short-time Fourier transforms. We previously proposed a CNN architecture which learns representations using sample-level filters beyond typical frame-level input representations. The architecture showed comparable performance to the spectrogram-based CNN model in music auto-tagging. In this paper, we extend the previous work in three ways. First, considering the sample-level model requires much longer training time, we progressively downsample the input signals and examine how it affects the performance. Second, we extend the model using multi-level and multi-scale feature aggregation technique and subsequently conduct transfer learning for several music classification tasks. Finally, we visualize filters learned by the sample-level CNN in each layer to identify hierarchically learned features and show that they are sensitive to log-scaled frequency.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Music and Audio Processing · Speech and Audio Processing · Music Technology and Sound Studies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Graduate School of Culture Technology, KAIST, 291 Daehak-ro, Yuseong-gu, Daejeon 34141, Korea
References (41)
Show source-provided references
1. Krizhevsky, A., Sutskever, I., and Hinton, G.E. (2012, January 3–6). Imagenet classification with deep convolutional neural networks. Proceedings of the 25th International Conference on Neural Information Processing Systems, Lake Tahoe, NV, USA.
2. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., and Dean, J. (2013, January 5–10). Distributed representations of words and phrases and their compositionality. Proceedings of the 26th International Conference on Neural Information Processing Systems, Lake Tahoe, NV, USA.
3. Zhang, X., Zhao, J., and Yann, L. (2015, January 7–12). Character-level convolutional networks for text classification. Proceedings of the 28th International Conference on Neural Information Processing Systems, Montreal, QC, Canada.
4. Kim, Y., Jernite, Y., Sontag, D., and Rush, A.M. (arXiv, 2016). Character-aware neural language models, arXiv.10.1609/aaai.v30i1.10362 ↗
5. Sainath, T.N., Weiss, R.J., Senior, A.W., Wilson, K.W., and Vinyals, O. (2015, January 6–10). Learning the speech front-end with raw waveform CLDNNs. Proceedings of the 16th Annual Conference of the International Speech Communication Association, Dresden, Germany.10.21437/Interspeech.2015-1 ↗
6. Collobert, R., Puhrsch, C., and Synnaeve, G. (arXiv, 2016). Wav2letter: An end-to-end convnet-based speech recognition system, arXiv.
7. Zhu, Z., Engel, J.H., and Hannun, A. (2016, January 8–12). Learning multiscale features directly from waveforms. Proceedings of the Annual Conference of the International Speech Communication Association, San Francisco, CA, USA.10.21437/Interspeech.2016-256 ↗
8. Dieleman, S., and Schrauwen, B. (2014, January 4–9). End-to-end learning for music audio. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Florence, Italy.10.1109/ICASSP.2014.6854950 ↗
9. Ardila, D., Resnick, C., Roberts, A., and Eck, D. (2016, January 7–11). Audio deepdream: Optimizing raw audio with convolutional networks. Proceedings of the International Society for Music Information Retrieval Conference, New York, NY, USA.
10. Thickstun, J., Harchaoui, Z., and Kakade, S. (arXiv, 2017). Learning features of music from scratch, arXiv.
11. Dai, W., Dai, C., Qu, S., Li, J., and Das, S. (2017, January 5–9). Very deep convolutional neural networks for raw waveforms. Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing, New Orleans, LA, USA.10.1109/ICASSP.2017.7952190 ↗
12. Aytar, Y., Vondrick, C., and Torralba, A. (2016, January 5–10). Soundnet: Learning sound representations from unlabeled video. Proceedings of the International Conference on Neural Information Processing Systems, Barcelona, Spain.10.1109/CVPR.2016.18 ↗
13. Lee, J., Park, J., Kim, K.L., and Nam, J. (2017, January 5–8). Sample-level deep convolutional neural networks for music auto-tagging using raw waveforms. Proceedings of the Sound Music Computing Conference (SMC), Espoo, Finland.
14. Simonyan, K., and Zisserman, A. (2015, January 7–9). Very deep convolutional networks for large-scale image recognition. Proceedings of the International Conference on Learning Representations, San Diego, CA, USA.
15. Multi-level and multi-scale feature aggregation using pre-trained convolutional neural networks for music auto-tagging10.1109/LSP.2017.2713830 ↗
16. Tokozume, Y., and Harada, T. (2017, January 5–9). Learning environmental sounds with end-to-end convolutional neural network. Proceedings of the IEEE 2017 IEEE International Conference on Acoustics, Speech and Signal Processing, New Orleans, LA, USA.10.1109/ICASSP.2017.7952651 ↗
17. Palaz, D., Doss, M.M., and Collobert, R. (2015, January 19–24). Convolutional neural networks-based continuous speech recognition using raw speech signal. Proceedings of the IEEE 2015 IEEE International Conference on Acoustics, Speech and Signal Processing, South Brisbane, Queensland, Australia.10.1109/ICASSP.2015.7178781 ↗
18. Palaz, D., Collobert, R., and Magimai-Doss, M. (2015, January 6–10). Analysis of CNN-based speech recognition system using raw speech as input. Proceedings of the 16th Annual Conference of the International Speech Communication Association, Dresden, Germany.10.21437/Interspeech.2015-3 ↗
19. Choi, K., Fazekas, G., and Sandler, M. (2016, January 7–11). Automatic tagging using deep convolutional neural networks. Proceedings of the 17th International Society of Music Information Retrieval Conference, New York, NY, USA.
20. Choi, K., Fazekas, G., Sandler, M., and Cho, K. (2017, January 5–9). Convolutional recurrent neural networks for music classification. Proceedings of the IEEE 2017 IEEE International Conference on Acoustics, Speech and Signal Processing, New Orleans, LA, USA.10.1109/ICASSP.2017.7952585 ↗
21. Pons, J., Lidy, T., and Serra, X. (2016, January 15–17). Experimenting with musically motivated convolutional neural networks. Proceedings of the IEEE International Workshop on Content-Based Multimedia Indexing (CBMI), Bucharest, Romania.10.1109/CBMI.2016.7500246 ↗
22. Lee, J. (2017, January 22). Music Dataset Split. Available online: https://github.com/jongpillee/music_dataset_split.
23. Musical genre classification of audio signals10.1109/TSA.2002.800560 ↗
24. Deep learning and music adversaries10.1109/TMM.2015.2478068 ↗
25. Law, E., West, K., Mandel, M.I., Bay, M., and Downie, J.S. (2009, January 26–30). Evaluation of algorithms using games: The case of music tagging. Proceedings of the International Society for Music Information Retrieval Conference, Kobe, Japan.
26. Schreiber, H. (2015, January 26–30). Improving genre annotations for the million song dataset. Proceedings of the International Society for Music Information Retrieval Conference, Malaga, Spain.
27. Bertin-Mahieux, T., Ellis, D.P., Whitman, B., and Lamere, P. (2011, January 24–28). The million song dataset. Proceedings of the International Society for Music Information Retrieval Conference, Miami, FL, USA.
28. Ioffe, S., and Szegedy, C. (2015, January 6–11). Batch normalization: Accelerating deep network training by reducing internal covariate shift. Proceedings of the The 32nd International Conference on Machine Learning, Lille, France.
29. Dieleman, S., and Schrauwen, B. (2013, January 4–8). Multiscale approaches to music audio feature learning. Proceedings of the International Society for Music Information Retrieval Conference, Curitiba, Brazil.
30. Van Den Oord, A., Dieleman, S., and Schrauwen, B. (2014, January 27–31). Transfer learning by supervised pre-training for audio-based music classification. Proceedings of the International Society for Music Information Retrieval Conference, Taipei, Taiwan.
31. Liu, J.Y., Jeng, S.K., and Yang, Y.H. (arXiv, 2016). Applying topological persistence in convolutional neural network for music audio signals, arXiv.
32. Güçlü, U., Thielen, J., Hanke, M., van Gerven, M., and van Gerven, M.A. (2016, January 5–10). Brains on beats. Proceedings of the International Conference on Neural Information Processing Systems, Barcelona, Spain.
33. Pons, J., Slizovskaia, O., Gong, R., Gómez, E., and Serra, X. (September, January 28). Timbre analysis of music audio Signals with convolutional neural networks. Proceedings of the 2017 25th European Signal Processing Conference, Kos Island, Greece.10.23919/EUSIPCO.2017.8081710 ↗
34. Jeong, I.Y., and Lee, K. (2016, January 7–11). Learning temporal features using a deep neural network and its application to music genre classification. Proceedings of the International Society for Music Information Retrieval Conference, New York, NY, USA.
35. Park, J., Lee, J., Park, J., Ha, J.W., and Nam, J. (arXiv, 2017). Representation learning of music using artist labels, arXiv.
36. Choi, K., Fazekas, G., Sandler, M., and Kim, J. (2015, January 26–30). Auralisation of deep convolutional neural networks: Listening to learned features. Proceedings of the International Society for Music Information Retrieval Conference, Malaga, Spain.
37. Bittner, R.M., McFee, B., Salamon, J., Li, P., and Bello, J.P. (2017, January 23–28). Deep salience representations for f0 estimation in polyphonic music. Proceedings of the International Society for Music Information Retrieval Conference, Suzhou, China.
38. Choi, K., Fazekas, G., and Sandler, M. (arXiv, 2016). Explaining deep convolutional neural networks on music classification, arXiv.10.1109/ICASSP.2017.7952585 ↗
39. Erhan, D., Bengio, Y., Courville, A., and Vincent, P. (2009). Visualizing Higher-Layer Features of a Deep Network, University of Montreal.
40. Zeiler, M.D., and Fergus, R. (2014). Visualizing and Understanding Convolutional Networks, Springer.10.1007/978-3-319-10590-1_53 ↗
41. Nguyen, A., Yosinski, J., and Clune, J. (2015, January 7–12). Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. Proceedings of the Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298640 ↗