BassNet: A Variational Gated Autoencoder for Conditional Generation of Bass Guitar Tracks with Learned Interactive Control
Source abstract
Abstract
Deep learning has given AI-based methods for music creation a boost by over the past years. An important challenge in this field is to balance user control and autonomy in music generation systems. In this work, we present BassNet, a deep learning model for generating bass guitar tracks based on musical source material. An innovative aspect of our work is that the model is trained to learn a temporally stable two-dimensional latent space variable that offers interactive user control. We empirically show that the model can disentangle bass patterns that require sensitivity to harmony, instrument timbre, and rhythm. An ablation study reveals that this capability is because of the temporal stability constraint on latent space trajectories during training. We also demonstrate that models that are trained on pop/rock music learn a latent space that offers control over the diatonic characteristics of the output, among other things. Lastly, we present and discuss generated bass tracks for three different music fragments. The work that is presented here is a step toward the integration of AI-based technology in the workflow of musical content creators.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Music and Audio Processing · Music Technology and Sound Studies · Neuroscience and Music Perception
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Contractor for Sony Computer Science Laboratories, 75005 Paris, France
Sony Computer Science Laboratories, 75005 Paris, France
References (68)
Show source-provided references
1. Hiller, L.A., and Isaacson, L.M. (1979). Experimental Music; Composition with an Electronic Computer, Greenwood Publishing Group Inc.
2. AI and Music: From Composition to Expressive Performance
3. Rowe, L.A., Vin, H.M., Plagemann, T., Shenoy, P.J., and Smith, J.R. (2003, January 2–8). Polyphonic music modeling with random fields. Proceedings of the Eleventh ACM International Conference on Multimedia, Berkeley, CA, USA.
4. Boulanger-Lewandowski, N., Bengio, Y., and Vincent, P. (July, January 26). Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription. Proceedings of the Twenty-Nine International Conference on Machine Learning (ICML’12), Edinburgh, UK.
5. Dhariwal, P., Jun, H., Payne, C., Kim, J.W., Radford, A., and Sutskever, I. (2020). Jukebox: A Generative Model for Music. arXiv.
6. Imitative Leadsheet Generation with User Constraints
7. Lattner, S., Grachten, M., and Widmer, G. (2018). Imposing higher-level Structure in Polyphonic Music Generation using Convolutional Restricted Boltzmann Machines and Constraints. J. Creat. Music. Syst., 3.
8. Mehri, S., Kumar, K., Gulrajani, I., Kumar, R., Jain, S., Sotelo, J., Courville, A.C., and Bengio, Y. (2016). SampleRNN: An Unconditional End-to-End Neural Audio Generation Model. arXiv.
9. Hadjeres, G., Pachet, F., and Nielsen, F. (2017, January 6–11). DeepBach: A Steerable Model for Bach Chorales Generation. Proceedings of the 34th International Conference on Machine Learning (ICML 2017), Sydney, NSW, Australia.
10. The continuator: Musical interaction with style10.1076/jnmr.32.3.333.16861 ↗
11. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion
12. Makhzani, A., Shlens, J., Jaitly, N., and Goodfellow, I.J. (2015). Adversarial Autoencoders. arXiv.
13. Kingma, D.P., and Welling, M. (2013). Auto-encoding variational bayes. arXiv.
14. Memisevic, R. (2011, January 6–13). Gradient-based learning of higher-order image features. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Barcelona, Spain.10.1109/ICCV.2011.6126419 ↗
15. Langford, J., and Pineau, J. (July, January 26). On multi-view feature learning. Proceedings of the 29th International Conference on Machine Learning (ICML-12), Edinburgh, UK.
16. Memisevic, R., and Exarchakis, G. (2013, January 16–21). Learning invariant features by harnessing the aperture problem. Proceedings of the 30th International Conference on Machine Learning, Atlanta, GA, USA.
17. Memisevic, R., and Hinton, G. (2007, January 17–22). Unsupervised learning of image transformations. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, USA.10.1109/CVPR.2007.383036 ↗
18. Learning to represent spatial transformations with factored higher-order Boltzmann machines10.1162/neco.2010.01-09-953 ↗
19. Lattner, S., and Grachten, M. (2019, January 20–23). High-Level Control of Drum Track Generation Using Learned Patterns of Rhythmic Interaction. Proceedings of the IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, New Paltz, NY, USA.10.1109/WASPAA.2019.8937261 ↗
20. Lattner, S., Dörfler, M., and Arzt, A. (2019). Learning Complex Basis Functions for Invariant Representations of Audio. arXiv.
21. Lattner, S., Grachten, M., and Widmer, G. (2018, January 23–27). Learning Transposition-Invariant Interval Features from Symbolic Music and Audio. Proceedings of the 19th International Society for Music Information Retrieval Conference, Paris, France.
22. Slow feature analysis: Unsupervised learning of invariances10.1162/089976602317318938 ↗
23. Le, Q.V., Zou, W.Y., Yeung, S.Y., and Ng, A.Y. (2011, January 20–25). Learning hierarchical invariant spatio-temporal features for action recognition with independent subspace analysis. Proceedings of the 24th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2011), Colorado Springs, CO, USA.10.1109/CVPR.2011.5995496 ↗
24. Bilinear models of natural images
25. A First Look at Music Composition Using LSTM Recurrent Neural Networks
26. Liang, F.T., Gotham, M., Johnson, M., and Shotton, J. (2017, January 23–27). Automatic Stylistic Composition of Bach Chorales with Deep LSTM. Proceedings of the 18th International Society for Music Information Retrieval Conference, Suzhou, China.
27. This time with feeling: Learning expressive musical performance10.1007/s00521-018-3758-9 ↗
28. Lyu, Q., Wu, Z., Zhu, J., and Meng, H. (2015, January 25–31). Modelling High-Dimensional Sequences with LSTM-RTRBM: Application to Polyphonic Music Generation. Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, Buenos Aires, Argentina.
29. Cherla, S. (2016). Neural Probabilistic Models for Melody Prediction, Sequence Labelling and Classification. [Ph.D. Thesis, University of London].
30. Dong, H.W., Hsiao, W.Y., Yang, L.C., and Yang, Y.H. (2018, January 2–7). MuseGAN: Multi-track sequential generative adversarial networks for symbolic music generation and accompaniment. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.11312 ↗
31. Yang, L.C., Chou, S.Y., and Yang, Y.H. (2017). MidiNet: A convolutional generative adversarial network for symbolic-domain music generation. arXiv.
32. Liu, H., and Yang, Y. (2018, January 17–20). Lead Sheet Generation and Arrangement by Conditional Generative Adversarial Network. Proceedings of the 17th IEEE International Conference on Machine Learning and Applications (ICMLA), Orlando, FL, USA.10.1109/ICMLA.2018.00114 ↗
33. Huang, C.A., Vaswani, A., Uszkoreit, J., Simon, I., Hawthorne, C., Shazeer, N., Dai, A.M., Hoffman, M.D., Dinculescu, M., and Eck, D. (2019, January 6–9). Music Transformer: Generating Music with Long-Term Structure. Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA.
34. Payne, C. (2019, April 25). MuseNet. OpenAI. Available online: https://openai.com/blog/musenet/.
35. Van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A.W., and Kavukcuoglu, K. (2016, January 13–15). WaveNet: A Generative Model for Raw Audio. Proceedings of the 9th ISCA Speech Synthesis Workshop, Sunnyvale, CA, USA.
36. Engel, J., Agrawal, K.K., Chen, S., Gulrajani, I., Donahue, C., and Roberts, A. (2019, January 6–9). GANSynth: Adversarial Neural Audio Synthesis. Proceedings of the 7th International Conference on Learning Representations, New Orleans, LA, USA.
37. Donahue, C., McAuley, J., and Puckette, M. (2019, January 6–9). Adversarial Audio Synthesis. Proceedings of the 7th International Conference on Learning Representations, New Orleans, LA, USA.
38. Engel, J., Resnick, C., Roberts, A., Dieleman, S., Norouzi, M., Eck, D., and Simonyan, K. (2017, January 6–11). Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders. Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia.
39. Roberts, A., Engel, J., Raffel, C., Hawthorne, C., and Eck, D. (2018, January 10–15). A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music. Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden.
40. Simon, I., Roberts, A., Raffel, C., Engel, J., Hawthorne, C., and Eck, D. (2018). Learning a latent space of multitrack measures. arXiv.
41. Aouameur, C., Esling, P., and Hadjeres, G. (2019, January 17–21). Neural Drum Machine: An Interactive System for Real-time Synthesis of Drum Sounds. Proceedings of the 10th International Conference on Computational Creativity, North Carolina, CA, USA.
42. Esling, P., Masuda, N., Bardet, A., Despres, R., and Chemla-Romeu-Santos, A. (2019). Universal audio synthesizer control with normalizing flows. arXiv.10.3390/app10010302 ↗
43. Nistal, J., Lattner, S., and Richard, G. (2020, January 11–15). DrumGAN: Synthesis of drum sounds with timbral feature conditioning using Generative Adversarial Networks. Proceedings of the 21st International Society for Music Information Retrieval Conference, Montréal, QC, Canada.
44. Ramires, A., Chandna, P., Favory, X., Gómez, E., and Serra, X. (2020, January 4–8). Neural Percussive Synthesis Parameterised by High-Level Timbral Features. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Barcelona, Spain.10.1109/ICASSP40776.2020.9053128 ↗
45. Mao, H.H., Shin, T., and Cottrell, G. (February, January 31). DeepJ: Style-specific music generation. Proceedings of the 2018 IEEE 12th International Conference on Semantic Computing (ICSC), Laguna Hills, CA, USA.10.1109/ICSC.2018.00077 ↗
46. Anticipation-RNN: Enforcing unary constraints in sequence generation, with application to interactive music generation10.1007/s00521-018-3868-4 ↗
47. Lattner, S., and Grachten, M. (2017, January 11–13). Learning Transformations of Musical Material using Gated Autoencoders. Proceedings of the 2nd Conference on Computer Simulation of Musical Creativity, Milton Keynes, UK.
48. Lattner, S., Grachten, M., and Widmer, G. (2018, January 23–27). A predictive model for music based on learned relative pitch representations. Proceedings of the 19th International Society for Music Information Retrieval Conference, Paris, France.
49. Dehghan, A., Ortiz, E.G., Villegas, R., and Shah, M. (2014, January 24–27). Who do i look like? Determining parent-offspring resemblance via gated autoencoders. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.227 ↗
50. Rimell, L., Mabona, A., Bulat, L., and Kiela, D. (2017, January 3–7). Learning to Negate Adjectives with Bilinear Models. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, Valencia, Spain.10.18653/v1/E17-2012 ↗
51. Factored four way conditional restricted Boltzmann machines for activity recognition10.1016/j.patrec.2015.01.013 ↗
52. Droniou, A., Ivaldi, S., and Sigaud, O. (2014, January 13–16). Learning a repertoire of actions with deep neural networks. Proceedings of the IEEE International Joint Conferences on Development and Learning and Epigenetic Robotics (ICDL-Epirob), Genoa, Italy.10.1109/DEVLRN.2014.6982986 ↗
53. Deep unsupervised network for multimodal perception, representation and classification10.1016/j.robot.2014.11.005 ↗
54. Michalski, V., Memisevic, R., and Konda, K. (2014, January 8–13). Modeling Deep Temporal Dependencies with Recurrent Grammar Cells. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada.
55. Calculation of a constant Q spectral transform10.1121/1.400476 ↗
56. Cheuk, K.W., Anderson, H.H., Agres, K., and Herremans, D. (2019). nnAudio: An on-the-fly GPU Audio to Spectrogram Conversion Toolbox Using 1D Convolution Neural Networks. arXiv.10.1109/ACCESS.2020.3019084 ↗
57. Kim, J.W., Salamon, J., Li, P., and Bello, J.P. (2018, January 15–20). Crepe: A Convolutional Representation for Pitch Estimation. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8461329 ↗
58. Böck, S., Korzeniowski, F., Schlüter, J., Krebs, F., and Widmer, G. (2016, January 15–19). madmom: A new Python Audio and Music Signal Processing Library. Proceedings of the 24th ACM International Conference on Multimedia, Amsterdam, The Netherlands.10.1145/2964284.2973795 ↗
59. Schlüter, J., and Böck, S. (2013, January 23). Musical Onset Detection with Convolutional Neural Networks. Proceedings of the 6th International Workshop on Machine Learning and Music (MML), Prague, Czech Republic.
60. Kathryn, H., and James, B. (2015, January 6–12). librosa: Audio and Music Signal Analysis in Python. Proceedings of the 14th Python in Science Conference, Austin, TX, USA.
61. Yu, F., Koltun, V., and Funkhouser, T.A. (2017). Dilated Residual Networks. arXiv.10.1109/CVPR.2017.75 ↗
62. He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition CVPR, Las Vegas, NV, USA.
63. Chabra, R., Straub, J., Sweeney, C., Newcombe, R., and Fuchs, H. (2019, January 16–20). StereoDRNet: Dilated Residual StereoNet. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.01206 ↗
64. Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press.
65. Lauw, H.W., Wong, R.C.W., Ntoulas, A., Lim, E.P., Ng, S.K., and Pan, S.J. (2020). Deep Multimodal Clustering with Cross Reconstruction. Advances in Knowledge Discovery and Data Mining, Springer International Publishing.10.1007/978-3-030-47436-2 ↗
66. Schönfeld, E., Ebrahimi, S., Sinha, S., Darrell, T., and Akata, Z. (2019, January 6–9). Cross-Linked Variational Autoencoders for Generalized Zero-Shot Learning. Proceedings of the International Conference on Learning Representations (ICLR 2019), New Orleans, LA, USA.
67. Lattner, S., and Grachten, M. (2017). Improving Content-Invariance in Gated Autoencoders for 2D and 3D Object Rotation. arXiv.
68. Kingma, D.P., and Ba, J. (2015, January 7–9). Adam: A Method for Stochastic Optimization. Proceedings of the 3rd International Conference on Learning Representations, San Diego, CA, USA.