A Comprehensive Review on Music Transcription
Source abstract
Abstract
Music transcription is the process of transforming recorded sound of musical performances into symbolic representations such as sheet music or MIDI files. Extensive research and development have been carried out in the field of music transcription and technology. This comprehensive review paper surveys the diverse methodologies, techniques, and advancements that have shaped the landscape of music transcription. The paper outlines the significance of music transcription in preserving, analyzing, and disseminating musical compositions across various genres and cultures. It also provides a historical perspective by tracing the evolution of music transcription from traditional manual methods to modern automated approaches. It also highlights the challenges in transcription posed by complex singing techniques, variations in instrumentation, ambiguity in pitch, tempo changes, rhythm, and dynamics. The review also categorizes four different types of transcription techniques, frame-level, note-level, stream-level, and notation-level, discussing their strengths and limitations. It also encompasses the various research domains of music transcription from general melody extraction to vocal melody, note-level monophonic to polyphonic vocal transcription, single-instrument to multi-instrument transcription, and multi-pitch estimation. The survey further covers a broad spectrum of music transcription applications in music production and creation. It also reviews state-of-the-art open-source as well as commercial music transcription tools for pitch estimation, onset and offset detection, general melody detection, and vocal melody detection. In addition, it also encompasses the currently available python libraries that can be used for music transcription. Furthermore, the review highlights the various open-source benchmark datasets for different areas of music transcription. It also provides a wide range of references supporting the historical context, theoretical frameworks, and foundational concepts to help readers understand the background of music transcription and the context of our paper.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Music and Audio Processing · Music Technology and Sound Studies · Speech and Audio Processing
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Center for Advanced Image and Information Technology, Jeonbuk National University, Jeonju 54896, Republic of Korea
Funding
National Research Foundation of Korea (NRF)
References (100+)
Show source-provided references
1. The discovery of an ancient Mesopotamian theory of music
2. West, M.L. (1992). Ancient Greek Music, Clarendon Press.10.1093/oso/9780198148975.001.0001 ↗
3. Barker, A. (1984). Greek Musical Writings: Volume 2, Harmonic and Acoustic Theory, Cambridge University Press.
4. Hagel, S. (2009). Ancient Greek Music: A New Technical History, Cambridge University Press.10.1017/CBO9780511691591 ↗
5. Randel, D.M. (2003). The Harvard Dictionary of Music, Harvard University Press.
6. Leech-Wilkinson, D. (2009). The Changing Sound of Music: Approaches to Studying Recorded Musical Performances, Centre for the History and Analysis of Recorded Music.
7. Gould, E. (2016). Behind Bars: The Definitive Guide to Music Notation, Faber Music Ltd.
8. Nettl, B. (2015). The Study of Ethnomusicology: Thirty-Three Discussions, University of Illinois Press.
9. Titon, J.T. (2016). Worlds of Music: An Introduction to the Music of the World’s Peoples, Cengage Learning.
10. Waring, W., and Rousseau, J.J. (1775). A Dictionary of Music. Translated from the French of Mons. J.J. Rousseau, J. French.
11. Roads, C. (2015). Composing Electronic Music: A New Aesthetic, Oxford University Press.10.1093/acprof:oso/9780195373233.001.0001 ↗
12. An End-to-End Neural Network for Polyphonic Piano Music Transcription10.1109/TASLP.2016.2533858 ↗
13. Klapuri, A., and Davy, M. (2006). Signal Processing Methods for Music Transcription, Springer-Verlag.10.1007/0-387-32845-9 ↗
14. Klapuri, A. (2006). Signal Processing Methods for Music Transcription, Springer.10.1007/0-387-32845-9 ↗
15. A generative model for music transcription10.1109/TSA.2005.852985 ↗
16. Ryynanen, M.P., and Klapuri, A. (2005, January 16–19). Polyphonic music transcription using note event modeling. Proceedings of the IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, New Paltz, NY, USA.
17. Dessein, A., Cont, A., and Lemaitre, G. (2010, January 9–13). Real-time polyphonic music transcription with non-negative matrix factorization and beta-divergence. Proceedings of the ISMIR—11th International Society for Music Information Retrieval Conference, Utrecht, The Netherlands.
18. Mauch, M., Cannam, C., Bittner, R., Fazekas, G., Salamon, J., Dai, J., Bello, J., and Dixon, S. (2015). Computer-Aided Melody Note Transcription Using the Tony Software: Accuracy and Efficiency, University of London.
19. Mauch, M., and Dixon, S. (2014, January 4–9). pYIN: A fundamental frequency estimator using probabilistic threshold distributions. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2014), Florence, Italy.10.1109/ICASSP.2014.6853678 ↗
20. YIN, a fundamental frequency estimator for speech and music10.1121/1.1458024 ↗
21. Dong, H.W., Hsiao, W.Y., and Yang, Y.H. (2018, January 23–27). Pypianoroll: Open source Python package for handling multitrack pianoroll. Proceedings of the ISMIR—19th International Society for Music Information Retrieval Conference, Paris, France.
22. Hsu, J.Y., and Su, L. (2021, January 7–12). VOCANO: A Note Transcription Framework for Singing Voice in Polyphonic Music. Proceedings of the ISMIR—22nd International Society for Music Information Retrieval Conference, Online.
23. Multipitch Estimation of Piano Sounds Using a New Probabilistic Spectral Smoothness Principle10.1109/TASL.2009.2038819 ↗
24. Combining Spectral and Temporal Representations for Multipitch Estimation of Polyphonic Music10.1109/TASLP.2015.2442411 ↗
25. Kelz, R., Dorfer, M., Korzeniowski, F., Böck, S., Arzt, A., and Widmer, G. (2016). On the potential of simple framewise approaches to piano transcription. arXiv.
26. Unsupervised transcription of piano music
27. Ewert, S., Plumbley, M.D., and Sandler, M. (2015, January 19–24). A dynamic programming variant of non-negative matrix deconvolution for the transcription of struck string instruments. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2015), South Brisbane, Australia.10.1109/ICASSP.2015.7178033 ↗
28. A Multipitch Analyzer Based on Harmonic Temporal Structured Clustering10.1109/TASL.2006.885248 ↗
29. Nam, J., Ngiam, J., Lee, H., and Slaney, M. (2011, January 24–28). A Classification-Based Polyphonic Piano Transcription Approach Using Learned Feature Representations. Proceedings of the ISMIR—12th International Society for Music Information Retrieval Conference, Miami, FL, USA.
30. Boulanger-Lewandowski, N., Bengio, Y., and Vincent, P. (2012). Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription. arXiv.10.1109/ICASSP.2013.6638244 ↗
31. Gardner, J., Simon, I., Manilow, E., Hawthorne, C., and Engel, J. (2022). MT3: Multi-task multitrack music transcription. arXiv.
32. Multi-pitch Streaming of Harmonic Sound Mixtures10.1109/TASLP.2013.2285484 ↗
33. Multiple-instrument polyphonic music transcription using a temporally constrained shift-invariant model10.1121/1.4790351 ↗
34. Multiple F0 Estimation and Source Clustering of Polyphonic Music Audio Using PLCA and HMRFs10.1109/TASLP.2014.2387388 ↗
35. (2023, October 05). Music Transcription with Convolutional Neural Networks. Available online: https://www.lunaverus.com/cnn.
36. Hsieh, T.H., Su, L., and Yang, Y.H. (2019, January 12–17). A streamlined encoder/decoder architecture for melody extraction. Proceedings of the International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2019), Brighton, UK.10.1109/ICASSP.2019.8682389 ↗
37. Eggink, J., and Brown, G.J. (2004, January 10–14). Extracting Melody Lines from Complex Audio. Proceedings of the ISMIR—5th International Conference on Music Information Retrieval, Barcelona, Spain.
38. Bittner, R.M., McFee, B., Salamon, J., Li, P., and Bello, J.P. (2017, January 23–27). Deep Salience Representations for F0 Estimation in Polyphonic Music. Proceedings of the ISMIR—International Society for Music Information Retrieval Conference, Suzhou, China.
39. Melody Extraction from Polyphonic Music Signals: Approaches, applications, and challenges10.1109/MSP.2013.2271648 ↗
40. Salamon, J., and Urbano, J. (2012, January 8–12). Current Challenges in the Evaluation of Predominant Melody Extraction Algorithms. Proceedings of the ISMIR–13th International Society for Music Information Retrieval Conference, Porto, Portugal.
41. Gao, Y., Zhang, X., and Li, W. (2021). Vocal Melody Extraction via HRNet-Based Singing Voice Separation and Encoder-Decoder-Based F0 Estimation. Electronics, 10.10.3390/electronics10030298 ↗
42. Lu, W.T., and Su, L. (2018, January 23–27). Vocal Melody Extraction with Semantic Segmentation and Audio-symbolic Domain Transfer Learning. Proceedings of the ISMIR—19th International Society for Music Information Retrieval Conference, Paris, France.
43. Su, L. (2018, January 15–20). Vocal melody extraction using patch-based CNN. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2018), Calgary, AB, Canada.10.1109/ICASSP.2018.8462420 ↗
44. Kum, S., Lin, J.H., Su, L., and Nam, J. (2020). Semi-supervised learning using teacher-student models for vocal melody extraction. arXiv.
45. Kum, S., and Nam, J. (2019). Joint Detection and Classification of Singing Voice Melody Using Convolutional Recurrent Neural Networks. Appl. Sci., 9.10.3390/app9071324 ↗
46. Yeh, C., and Roebel, A. (2009, January 26–30). Multiple-F0 Estimation for Mirex 2009. Proceedings of the ISMIR—10th International Society for Music Information Retrieval Conference, Utrecht, The Netherlands.
47. Klapuri, A.P. (2005, January 16–19). A perceptually motivated multiple-f0 estimation method. Proceedings of the IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, New Paltz, NY, USA.
48. A multiple-F0 estimation approach based on Gaussian spectral modelling for polyphonic music transcription10.1080/09298211003695579 ↗
49. Chang, W.C., Su, A.W., Yeh, C., Roebel, A., and Rodet, X. (2008). Digital Audio Effects (DAFx-08), HAL.
50. Christensen, M., and Jakobsson, A. (2009). Multi-Pitch Estimation, Morgan and Claypool.10.1007/978-3-031-02558-7 ↗
51. Cuesta, H., McFee, B., and Gómez, E. (2020, January 11–16). Multiple f0 estimation in vocal ensembles using convolutional neural networks. Proceedings of the ISMIR—21th International Society for Music Information Retrieval Conference, virtual conference.
52. Yang, L., Maezawa, A., Smith, J.B., and Chew, E. (2017, January 5–9). Probabilistic transcription of sung melody using a pitch dynamic model. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2017), New Orlean, LA, USA.10.1109/ICASSP.2017.7952166 ↗
53. Meseguer-Brocal, G., Bittner, R., Durand, S., and Brost, B. (2020). Data cleansing with contrastive learning for vocal note event annotations. arXiv.
54. SiPTH: Singing Transcription Based on Hysteresis Defined on the Pitch-Time Curve10.1109/TASLP.2014.2331102 ↗
55. Bittner, R.M., Pasalo, K., Bosch, J.J., Meseguer-Brocal, G., and Rubinstein, D. (2021). vocadito: A dataset of solo vocals with f0, note, and lyric annotations. arXiv.
56. Choir acoustics: An overview of scientific research published to date
57. Dai, J., and Dixon, S. (2017, January 23–27). Analysis of Interactive Intonation in Unaccompanied SATB Ensembles. Proceedings of the ISMIR—18th International Society for Music Information Retrieval Conference, Suzhou, China.
58. Schramm, R., and Benetos, E. (2017, January 15–17). Automatic Transcription of a Cappella Recordings from Multiple Singers. Proceedings of the Audio Engineering Society Conference, Arlington, VA, USA. Available online: https://www.proceedings.com/audio-engineering-society-aes/.
59. Perceptual evaluations of voice scatter in unison choir sounds10.1016/S0892-1997(05)80342-X ↗
60. McLeod, A., Schramm, R., Steedman, M., and Benetos, E. (2017). Automatic Transcription of Polyphonic Vocal Music. Appl. Sci., 7.10.3390/app7121285 ↗
61. Schramm, R. (2021). Handbook of Artificial Intelligence for Music: Foundations, Advanced Approaches, and Developments for Creativity, Springer.
62. Hawthorne, C., Elsen, E., Song, J., Roberts, A., Simon, I., Raffel, C., Engel, J., Oore, S., and Eck, D. (2017). Onsets and frames: Dual-objective piano transcription. arXiv.
63. High-Resolution Piano Transcription with Pedals by Regressing Onset and Offset Times10.1109/TASLP.2021.3121991 ↗
64. Cheng, T., Mauch, M., Benetos, E., and Dixon, S. (2016, January 7–11). An attack/decay model for piano transcription. Proceedings of the ISMIR–17th International Society for Music Information Retrieval Conference, New York, NY, USA.
65. Koepke, A.S., Wiles, O., Moses, Y., and Zisserman, A. (2020, January 4–8). Sight to sound: An end-to-end approach for visual piano transcription. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2020), Barcelona, Spain.10.1109/ICASSP40776.2020.9053115 ↗
66. Hawthorne, C., Stasyuk, A., Roberts, A., Simon, I., Huang, C.Z., Dieleman, S., Elsen, E., Engel, J., and Eck, D. (2019). Enabling factorized piano music modeling and generation with the MAESTRO dataset. In ICLR. arXiv.
67. Fiss, X., and Kwasinski, A. (2011, January 22–27). Automatic real-time electric guitar audio transcription. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2011), Prague, Czech Republic.10.1109/ICASSP.2011.5946418 ↗
68. Kim, S., Hayashi, T., and Toda, T. (September, January 29). Note-level automatic guitar transcription using attention mechanism. Proceedings of the 30th European Signal Processing Conference (EUSIPCO), Belgrade, Serbia.
69. Barbancho, I., de la Bandera, C., Barbancho, A.M., and Tardon, L.J. (2009, January 19–24). Transcription and expressiveness detection system for violin music. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Taipei, Taiwan.10.1109/ICASSP.2009.4959552 ↗
70. Automated Violin Fingering Transcription Through Analysis of an Audio Recording10.1162/COMJ_a_00129 ↗
71. Al-Ghawanmeh, F.M., Jafar, I.F., Al-taee, M.A., Al-ghawanmeh, M.T., and Muhsin, Z.J. (2011, January 22–25). Development of Improved automatic music transcription system for the Arabian flute (nay). Proceedings of the 8th International Multi-Conference on Systems, Signals & Devices, Sousse, Tunisia.10.1109/SSD.2011.5993561 ↗
72. Al-Taee, M.A., Al-Rawi, M.S., and Al-Ghawanmeh, F.M. (April, January 31). Time-frequency analysis of the Arabian flute (Nay) tone applied to automatic music transcription. Proceedings of the IEEE/ACS International Conference on Computer Systems and Applications, Doha, Qatar.
73. Cheuk, K.W., Choi, K., Kong, Q., Li, B., Won, M., Hung, A., Wang, J.C., and Herremans, D. (2022). Jointist: Joint learning for multi-instrument transcription and its applications. arXiv.
74. Multi-Instrument Automatic Music Transcription with Self-Attention-Based Instance Segmentation10.1109/TASLP.2020.3030482 ↗
75. Tanaka, K., Nakatsuka, T., Nishikimi, R., Yoshii, K., and Morishima, S. (2020, January 11–16). Multi-Instrument Music Transcription Based on Deep Spherical Clustering of Spectrograms and Pitchgrams. Proceedings of the ISMIR—21th International Society for Music Information Retrieval Conference, virtual conference.
76. Wu, Y.T., Chen, B., and Su, L. (2019, January 12–17). Polyphonic music transcription with semantic segmentation. Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2018), Brighton, UK.10.1109/ICASSP.2019.8682605 ↗
77. Vogl, R., Widmer, G., and Knees, P. (2018). Towards multi-instrument drum transcription. arXiv.
78. Cheuk, K.W., Choi, K., Kong, Q., Li, B., Won, M., Wang, J.C., and Herremans, Y.N. (2023). Jointist: Simultaneous Improvement of Multi-instrument Transcription and Music Source Separation via Joint Training. arXiv.
79. Manilow, E., Seetharaman, P., and Pardo, B. (2020, January 4–8). Simultaneous separation and transcription of mixtures with multiple polyphonic and percussive instruments. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2020), Barcelona, Spain.10.1109/ICASSP40776.2020.9054340 ↗
80. Yamamoto, Y., Nam, J., and Terasawa, H. (2023). PrimaDNN’: A Characteristics-aware DNN Customization for Singing Technique Detection. European signal processing conference (EUSIPCO). arXiv.10.23919/EUSIPCO58844.2023.10290004 ↗
81. Yamamoto, Y., Nam, J., and Terasawa, H. (2022, January 4–8). Analysis and Detection of Singing Techniques in Repertoires of J-POP Solo Singers. Proceedings of the ISMIR—23th International Society for Music Information Retrieval Conference, Bengaluru, India.
82. System for Describing Vocal Timbre in Popular Song
83. Emmons, S., and Chase, C. (2006). Prescriptions for Choral Excellence, Oxford University Press.10.1093/oso/9780195182422.001.0001 ↗
84. Doscher, B.M. (1993). The Functional Unity of the Singing Voice, Scarecrow Press.
85. Thoughts on how to play in tune: Pitch and intonation10.1080/0749446032000134733 ↗
86. She Sang as She Spoke: Billie Holiday and Aspects of Speech Intonation and Diction10.1080/17494060.2014.903055 ↗
87. McAdams, S., and Goodchild, M. (2017). Routledge Companion to Music Cognition, Routledge.
88. Polyphonic piano transcription based on graph convolutional network10.1016/j.sigpro.2023.109134 ↗
89. Piano sheet music identification using dynamic n-gram fingerprinting
90. Multi-Dimensional Spectral Process for Cepstral Feature Engineering & Formant Coding
91. Korzeniowski, F., and Widmer, G. (2016). Feature learning for chord recognition: The deep chroma extractor. arXiv.
92. Effects of tempo on the timing of simple musical rhythms10.1525/mp.2002.19.4.565 ↗
93. The perception of tempo in music10.1068/p5353 ↗
94. Annotator subjectivity in harmony annotations of popular music10.1080/09298215.2019.1613436 ↗
95. Kurth, F., Müller, M., Fremerey, C., Chang, Y.H., and Clausen, M. (2007, January 23–27). Automated Synchronization of Scanned Sheet Music with Audio Recordings. Proceedings of the ISMIR—8th International Society for Music Information Retrieval Conference, Vienna, Austria.
96. Bereket, M., and Shi, K. (2017). An AI Approach to AutZomatic Natural Music Transcription, Stanford University.
97. (2023, October 05). Doremir Music Research AB, ScoreCloud. Available online: https://www.audacityteam.org/.
98. (2023, October 05). Audacity Software. Available online: http://thurs3.pbworks.com/f/audacity.pdf.
99. McFee, B., Raffel, C., Liang, D., Ellis, D.P., McVicar, M., Battenberg, E., and Nieto, O. (2015, January 6–12). librosa: Audio and music signal analysis in python. Proceedings of the 14th Python in Science Conference 2015, Austin, TX, USA.10.25080/Majora-7b98e3ed-003 ↗
100. Cuthbert, M.S., and Ariza, C. (2010, January 9–13). music21: A toolkit for computer-aided musicology and symbolic music data. Proceedings of the ISMIR—11th International Society for Music Information Retrieval Conference, Utrecht, The Netherlands.