End-to-End Neural Optical Music Recognition of Monophonic Scores
Source abstract
Abstract
Optical Music Recognition is a field of research that investigates how to computationally decode music notation from images. Despite the efforts made so far, there are hardly any complete solutions to the problem. In this work, we study the use of neural networks that work in an end-to-end manner. This is achieved by using a neural model that combines the capabilities of convolutional neural networks, which work on the input image, and recurrent neural networks, which deal with the sequential nature of the problem. Thanks to the use of the the so-called Connectionist Temporal Classification loss function, these models can be directly trained from input images accompanied by their corresponding transcripts into music symbol sequences. We also present the Printed Music Scores dataset, containing more than 80,000 monodic single-staff real scores in common western notation, that is used to train and evaluate the neural approach. In our experiments, it is demonstrated that this formulation can be carried out successfully. Additionally, we study several considerations about the codification of the output musical sequences, the convergence and scalability of the neural models, as well as the ability of this approach to locate symbols in the input score.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Music and Audio Processing · Speech and Audio Processing · Music Technology and Sound Studies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Schulich School of Music, McGill University, Montreal, QC H3A 1E3, Canada
PRHLT Research Center, Universitat Politècnica de València, 46022 Valencia, Spain
Instituto Superior de Enseñanzas Artísticas, 03690 Alicante, Spain
Departamento de Lenguajes y Sistemas Informáticos, Universidad de Alicante, 03690 Alicante, Spain
Funding
Social Sciences and Humanities Research Council of Canada · Spanish Ministerio de Economía y Competitividad
References (56)
Show source-provided references
1. Content-Based Music Information Retrieval: Current Directions and Future Challenges10.1109/JPROC.2008.916370 ↗
2. Roland, P. (2002, January 19–20). The music encoding initiative (MEI). Proceedings of the First International Conference on Musical Applications Using XML, Milan, Italy.
3. Good, M., and Actor, G. (2003, January 15–17). Using MusicXML for File Interchange. Proceedings of the International Conference on Web Delivering of Music (WEDELMUSIC), Leeds, UK.
4. Meredith, D. (2015). Computational Music Analysis, Springer. [1st ed.].10.1007/978-3-319-25931-4 ↗
5. Applications of RISM data in digital libraries and digital musicology
6. The Challenge of Optical Music Recognition10.1023/A:1002485918032 ↗
7. Liwicki, M., Graves, A., Bunke, H., and Schmidhuber, J. (2007, January 23–26). A novel approach to on-line handwriting recognition based on bidirectional long short-term memory networks. Proceedings of the 9th International Conference on Document Analysis and Recognition, Curitiba, Brazil.
8. Graves, A., Mohamed, A.R., and Hinton, G. (2013, January 26–31). Speech recognition with deep recurrent neural networks. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, BC, Canada.10.1109/ICASSP.2013.6638947 ↗
9. Ng, K., McLean, A., and Marsden, A. (2014, January 8–10). Big Data Optical Music Recognition with Multi Images and Multi Recognisers. Proceedings of the Electronic Visualisation and the Arts, London, UK.10.14236/ewic/EVA2014.50 ↗
10. Towards a Standard Testbed for Optical Music Recognition: Definitions, Metrics, and Page Images10.1080/09298215.2015.1045424 ↗
11. Optical music recognition: State-of-the-art and open issues10.1007/s13735-012-0004-6 ↗
12. Deep learning10.1038/nature14539 ↗
13. Amodei, D., Anubhai, R., Battenberg, E., Case, C., Casper, J., Catanzaro, B., Chen, J., Chrzanowski, M., Coates, A., and Diamos, G. (2016, January 19–24). Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin. Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA.
14. Voigtlaender, P., Doetsch, P., and Ney, H. (2016, January 23–26). Handwriting Recognition with Large Multidimensional Long Short-Term Memory Recurrent Neural Networks. Proceedings of the 15th International Conference on Frontiers in Handwriting Recognition ICFHR 2016, Shenzhen, China.10.1109/ICFHR.2016.0052 ↗
15. Graves, A., Fernández, S., Gomez, F., and Schmidhuber, J. (2006, January 25–29). Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks. Proceedings of the 23rd International Conference on Machine Learning—ICML ’06, Pittsburg, PA, USA.10.1145/1143844.1143891 ↗
16. Selfridge-Field, E. (1997). Beyond MIDI: The Handbook of Musical Codes, MIT Press.
17. CVC-MUSCIMA: A ground truth of handwritten music score images for writer identification and staff removal
18. Hajic, J., Novotný, J., Pecina, P., and Pokorný, J. (2016, January 7–11). Further Steps Towards a Standard Testbed for Optical Music Recognition. Proceedings of the 17th International Society for Music Information Retrieval Conference, New York City, NY, USA.
19. Calvo-Zaragoza, J., Valero-Mas, J.J., and Pertusa, A. (2017, January 23–27). End-to-End Optical Music Recognition Using Neural Networks. Proceedings of the 18th International Society for Music Information Retrieval Conference, Suzhou, China.
20. Pinto, T., Rebelo, A., Giraldi, G.A., and Cardoso, J.S. (2011, January 8–10). Music Score Binarization Based on Domain Knowledge. Proceedings of the 5th Iberian Conference—Pattern Recognition and Image Analysis, IbPRIA 2011, Las Palmas de Gran Canaria, Spain.
21. Campos, V.B., Calvo-Zaragoza, J., Toselli, A.H., and Vidal-Ruiz, E. (2016, January 23–26). Sheet Music Statistical Layout Analysis. Proceedings of the 15th International Conference on Frontiers in Handwriting Recognition ICFHR 2016, Shenzhen, China.
22. Vigliensoni, G., Burlet, G., and Fujinaga, I. (2013, January 4–8). Optical Measure Recognition in Common Music Notation. Proceedings of the 14th International Society for Music Information Retrieval Conference, Curitiba, Brazil.
23. Burgoyne, J.A., Ouyang, Y., Himmelman, T., Devaney, J., Pugin, L., and Fujinaga, I. (2009, January 26–30). Lyric extraction and recognition on digital images of early music sources. Proceedings of the 10th International Society for Music Information Retrieval Conference, Kobe, Japan.
24. A Comparative Study of Staff Removal Algorithms10.1109/TPAMI.2007.70749 ↗
25. Staff Detection with Stable Paths10.1109/TPAMI.2009.34 ↗
26. Géraud, T. (2014, January 19–20). A morphological method for music score staff removal. Proceedings of the 21st International Conference on Image Processing (ICIP), Paris, France.10.1109/ICIP.2014.7025526 ↗
27. Staff-line detection and removal using a convolutional neural network10.1007/s00138-017-0844-4 ↗
28. Staff-line removal with selectional auto-encoders10.1016/j.eswa.2017.07.002 ↗
29. Optical recognition of music symbols: A comparative study10.1007/s10032-009-0100-1 ↗
30. Calvo-Zaragoza, J., Valero-Mas, J.J., and Rico-Juan, J.R. (2017, January 24–26). Recognition of Handwritten Music Symbols using Meta-features Obtained from Weak Classifiers based on Nearest Neighbor. Proceedings of the 6th International Conference on Pattern Recognition Applications and Methods ICPRAM, Porto, Portugal.10.5220/0006120200960104 ↗
31. Pinheiro Pereira, R.M., Matos, C.E., Braz Junior, G., de Almeida, J.a.D., and de Paiva, A.C. (2016, January 8–11). A Deep Approach for Handwritten Musical Symbols Recognition. Proceedings of the 22nd Brazilian Symposium on Multimedia and the Web 2016—Webmedia ’16, Teresina, Brazil.10.1145/2976796.2988171 ↗
32. Lee, S., Son, S.J., Oh, J., and Kwak, N. (2016, January 8–10). Handwritten Music Symbol Classification Using Deep Convolutional Neural Networks. Proceedings of the 3rd International Conference on Information Science and Security, Beijing, China.10.1109/ICISSEC.2016.7885856 ↗
33. Calvo-Zaragoza, J., Sánchez, A.J.G., and Pertusa, A. (2017, January 9–15). Recognition of Handwritten Music Symbols with Convolutional Neural Codes. Proceedings of the 14th IAPR International Conference on Document Analysis and Recognition, ICDAR 2017, Kyoto, Japan.10.1109/ICDAR.2017.118 ↗
34. Pacha, A., and Eidenberger, H. (2017, January 9–15). Towards a Universal Music Symbol Classifier. Proceedings of the 12th International Workshop on Graphics Recognition, 14th IAPR International Conference on Document Analysis and Recognition, GREC@ICDAR 2017, Kyoto, Japan.10.1109/ICDAR.2017.265 ↗
35. Couasnon, B. (2001, January 13). Dmos: A generic document recognition method, application to an automatic generator of musical scores, mathematical formulae and table structures recognition systems. Proceedings of the Sixth International Conference on Document Analysis and Recognition, Bangalore, India.
36. Szwoch, M. (2007, January 23–26). Guido: A Musical Score Recognition System. Proceedings of the 9th International Conference on Document Analysis and Recognition (ICDAR 2007), Curitiba, Brazil.10.1109/ICDAR.2007.4377027 ↗
37. Robust and adaptive OMR system including fuzzy modeling, fusion of musical rules, and possible error detection10.1155/2007/81541 ↗
38. Raphael, C., and Wang, J. (2011, January 24–28). New Approaches to Optical Music Recognition. Proceedings of the 12th International Society for Music Information Retrieval Conference ISMIR 2011, Miami, FL, USA.
39. Bitteur, H. (2018, March 21). Audiveris. Available online: https://github.com/Audiveris/audiveris.
40. Pugin, L. (2006, January 8–12). Optical Music Recognition of Early Typographic Prints using Hidden Markov Models. Proceedings of the 7th International Conference on Music Information Retrieval, Victoria, BC, Canada.
41. Tardón, L.J., Sammartino, S., Barbancho, I., Gómez, V., and Oliver, A. (2009). Optical Music Recognition for Scores Written in White Mensural Notation. EURASIP J. Image Video Process., 2009.10.1155/2009/843401 ↗
42. Avoiding staff removal stage in optical music recognition: Application to scores written in white mensural notation10.1007/s10044-014-0415-5 ↗
43. Calvo-Zaragoza, J., Toselli, A.H., and Vidal, E. (2016, January 23–26). Early Handwritten Music Recognition with Hidden Markov Models. Proceedings of the 15th International Conference on Frontiers in Handwriting Recognition ICFHR 2016, Shenzhen, China.10.1109/ICFHR.2016.0067 ↗
44. The Simplified ’Plaine and Easie Code System’ for Notating Music: A Proposal for International Adoption
45. Pugin, L., Zitellini, R., and Roland, P. (2014, January 27–31). Verovio—A library for Engraving MEI Music Notation into SVG. Proceedings of the 15th International Conferencefor Music Information Retrieval Conference, Taipei, Taiwan.
46. Graves, A. (2008). Supervised Sequence Labelling with Recurrent Neural Networks. [Ph.D. Thesis, Technical University of Munich].
47. Zeiler, M.D., and Fergus, R. (2014, January 6–12). Visualizing and Understanding Convolutional Networks. Proceedings of the 13th European Conference on Computer Vision—ECCV 2014, Zurich, Switzerland. Part I.10.1007/978-3-319-10590-1_53 ↗
48. Rabiner, L., and Juang, B.H. (1993). Fundamentals of Speech Recognition, Prentice Hall, Inc.
49. Simonyan, K., and Zisserman, A. (arXiv, 2014). Very deep convolutional networks for large-scale image recognition, arXiv, preprint arXiv:1409.1556.
50. Ioffe, S., and Szegedy, C. (2015, January 6–11). Batch normalization: Accelerating deep network training by reducing internal covariate shift. Proceedings of the 32nd International Conference on Machine Learning—ICML 2015, Lille, France.
51. Glorot, X., Bordes, A., and Bengio, Y. (2011, January 11–13). Deep sparse rectifier neural networks. Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2011, Fort Lauderdale, FL, USA.
52. Long Short-Term Memory10.1162/neco.1997.9.8.1735 ↗
53. Links Between Markov Models and Multilayer Perceptrons10.1109/34.62605 ↗
54. Bottou, L. (2010, January 22–27). Large-scale machine learning with stochastic gradient descent. Proceedings of the COMPSTAT’ 2010, Paris, France.10.1007/978-3-7908-2604-3_16 ↗
55. Zeiler, M.D. (arXiv, 2012). ADADELTA: An adaptive learning rate method, arXiv, preprint arXiv:1212.5701.
56. Calvo-Zaragoza, J. (2018, April 09). TensorFlow Code to Perform End-to-End Optical Music Recognition on Monophonic Scores Through Convolutional Recurrent Neural Networks And CTC-Based Training. Available online: http://github.com/calvozaragoza/tf-deep-omr.10.3390/app8040606 ↗