A Survey of Handwritten Character Recognition with MNIST and EMNIST
Source abstract
Abstract
This paper summarizes the top state-of-the-art contributions reported on the MNIST dataset for handwritten digit recognition. This dataset has been extensively used to validate novel techniques in computer vision, and in recent years, many authors have explored the performance of convolutional neural networks (CNNs) and other deep learning techniques over this dataset. To the best of our knowledge, this paper is the first exhaustive and updated review of this dataset; there are some online rankings, but they are outdated, and most published papers survey only closely related works, omitting most of the literature. This paper makes a distinction between those works using some kind of data augmentation and works using the original dataset out-of-the-box. Also, works using CNNs are reported separately; as they are becoming the state-of-the-art approach for solving this problem. Nowadays, a significant amount of works have attained a test error rate smaller than 1% on this dataset; which is becoming non-challenging. By mid-2017, a new dataset was introduced: EMNIST, which involves both digits and letters, with a larger amount of data acquired from a database different than MNIST’s. In this paper, EMNIST is explained and some results are surveyed.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Handwritten Text Recognition Techniques · Advanced Neural Network Applications · Vehicle License Plate Recognition
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Computer Science Department, Universidad Carlos III of Madrid, 28911 Leganés, Madrid, Spain
Funding
Ministerio de Educación, Cultura y Deporte
References (91)
Show source-provided references
1. TensorFlow (2018, April 20). MNIST for ML Beginners. Available online: https://www.tensorflow.org/get_started/mnist/beginners.
2. LeCun, Y., Cortes, C., and Burges, C.J.C. (2018, April 25). The MNIST Database of Handwritten Digits. Available online: http://yann.lecun.com/exdb/mnist/.
3. Benenson, R. (2018, May 21). Classification Datasets Results. Available online: http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html.
4. Gradient-based learning applied to document recognition10.1109/5.726791 ↗
5. Shape matching and object recognition using shape contexts10.1109/34.993558 ↗
6. Deformation models for image recognition10.1109/TPAMI.2007.1153 ↗
7. Kégl, B., and Busa-Fekete, R. (2009, January 14–18). Boosting products of base classifiers. Proceedings of the 26th Annual International Conference on Machine Learning, Montreal, QC, Canada.10.1145/1553374.1553439 ↗
8. Training invariant support vector machines10.1023/A:1012454411458 ↗
9. Simard, P., Steinkraus, D., and Platt, J.C. (2003, January 3–6). Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis. Proceedings of the 7th International Conference on Document Analysis and Recognition, Edinburgh, UK.
10. Deng, L., and Yu, D. (2011, January 27–31). Deep Convex Net: A Scalable Architecture for Speech Pattern Classification. Proceedings of the 12th Annual Conference of the International Speech Communication Association, Florence, Italy.10.21437/Interspeech.2011-607 ↗
11. Meier, U., Cireşan, D.C., Gambardella, L.M., and Schmidhuber, J. (2011, January 18–21). Better Digit Recognition with a Committee of Simple Neural Nets. Proceedings of the 2011 International Conference on Document Analysis and Recognition, Beijing, China.10.1109/ICDAR.2011.252 ↗
12. Deep, big, simple neural nets for handwritten digit recognition10.1162/NECO_a_00052 ↗
13. Martin, C.H. (2018, May 10). TensorFlow Reproductions: Big Deep Simple MNIST. Available online: https://calculatedcontent.com/2016/06/08/tensorflow-reproductions-big-deep-simple-mnist/.
14. A trainable feature extractor for handwritten digit recognition10.1016/j.patcog.2006.10.011 ↗
15. Simple Method for High-Performance Digit Recognition Based on Sparse Coding10.1109/TNN.2008.2005830 ↗
16. Ranzato, M.A., Poultney, C., Chopra, S., and LeCun, Y. (2006). Efficient Learning of Sparse Representations with an Energy-Based Model. Advances in Neural Information Processing Systems 19, NIPS Proceedings, MIT Press.10.7551/mitpress/7503.003.0147 ↗
17. Jarrett, K., Kavukcuoglu, K., Ranzato, M.A., and LeCun, Y. (October, January 29). What is the Best Multi-Stage Architecture for Object Recognition?. Proceedings of the 2011 International Conference on Computer Vision, Kyoto, Japan.
18. Cireşan, D.C., Meier, U., Masci, J., Gambardella, L.M., and Schmidhuber, J. (2011, January 19–22). Flexible, High Performance Convolutional Neural Networks for Image Classification. Proceedings of the 22nd International Joint Conference on Artificial Intelligence, Barcelona, Spain.
19. Cireşan, D.C., Meier, U., Gambardella, L.M., and Schmidhuber, J. (2011, January 18–21). Convolutional Neural Network Committees for Handwritten Character Classification. Proceedings of the 2011 International Conference on Document Analysis and Recognition, Beijing, China.10.1109/ICDAR.2011.229 ↗
20. Cireşan, D.C., Meier, U., and Schmidhuber, J. (2012, January 16–21). Multi-column deep neural networks for image classification. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.10.1109/CVPR.2012.6248110 ↗
21. McDonnell, M.D., Tissera, M.D., Vladusich, T., van Schaik, A., and Tapson, J. (2015). Fast, simple and accurate handwritten digit classification by training shallow neural network classifiers with the ‘extreme learning machine’ algorithm. PLoS ONE, 10.10.1371/journal.pone.0134254 ↗
22. Extreme learning machine: Theory and applications10.1016/j.neucom.2005.12.126 ↗
23. Representational learning with extreme learning machine for big data
24. Wan, L., Zeiler, M., Zhang, S., LeCun, Y., and Fergus, R. (2013, January 16–21). Regularization of neural networks using DropConnect. Proceedings of the 30th International Conference on Machine Learning, Atlanta, GA, USA.
25. Zeiler, M.D., and Fergus, R. (2013). Stochastic Pooling for Regularization of Deep Convolutional Neural Networks. arXiv.
26. Goodfellow, I.J., Warde-Farley, D., Mirza, M., Courville, A., and Bengio, Y. (2013, January 16–21). Maxout networks. Proceedings of the 30th International Conference on Machine Learning, Atlanta, GA, USA.
27. Lee, C.Y., Xie, S., Gallagher, P.W., Zhang, Z., and Tu, Z. (2015, January 9–12). Deeply supervised nets. Proceedings of the 18th International Conference on Artificial Intelligence and Statistics, San Diego, CA, USA.
28. Sato, I., Nishimura, H., and Yokoi, K. (2015). APAC: Augmented PAttern Classification with Neural Networks. arXiv.
29. Chang, J.R., and Chen, Y.S. (2015). Batch-normalized Maxout Network in Network. arXiv.
30. Lee, C.Y., Gallagher, P.W., and Tu, Z. (2016, January 9–11). Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree. Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, Cadiz, Spain.
31. Liang, M., and Hu, X. (2015, January 7–12). Recurrent convolutional neural network for object recognition. Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.
32. Liao, Z., and Carneiro, G. (2015). On the Importance of Normalisation Layers in Deep Learning with Piecewise Linear Activation Units. arXiv.10.1109/WACV.2016.7477624 ↗
33. Liao, Z., and Carneiro, G. (2015). Competitive Multi-scale Convolution. arXiv.
34. Graham, B. (2015). Fractional Max-Pooling. arXiv.
35. McFonnell, M.D., and Vladusich, T. (2015, January 12–17). Enhanced Image Classification With a Fast-Learning Shallow Convolutional Neural Network. Proceedings of the 2015 International Joint Conference on Neural Networks, Killarney, Ireland.10.1109/IJCNN.2015.7280796 ↗
36. Mairal, J., Koniusz, P., Harchaoui, Z., and Schmid, C. (2014, January 8–13). Convolutional kernel networks. Proceedings of the 27th International Conference on Neural Information Processing Systems, Montreal, QC, Canada.
37. Multi-loss Regularized Deep Neural Network10.1109/TCSVT.2015.2477937 ↗
38. Srivastava, R.K., Greff, K., and Schmidhuber, J. (2015, January 7–12). Training Very Deep Networks. Proceedings of the 28th International Conference on Neural Information Processing Systems, Montreal, QC, Canada.
39. Lin, M., Chen, Q., and Yan, S. (2014, January 14–16). Network In Network. Proceedings of the 2nd International Conference on Learning Representations, Banff, AB, Canada.
40. Ranzato, M.A., Huang, F.J., Boureau, Y.L., and LeCun, Y. (2007, January 17–22). Unsupervised Learning of Invariant Feature Hierarchies with Applications to Object Recognition. Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, USA.10.1109/CVPR.2007.383157 ↗
41. Invariant Scattering Convolution Networks10.1109/TPAMI.2012.230 ↗
42. Calderón, A., Roa-Valle, S., and Victorino, J. (2003, January 19–21). Handwritten digit recognition using convolutional neural networks and Gabor filters. Proceedings of the 2003 International Conference on Computational Intelligence, Cancun, Mexico.
43. Le, Q.V., Ngiam, J., Coates, A., Prochnow, B., and Ng, A.Y. (July, January 28). On Optimization Methods for Deep Learning. Proceedings of the 28th International Conference on Machine Learning, Washington, DC, USA.
44. Yang, Z., Moczulski, M., Denil, M., de Freitas, N., Smola, A., Song, L., and Wang, Z. (2015, January 7–13). Deep Fried Convnets. Proceedings of the 2015 IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.173 ↗
45. Hertel, L., Barth, E., Käster, T., and Martinetz, T. (2015, January 12–16). Deep convolutional neural networks as generic feature extractors. Proceedings of the 2015 International Joint Conference on Neural Networks, Killarney, Ireland.10.1109/IJCNN.2015.7280683 ↗
46. Unsupervised feature learning with C-SVDDNet10.1016/j.patcog.2016.06.001 ↗
47. Hybrid Orthogonal Projection and Estimation (HOPE): A New Framework to Learn Neural Networks
48. Visin, F., Kastner, K., Cho, K., Matteucci, M., Courville, A., and Bengio, Y. (2015). ReNet: A Recurrent Neural Network Based Alternative to Convolutional Networks. arXiv.
49. Trainable COSFIRE Filters for Keypoint Detection and Pattern Recognition10.1109/TPAMI.2012.106 ↗
50. PCANet: A Simple Deep Learning Baseline for Image Classification?10.1109/TIP.2015.2475625 ↗
51. Task-Driven Dictionary Learning10.1109/TPAMI.2011.156 ↗
52. Jia, Y., Huang, C., and Darrell, T. (2012, January 16–21). Beyond spatial pyramids: Receptive field learning for pooled image features. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA.
53. Sparse Activity and Sparse Connectivity in Supervised Learning
54. Lee, H., Grosse, R., Ranganath, R., and Ng, A.Y. (2009, January 14–18). Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Proceedings of the 26th Annual International Conference on Machine Learning, Montreal, QC, Canada.10.1145/1553374.1553453 ↗
55. Min, R., Stanley, D.A., Yuan, Z., Bonner, A., and Zhang, Z. (2009). A Deep Non-Linear Feature Mapping for Large-Margin kNN Classification. arXiv.10.1109/ICDM.2009.27 ↗
56. Yang, J., Yu, K., and Huang, T. (2010, January 13–18). Supervised translation-invariant sparse coding. Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition, San Francisco, CA, USA.10.1109/CVPR.2010.5539958 ↗
57. Salakhutdinov, R., and Hinton, G. (2009, January 16–18). Deep Boltzmann Machines. Proceedings of the 12th International Conference on Artificial Intelligence and Statistics, Clearwater Beach, FL, USA.
58. Goodfellow, I.J., Mirza, M., Courville, A., and Bengio, Y. (2013). Multi-Prediction Deep Boltzmann Machines. Advances in Neural Information Processing Systems 26, NIPS Proceedings, Neural Information Processing Systems Foundation, Inc.
59. Mishkin, D., and Matas, J. (2016, January 2–4). All you need is a good init. Proceedings of the 4th International Conference on Learning Representations, Scottsdale, AZ, USA.
60. Alom, M.Z., Hasan, M., Yakopcic, C., and Taha, T.M. (2017). Inception Recurrent Convolutional Neural Network for Object Recognition. arXiv.10.1109/IJCNN.2018.8489635 ↗
61. Baker, B., Gupta, O., Naik, N., and Raskar, R. (2017, January 24–26). Designing Neural Network Architectures using Reinforcement Learning. Proceedings of the 5th International Conference on Learning Representations, Toulon, France.
62. Davison, J. (2018, May 04). DEvol: Automated Deep Neural Network Design via Genetic Programming. Available online: https://github.com/joeddav/devol.
63. Evolutionary Convolutional Neural Networks: An Application to Handwriting Recognition10.1016/j.neucom.2017.12.049 ↗
64. Hybridizing Evolutionary Computation and Deep Neural Networks: An Approach to Handwriting Recognition Using Committees and Transfer Learning10.1155/2019/2952304 ↗
65. Bochinski, E., Senst, T., and Sikora, T. (2017, January 17–20). Hyper-parameter optimization for convolutional neural network committees based on evolutionary algorithms. Proceedings of the 2017 IEEE International Conference on Image Processing, Beijing, China.10.1109/ICIP.2017.8297018 ↗
66. Model Selection in Committees of Evolved Convolutional Neural Networks Using Genetic Algorithms10.1007/978-3-030-03493-1_39 ↗
67. Cohen, G., Afshar, S., Tapson, J., and van Schaik, A. (2017). EMNIST: An extension of MNIST to handwritten letters. arXiv.10.1109/IJCNN.2017.7966217 ↗
68. NIST (2018, April 28). NIST Special Database 19, Available online: https://www.nist.gov/srd/nist-special-database-19.
69. Grother, P.J., and Hanaoka, K.K. (2016). NIST Special Database 19 Handprinted Forms and Characters Database, Technical Report.
70. Online and adaptive pseudoinverse solutions for ELM weights
71. Ghadekar, P., Ingole, S., and Sonone, D. (2018, January 16–18). Handwritten Digit and Letter Recognition Using Hybrid DWT-DCT with KNN and SVM Classifier. Proceedings of the 4th International Conference on Computing Communication Control and Automation, Pune, India.10.1109/ICCUBEA.2018.8697684 ↗
72. Botalb, A., Moinuddin, M., Al-Saggaf, U.M., and Ali, S.S.A. (2018, January 13–14). Contrasting Convolutional Neural Network (CNN) with Multi-Layer Perceptron (MLP) for Big Data Analysis. Proceedings of the 2018 International Conference on Intelligent and Advanced System, Kuala Lumpur, Malaysia.10.1109/ICIAS.2018.8540626 ↗
73. Markov Random Field Based Convolutional Neuralx Networks for Image Classification
74. Singh, S., Paul, A., and Arun, M. (2017, January 4–5). Parallelization of digit recognition system using Deep Convolutional Neural Network on CUDA. Proceedings of the 2017 Third International Conference on Sensing, Signal Processing and Security, Chennai, India.10.1109/SSPS.2017.8071623 ↗
75. Handwritten text recognition: With deep learning and Android
76. Sabour, S., Frosst, N., and Hinton, G.E. (2017). Dynamic Routing Between Capsules. Advances in Neural Information Processing Systems 30, Neural Information Processing Systems Foundation, Inc.. NIPS Proceedings.
77. Jayasundara, V., Jayasekara, S., Jayasekara, N.H., Rajasegaran, J., Seneviratne, S., and Rodrigo, R. (January, January 7–11). TextCaps: Handwritten character recognition with very small datasets. Proceedings of the 2019 IEEE Winter Conference on Applications of Computer Vision, Waikoloa Village, HI, USA.10.1109/WACV.2019.00033 ↗
78. Deep convolutional extreme learning machines: Filters combination and error model validation10.1016/j.neucom.2018.10.063 ↗
79. Confusion Matrix-Based Building of Hierarchical Classification
80. Dufourq, E., and Bassett, B.A. (2017). EDEN: Evolutionary Deep Networks for Efficient Machine Learning. arXiv.10.1109/RoboMech.2017.8261132 ↗
81. Event-Driven Random Back-Propagation: Enabling Neuromorphic Deep Learning Machines10.3389/fnins.2017.00324 ↗
82. Shu, L., Xu, H., and Liu, B. (2018). Unseen Class Discovery in Open-world Classification. arXiv.
83. Optical Character Recognition on Bank Cheques Using 2D Convolution Neural Network
84. Sharma, A.S., Mridul, M.A., Jannat, M.E., and Islam, M.S. (2018, January 21–22). A Deep CNN Model for Student Learning Pedagogy Detection Data Collection Using OCR. Proceedings of the 2018 International Conference on Bangla Speech and Language Processing, Sylhet, Bangladesh.
85. Shawon, A., Rahman, M.J.U., Mahmud, F., and Zaman, M.A. (2018, January 21–22). Bangla Handwritten Digit Recognition Using Deep CNN for Large and Unbiased Dataset. Proceedings of the 2018 International Conference on Bangla Speech and Language Processing, Sylhet, Bangladesh.10.1109/ICBSLP.2018.8554900 ↗
86. Milgram, J., Cheriet, M., and Sabourin, R. (August, January 31). Estimating accurate multi-class probabilities with support vector machines. Proceedings of the 2005 IEEE International Joint Conference on Neural Networks, Montreal, QC, Canada.
87. Supervised Learning of Fuzzy ARTMAP Neural Networks Through Particle Swarm Optimisation
88. Automatic Recognition of Handwritten Numerical Strings: A Recognition and Verification Strategy10.1109/TPAMI.2002.1046154 ↗
89. Radtke, P.V.W., Sabourin, R., and Wong, T. (2008, January 16–20). Using the RRT algorithm to optimize classification systems for handwritten digits and letters. Proceedings of the 2008 ACM Symposium on Applied Computing, Fortaleza, Brazil.10.1145/1363686.1364103 ↗
90. Koerich, A.L., and Kalva, P.R. (2005, January 11–14). Unconstrained handwritten character recognition using metaclasses of characters. Proceedings of the 2005 IEEE International Conference on Image Processing, Genoa, Italy.10.1109/ICIP.2005.1530112 ↗
91. Cavalin, P.R., Britto, A.S., Bortolozzi, F., Sabourin, R., and Oliveira, L.E.S. (2006, January 23–27). An implicit segmentation-based method for recognition of handwritten strings of characters. Proceedings of the 2006 ACM Symposium on Applied Computing, Dijon, France.10.1145/1141277.1141468 ↗