Text Normalization Using Encoder–Decoder Networks Based on the Causal Feature Extractor
Source abstract
Abstract
The encoder–decoder architecture is a well-established, effective and widely used approach in many tasks of natural language processing (NLP), among other domains. It consists of two closely-collaborating components: An encoder that transforms the input into an intermediate form, and a decoder producing the output. This paper proposes a new method for the encoder, named Causal Feature Extractor (CFE), based on three main ideas: Causal convolutions, dilatations and bidirectionality. We apply this method to text normalization, which is a ubiquitous problem that appears as the first step of many text-to-speech (TTS) systems. Given a text with symbols, the problem consists in writing the text exactly as it should be read by the TTS system. We make use of an attention-based encoder–decoder architecture using a fine-grained character-level approach rather than the usual word-level one. The proposed CFE is compared to other common encoders, such as convolutional neural networks (CNN) and long-short term memories (LSTM). Experimental results show the feasibility of CFE, achieving better results in terms of accuracy, number of parameters, convergence time, and use of an attention mechanism based on attention matrices. The obtained accuracy ranges from 83.5% to 96.8% correctly normalized sentences, depending on the dataset. Moreover, the proposed method is generic and can be applied to different types of input such as text, audio and images.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Natural Language Processing Techniques · Topic Modeling · Speech Recognition and Synthesis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Max Planck Institute for Intelligent Systems, 72076 Tübingen, Germany
Department of Computer Science and Systems, University of Murcia, 30100 Murcia, Spain
References (29)
Show source-provided references
1. Dabre, R., Chu, C., and Kunchukuttan, A. (2020). A Comprehensive Survey of Multilingual Neural Machine Translation. arXiv.10.18653/v1/2020.coling-tutorials.3 ↗
2. Recent automatic text summarization techniques: A survey10.1007/s10462-016-9475-9 ↗
3. Survey of the state of the art in natural language generation: Core tasks, applications and evaluation10.1613/jair.5477 ↗
4. Minaee, S., Kalchbrenner, N., Cambria, E., Nikzad, N., Chenaghlu, M., and Gao, J. (2020). Deep Learning Based Text Classification: A Comprehensive Review. arXiv.10.1145/3439726 ↗
5. Van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A.W., and Kavukcuoglu, K. (2016, January 13–15). WaveNet: A Generative Model for Raw Audio. Proceedings of the 9th ISCA Speech Synthesis Workshop, Sunnyvale, CA, USA.
6. Sproat, R., and Jaitly, N. (2016). RNN Approaches to Text Normalization: A Challenge. arXiv.10.21437/Interspeech.2017-35 ↗
7. Multilingual text analysis for text-to-speech synthesis10.1017/S1351324997001654 ↗
8. Sodimana, K., Silva, P.D., Sproat, R., Theeraphol, A., Li, C.F., Gutkin, A., Sarin, S., and Pipatsrisawat, K. (2018, January 29–31). Text Normalization for Bangla, Khmer, Nepali, Javanese, Sinhala, and Sundanese TTS Systems. Proceedings of the 6th International Workshop on Spoken Language Technologies for Under-Resourced Languages (SLTU-2018), Gurugram, India.10.21437/SLTU.2018-31 ↗
9. Ikeda, T., Shindo, H., and Matsumoto, Y. (2016, January 11). Japanese Text Normalization with Encoder–Decoder Model. Proceedings of the 2nd Workshop on Noisy User-generated Text, NUT@COLING 2016, Osaka, Japan.
10. Character level embedding with deep convolutional neural network for text normalization of unstructured data for Twitter sentiment analysis10.1007/s13278-019-0557-y ↗
11. Neural models of text normalization for speech applications10.1162/coli_a_00349 ↗
12. Roark, B., Sproat, R., Allauzen, C., Riley, M., Sorensen, J., and Tai, T. (2012, January 10). The OpenGrm open-source finite-state grammar software libraries. Proceedings of the ACL 2012 System Demonstrations, Jeju Island, Korea.
13. Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A.C., Salakhutdinov, R., Zemel, R.S., and Bengio, Y. (2015, January 6–11). Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France.
14. Chung, J., Cho, K., and Bengio, Y. (2016, January 7–12). A Character-level Decoder without Explicit Segmentation for Neural Machine Translation. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, Berlin, Germany.10.18653/v1/P16-1160 ↗
15. Fully Character-Level Neural Machine Translation without Explicit Segmentation10.1162/tacl_a_00067 ↗
16. Cho, K., Van Merriënboer, B., Bahdanau, D., and Bengio, Y. (2014). On the properties of neural machine translation: Encoder–decoder approaches. arXiv.10.3115/v1/W14-4012 ↗
17. Sutskever, I., Vinyals, O., and Le, Q.V. (2014, January 8–13). Sequence to Sequence Learning with Neural Networks. Proceedings of the Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, Montreal, QC, Canada.
18. Dropout: A simple way to prevent neural networks from overfitting
19. The dropout learning algorithm10.1016/j.artint.2014.02.004 ↗
20. Salimans, T., and Kingma, D.P. (2016, January 5–10). Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Proceedings of the 30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain.
21. Pascanu, R., Mikolov, T., and Bengio, Y. (2013, January 16–21). On the difficulty of training recurrent neural networks. Proceedings of the International Conference on Machine Learning (ICML 2013), Atlanta, GA, USA.
22. Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.
23. Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural Machine Translation by Jointly Learning to Align and Translate. arXiv.
24. Riezler, S., and Maxwell, J.T. (2005, January 29). On Some Pitfalls in Automatic Evaluation and Significance Testing for MT. Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization@ACL 2005, Ann Arbor, MI, USA.
25. Permutation tests for studying classifier performance
26. Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. (2017, January 4–9). Automatic differentiation in pytorch. Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.
27. Klein, G., Kim, Y., Deng, Y., Senellart, J., and Rush, A.M. (2017). Opennmt: Open-source toolkit for neural machine translation. arXiv.10.18653/v1/P17-4012 ↗
28. Glorot, X., and Bengio, Y. (2010, January 13–15). Understanding the difficulty of training deep feedforward neural networks. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Sardinia, Italy.
29. Binary codes capable of correcting deletions, insertions, and reversals