Temporal Convolution Network Based Joint Optimization of Acoustic-to-Articulatory Inversion
Source abstract
Abstract
Articulatory features are proved to be efficient in the area of speech recognition and speech synthesis. However, acquiring articulatory features has always been a difficult research hotspot. A lightweight and accurate articulatory model is of significant meaning. In this study, we propose a novel temporal convolution network-based acoustic-to-articulatory inversion system. The acoustic feature is converted into a high-dimensional hidden space feature map through temporal convolution with frame-level feature correlations taken into account. Meanwhile, we construct a two-part target function combining prediction’s Root Mean Square Error (RMSE) and the sequences’ Pearson Correlation Coefficient (PCC) to jointly optimize the performance of the specific inversion model from both aspects. We also further conducted an analysis on the impact of the weight between the two parts on the final performance of the inversion model. Extensive experiments have shown that our, temporal convolution networks (TCN) model outperformed the Bi-derectional Long Short Term Memory model by 1.18 mm in RMSE and 0.845 in PCC with 14 model parameters when optimizing evenly with RMSE and PCC aspects.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech and Audio Processing · Speech Recognition and Synthesis · Music and Audio Processing
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Key Laboratory of Speech Acoustics and Content Understanding, Institute of Acoustics, Chinese Academy of Sciences, Beijing 100190, China
University of Chinese Academy of Sciences, Beijing 100049, China
The Xinjiang Technical Institute of Physics & Chemistry, Chinese Academy of Sciences, Urumqi 830011, China
References (30)
Show source-provided references
1. Qin, C., and Carreira-Perpiñán, M.Á. (2007, January 27–31). An empirical investigation of the nonuniqueness in the acoustic-to-articulatory mapping. Proceedings of the Eighth Annual Conference of the International Speech Communication Association, Antwerp, Belgium.10.21437/Interspeech.2007-16 ↗
2. Combining acoustic and articulatory feature information for robust speech recognition10.1016/S0167-6393(01)00020-6 ↗
3. Mitra, V., Sivaraman, G., Bartels, C., Nam, H., Wang, W., Espy-Wilson, C., Vergyri, D., and Franco, H. (2017, January 5–9). Joint modeling of articulatory and acoustic spaces for continuous speech recognition tasks. Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA.10.1109/ICASSP.2017.7953149 ↗
4. Articulatory and spectrum information fusion based on deep recurrent neural networks10.1109/TASLP.2019.2894554 ↗
5. Hybrid convolutional neural networks for articulatory and acoustic information based speech recognition10.1016/j.specom.2017.03.003 ↗
6. The use of articulatory movement data in speech synthesis applications: An overview—Application of articulatory movements using machine learning algorithms10.1250/ast.36.467 ↗
7. Cao, B., Kim, M.J., van Santen, J.P.H., Mau, T., and Wang, J. (2017, January 20–24). Integrating Articulatory Information in Deep Learning-Based Text-to-Speech Synthesis. Proceedings of the Interspeech 2017, Stockholm, Sweden.10.21437/Interspeech.2017-1762 ↗
8. Aryal, S., and Gutierrez-Osuna, R. (2014, January 4–9). Accent conversion through cross-speaker articulatory synthesis. Proceedings of the 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Florence, Italy.10.1109/ICASSP.2014.6855097 ↗
9. Articulatory and bottleneck features for speaker-independent ASR of dysarthric speech10.1016/j.csl.2019.05.002 ↗
10. Hahm, S., and Wang, J. (2015, January 6–10). Parkinson’s condition estimation using speech acoustic and inversely mapped articulatory data. Proceedings of the Sixteenth Annual Conference of the International Speech Communication Association, Dresden, Germany.10.21437/Interspeech.2015-186 ↗
11. Jones, D.K. (2017). Development of Kinematic Templates for Automatic Pronunciation Assessment Using Acoustic-to-Articulatory Inversion, Marquette University.
12. Improving Mandarin tone recognition based on DNN by combining acoustic and articulatory features using extended recognition networks10.1007/s11265-018-1334-2 ↗
13. Articulatory modeling for pronunciation error detection without non-native training data based on DNN transfer learning10.1587/transinf.2017EDP7019 ↗
14. Quantifying kinematic aspects of reduction in a contrasting rate production task10.1121/1.4987629 ↗
15. Westbury, J.R., Turner, G., and Dembowski, J. (1994). X-ray Microbeam Speech Production Database User’s Handbook, Waisman Center on Mental Retardation and Human Development.
16. An approach to real-time magnetic resonance imaging for speech production10.1121/1.1652588 ↗
17. Inversion of articulatory-to-acoustic transformation in the vocal tract by a computer-sorting technique10.1121/1.381848 ↗
18. Dusan, S., and Deng, L. (2000, January 1–4). Acoustic-to-articulatory inversion using dynamical and phonological constraints. Proceedings of the 5th Seminar on Speech Production, Kloster Seeon, Germany.
19. Toda, T., Black, A., and Tokuda, K. (2004, January 4–8). Acoustic-to-articulatory inversion mapping with Gaussian mixture model. Proceedings of the Eighth International Conference on Spoken Language Processing, Jeju-si, Korea.10.21437/Interspeech.2004-410 ↗
20. Acoustic-articulatory modeling with the trajectory HMM10.1109/LSP.2008.917004 ↗
21. Tobing, P.L., Kameoka, H., and Toda, T. (2017, January 12–15). Deep acoustic-to-articulatory inversion mapping with latent trajectory modeling. Proceedings of the 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), Kuala Lumpur, Malaysia.10.1109/APSIPA.2017.8282219 ↗
22. Uria, B., Murray, I., Renals, S., and Richmond, K. (2012, January 9–13). Deep architectures for articulatory inversion. Proceedings of the Thirteenth Annual Conference of the International Speech Communication Association, Portland, Oregon.10.21437/Interspeech.2012-263 ↗
23. Richmond, K. (2007, January 22–25). Trajectory mixture density networks with multiple mixtures for acoustic-articulatory inversion. Proceedings of the International Conference on Nonlinear Speech Processing, Paris, France.
24. Liu, P., Yu, Q., Wu, Z., Kang, S., Meng, H., and Cai, L. (2015, January 19–24). A deep recurrent approach for acoustic-to-articulatory inversion. Proceedings of the 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), South Brisbane, Australia.10.1109/ICASSP.2015.7178812 ↗
25. Bai, S., Kolter, J.Z., and Koltun, V. (2018). An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv.
26. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
27. Nair, V., and Hinton, G.E. (2010, January 21–24). Rectified linear units improve restricted boltzmann machines. Proceedings of the 27th International Conference on Machine Learning, Haifa, Israel.
28. Parrot, M., Millet, J., and Dunbar, E. (2020, January 25–29). Independent and Automatic Evaluation of Speaker-Independent Acoustic-to-Articulatory Reconstruction. Proceedings of the Interspeech 2020, Shanghai, China.10.21437/Interspeech.2020-1746 ↗
29. Unsupervised speaker adaptation for speaker independent acoustic to articulatory speech inversion10.1121/1.5116130 ↗
30. Real-time magnetic resonance imaging and electromagnetic articulography database for speech production research (TC)10.1121/1.4890284 ↗