TASE: Task-Aware Speech Enhancement for Wake-Up Word Detection in Voice Assistants
Source abstract
Abstract
Wake-up word spotting in noisy environments is a critical task for an excellent user experience with voice assistants. Unwanted activation of the device is often due to the presence of noises coming from background conversations, TVs, or other domestic appliances. In this work, we propose the use of a speech enhancement convolutional autoencoder, coupled with on-device keyword spotting, aimed at improving the trigger word detection in noisy environments. The end-to-end system learns by optimizing a linear combination of losses: a reconstruction-based loss, both at the log-mel spectrogram and at the waveform level, as well as a specific task loss that accounts for the cross-entropy error reported along the keyword spotting detection. We experiment with several neural network classifiers and report that deeply coupling the speech enhancement together with a wake-up word detector, e.g., by jointly training them, significantly improves the performance in the noisiest conditions. Additionally, we introduce a new publicly available speech database recorded for the Telefónica’s voice assistant, Aura. The OK Aura Wake-up Word Dataset incorporates rich metadata, such as speaker demographics or room conditions, and comprises hard negative examples that were studiously selected to present different levels of phonetic similarity with respect to the trigger words “OK Aura”.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech Recognition and Synthesis · Speech and Audio Processing · Music and Audio Processing
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Telefónica I+D, Research, 08019 Barcelona, Spain
TALN Natural Language Processing Group, Universitat Pompeu Fabra, 08018 Barcelona, Spain
Telefónica I+D, Digital Home, 28050 Madrid, Spain
AUDIAS Audio, Data Intelligence and Speech Group, Universidad Autónoma de Madrid, 28049 Madrid, Spain
TSC Signal Theory and Communications Department, Universitat Politècnica de Catalunya, 08034 Barcelona, Spain
CLiC Language and Computation Center, UBICS UB Institute of Complex Systems, Universitat de Barcelona, 08007 Barcelona, Spain
Funding
European Union · Agencia Estatal de Investigación, Ministerio de Ciencia, Innovación y Universidades and the Fondo Social Europeo
References (55)
Show source-provided references
1. Bonet, D., Cámbara, G., López, F., Gómez, P., Segura, C., Luque, J., and Farrús, M. (2021, January 24–25). Speech Enhancement for Wake-Up-Word detection in Voice Assistants. Proceedings of the IberSPEECH 2021, Valladolid, Spain.10.21437/IberSPEECH.2021-9 ↗
2. Ge, F., and Yan, Y. (2017, January 5–9). Deep neural network based wake-up-word speech recognition with two-stage detection. Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA.10.1109/ICASSP.2017.7952659 ↗
3. Kumar, R., Rodehorst, M., Wang, J., Gu, J., and Kulis, B. (2020, January 25–29). Building a Robust Word-Level Wakeword Verification Network. Proceedings of the INTERSPEECH, Shanghai, China.10.21437/Interspeech.2020-2018 ↗
4. Michaely, A.H., Zhang, X., Simko, G., Parada, C., and Aleksic, P. (2017, January 16–20). Keyword spotting for Google assistant using contextual speech recognition. Proceedings of the 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), Okinawa, Japan.10.1109/ASRU.2017.8268946 ↗
5. (2021, December 29). Hey Siri: An On-device DNN-Powered Voice Trigger for Apple’s Personal Assistant. Available online: https://machinelearning.apple.com/research/hey-siri.
6. Loizou, P.C. (2013). Speech Enhancement: Theory and Practice, CRC Press.10.1201/b14529 ↗
7. An experimental study on speech enhancement based on deep neural networks10.1109/LSP.2013.2291240 ↗
8. Reddy, C.K., Beyrami, E., Pool, J., Cutler, R., Srinivasan, S., and Gehrke, J. (2019). A scalable noisy speech dataset and online subjective test framework. arXiv.10.21437/Interspeech.2019-3087 ↗
9. Spectral subtraction-based speech enhancement for cochlear implant patients in background noise10.1121/1.1852873 ↗
10. Zorilă, C., Boeddeker, C., Doddipatla, R., and Haeb-Umbach, R. (2019, January 14–18). An investigation into the effectiveness of enhancement in ASR training and test for chime-5 dinner party transcription. Proceedings of the 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), Singapore.10.1109/ASRU46091.2019.9003785 ↗
11. Maas, A.L., Le, Q.V., O’Neil, T.M., Vinyals, O., Nguyen, P., and Ng, A.Y. (2012, January 9–13). Recurrent Neural Networks for Noise Reduction in Robust ASR. Proceedings of the Thirteenth Annual Conference of the International Speech Communication Association, Portland, OR, USA.10.21437/Interspeech.2012-6 ↗
12. Weninger, F., Erdogan, H., Watanabe, S., Vincent, E., Le Roux, J., Hershey, J.R., and Schuller, B. (2015). Speech enhancement with LSTM recurrent neural networks and its application to noise-robust ASR. International Conference on Latent Variable Analysis and Signal Separation, Springer.10.1007/978-3-319-22482-4_11 ↗
13. Sainath, T.N., and Parada, C. (2015, January 6–10). Convolutional neural networks for small-footprint keyword spotting. Proceedings of the Sixteenth Annual Conference of the International Speech Communication Association, Dresden, Germany.10.21437/Interspeech.2015-352 ↗
14. Kumar, R., Yeruva, V., and Ganapathy, S. (2018, January 2–6). On Convolutional LSTM Modeling for Joint Wake-Word Detection and Text Dependent Speaker Verification. Proceedings of the Interspeech, Hyderabad, India.10.21437/Interspeech.2018-1759 ↗
15. Arik, S.O., Kliegl, M., Child, R., Hestness, J., Gibiansky, A., Fougner, C., Prenger, R., and Coates, A. (2017). Convolutional recurrent neural networks for small-footprint keyword spotting. arXiv.10.21437/Interspeech.2017-1737 ↗
16. Yamamoto, T., Nishimura, R., Misaki, M., and Kitaoka, N. (2019, January 15–19). Small-Footprint Magic Word Detection Method Using Convolutional LSTM Neural Network. Proceedings of the INTERSPEECH, Graz, Austria.10.21437/Interspeech.2019-1662 ↗
17. Shan, C., Zhang, J., Wang, Y., and Xie, L. (2018). Attention-based end-to-end models for small-footprint keyword spotting. arXiv.10.21437/Interspeech.2018-1777 ↗
18. Raju, A., Panchapagesan, S., Liu, X., Mandal, A., and Strom, N. (2018). Data augmentation for robust keyword spotting under playback interference. arXiv.
19. Hsiao, R., Ma, J., Hartmann, W., Karafiát, M., Grézl, F., Burget, L., Szöke, I., Černockỳ, J.H., Watanabe, S., and Chen, Z. (2015, January 13–17). Robust speech recognition in unknown reverberant and noisy conditions. Proceedings of the 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), Scottsdale, AZ, USA.10.1109/ASRU.2015.7404841 ↗
20. Hannun, A., Case, C., Casper, J., Catanzaro, B., Diamos, G., Elsen, E., Prenger, R., Satheesh, S., Sengupta, S., and Coates, A. (2014). Deep speech: Scaling up end-to-end speech recognition. arXiv.
21. Yoon, K.M., and Kim, W. (2020). Small-Footprint Wake Up Word Recognition in Noisy Environments Employing Competing-Words-Based Feature. Electronics, 9.10.3390/electronics9122202 ↗
22. Meyer, J., and Simmer, K.U. (1997, January 21–24). Multi-channel speech enhancement in a car environment using Wiener filtering and spectral subtraction. Proceedings of the 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing, Munich, Germany.
23. A signal subspace approach for speech enhancement10.1109/89.397090 ↗
24. Park, S.R., and Lee, J. (2016). A fully convolutional neural network for speech enhancement. arXiv.10.21437/Interspeech.2017-1465 ↗
25. Rethage, D., Pons, J., and Serra, X. (2018, January 15–20). A wavenet for speech denoising. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8462417 ↗
26. Phan, H., McLoughlin, I.V., Pham, L., Chén, O.Y., Koch, P., De Vos, M., and Mertins, A. (2020). Improving GANs for speech enhancement. arXiv.10.1109/LSP.2020.3025020 ↗
27. Pascual, S., Bonafonte, A., and Serra, J. (2017). SEGAN: Speech enhancement generative adversarial network. arXiv.10.21437/Interspeech.2017-1428 ↗
28. Generative adversarial nets
29. Ronneberger, O., Fischer, P., and Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer.10.1007/978-3-319-24574-4_28 ↗
30. Défossez, A., Synnaeve, G., and Adi, Y. (2020). Real time speech enhancement in the waveform domain. arXiv.10.21437/Interspeech.2020-2409 ↗
31. Long short-term memory10.1162/neco.1997.9.8.1735 ↗
32. Fu, S.W., Tsao, Y., Hwang, H.T., and Wang, H.M. (2018). Quality-Net: An end-to-end non-intrusive speech quality assessment model based on BLSTM. arXiv.10.21437/Interspeech.2018-1802 ↗
33. Learning with learned loss function: Speech enhancement with quality-net to improve perceptual evaluation of speech quality10.1109/LSP.2019.2953810 ↗
34. Progressive loss functions for speech enhancement with deep neural networks10.1186/s13636-020-00191-3 ↗
35. Braun, S., and Tashev, I. (2020). Data augmentation and loss normalization for deep noise suppression. International Conference on Speech and Computer, Springer.10.1007/978-3-030-60276-5_8 ↗
36. Yamamoto, R., Song, E., and Kim, J.M. (2020, January 4–8). Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram. Proceedings of the ICASSP 2020—2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.10.1109/ICASSP40776.2020.9053795 ↗
37. Ardila, R., Branson, M., Davis, K., Henretty, M., Kohler, M., Meyer, J., Morais, R., Saunders, L., Tyers, F.M., and Weber, G. (2019). Common Voice: A massively-multilingual speech corpus. arXiv.
38. Cámbara, G., Luque, J., Bonet, D., López, F., Farrús, M., Gómez, P., and Segura, C. (2021). OK Aura Wake-up Word Dataset. Zenodo.
39. Hou, J., Shi, Y., Ostendorf, M., Hwang, M.Y., and Xie, L. (2020, January 4–8). Mining effective negative training samples for keyword spotting. Proceedings of the ICASSP 2020—2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.10.1109/ICASSP40776.2020.9053009 ↗
40. Dean, D.B., Sridharan, S., Vogt, R.J., and Mason, M.W. (2010, January 26–30). The QUT-NOISE-TIMIT corpus for the evaluation of voice activity detection algorithms. Proceedings of the 11th Annual Conference of the International Speech Communication Association, Chiba, Japan.10.21437/Interspeech.2010-774 ↗
41. Lleida, E., Ortega, A., Miguel, A., Bazán-Gil, V., Pérez, C., Gómez, M., and de Prada, A. (2019). Albayzin 2018 evaluation: The IberSpeech-RTVE challenge on speech technologies for Spanish broadcast media. Appl. Sci., 9.10.3390/app9245412 ↗
42. Bredin, H., Yin, R., Coria, J.M., Gelly, G., Korshunov, P., Lavechin, M., Fustes, D., Titeux, H., Bouaziz, W., and Gill, M.P. (2020, January 4–8). Pyannote.audio: Neural building blocks for speaker diarization. Proceedings of the ICASSP 2020, IEEE International Conference on Acoustics, Speech, and Signal Processing, Speech, and Signal Processing, Barcelona, Spain.10.1109/ICASSP40776.2020.9052974 ↗
43. Unleashing the killer corpus: Experiences in creating the multi-everything AMI Meeting Corpus10.1007/s10579-007-9040-x ↗
44. Image method for efficiently simulating small-room acoustics10.1121/1.382599 ↗
45. LeCun, Y. (2022, February 01). LeNet-5, Convolutional Neural Networks. Available online: http://yann.lecun.com/exdb/lenet.
46. Tang, R., and Lin, J. (2017). Honk: A PyTorch Reimplementation of Convolutional Neural Networks for Keyword Spotting. arXiv.
47. Tang, R., and Lin, J. (2018, January 15–20). Deep Residual Learning for Small-Footprint Keyword Spotting. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8462688 ↗
48. Scholefield, M.D. (2022, February 01). Mycroft Precise. Available online: https://github.com/MycroftAI/mycroft-precise.
49. mhiro2 (2022, February 01). Freesound Audio Tagging 2019: Simple 2D-CNN Classifier with PyTorch. Available online: https://www.kaggle.com/mhiro2/simple-2d-cnn-classifier-with-pytorch/.
50. Fonseca, E., Plakal, M., Font, F., Ellis, D.P., and Serra, X. (2019). Audio tagging with noisy labels and minimal supervision. arXiv.10.33682/w13e-5v06 ↗
51. Cámbara, G., Luque, J., and Farrús, M. (2020, January 4–8). Detection of speech events and speaker characteristics through photo-plethysmographic signal neural processing. Proceedings of the ICASSP 2020—2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain.10.1109/ICASSP40776.2020.9052972 ↗
52. Index for rating diagnostic tests10.1002/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3 ↗
53. Rix, A.W., Beerends, J.G., Hollier, M.P., and Hekstra, A.P. (2001, January 7–11). Perceptual evaluation of speech quality (PESQ)—A new method for speech quality assessment of telephone networks and codecs. Proceedings of the 2001 IEEE International Conference on Acoustics, Speech, and Signal Processing, Proceedings (Cat. No. 01CH37221), Salt Lake City, UT, USA.
54. Taal, C.H., Hendriks, R.C., Heusdens, R., and Jensen, J. (2010, January 14–19). A short-time objective intelligibility measure for time-frequency weighted noisy speech. Proceedings of the 2010 IEEE international conference on acoustics, speech and signal processing, Dallas, TX, USA.10.1109/ICASSP.2010.5495701 ↗
55. Valentini Botinhao, C., Wang, X., Takaki, S., and Yamagishi, J. (2016, January 8–12). Speech Enhancement for a Noise-Robust Text-to-Speech Synthesis System using Deep Recurrent Neural Networks. Proceedings of the 7th Annual Conference of the International Speech Communication Association, San Francisco, CA, USA.10.21437/Interspeech.2016-159 ↗