Unsupervised Adaptation of Deep Speech Activity Detection Models to Unseen Domains
Source abstract
Abstract
Speech Activity Detection (SAD) aims to accurately classify audio fragments containing human speech. Current state-of-the-art systems for the SAD task are mainly based on deep learning solutions. These applications usually show a significant drop in performance when test data are different from training data due to the domain shift observed. Furthermore, machine learning algorithms require large amounts of labelled data, which may be hard to obtain in real applications. Considering both ideas, in this paper we evaluate three unsupervised domain adaptation techniques applied to the SAD task. A baseline system is trained on a combination of data from different domains and then adapted to a new unseen domain, namely, data from Apollo space missions coming from the Fearless Steps Challenge. Experimental results demonstrate that domain adaptation techniques seeking to minimise the statistical distribution shift provide the most promising results. In particular, Deep CORAL method reports a 13% relative improvement in the original evaluation metric when compared to the unadapted baseline model. Further experiments show that the cascaded application of Deep CORAL and pseudo-labelling techniques can improve even more the results, yielding a significant 24% relative improvement in the evaluation metric when compared to the baseline system.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech and Audio Processing · Speech Recognition and Synthesis · Music and Audio Processing
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
ViVoLab, Aragón Institute for Engineering Research (I3A), University of Zaragoza, 50018 Zaragoza, Spain
Funding
European Union’s Horizon 2020 · MCIN/AEI · European Union “NextGenerationEU”/PRTR · Government of Aragón
References (73)
Show source-provided references
1. Gerven, S.V., and Xie, F. (1997, January 22–25). A comparative study of speech detection methods. Proceedings of the Fifth European Conference on Speech Communication and Technology, Rhodes, Greece.10.21437/Eurospeech.1997-108 ↗
2. A robust algorithm for word boundary detection in the presence of noise10.1109/89.294354 ↗
3. Voice activity detection based on multiple statistical models10.1109/TSP.2006.874403 ↗
4. Ng, T., Zhang, B., Nguyen, L., Matsoukas, S., Zhou, X., Mesgarani, N., Veselỳ, K., and Matějka, P. (2012, January 9–13). Developing a Speech Activity Detection system for the DARPA RATS program. Proceedings of the Interspeech, Portland, OR, USA.10.21437/Interspeech.2012-527 ↗
5. Robust voice activity detection algorithm for estimating noise spectrum10.1049/el:20000192 ↗
6. Efficient voice activity detection algorithms using long-term speech information10.1016/j.specom.2003.10.002 ↗
7. An energy-efficient voice activity detector using deep neural networks and approximate computing10.1016/j.mejo.2019.03.009 ↗
8. Vesperini, F., Vecchiotti, P., Principi, E., Squartini, S., and Piazza, F. (2016, January 24–29). Deep neural networks for multi-room voice activity detection: Advancements and comparative evaluation. Proceedings of the 2016 International Joint Conference on Neural Networks (IJCNN), Vancouver, BC, Canada.10.1109/IJCNN.2016.7727633 ↗
9. Fan, Z.C., Bai, Z., Zhang, X.L., Rahardja, S., and Chen, J. (2019, January 12–17). AUC Optimization for Deep Learning Based Voice Activity Detection. Proceedings of the IEEE ICASSP, Brighton, UK.10.1109/ICASSP.2019.8682803 ↗
10. Long short-term memory10.1162/neco.1997.9.8.1735 ↗
11. Kim, J., Kim, J., Lee, S., Park, J., and Hahn, M. (2016, January 21–24). Vowel based voice activity detection with LSTM recurrent neural network. Proceedings of the 8th International Conference on Signal Processing Systems, Auckland, New Zealand.10.1145/3015166.3015207 ↗
12. Exploring convolutional, recurrent, and hybrid deep neural networks for speech and music detection in a large audio dataset10.1186/s13636-019-0152-1 ↗
13. Viñals, I., Gimeno, P., Ortega, A., Miguel, A., and Lleida, E. (2018, January 2–6). Estimation of the Number of Speakers with Variational Bayesian PLDA in the DIHARD Diarization Challenge. Proceedings of the Interspeech, Hyderabad, India.10.21437/Interspeech.2018-1841 ↗
14. Vinals, I., Gimeno, P., Ortega, A., Miguel, A., and Lleida, E. (2018, January 21–23). In-domain Adaptation Solutions for the RTVE 2018 Diarization Challenge. Proceedings of the IberSPEECH, Barcelona, Spain.10.21437/IberSPEECH.2018-45 ↗
15. Sainath, T.N., Vinyals, O., Senior, A., and Sak, H. (2015, January 19–24). Convolutional, long short-term memory, fully connected deep neural networks. Proceedings of the 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), South Brisbane, QLD, Australia.10.1109/ICASSP.2015.7178838 ↗
16. End-to-end sequence labeling via convolutional recurrent neural network with a connectionist temporal classification layer10.2991/ijcis.d.200316.001 ↗
17. Vafeiadis, A., Fanioudakis, E., Potamitis, I., Votis, K., Giakoumis, D., Tzovaras, D., Chen, L., and Hamzaoui, R. (2019, January 15–19). Two-Dimensional Convolutional Recurrent Neural Networks for Speech Activity Detection. Proceedings of the Interspeech 2019, Graz, Austria.10.21437/Interspeech.2019-1354 ↗
18. Byers, F., Byers, F., and Sadjadi, O. (2019). 2017 Pilot Open Speech Analytic Technologies Evaluation (2017 NIST Pilot OpenSAT): Post Evaluation Summary.10.6028/NIST.IR.8242 ↗
19. Hansen, J.H., Joglekar, A., Shekhar, M.C., Kothapally, V., Yu, C., Kaushik, L., and Sangwan, A. (2019, January 15–19). The 2019 Inaugural Fearless Steps Challenge: A Giant Leap for Naturalistic Audio. Proceedings of the Interspeech, Graz, Austria.10.21437/Interspeech.2019-2301 ↗
20. Hansen, J.H., Sangwan, A., Joglekar, A., Bulut, A.E., Kaushik, L., and Yu, C. (2018, January 2–6). Fearless Steps: Apollo-11 Corpus Advancements for Speech Technologies from Earth to the Moon. Proceedings of the Interspeech, Hyderabad, India.10.21437/Interspeech.2018-1942 ↗
21. Mezza, A.I., Habets, E.A., Müller, M., and Sarti, A. (2021, January 18–22). Unsupervised domain adaptation for acoustic scene classification using band-wise statistics matching. Proceedings of the 2020 28th European Signal Processing Conference (EUSIPCO), Virtual.10.23919/Eusipco47968.2020.9287533 ↗
22. Anoop, C., Prathosh, A., and Ramakrishnan, A. (2021, January 15–17). Unsupervised domain adaptation schemes for building ASR in low-resource languages. Proceedings of the Workshop on Automatic Speech Recognition and Understanding, ASRU, Cartagena, Colombia.
23. Convolutional recurrent neural networks for Speech Activity Detection in naturalistic audio from apollo missions10.21437/IberSPEECH.2021-6 ↗
24. Gimeno, P., Ortega, A., Miguel, A., and Lleida, E. (2021). Unsupervised Representation Learning for Speech Activity Detection in the Fearless Steps Challenge 2021. Interspeech 2021, ISCA.10.21437/Interspeech.2021-309 ↗
25. Zhang, X.L. (2014, January 4–9). Unsupervised domain adaptation for deep neural network based voice activity detection. Proceedings of the 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Florence, Italy.10.1109/ICASSP.2014.6854930 ↗
26. Luo, Y., Zheng, L., Guan, T., Yu, J., and Yang, Y. (2019, January 15–20). Taking a closer look at domain shift: Category-level adversaries for semantics consistent domain adaptation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00261 ↗
27. Chu, C., and Wang, R. (2018). A survey of domain adaptation for neural machine translation. arXiv.
28. A survey of unsupervised deep domain adaptation10.1145/3400066 ↗
29. An unsupervised deep domain adaptation approach for robust speech recognition10.1016/j.neucom.2016.11.063 ↗
30. Wang, Q., Rao, W., Sun, S., Xie, L., Chng, E.S., and Li, H. (2018, January 15–20). Unsupervised domain adaptation via domain adversarial training for speaker recognition. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8461423 ↗
31. A novel speech enhancement method by learnable sparse and low-rank decomposition and domain adaptation10.1016/j.specom.2015.11.003 ↗
32. Deep visual domain adaptation: A survey10.1016/j.neucom.2018.05.083 ↗
33. Tan, B., Zhang, Y., Pan, S., and Yang, Q. (2017, January 4–9). Distant domain transfer learning. Proceedings of the AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.10.1609/aaai.v31i1.10826 ↗
34. Csurka, G. (2017). Domain adaptation for visual applications: A comprehensive survey. arXiv.10.1007/978-3-319-58347-1 ↗
35. Hu, J., Lu, J., and Tan, Y.P. (2015, January 1–7). Deep transfer metric learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298629 ↗
36. Long, M., Zhu, H., Wang, J., and Jordan, M.I. (2016). Unsupervised domain adaptation with residual transfer networks. arXiv.
37. Sun, B., Feng, J., and Saenko, K. (2016, January 12–17). Return of frustratingly easy domain adaptation. Proceedings of the AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.10.1609/aaai.v30i1.10306 ↗
38. Li, Y., Wang, N., Shi, J., Liu, J., and Hou, X. (2016). Revisiting batch normalization for practical domain adaptation. arXiv.
39. Coupled generative adversarial networks
40. Domain-adversarial training of neural networks
41. Zhuang, F., Cheng, X., Luo, P., Pan, S.J., and He, Q. (2015, January 25–31). Supervised representation learning: Transfer learning with deep autoencoders. Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, Buenos Aires, Argentina.
42. Zhu, J.Y., Park, T., Isola, P., and Efros, A.A. (2017, January 22–29). Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.244 ↗
43. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks
44. Iscen, A., Tolias, G., Avrithis, Y., and Chum, O. (2019, January 15–20). Label propagation for deep semi-supervised learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00521 ↗
45. Shi, W., Gong, Y., Ding, C., Tao, Z.M., and Zheng, N. (2018, January 8–14). Transductive semi-supervised deep learning using min-max features. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01228-1_19 ↗
46. Wang, G.H., and Wu, J. (2020, January 7–12). Repetitive reprediction deep decipher for semi-supervised learning. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i04.6082 ↗
47. Rizve, M.N., Duarte, K., Rawat, Y.S., and Shah, M. (2021). In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. arXiv.
48. Multispecies bioacoustic classification using transfer learning of deep convolutional neural networks with pseudo-labeling10.1016/j.apacoust.2020.107375 ↗
49. Takashima, Y., Fujita, Y., Horiguchi, S., Watanabe, S., García, P., and Nagamatsu, K. (2021). Semi-Supervised Training with Pseudo-Labeling for End-to-End Neural Diarization. arXiv.10.21437/Interspeech.2021-384 ↗
50. Xu, Q., Likhomanenko, T., Kahn, J., Hannun, A., Synnaeve, G., and Collobert, R. (2020). Iterative pseudo-labeling for speech recognition. arXiv.10.21437/Interspeech.2020-1800 ↗
51. Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the knowledge in a neural network. arXiv.
52. Shen, P., Lu, X., Li, S., and Kawai, H. (2018, January 2–6). Feature Representation of Short Utterances Based on Knowledge Distillation for Spoken Language Identification. Proceedings of the Interspeech, Hyderabad, India.10.21437/Interspeech.2018-1519 ↗
53. Li, J., Zhao, R., Chen, Z., Liu, C., Xiao, X., Ye, G., and Gong, Y. (2018, January 15–20). Developing far-field speaker system via teacher-student learning. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8462209 ↗
54. Korattikara, A., Rathod, V., Murphy, K., and Welling, M. (2015). Bayesian dark knowledge. arXiv.
55. Asami, T., Masumura, R., Yamaguchi, Y., Masataki, H., and Aono, Y. (2017, January 5–9). Domain adaptation of dnn acoustic models using knowledge distillation. Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA.10.1109/ICASSP.2017.7953145 ↗
56. Li, J., Seltzer, M.L., Wang, X., Zhao, R., and Gong, Y. (2017). Large-scale domain adaptation via teacher-student learning. arXiv.10.21437/Interspeech.2017-519 ↗
57. Voice Activity Detection with Teacher-Student Domain Emulation10.21437/Interspeech.2021-1234 ↗
58. Voice activity detection in the wild: A data-driven approach using teacher-student training10.1109/TASLP.2021.3073596 ↗
59. Sun, B., and Saenko, K. (2016, January 8–16). Deep coral: Correlation alignment for deep domain adaptation. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-49409-8_35 ↗
60. Morerio, P., and Murino, V. (2017). Correlation alignment by riemannian metric for domain adaptation. arXiv.
61. Huang, Z., and Gool, L.V. (2016). A Riemannian Network for SPD Matrix Learning. arXiv.10.1609/aaai.v31i1.10866 ↗
62. MacDuffee, C.C. (2012). The Theory of Matrices, Springer Science & Business Media.
63. Audio segmentation of broadcast news in the Albayzin-2010 evaluation: Overview, results, and discussion10.1186/1687-4722-2011-1 ↗
64. McCowan, I., Carletta, J., Kraaij, W., Ashby, S., Bourban, S., Flynn, M., Guillemot, M., Hain, T., Kadlec, J., and Karaiskos, V. (September, January 30). The AMI meeting corpus. Proceedings of the 5th International Conference on Methods and Techniques in Behavioral Research, Wageningen, The Netherlands.
65. Lleida, E., Ortega, A., Miguel, A., Bazán-Gil, V., Pérez, C., Gómez, M., and De Prada, A. (2019). Albayzin 2018 evaluation: The iberspeech-rtve challenge on speech technologies for spanish broadcast media. Appl. Sci., 9.10.3390/app9245412 ↗
66. Janin, A., Baron, D., Edwards, J., Ellis, D., Gelbart, D., Morgan, N., Peskin, B., Pfau, T., Shriberg, E., and Stolcke, A. (2003, January 6–10). The ICSI meeting corpus. Proceedings of the 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, Proceedings (ICASSP’03), Hong Kong, China.
67. Bell, P., Gales, M.J., Hain, T., Kilgour, J., Lanchantin, P., Liu, X., McParland, A., Renals, S., Saz, O., and Wester, M. (2015, January 13–17). The MGB challenge: Evaluating multi-genre broadcast media recognition. Proceedings of the 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), Scottsdale, AZ, USA.10.1109/ASRU.2015.7404863 ↗
68. Ortega, A., Miguel, A., Lleida, E., Bazán, V., Pérez, C., Gómez, M., and de Prada, A. (2022, February 07). Albayzin Evaluation: IberSPEECH-RTVE 2020 Speaker Diarization and Identity Assignment. Available online: http://catedrartve.unizar.es/reto2020/EvalPlan-SD-2020-v1.pdf.
69. Canavan, A., Graff, D., and Zipperlen, G. (1997). Callhome American English Speech, Linguistic Data Consortium.
70. Ioffe, S., and Szegedy, C. (2015, January 7–9). Batch normalization: Accelerating deep network training by reducing internal covariate shift. Proceedings of the International Conference on Machine Learning, PMLR, Lille, France.
71. Nair, V., and Hinton, G.E. (2010, January 21–24). Rectified linear units improve restricted boltzmann machines. Proceedings of the 27th International Conference on Machine Learning (ICML-10), Haifa, Israel.
72. Joglekar, A., Hansen, J.H., Shekar, M.C., and Sangwan, A. (2020). Fearless Steps Challenge (fs-2): Supervised learning with massive naturalistic apollo data. arXiv.10.21437/Interspeech.2020-3054 ↗
73. Chen, C., Fu, Z., Chen, Z., Jin, S., Cheng, Z., Jin, X., and Hua, X.S. (2020, January 7–12). HoMM: Higher-order moment matching for unsupervised domain adaptation. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i04.5745 ↗