A Self-Supervised Method for Speaker Recognition in Real Sound Fields with Low SNR and Strong Reverberation
Source abstract
Abstract
Speaker recognition is essential in smart voice applications for personal identification. Current state-of-the-art techniques primarily focus on ideal acoustic conditions. However, the traditional spectrogram struggles to differentiate between noise, reverberation, and speech. To overcome this challenge, MFCC can be replaced with the output from a self-supervised learning model. This study introduces a TDNN enhanced with a pre-trained model for robust performance in noisy and reverberant environments, referred to as PNR-TDNN. The PNR-TDNN employs HuBERT as its backbone, while the TDNN is an improved ECAPA-TDNN. The pre-trained model employs the Canopy/Mini Batch k-means++ strategy. In the TDNN architecture, several enhancements are implemented, including a cross-channel fusion mechanism based on Res2Net. Additionally, a non-average attention mechanism is applied to the pooling operation, focusing on the weight information of each channel within the Squeeze-and-Excitation Net. Furthermore, the contribution of individual channels to the pooling of time-domain frames is enhanced by substituting attentive statistics with multi-head attention statistics. Validated by zhvoice in noisy conditions, the minimized PNR-TDNN demonstrates a 5.19% improvement in EER compared to CAM++. In more challenging environments with noise and reverberation, the minimized PNR-TDNN further improves EER by 3.71% and 9.6%, respectively, and MinDCF by 3.14% and 3.77%, respectively. The proposed method has also been validated on the VoxCeleb1 and cn-celeb_v2 datasets, representing a significant breakthrough in the field of speaker recognition under challenging conditions. This advancement is particularly crucial for enhancing safety and protecting personal identification in voice-enabled microphone applications.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech and Audio Processing · Speech Recognition and Synthesis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Instrument and Electronics, North University of China, Taiyuan 030051, China
School of Information and Communication Engineering, North University of China, Taiyuan 030051, China
Funding
Innovative Research Group Project of National Natural Science Foundation of China · Excellent Youth foundation of Shanxi Province · Key research and development project of Shanxi Province · fundamental research program of Shanxi Province · Aviation Science Foundation · Shanxi province key laboratory of quantum sensing and precision measurement · 173 Foundation · ZBZDJCYJKT
References (63)
Show source-provided references
1. Efficient and Privacy-Preserving Speaker Recognition for Cybertwin-Driven 6G10.1109/JIOT.2021.3097266 ↗
2. From Attack to Identification: MEMS Sensor Fingerprinting Using Acoustic Signals10.1109/JIOT.2022.3221930 ↗
3. A Continuous Articulatory-Gesture-Based Liveness Detection for Voice Authentication on Smart Devices10.1109/JIOT.2022.3199995 ↗
4. An overview of text-independent speaker recog-nition: From features to supervectors10.1016/j.specom.2009.08.009 ↗
5. Enrollment-Stage Backdoor Attacks on Speaker Recognition Systems via Adversarial Ultrasound10.1109/JIOT.2023.3328253 ↗
6. Secure mmWave-Radar-Based Speaker Verification for IoT Smart Home10.1109/JIOT.2020.3023101 ↗
7. Stop Deceiving! An Effective Defense Scheme Against Voice Impersonation Attacks on Smart Devices10.1109/JIOT.2021.3110588 ↗
8. Machine Learning in Real-Time Internet of Things (IoT) Systems: A Survey10.1109/JIOT.2022.3161050 ↗
9. A review on speaker recognition: Technology and challenges10.1016/j.compeleceng.2021.107005 ↗
10. Snyder, D., Garcia-Romero, D., Sell, G., Povey, D., and Khudanpur, S. (2018, January 15–20). X-vectors: Robust DNN embeddings for speaker recognition. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8461375 ↗
11. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning forimage recognition. Proceedings of the Computer Vision and Pattern Recognition (CVPR) 2016, IEEE Computer Society, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
12. Variani, E., Lei, X., McDermott, E., Moreno, I.L., and Dominguez, J.G. (2014, January 4–9). Deep neural networks for small footprint text-dependent speaker verification. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP, Florence, Italy.10.1109/ICASSP.2014.6854363 ↗
13. Snyder, D., Garcia-Romero, D., Sell, G., McCree, A., Povey, D., and Khudanpur, S. (2019, January 12–17). Speaker recognition for multi-speaker conversations using x-vectors. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP, Brighton, UK.10.1109/ICASSP.2019.8683760 ↗
14. Garcia-Romero, D., McCree, A., Snyder, D., and Sell, G. (2020, January 4–9). JHUHLTCOE system for the VoxSRC speaker recognition challenge. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP, Virtual.10.1109/ICASSP40776.2020.9053209 ↗
15. H-VECTORS: Improving the robustness in utterance-level speaker embeddings using a hierarchical attention model10.1016/j.neunet.2021.05.024 ↗
16. Zhou, T., Zhao, Y., and Wu, J. (2021, January 19–22). Resnext and res2net structures for speaker verification. Proceedings of the IEEE Spoken Language Technology Workshop 2021, Shenzhen, China.10.1109/SLT48900.2021.9383531 ↗
17. Ko, H., and Hansen, J.H.L. (2022, January 18–22). Dfresnet: Boosting speaker verification performance with depth-first design. Proceedings of the Interspeech 2022, Incheon, Republic of Korea.
18. Gao, Z., Song, Y., McLoughlin, I.V., Guo, W., and Dai, L. (2018, January 2–6). An improved deep embedding learning method for short duration speaker verification. Proceedings of the Interspeech, Hyderabad, India.10.21437/Interspeech.2018-1515 ↗
19. Chen, C.P., Zhang, S.Y., Yeh, C.T., Wang, J.C., Wang, T., and Huang, C.L. (2019, January 12–17). Speaker characterization using TDNN-LSTM based speaker embedding. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK.10.1109/ICASSP.2019.8683185 ↗
20. Desplanques, B., Thienpondt, J., and Demuynck, K. (2020, January 25–29). ECAPATDNN: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification. Proceedings of the 21st Annual conference of the International Speech Communication Association INTER SPEECH, Shanghai, China.10.21437/Interspeech.2020-2650 ↗
21. An overview of speaker identification: Accuracy and robustness issues10.1109/MCAS.2011.941079 ↗
22. On the Detection of Adaptive Adversarial Attacks in Speaker Verification Systems10.1109/JIOT.2023.3267619 ↗
23. Toward Pitch-Insensitive Speaker Verification via Soundfield10.1109/JIOT.2023.3290001 ↗
24. Kheder, W.B., and Matrouf, D. (2015, January 19–24). Additive noise compensation in the i-vector space for speaker recognition. Proceedings of the International Conference on Acoustics, Speech and Signal Processing ICASSP, Brisbane, Australia.
25. Two level Noise Robust and Block Featured PNN Model for Speaker Recognition in Real Environment10.1007/s11277-022-09734-7 ↗
26. O’Malley, T., Ding, S., Narayanan, A., Wang, Q., Rikhye, K., Liang, Q., He, Y., and McGraw, I. (2023, January 4–10). Conditional Conformer: Improving Speaker Modulation For Single And Multi-User Speech Enhancement. Proceedings of the 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece.10.1109/ICASSP49357.2023.10095369 ↗
27. Joint estimation of pose, depth, and optical flow with a competition–cooperation transformer network10.1016/j.neunet.2023.12.020 ↗
28. UDF-GAN: Unsupervised dense optical-flow estimation using cycle Generative Adversarial Networks10.1016/j.knosys.2023.110568 ↗
29. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019, January 2–7). Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL), Minneapolis, MN, USA.
30. Wallach, H., Larochelle, H., Beygelzimer, A., d’Alche-Buc, F., Fox, E., and Garnett, R. (2019). Xlnet: Generalized autoregressive pretraining for language understanding. Advances in Neural Information Processing Systems (NeurIPS), Curran Associates, Inc.
31. Exploring the limits of transfer learning with a unified text-to-text transformer
32. Hsu, W.-N., Bolte, B., Tsai, Y.-H.H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A. (2021). Hubert: Self-supervised speech representation learning by masked prediction of hidden units. arXiv.10.1109/TASLP.2021.3122291 ↗
33. Chung, Y.-A., Hsu, W.-N., Tang, H., and Glass, J. (2019). An unsupervised autoregressive model for speech representation learning. arXiv.10.21437/Interspeech.2019-1473 ↗
34. Baevski, A., Schneider, S., and Auli, M. (2019). vq-wav2vec: Selfsupervised learning of discrete speech representations. arXiv.
35. Baevski, A., Zhou, H., Mohamed, A., and Auli, M. (2020). wav2vec 2.0: A framework for self-supervised learning of speech representations. arXiv.
36. Chen, Z., and Chen, S. (2022, January 7–13). Large-scale self-supervised speech representation learning for automatic speaker verification. Proceedings of the ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Virtual.10.1109/ICASSP43922.2022.9747814 ↗
37. A Novel Exploitative and Explorative GWO-SVM Algorithm for Smart Emotion Recognition10.1109/JIOT.2023.3235356 ↗
38. Chen, S., and Wang, C. (2022). WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing. arXiv.10.1109/JSTSP.2022.3188113 ↗
39. Wang, H., Zheng, S., Chen, Y., Cheng, L., and Chen, Q. (2023). CAM++: A Fast and Efficient Network for Speaker Verification Using Context-Aware Masking. arXiv.10.21437/Interspeech.2023-1513 ↗
40. Neural Inference Search for Multiloss Segmentation Models10.1109/TNNLS.2023.3282799 ↗
41. Semantic segmentation using Firefly Algorithm-based evolving ensemble deep neural networks10.1016/j.knosys.2023.110828 ↗
42. Semantic Segmentation of Crop and Weed using an Encoder-Decoder Network and Image Enhancement Method under Uncontrolled Outdoor Illumination10.1109/ACCESS.2020.2991354 ↗
43. Wang, A., Pen, T., Cao, H., Xu, Y., Wei, X., and Cui, B.B. (2022). TIA-YOLOv5: An improved YOLOv5 network for real-time detection of crop and weed in the field. Front. Plant Sci., 13.10.3389/fpls.2022.1091655 ↗
44. Intelligent Bionic Polarization Orientation Method Using Biological Neuron Model for Harsh Conditions10.1109/TPAMI.2024.3484183 ↗
45. Wu, X., Liu, J., Shen, C., Cao, H., Wang, C., and Tang, J. (2024). Vehicle-Mounted Polarization Orientation Strategy Considering Inclined Pavement and Occlusion Conditions. IEEE Trans. Intell. Veh., early access.10.1109/TIV.2024.3385396 ↗
46. Multiaperture Visual Velocity Measurement Method Based on Biomimetic Compound-Eye for UAVs10.1109/JIOT.2023.3324966 ↗
47. Brain-like position measurement method based on improved optical flow algorithm10.1016/j.isatra.2023.09.005 ↗
48. Cascaded Speech Separation Denoising and Dereverberation Using Attention and TCN-WPE Networks for Speech Devices10.1109/JIOT.2024.3361563 ↗
49. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS), MIT Press.
50. Yu, Y.-Q., and Li, W.-J. (2020, January 25–29). Densely connected time delay neural network for speaker verification. Proceedings of the Annual Conference of the International Speech Communication Association (INTER SPEECH), Shanghai, China.10.21437/Interspeech.2020-1275 ↗
51. Least squares quantization in pcm10.1109/TIT.1982.1056489 ↗
52. Improved K-means Algorithm Based on Density Canopy10.1016/j.knosys.2018.01.031 ↗
53. Arthur, D., and Vassilvitskii, S. (2007, January 7–9). K-Means++: The Advantages of Careful Seeding. Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2007, New Orleans, LA, USA.
54. Multi-rate strong tracking square-root cubature Kalman filter for MEMS-INS/GPS/polarization compass integrated navigation system10.1016/j.ymssp.2021.108146 ↗
55. India, M., Safari, P., and Hernando, J. (2019). Self Multi-Head Attention for Speaker Recognition. arXiv.10.21437/Interspeech.2019-2616 ↗
56. Scheibler, R., Bezzam, E., and Dokmanic, I. (2018, January 15–20). Pyroomacoustics: A python package for audio room simulation and array processing algorithms. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8461310 ↗
57. Deng, J., Guo, J., Xue, N., and Zafeiriou, S. (2019, January 15–20). Arc-face: Additive angular margin loss for deep face recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00482 ↗
58. Zhang, Y., Lv, Z., Wu, H., Zhang, S., Hu, P., Wu, Z., Lee, H., and Meng, H. (2022). Mfa-conformer: Multi-scale feature aggregation conformer for automatic speaker verification. arXiv.10.21437/Interspeech.2022-563 ↗
59. Wang, H., Liang, C., Wang, S., Chen, Z., Zhang, B., Xiang, X., Deng, Y., and Qian, Y. (2023, January 4–10). Wespeaker: A Research and Production Oriented Speaker Embedding Learning Toolkit. Proceedings of the ICASSP 2023 IEEE, Rhodes Island, Greece.10.1109/ICASSP49357.2023.10096626 ↗
60. Yakovlev, I., Makarov, R., Balykin, A., Malov, P., Okhotnikov, A., and Torgashov, N. (2024). Reshape Dimensions Network for Speaker Recognition. arXiv.10.21437/Interspeech.2024-2116 ↗
61. Golden Gemini is All You Need: Finding the Sweet Spots for Speaker Verification10.1109/TASLP.2024.3385277 ↗
62. Nagrani, A., Chung, J.S., and Zisserman, A. (2017, January 20–24). VoxCeleb: A largescale speaker identification dataset. Proceedings of the Interspeech, Stockholm, Sweden.10.21437/Interspeech.2017-950 ↗
63. Fan, Y., Kang, J.W., Li, L.T., Li, K.C., Chen, H.L., Cheng, S.T., Zhang, P.Y., Zhou, Z.Y., Cai, Y.Q., and Wang, D. (2020, January 4–8). CN-Celeb: A Challenging Chinese Speaker Recognition Dataset. Proceedings of the International Conference on Acoustics, Speech, and Signal Processing IEEE, Barcelona, Spain.10.1109/ICASSP40776.2020.9054017 ↗