Voice Deepfake Detection Using the Self-Supervised Pre-Training Model HuBERT
Source abstract
Abstract
In recent years, voice deepfake technology has developed rapidly, but current detection methods have the problems of insufficient detection generalization and insufficient feature extraction for unknown attacks. This paper presents a forged speech detection method (HuRawNet2_modified) based on a self-supervised pre-trained model (HuBERT) to improve detection (and address the above problems). A combination of impulsive signal-dependent additive noise and additive white Gaussian noise was adopted for data boosting and augmentation, and the HuBERT model was fine-tuned on different language databases. On this basis, the size of the extracted feature maps was modified independently by the α-feature map scaling (α-FMS) method, with a modified end-to-end method using the RawNet2 model as the backbone structure. The results showed that the HuBERT model could extract features more comprehensively and accurately. The best evaluation indicators were an equal error rate (EER) of 2.89% and a minimum tandem detection cost function (min t-DCF) of 0.2182 on the database of the ASVspoof2021 LA challenge, which verified the effectiveness of the detection method proposed in this paper. Compared with the baseline systems in databases of the ASVspoof 2021 LA challenge and the FMFCC-A, the values of EER and min t-DCF decreased. The results also showed that the self-supervised pre-trained model with fine-tuning can extract acoustic features across languages. And the detection can be slightly improved when the languages of the pre-trained database, and the fine-tuned and tested database are the same.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech Recognition and Synthesis · Speech and Audio Processing · Music and Audio Processing
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Information and Cyber Security, People’s Public Security University of China, Beijing 100038, China
Funding
Double First-Class Innovation Research Project for People’s Public Security University of China
References (31)
Show source-provided references
1. Wang, X., and Yamagishi, J. (2022). Frontiers in Fake Media Generation and Detection, Springer Nature.
2. Towards End-to-End Synthetic Speech Detection10.1109/LSP.2021.3089437 ↗
3. Tak, H., Patino, J., Todisco, M., Nautsch, A., Evans, N., and Larcher, A. (2021, January 6–11). End-to-End Anti-Spoofing with RawNet2. Proceedings of the ICASSP 2021—2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, ON, Canada.10.1109/ICASSP39728.2021.9414234 ↗
4. Tak, H., Jung, J., Patino, J., Kamble, M., Todisco, M., and Evans, N. (2021). End-to-End Spectro-Temporal Graph Attention Networks for Speaker Verification Anti-Spoofing and Speech Deepfake Detection. arXiv.10.21437/ASVSPOOF.2021-1 ↗
5. Wu, Z., Kinnunen, T., Evans, N., Yamagishi, J., Hanilçi, C., Sahidullah, M., and Sizov, A. (2015, January 4–5). ASVspoof 2015: The First Automatic Speaker Verification Spoofing and Countermeasures Challenge. Proceedings of the Interspeech 2015, 16th Annual Conference of the International Speech Communication Association, Dresden, Germany.10.21437/Interspeech.2015-462 ↗
6. ASVspoof 2019: Spoofing Countermeasures for the Detection of Synthesized, Converted and Replayed Speech10.1109/TBIOM.2021.3059479 ↗
7. Yamagishi, J., Wang, X., Todisco, M., Sahidullah, M., Patino, J., Nautsch, A., Liu, X., Lee, K.A., Kinnunen, T., and Evans, N. (2021, January 16). ASVspoof 2021: Accelerating Progress in Spoofed and Deepfake Speech Detection. Proceedings of the 2021 Edition of the Automatic Speaker Verification and Spoofing Countermeasures Challenge, Online.10.21437/ASVSPOOF.2021-8 ↗
8. Zhang, Z., Gu, Y., Yi, X., and Zhao, X. (2021, January 20–22). FMFCC-a: A Challenging Mandarin Dataset for Synthetic Speech Detection. Proceedings of the Digital Forensics and Watermarking: 20th International Workshop, Beijing, China.10.1007/978-3-030-95398-0_9 ↗
9. HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units10.1109/TASLP.2021.3122291 ↗
10. Kenton, J.D.M.-W.C., and Toutanova, L.K. (2019, January 2–7). Bert: Pre-Training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA.
11. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., and Dean, J. (2013, January 5–10). Distributed Representations of Words and Phrases and Their Compositionality. Proceedings of the Advances in Neural Information Processing Systems 26 (NIPS 2013), Lake Tahoe, NV, USA.
12. Van den Oord, A., Li, Y., and Vinyals, O. (2018). Representation Learning with Contrastive Predictive Coding. arXiv.
13. Schneider, S., Baevski, A., Collobert, R., and Auli, M. (2019, January 15–19). Wav2vec: Unsupervised Pre-Training for Speech Recognition. Proceedings of the Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria.10.21437/Interspeech.2019-1873 ↗
14. Wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
15. WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing10.1109/JSTSP.2022.3188113 ↗
16. Zhang, B., Lv, H., Guo, P., Shao, Q., Yang, C., Xie, L., Xu, X., Bu, H., Chen, X., and Zeng, C. (2022, January 7–13). WENETSPEECH: A 10,000+ Hours Multi-Domain Mandarin Corpus for Speech Recognition. Proceedings of the ICASSP 2022—2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Virtual.10.1109/ICASSP43922.2022.9746682 ↗
17. Wang, X., and Yamagishi, J. (July, January 28). Investigating Self-Supervised Front Ends for Speech Spoofing Countermeasures. Proceedings of the Speaker and Language Recognition Workshop (Odyssey 2022), Beijing, China.10.21437/Odyssey.2022-14 ↗
18. Tak, H., Todisco, M., Wang, X., Jung, J., Yamagishi, J., and Evans, N. (July, January 28). Automatic Speaker Verification Spoofing and Deepfake Detection Using Wav2vec 2.0 and Data Augmentation. Proceedings of the Speaker and Language Recognition Workshop (Odyssey 2022), Beijing, China.10.21437/Odyssey.2022-16 ↗
19. Park, D.S., Chan, W., Zhang, Y., Chiu, C.-C., Zoph, B., Cubuk, E.D., and Le, Q.V. (2019, January 15–19). SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition. Proceedings of the Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria.10.21437/Interspeech.2019-2680 ↗
20. An Efficient Real-Valued Sparse Bayesian Learning for Non-Circular Signal’s DOA Estimation in the Presence of Impulsive Noise10.1016/j.dsp.2020.102838 ↗
21. Jung, J., Kim, S., Shim, H., Kim, J., and Yu, H.-J. (2020, January 25–29). Improved Rawnet with Feature Map Scaling for Text-Independent Speaker Verification Using Raw Waveforms. Proceedings of the Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Shanghai, China.10.21437/Interspeech.2020-1011 ↗
22. Desplanques, B., Thienpondt, J., and Demuynck, K. (2020, January 25–29). ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification. Proceedings of the Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Shanghai, China.10.21437/Interspeech.2020-2650 ↗
23. Ravanelli, M., Parcollet, T., Plantinga, P., Rouhe, A., Cornell, S., Lugosch, L., Subakan, C., Dawalatabad, N., Heba, A., and Zhong, J. (2021). SpeechBrain: A General-Purpose Speech Toolkit. arXiv.
24. α-feature map scaling for raw waveform speaker verification
25. Zhang, J., Inoue, N., and Shinoda, K. (2018, January 2–6). I-Vector Transformation Using Conditional Generative Adversarial Networks for Short Utterance Speaker Verification. Proceedings of the Interspeech 2018, 19th Annual Conference of the International Speech Communication Association, Hyderabad, India.10.21437/Interspeech.2018-1680 ↗
26. Ma, H., Yi, J., Wang, C., Yan, X., Tao, J., Wang, T., Wang, S., Xu, L., and Fu, R. (2022). FAD: A Chinese Dataset for Fake Audio Detection. arXiv.
27. Jung, J., Heo, H.-S., Tak, H., Shim, H., Chung, J.S., Lee, B.-J., Yu, H.-J., and Evans, N. (2022, January 7–13). AASIST: Audio Anti-Spoofing Using Integrated Spectro-Temporal Graph Attention Networks. Proceedings of the ICASSP 2022—2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Virtual.10.1109/ICASSP43922.2022.9747766 ↗
28. Constant Q Cepstral Coefficients: A Spoofing Countermeasure for Automatic Speaker Verification10.1016/j.csl.2017.01.001 ↗
29. Sahidullah, M., Kinnunen, T., and Hanilçi, C. (2015, January 4–5). A Comparison of Features for Synthetic Speech Detection. Proceedings of the Interspeech 2015, 16th Annual Conference of the International Speech Communication Association, Dresden, Germany.10.21437/Interspeech.2015-472 ↗
30. Wang, X., and Yamagishi, J. (September, January 30). A Comparative Study on Recent Neural Spoofing Countermeasures for Synthetic Speech Detection. Proceedings of the Interspeech 2021, 22nd Annual Conference of the International Speech Communication Association, Brno, Czechia.10.21437/Interspeech.2021-702 ↗
31. Todisco, M., Delgado, H., and Evans, N. (2016, January 21–24). A New Feature for Automatic Speaker Verification Anti-Spoofing: Constant Q Cepstral Coefficients. Proceedings of the Speaker and Language Recognition Workshop, Bilbao, Spain.10.21437/Odyssey.2016-41 ↗