AudioVAE-MASR: A Continuous-Latent Masked Autoregressive Framework for Multi-Distortion Speech Restoration
Source abstract
Abstract
Real-world speech restoration must handle coupled distortions, including acoustic noise and reverberation, codec artifacts, clipping, and artifacts left by upstream enhancement systems. Token-based generative systems offer a flexible route for such universal restoration, but discrete audio tokens can discard fine acoustic detail, and aggressive generative decoding may over-process inputs that are already close to clean speech. We propose AudioVAE-MASR, a continuous-latent masked autoregressive framework for multi-distortion speech restoration. A frozen AudioVAE maps clean and degraded speech into paired continuous latent sequences; a Conformer-based branch extracts the degraded-condition sequence Cy from degraded latents; a two-stream masked autoregressive encoder-decoder conditions masked clean-latent recovery on both degraded context and visible clean tokens; and a lightweight diffusion head models the masked clean tokens in the continuous latent space. On the released CCF AATC 2025 blind test set, the main inference setting (K=16, temperature 0.5) achieved WAcc 0.793, SIG 3.401, BAK 3.987, OVRL 3.111, PESQ 1.780, and ESTOI 0.798. Relative to the degraded input, these results improved WAcc and DNSMOS but did not improve PESQ; relative to the organizer baseline, they improved WAcc, SIG, OVRL, and PESQ but remained lower in BAK. A local subjective MOS evaluation with five listeners gave an overall mean score of 4.08 for AudioVAE-MASR, compared with 3.70 for the degraded input and 4.59 for the clean reference. Distortion-type, ablation, and parameter-sensitivity analyses further show that codec inputs remain vulnerable to over-restoration and that longer iterative decoding does not provide a consistent gain. The study therefore presents AudioVAE-MASR as a transparent continuous-latent restoration framework and identifies the fidelity-control problems that must be solved before such generative restoration can match the strongest lightweight discriminative systems.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech and Audio Processing · Hearing Loss and Rehabilitation · Speech Recognition and Synthesis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Electrical Engineering and Electronic Information, Xihua University, Chengdu 610039, China
References (44)
Show source-provided references
1. Zhang, W., Scheibler, R., Saijo, K., Cornell, S., Li, C., Ni, Z., Kumar, A., Pirklbauer, J., Sach, M., and Watanabe, S. (2024). Urgent challenge: Universality, robustness, and generalizability for speech enhancement. arXiv.10.21437/Interspeech.2024-1239 ↗
2. Saijo, K., Zhang, W., Cornell, S., Scheibler, R., Li, C., Ni, Z., Kumar, A., Sach, M., Fu, Y., and Wang, W. (2025). Interspeech 2025 URGENT speech enhancement challenge. arXiv.10.21437/Interspeech.2025-1363 ↗
3. Zhang, J., Zhu, M., Xu, X., Bu, H., Ling, Z., and Wu, Z. (2025). The CCF AATC 2025 Speech Restoration Challenge: A Retrospective. arXiv.
4. Serbest, S., Stojkovic, T., Cernak, M., and Harper, A. (2025, January 17–21). DeepFilterGAN: A Full-band Real-time Speech Enhancement System with GAN-based Stochastic Regeneration. Proceedings of the Interspeech 2025, Rotterdam, The Netherlands.10.21437/Interspeech.2025-1889 ↗
5. Li, X., Wang, Q., and Liu, X. (2024). Masksr: Masked language model for full-band speech restoration. arXiv.10.21437/Interspeech.2024-1584 ↗
6. Neural codec language models are zero-shot text to speech synthesizers10.1109/TASLPRO.2025.3530270 ↗
7. Nair, A.A., and Koishida, K. (2021). Cascaded time+ time-frequency unet for speech enhancement: Jointly addressing clipping, codec distortions, and gaps. Proceedings of the ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE.10.1109/ICASSP39728.2021.9414721 ↗
8. Gulati, A., Qin, J., Chiu, C.C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., and Wu, Y. (2020). Conformer: Convolution-augmented transformer for speech recognition. arXiv.10.21437/Interspeech.2020-3015 ↗
9. Enhancement of speech corrupted by acoustic noise10.1109/ICASSP.1979.1170788 ↗
10. All-pole modeling of degraded speech10.1109/TASSP.1978.1163086 ↗
11. Lu, Y.X., Ai, Y., and Ling, Z.H. (2023). MP-SENet: A speech enhancement model with parallel denoising of magnitude and phase spectra. arXiv.10.21437/Interspeech.2023-1441 ↗
12. High-fidelity audio compression with improved rvqgan10.52202/075280-1214 ↗
13. Défossez, A., Copet, J., Synnaeve, G., and Adi, Y. (2022). High fidelity neural audio compression. arXiv.
14. Audiolm: A language modeling approach to audio generation10.1109/TASLP.2023.3288409 ↗
15. Wei, C., Mangalam, K., Huang, P.Y., Li, Y., Fan, H., Xu, H., Wang, H., Xie, C., Yuille, A., and Feichtenhofer, C. (2023). Diffusion models as masked autoencoders. Proceedings of the IEEE/CVF International Conference on Computer Vision, IEEE.10.1109/ICCV51070.2023.01492 ↗
16. Autoregressive image generation without vector quantization10.52202/079017-1797 ↗
17. Zhang, C., Chen, Z., Zheng, K., and Zhu, J. (2025). VoiceBridge: Designing Latent Bridge Models for General Speech Restoration at Scale. arXiv.
18. Jukić, A., Korostik, R., Balam, J., and Ginsburg, B. (2024, January 1–5). Schrödinger Bridge for Generative Speech Enhancement. Proceedings of the Interspeech 2024, Kos, Greece.10.21437/Interspeech.2024-579 ↗
19. Han, S., Lee, S., Lee, J., and Lee, K. (2025, January 17–21). Few-step Adversarial Schrödinger Bridge for Generative Speech Enhancement. Proceedings of the Interspeech 2025, Rotterdam, The Netherlands.10.21437/Interspeech.2025-673 ↗
20. Wang, S., Liu, S., Harper, A., Kendrick, P., Salzmann, M., and Cernak, M. (2024). Diffusion-based Speech Enhancement with Schrödinger Bridge and Symmetric Noise Schedule. arXiv.
21. Van Den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K. (2016). Wavenet: A generative model for raw audio. arXiv.
22. Language models are few-shot learners
23. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics.
24. He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. (2022). Masked autoencoders are scalable vision learners. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE.10.1109/CVPR52688.2022.01553 ↗
25. Chang, H., Zhang, H., Jiang, L., Liu, C., and Freeman, W.T. (2022). Maskgit: Masked generative image transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE.10.1109/CVPR52688.2022.01103 ↗
26. Garcia, H.F., Seetharaman, P., Kumar, R., and Pardo, B. (2023). Vampnet: Music generation via masked acoustic token modeling. arXiv.
27. Zhou, Y., Zeng, G., Liu, X., Li, X., Yu, R., Wang, Z., Ye, R., Sun, W., Gui, J., and Li, K. (2025). VoxCPM: Tokenizer-Free TTS for Context-Aware Speech Generation and True-to-Life Voice Cloning. arXiv.
28. Peebles, W., and Xie, S. (2023). Scalable diffusion models with transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, IEEE.10.1109/ICCV51070.2023.00387 ↗
29. Zhang, J., Yang, J., Fang, Z., Wang, Y., Zhang, Z., Wang, Z., Fan, F., and Wu, Z. (2025). Anyenhance: A unified generative model with prompt-guidance and self-critic for voice enhancement. arXiv.10.1109/TASLPRO.2025.3587393 ↗
30. Yamagishi, J., Veaux, C., and MacDonald, K. (2019). CSTR VCTK Corpus: English Multi-Speaker Corpus for CSTR Voice Cloning Toolkit, Version 0.92; University of Edinburgh, The Centre for Speech Technology Research (CSTR).
31. Shi, Y., Bu, H., Xu, X., Zhang, S., and Li, M. (2020). Aishell-3: A multi-speaker mandarin tts corpus and the baselines. arXiv.10.21437/Interspeech.2021-755 ↗
32. Richter, J., Wu, Y.C., Krenn, S., Welker, S., Lay, B., Watanabe, S., Richard, A., and Gerkmann, T. (2024). EARS: An anechoic fullband speech dataset benchmarked for speech enhancement and dereverberation. arXiv.10.21437/Interspeech.2024-153 ↗
33. Liu, H., Liu, X., Kong, Q., Tian, Q., Zhao, Y., Wang, D., Huang, C., and Wang, Y. (2022). Voicefixer: A unified framework for high-fidelity speech restoration. arXiv.10.21437/Interspeech.2022-11026 ↗
34. Defossez, A., Synnaeve, G., and Adi, Y. (2020). Real time speech enhancement in the waveform domain. arXiv.10.21437/Interspeech.2020-2409 ↗
35. Zhao, S., Ma, B., Watcharasupat, K.N., and Gan, W.S. (2022). FRCRN: Boosting feature representation using frequency recurrence for monaural speech enhancement. Proceedings of the ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE.10.1109/ICASSP43922.2022.9747578 ↗
36. Reddy, C.K., Dubey, H., Gopal, V., Cutler, R., Braun, S., Gamper, H., Aichner, R., and Srinivasan, S. (2021). ICASSP 2021 deep noise suppression challenge. Proceedings of the ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE.10.1109/ICASSP39728.2021.9415105 ↗
37. TF-GridNet: Integrating full-and sub-band modeling for speech separation10.1109/TASLP.2023.3304482 ↗
38. StoRM: A diffusion-based stochastic regeneration model for speech enhancement and dereverberation10.1109/TASLP.2023.3294692 ↗
39. Speech enhancement and dereverberation with diffusion-based generative models10.1109/TASLP.2023.3285241 ↗
40. Kang, B., Zhu, X., Zhang, Z., Ye, Z., Liu, M., Wang, Z., Zhu, Y., Ma, G., Chen, J., and Xiao, L. (2025). LLaSE-G1: Incentivizing generalization capability for llama-based speech enhancement. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics.10.18653/v1/2025.acl-long.651 ↗
41. Nichol, A.Q., and Dhariwal, P. (2021). Improved denoising diffusion probabilistic models. Proceedings of the International Conference on Machine Learning, PMLR.
42. Rec, I. (2005). P.862.2: Wideband Extension to Recommendation p.862 for the Assessment of Wideband Telephone Networks and Speech Codecs, International Telecommunication Union.
43. Radford, A., Kim, J.W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. (2023). Robust speech recognition via large-scale weak supervision. Proceedings of the International Conference on Machine Learning, PMLR.
44. Binary codes capable of correcting deletions, insertions, and reversals