Parallel Enhancement and Bandwidth Extension of Coded Speech
Source abstract
Abstract
An important use case of speech bandwidth extension (BWE) is generating high-frequency components from band-limited speech processed by a speech codec. Recent works on BWE have demonstrated remarkable capabilities in generating high-quality, high-band components using deep learning techniques. Among them, Streaming SEANet (StrmSEANet) has also been shown to be effective for BWE with reduced delay and computational complexity, making it suitable for real-time speech processing. However, the effect of the coding artifact in the lower band of the input signal has not been sufficiently considered in many deep learning-based BWE methods. In this work, we propose Parallel Enhancement and Bandwidth Extension of coded speech (PEBE), where two lightweight networks, referred to as Compact Streaming SEANet (CompSEANet), for coded speech enhancement (CSE) and BWE are configured in parallel. The CSE and BWE models are separately trained with the task-specific training settings, thereby effectively improving the reconstruction quality of the band-limited speech signals degraded by coding artifacts. Experimental results demonstrate that the proposed PEBE significantly outperforms the baseline AP-BWE, StrmSEANet, and standalone CompSEANet in reconstructing wideband (WB) and fullband speech from Opus-coded narrowband and WB signals. The proposed method achieves the highest scores in the subjective MUSHRA test while providing the fastest inference among all compared methods, with real-time factors (RTF) of 33.95× and 18.38× measured on a Samsung SM-F711 mobile device under single-thread execution.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech and Audio Processing · Advanced Data Compression Techniques · Speech Recognition and Synthesis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of AI Convergence, Gwangju Institute of Science and Technology, Gwangju 61005, Republic of Korea
Department of Electrical Engineering and Computer Science, Gwangju Institute of Science and Technology, Gwangju 61005, Republic of Korea
Media Coding Research Section, Electronics and Telecommunications Research Institute, Daejeon 34129, Republic of Korea
School of Electrical Engineering, Korea Advanced Institute of Science and Technology, Daejeon 34141, Republic of Korea
Funding
korean government
References (45)
Show source-provided references
1. (1999). Mandatory Speech Codec Speech Processing Functions; Adaptive Multi-Rate (AMR) Speech Codec; Transcoding Functions (Standard No. 3GPP TS 26.090).
2. (2002). Wideband Coding of Speech at Around 16 kbit/s Using Adaptive Multi-Rate Wideband (AMR-WB) (Standard No. ITU-T Recommendation G.722.2).
3. Valin, J.M., Vos, K., and Terriberry, T. (2012). Definition of the Opus Audio Codec, Internet Engineering Task Force (IETF). IETF RFC 6716.
4. (2022). Codec for Enhanced Voice Services (EVS); ANSI C Code (Floating-Point) (Standard No. 3GPP TS 26.443 V17.0.0).
5. Wang, H., and Wang, D. (2020, January 4–8). Time-Frequency Loss for CNN Based Speech Super-Resolution. Proceedings of the IEEE ICASSP, Barcelona, Spain.
6. Mandel, M., Tal, O., and Adi, Y. (2023, January 4–10). AERO: Audio Super Resolution in the Spectral Domain. Proceedings of the IEEE ICASSP, Rhodes Island, Greece.10.1109/ICASSP49357.2023.10095382 ↗
7. Li, Y., Tagliasacchi, M., Rybakov, O., Ungureanu, V., and Roblek, D. (2021, January 6–11). Real-time speech frequency bandwidth extension. Proceedings of the IEEE ICASSP, Toronto, ON, Canada.10.1109/ICASSP39728.2021.9413439 ↗
8. Towards High-Quality and Efficient Speech Bandwidth Extension with Parallel Amplitude and Phase Prediction10.1109/TASLP.2024.3519881 ↗
9. A New Framework for CNN-Based Speech Enhancement in the Time Domain10.1109/TASLP.2019.2913512 ↗
10. Tagliasacchi, M., Li, Y., Misiunas, K., and Roblek, D. (2020, January 25–29). SEANet: A Multi-Modal Speech Enhancement Network. Proceedings of the Interspeech 2020, Shanghai, China.10.21437/Interspeech.2020-1563 ↗
11. Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., and Xie, S. (2022, January 18–24). A ConvNet for the 2020s. Proceedings of the IEEE/CVF CVPR, New Orleans, LA, USA.10.1109/CVPR52688.2022.01167 ↗
12. Jang, W., Lim, D., Yoon, J., Kim, B., and Kim, J. (September, January 30). UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation. Proceedings of the Interspeech 2021, Brno, Czech Republic.10.21437/Interspeech.2021-1016 ↗
13. Wen, L., Wang, L., Zheng, Y., Shi, W., and Choi, K.P. (2024, January 15–19). FT-CSR: Cascaded Frequency-Time Method for Coded Speech Restoration. Proceedings of the IEEE ICME, Niagara Falls, ON, Canada.10.1109/ICME57554.2024.10687650 ↗
14. Lin, J., Kalgaonkar, K., He, Q., and Lei, X. (2022, January 22–27). Speech Enhancement for Low Bit Rate Speech Codec. Proceedings of the IEEE ICASSP, Singapore.10.1109/ICASSP43922.2022.9746670 ↗
15. Rowe, D. (2011, January 16–18). Codec 2-open source speech coding at 2400 bits/s and below. Proceedings of the ARRL and TAPR Digital Communications Conference, Baltimore, MD, USA.
16. Rong, X., Wang, D., Hu, Q., Wang, Y., Hu, Y., and Lu, J. (2025, January 17–21). TS-URGENet: A Three-stage Universal Robust and Generalizable Speech Enhancement Network. Proceedings of the Interspeech 2025, Rotterdam, The Netherlands.10.21437/Interspeech.2025-734 ↗
17. Le, X., Chen, Z., Sun, S., Xia, X., and Huang, C. (2025, January 17–21). Multistage Universal Speech Enhancement System for URGENT Challenge. Proceedings of the Interspeech 2025, Rotterdam, The Netherlands.10.21437/Interspeech.2025-749 ↗
18. Sun, Z., Li, A., Lei, T., Chen, R., Yu, M., Zheng, C., Zhou, Y., and Yu, D. (2025, January 17–21). Scaling beyond Denoising: Submitted System and Findings in URGENT Challenge 2025. Proceedings of the Interspeech 2025, Rotterdam, The Netherlands.10.21437/Interspeech.2025-795 ↗
19. Serbest, S., Stojkovic, T., Cernak, M., and Harper, A. (2025, January 17–21). DeepFilterGAN: A Full-band Real-time Speech Enhancement System with GAN-based Stochastic Regeneration. Proceedings of the Interspeech 2025, Rotterdam, The Netherlands.10.21437/Interspeech.2025-1889 ↗
20. Goswami, N., and Harada, T. (2025, January 17–21). FUSE: Universal Speech Enhancement using Multi-Stage Fusion of Sparse Compression and Token Generation Models for the URGENT 2025 Challenge. Proceedings of the Interspeech 2025, Rotterdam, The Netherlands.10.21437/Interspeech.2025-251 ↗
21. Goswami, N., and Harada, T. (2025, January 17–21). Universal Speech Enhancement with Regression and Generative Mamba. Proceedings of the Interspeech 2025, Rotterdam, The Netherlands.
22. Saijo, K., Zhang, W., Cornell, S., Scheibler, R., Li, C., Ni, Z., Kumar, A., Sach, M., Fu, Y., and Wang, W. (2025, January 17–21). Interspeech 2025 URGENT Speech Enhancement Challenge. Proceedings of the Interspeech 2025, Rotterdam, The Netherlands.10.21437/Interspeech.2025-1363 ↗
23. Alias-and-Separate: Wideband Speech Coding Using Sub-Nyquist Sampling and Speech Separation10.1109/LSP.2022.3207381 ↗
24. Improved Alias-and-Separate Speech Coding Framework with Minimal Algorithmic Delay10.1109/JSTSP.2024.3501681 ↗
25. Büthe, J., Valin, J.M., and Mustafa, A. (2023, January 22–25). LACE: A Light-Weight, Causal Model for Enhancing Coded Speech Through Adaptive Convolutions. Proceedings of the IEEE WASPAA, New Paltz, NY, USA.10.1109/WASPAA58266.2023.10248150 ↗
26. Büthe, J., Mustafa, A., Valin, J.M., Helwani, K., and Goodwin, M.M. (2024, January 14–19). NOLACE: Improving Low-Complexity Speech Codec Enhancement Through Adaptive Temporal Shaping. Proceedings of the IEEE ICASSP, Seoul, Republic of Korea.10.1109/ICASSP48485.2024.10448332 ↗
27. Enhancement of Coded Speech Using Neural Network-Based Side Information10.1109/ACCESS.2021.3108784 ↗
28. Biswas, A., and Jia, D. (2020, January 4–8). Audio Codec Enhancement with Generative Adversarial Networks. Proceedings of the IEEE ICASSP, Barcelona, Spain.10.1109/ICASSP40776.2020.9053113 ↗
29. Korse, S., Pia, N., Gupta, K., and Fuchs, G. (2022, January 22–27). PostGAN: A GAN-Based Post-Processor to Enhance the Quality of Coded Speech. Proceedings of the IEEE ICASSP, Singapore.10.1109/ICASSP43922.2022.9747733 ↗
30. Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014, January 8–13). Generative Adversarial Nets. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS 2014), Montreal, QC, Canada.
31. Kumar, K., Kumar, R., De Boissiere, T., Gestin, L., Teoh, W.Z., Sotelo, J., de Brébisson, A., Bengio, Y., and Courville, A.C. (2019, January 8–14). MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS 2019), Vancouver, BC, Canada.
32. Kaneko, T., Tanaka, K., Kameoka, H., and Seki, S. (2022, January 22–27). iSTFTNet: Fast and Lightweight Mel-Spectrogram Vocoder Incorporating Inverse Short-Time Fourier Transform. Proceedings of the IEEE ICASSP, Singapore.10.1109/ICASSP43922.2022.9746713 ↗
33. Mao, X., Li, Q., Xie, H., Lau, R.Y.K., Wang, Z., and Smolley, S.P. (2017, January 22–29). Least Squares Generative Adversarial Networks. Proceedings of the IEEE ICCV, Venice, Italy.10.1109/ICCV.2017.304 ↗
34. Kong, J., Kim, J., and Bae, J. (2020, January 6–12). HiFi-GAN: Generative Adversarial Networks for Efficient and High-Fidelity Speech Synthesis. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS 2020), Virtual.
35. Yamamoto, R., Song, E., and Kim, J.M. (2020, January 4–8). Parallel WaveGAN: A Fast Waveform Generation Model Based on Generative Adversarial Networks with Multi-Resolution Spectrogram. Proceedings of the IEEE ICASSP, Barcelona, Spain.10.1109/ICASSP40776.2020.9053795 ↗
36. (2007). Wideband Extension to Recommendation P.862 for the Assessment of Wideband Telephone Networks and Speech Codecs. Standard No. ITU-T Recommendation P.862.2.
37. Hines, A., Skoglund, J., Kokaram, A., and Harte, N. (2012, January 4–6). ViSQOL: The Virtual Speech Quality Objective Listener. Proceedings of the IWAENC, Aachen, Germany.
38. Chinen, M., Lim, F.S., Skoglund, J., Gureev, N., O’Gorman, F., and Hines, A. (2020, January 26–28). ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metric. Proceedings of the QoMEX, Athlone, Ireland.10.1109/QoMEX48832.2020.9123150 ↗
39. Valentini-Botinhao, C., Wang, X., Takaki, S., and Yamagishi, J. (2016, January 8–12). Speech Enhancement for a Noise-Robust Text-to-Speech Synthesis System Using Deep Recurrent Neural Networks. Proceedings of the Interspeech 2016, San Francisco, CA, USA.10.21437/Interspeech.2016-159 ↗
40. Veaux, C., Yamagishi, J., and King, S. (2013, January 1–3). The voice bank corpus: Design, collection and data analysis of a large regional accent speech database. Proceedings of the O-COCOSDA, Seoul, Republic of Korea.10.1109/ICSDA.2013.6709856 ↗
41. Kingma, D.P., and Ba, J. (2015, January 7–9). Adam: A Method for Stochastic Optimization. Proceedings of the ICLR, San Diego, CA, USA.
42. Ba, J.L., Kiros, J.R., and Hinton, G.E. (2016). Layer Normalization. arXiv.
43. Conv-TasNet: Surpassing Ideal Time–Frequency Magnitude Masking for Speech Separation10.1109/TASLP.2019.2915167 ↗
44. Mittag, G., Naderi, B., Chehadi, A., and Möller, S. (September, January 30). NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasets. Proceedings of the Interspeech 2021, Brno, Czech Republic.10.21437/Interspeech.2021-299 ↗
45. (2015). Method for the Subjective Assessment of Intermediate Quality Level of Audio Systems. Standard No. ITU-R Recommendation BS.1534-3.