A Lightweight Framework for Audio-Visual Segmentation with an Audio-Guided Space–Time Memory Network
Source abstract
Abstract
As a multimodal fusion task, audio-visual segmentation (AVS) aims to locate sounding objects at the pixel level within a given image. This capability holds significant importance and practical value in applications such as intelligent surveillance, multimedia content analysis, and human–robot interaction. However, existing AVS models typically feature complex architectures, require a large number of parameters, and are challenging to deploy on embedded platforms. Furthermore, these models often lack integration with object tracking mechanisms and fail to address the issue of the mis-segmentation of unvoiced objects caused by environmental noise in real-world scenarios. To address these challenges, this research proposes a lightweight audio-visual segmentation framework incorporating an audio-guided space–time memory network (AG-STMNet). First, a mask generator with a scoring mechanism was developed to identify sounding objects from generated masks. This component integrates Fastsam, a lightweight, pre-trained, object-aware segmentation model, with WAV2CLIP, a parameter-efficient audio-visual alignment model. Subsequently, AG-STMNet, an audio-guided video object segmentation network, was introduced to track sounding objects using video object segmentation techniques while mitigating environmental noise. Finally, the mask generator and AG-STMNet were combined to form the complete framework. The experimental results demonstrate that the framework achieves a mean Intersection over Union (mIoU) score of 41.5, indicating its potential as a viable lightweight solution for practical applications.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Music and Audio Processing · Speech and Audio Processing · Hearing Loss and Rehabilitation
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming 650500, China
Funding
Key Laboratory of Industrial Intelligence and System of Yunnan Province
References (23)
Show source-provided references
1. Zhou, J., Wang, J., Zhang, J., Sun, W., Zhang, J., Birchfield, S., Guo, D., Kong, L., Wang, M., and Zhong, Y. (2022, January 23–27). Audio–visual segmentation. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.10.1007/978-3-031-19836-6_22 ↗
2. Polyphonic sound event localization and detection based on Multiple Attention Fusion ResNet10.3934/mbe.2024089 ↗
3. Liu, J., Liu, Y., Zhang, F., Ju, C., Zhang, Y., and Wang, Y. (2024, January 16–22). Audio-visual segmentation via unlabeled frame exploitation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.02487 ↗
4. Chen, T., Tan, Z., Gong, T., Chu, Q., Wu, Y., Liu, B., Lu, L., Ye, J., and Yu, N. (2024). Bootstrapping audio-visual segmentation by strengthening audio cues. arXiv.10.1109/TCSVT.2024.3486344 ↗
5. Enhance audio-visual segmentation with hierarchical encoder and audio guidance10.1016/j.neucom.2024.127885 ↗
6. Hao, D., Mao, Y., He, B., Han, X., Dai, Y., and Zhong, Y. (2024, January 20–27). Improving audio-visual segmentation with bidirectional generation. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.10.1609/aaai.v38i3.27978 ↗
7. Yang, Q., Nie, X., Li, T., Gao, P., Guo, Y., Zhen, C., Yan, P., and Xiang, S. (2024, January 17–21). Cooperation does matter: Exploring multi-order bilateral relations for audio-visual segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR52733.2024.02562 ↗
8. Mao, Y., Zhang, J., Xiang, M., Zhong, Y., and Dai, Y. (2023, January 2–3). Multimodal variational auto-encoder based audio-visual segmentation. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.00094 ↗
9. Liu, C., Li, P.P., Qi, X., Zhang, H., Li, L., Wang, D., and Yu, X. (November, January 29). Audio-visual segmentation by exploring cross-modal mutual semantics. Proceedings of the 31st ACM International Conference on Multimedia, Ottawa, ON, Canada.
10. Wang, Y., Sun, P., Li, Y., Zhang, H., and Hu, D. (2022, January 23–27). Can Textual Semantics Mitigate Sounding Object Segmentation Preference?. Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel.
11. Wang, Y., Zhu, J., Dong, F., and Zhu, S. (2024). Progressive Confident Masking Attention Network for Audio-Visual Segmentation. arXiv.10.2139/ssrn.4656630 ↗
12. Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., and Lo, W.Y. (2023, January 2–3). Segment anything. Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France.10.1109/ICCV51070.2023.00371 ↗
13. Mo, S., and Tian, Y. (2023). AV-SAM: Segment anything model meets audio-visual localization and segmentation. arXiv.
14. Huang, S., Ling, R., Li, H., Hui, T., Tang, Z., Wei, X., Han, J., and Liu, S. (2024). Unleashing the temporal-spatial reasoning capacity of gpt for training-free audio and language referenced video object segmentation. arXiv.10.1609/aaai.v39i4.32387 ↗
15. Bhosale, S., Yang, H., Kanojia, D., Deng, J., and Zhu, X. (2025, January 10–14). Unsupervised audio-visual segmentation with modality alignment. Proceedings of the AAAI Conference on Artificial Intelligence, Edmonton, AB, Canada.10.1609/aaai.v39i15.33709 ↗
16. Liu, J., Wang, Y., Ju, C., Ma, C., Zhang, Y., and Xie, W. (2024, January 3–8). Annotation-free audio-visual segmentation. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA.10.1109/WACV57701.2024.00551 ↗
17. Nguyen, K.B., and Park, C.J. (2024). SAVE: Segment Audio-Visual Easy way using Segment Anything Model. arXiv.10.2139/ssrn.4882670 ↗
18. Lin, Y.B., Sung, Y.L., Lei, J., Bansal, M., and Bertasius, G. (2023, January 17–24). Vision transformers are parameter-efficient audio-visual learners. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.00228 ↗
19. Each Perform Its Functions: Task Decomposition and Feature Assignment for Audio-Visual Segmentation10.1109/TMM.2024.3394682 ↗
20. Zhao, X., Ding, W., An, Y., Du, Y., Yu, T., Li, M., Tang, M., and Wang, J. (2023). Fast segment anything. arXiv.
21. Wu, H.H., Seetharaman, P., Kumar, K., and Bello, J.P. (2022, January 7–13). Wav2clip: Learning robust audio representations from clip. Proceedings of the ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Virtual.10.1109/ICASSP43922.2022.9747669 ↗
22. Oh, S.W., Lee, J.Y., Xu, N., and Kim, S.J. (November, January 27). Video object segmentation using space-time memory networks. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Repulic of Korea.
23. Yu, J., Li, H., Hao, Y., Wu, J., Xu, T., Wang, S., and He, X. (2023, January 20–24). How Can Contrastive Pre-training Benefit Audio-Visual Segmentation? A Study from Supervised and Zero-shot Perspectives. Proceedings of the BMVC, Aberdeen, UK.