Improved Patch-Mix Transformer and Contrastive Learning Method for Sound Classification in Noisy Environments
Source abstract
Abstract
In urban environments, noise significantly impacts daily life and presents challenges for Environmental Sound Classification (ESC). The structural influence of urban noise on audio signals complicates feature extraction and audio classification for environmental sound classification methods. To address these challenges, this paper proposes a Contrastive Learning-based Audio Spectrogram Transformer (CL-Transformer) that incorporates a Patch-Mix mechanism and adaptive contrastive learning strategies while simultaneously improving and utilizing adaptive data augmentation techniques for model training. Firstly, a combination of data augmentation techniques is introduced to enrich environmental sounds. Then, the Patch-Mix feature fusion scheme randomly mixes patches of the enhanced and noisy spectrograms during the Transformer’s patch embedding. Furthermore, a novel contrastive learning scheme is introduced to quantify loss and improve model performance, synergizing well with the Transformer model. Finally, experiments on the ESC-50 and UrbanSound8K public datasets achieved accuracies of 97.75% and 92.95%, respectively. To simulate the impact of noise in real urban environments, the model is evaluated using the UrbanSound8K dataset with added background noise at different signal-to-noise ratios (SNR). Experimental results demonstrate that the proposed framework performs well in noisy environments.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Music and Audio Processing · Speech and Audio Processing · Music Technology and Sound Studies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Computer Science and Engineering, Guilin University of Technology, Guilin 541006, China
College of Physics and Electronic Information Engineering, Guilin University of Technology, Guilin 541006, China
School of Information and Communication, Guilin University of Electronic Technology, Guilin 541006, China
Funding
National Natural Science Foundation of China · Guangxi Science and Technology Major Program · Innovation Project of GUET Graduate Education
References (37)
Show source-provided references
1. Dynamic-Structured Reservoir Spiking Neural Network in Sound Localization10.1109/ACCESS.2024.3360491 ↗
2. Warnecke, M., and Litovsky, R.Y. (2021). Signal Envelope and Speech Intelligibility Differentially Impact Auditory Motion Perception. Sci. Rep., 11.10.1038/s41598-021-94662-y ↗
3. Yang, C., Gan, X., Peng, A., and Yuan, X. (2023). ResNet Based on Multi-Feature Attention Mechanism for Sound Classification in Noisy Environments. Sustainability, 15.10.3390/su151410762 ↗
4. Environmental Sound Classification: A Descriptive Review of the Literature
5. Zhang, D., Zhong, Z., Xia, Y., Wang, Z., and Xiong, W. (2023). An Automatic Classification System for Environmental Sound in Smart Cities. Sensors, 23.10.3390/s23156823 ↗
6. Environmental Sound Classification Using a Regularized Deep Convolutional Neural Network with Data Augmentation10.1016/j.apacoust.2020.107389 ↗
7. Attention-Based Convolutional Recurrent Neural Network for Environmental Sound Classification10.1016/j.neucom.2020.08.069 ↗
8. Rethinking Environmental Sound Classification Using Convolutional Neural Networks: Optimized Parameter Tuning of Single Feature Extraction10.1007/s00521-021-06091-7 ↗
9. MSARN: A Multi-Scale Attention Residual Network for End-to-End Environmental Sound Classification10.1007/s11063-023-11383-1 ↗
10. Das, J.K., Ghosh, A., Pal, A.K., Dutta, S., and Chakrabarty, A. (2020, January 27–29). Urban Sound Classification Using Convolutional Neural Network and Long Short Term Memory Based on Multiple Features. Proceedings of the 2020 Fourth International Conference on Intelligent Computing in Data Sciences (ICDS), Hong Kong, China.10.1109/ICDS50568.2020.9268723 ↗
11. Mushtaq, Z., and Su, S.F. (2020). Efficient Classification of Environmental Sounds through Multiple Features Aggregation and Data Enhancement Techniques for Spectrogram Images. Symmetry, 12.10.3390/sym12111822 ↗
12. Nasiri, A., and Hu, J. (2021). SoundCLR: Contrastive Learning of Representations for Improved Environmental Sound Classification. arXiv.
13. Qiao, T., Zhang, S., Cao, S., and Xu, S. (2021). High Accurate Environmental Sound Classification: Sub-Spectrogram Segmentation versus Temporal-Frequency Attention Mechanism. Sensors, 21.10.3390/s21165500 ↗
14. Guzhov, A., Raue, F., Hees, J., and Dengel, A. (2021, January 10–15). ESResNet: Environmental Sound Classification Based on Visual Domain Models. Proceedings of the 2020 25th International Conference on Pattern Recognition (ICPR), Milan, Italy.10.1109/ICPR48806.2021.9413035 ↗
15. Nogueira, A.F.R., Oliveira, H.S., Machado, J.J.M., and Tavares, J.M.R.S. (2022). Transformers for Urban Sound Classification—A Comprehensive Performance Evaluation. Sensors, 22.10.3390/s22228874 ↗
16. Tsalera, E., Papadakis, A., and Samarakou, M. (2021). Comparison of Pre-Trained CNNs for Audio Classification Using Transfer Learning. J. Sens. Actuator Netw., 10.10.3390/jsan10040072 ↗
17. Sound Event Detection of Weakly Labelled Data with CNN-Transformer and Automatic Threshold Optimization10.1109/TASLP.2020.3014737 ↗
18. VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
19. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017, January 4–9). Attention Is All You Need. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA.
20. Gong, Y., Chung, Y.A., and Glass, J. (2021). AST: Audio Spectrogram Transformer. arXiv.10.21437/Interspeech.2021-698 ↗
21. Elliott, D., Otero, C.E., Wyatt, S., and Martino, E. (2021). Tiny Transformers for Environmental Sound Classification at the Edge. arXiv.
22. Deep Convolutional Neural Networks and Data Augmentation for Environmental Sound Classification10.1109/LSP.2017.2657381 ↗
23. Park, D.S., and Chan, W. (2024, August 25). SpecAugment: A New Data Augmentation Method for Automatic Speech Recognition. Google AI Blog, 2019. Available online: https://towardsdatascience.com/state-of-the-art-audio-data-augmentation-with-google-brains-specaugment-and-pytorch-d3d1a3ce291e.
24. Investigation of the Effectiveness of Time-Frequency Domain Images and Acoustic Features in Urban Sound Classification10.1016/j.apacoust.2023.109564 ↗
25. Ashurov, A., Zhou, Y., and Shi, L. (2022). Environmental Sound Classification Based on Transfer-Learning Techniques with Multiple Optimizers. Electronics, 11.10.3390/electronics11152279 ↗
26. Concatenation-Based Pre-Trained Convolutional Neural Networks Using Attention Mechanism for Environmental Sound Classification10.1016/j.apacoust.2023.109759 ↗
27. Deep Learning-Based Environmental Sound Classification Using Feature Fusion and Data Enhancement
28. Bridging the Gap between the Short-Time Fourier Transform (STFT), Wavelets, the Constant-Q Transform and Multi-Resolution STFT10.1007/s11760-020-01701-8 ↗
29. Bae, S., Kim, J.W., Cho, W.Y., Baek, H., Son, S., Lee, B., Ha, C., Tae, K., Kim, S., and Yun, S.-Y. (2023). Patch-Mix Contrastive Learning with Audio Spectrogram Transformer on Respiratory Sound Classification. arXiv.10.21437/Interspeech.2023-1426 ↗
30. EnvGAN: A GAN-Based Augmentation to Improve Environmental Sound Classification10.1007/s10462-022-10153-0 ↗
31. CNN-RNN and Data Augmentation Using Deep Convolutional Generative Adversarial Network for Environmental Sound Classification10.1109/LSP.2022.3150258 ↗
32. Nogueira, A.F.R., Oliveira, H.S., Machado, J.J.M., and Tavares, J.M.R.S. (2022). Sound Classification and Processing of Urban Environments: A Systematic Literature Review. Sensors, 22.10.3390/s22228608 ↗
33. Piczak, K.J. (2015, January 26–30). ESC: Dataset for Environmental Sound Classification. Proceedings of the 23rd ACM International Conference on Multimedia, Brisbane, Australia.10.1145/2733373.2806390 ↗
34. Salamon, J., Jacoby, C., and Bello, J.P. (2014, January 3–7). A Dataset and Taxonomy for Urban Sound Research. Proceedings of the 22nd ACM International Conference on Multimedia, New York, NY, USA.10.1145/2647868.2655045 ↗
35. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20–25). Imagenet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
36. Gazneli, A., Zimerman, G., Ridnik, T., Sharir, G., and Noy, A. (2022). End-to-End Audio Strikes Back: Boosting Augmentations towards an Efficient Audio Classification Network. arXiv.
37. Chen, K., Du, X., Zhu, B., Ma, Z., Berg-Kirkpatrick, T., and Dubnov, S. (2022, January 23–27). HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection. Proceedings of the 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2022), Singapore.10.1109/ICASSP43922.2022.9746312 ↗