Spectrogram Features for Audio and Speech Analysis
Source abstract
Abstract
Spectrogram-based representations have grown to dominate the feature space for deep learning audio analysis systems, and are often adopted for speech analysis also. Initially, the primary motivation behind spectrogram-based representations was their ability to present sound as a two-dimensional signal in the time–frequency plane, which not only provides an interpretable physical basis for analysing sound, but also unlocks the use of a range of machine learning techniques such as convolutional neural networks, which had been developed for image processing. A spectrogram is a matrix characterised by the resolution and span of its dimensions, as well as by the representation and scaling of each element. Many possibilities for these three characteristics have been explored by researchers across numerous application areas, with different settings showing affinity for various tasks. This paper reviews the use of spectrogram-based representations and surveys the state-of-the-art to question how front-end feature representation choice allies with back-end classifier architecture for different tasks.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Music and Audio Processing · Speech Recognition and Synthesis · Emotion and Mood Recognition
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
ICT Cluster, Singapore Institute of Technology, 1 Punggol Coast Road, Singapore 828608, Singapore
Austrian Institute of Technology, 1210 Vienna, Austria
National Engineering Research Center of Speech and Language Information Processing (NERC-SLIP), The University of Science and Technology of China, Hefei 230026, China
Computer Science, Duke-Kunshan University, Kunshan, Suzhou 215316, China
Meta Inc., Reality Labs, 75002 Paris, France
References (100+)
Show source-provided references
1. The Sound Spectrograph10.1121/1.1916342 ↗
2. Zue, V.W., and Cole, R.A. (1979, January 2–4). Experiments on spectrogram reading. Proceedings of the Acoustics, Speech, and Signal Processing, IEEE International Conference on ICASSP’79, Washington, DC, USA.10.1109/ICASSP.1979.1170735 ↗
3. McLoughlin, I.V. (2016). Speech and Audio Processing: A MATLAB-Based Approach, Cambridge University Press.10.1017/CBO9781316084205 ↗
4. Fourier Series10.1038/059606a0 ↗
5. Ifeachor, E.C., and Jervis, B.W. (1993). Digital Signal Processing: A Practical Approach, Addison-Wesley.
6. Robust Sound Event Classification using Deep Neural Networks10.1109/TASLP.2015.2389618 ↗
7. CNN Explainer: Learning Convolutional Neural Networks with Interactive Visualization10.1109/TVCG.2020.3030418 ↗
8. Desplanques, B., Thienpondt, J., and Demuynck, K. (2020, January 25). ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification. Proceedings of the Interspeech, Shanghai, China.10.21437/Interspeech.2020-2650 ↗
9. Radford, A., Kim, J.W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. (2023, January 23). Robust speech recognition via large-scale weak supervision. Proceedings of the ICML, Honolulu, HI, USA.
10. Accuracy enhancement method for speech emotion recognition from spectrogram using temporal frequency correlation and positional information learning through knowledge transfer10.1109/ACCESS.2024.3447770 ↗
11. Ellis, D.P. (2020, January 01). Gammatone-Like Spectrograms. Available online: https://www.ee.columbia.edu/~dpwe/resources/matlab/gammatonegram/.
12. FitzGerald, D. (2012, January 28). Vocal separation using nearest neighbours and median filtering. Proceedings of the IET Irish Signals and Systems Conference (ISSC 2012), Maynooth, Ireland.10.1049/ic.2012.0225 ↗
13. Calculation of a constant Q spectral transform10.1121/1.400476 ↗
14. Yizhi, L., Yuan, R., Zhang, G., Ma, Y., Chen, X., Yin, H., Xiao, C., Lin, C., Ragni, A., and Benetos, E. (2023, January 1). MERT: Acoustic music understanding model with large-scale self-supervised training. Proceedings of the ICLR, Kigali, Rwanda.
15. Musical timbre style transfer with diffusion model10.7717/peerj-cs.2194 ↗
16. Exploiting correlogram structure for robust speech recognition with multiple speech sources10.1016/j.specom.2007.05.003 ↗
17. Time-domain modeling of peripheral auditory processing: A modular architecture and a software platform10.1121/1.414456 ↗
18. Walters, T.C. (2011). Auditory-Based Processing of Communication Sounds. [Ph.D. Thesis, University of Cambridge].
19. Sound retrieval and ranking using sparse auditory representations10.1162/NECO_a_00011 ↗
20. Lyon, R.F., Rehn, M., Walters, T., Bengio, S., and Chechik, G. (2013). Audio Classification for Information Retrieval Using Sparse Features. (8,463,719), U.S. Patent.
21. Lyon, R.F., Ponte, J., and Chechik, G. (2011, January 22). Sparse coding of auditory features for machine hearing in interference. Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), Prague, Czech Republic.10.1109/ICASSP.2011.5947698 ↗
22. Lyon, R.F. (2011, January 16). Machine hearing: Audio analysis by emulation of human hearing. Proceedings of the Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), New Palz, NY, USA.10.1109/ASPAA.2011.6082352 ↗
23. Zhang, H., McLoughlin, I., and Song, Y. (2015, January 19). Robust Sound Event Recognition using Convolutional Neural Networks. Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brisbane, Australia.10.1109/ICASSP.2015.7178031 ↗
24. Leonardis, A., Bischof, H., and Pinz, A. (2006). Sampling Strategies for Bag-of-Features Image Classification. Proceedings of the Computer Vision—ECCV 2006, Springer.10.1007/11744047 ↗
25. Gong, Y., Chung, Y.A., and Glass, J. (2021, January 15). AST: Audio Spectrogram Transformer. Proceedings of the Interspeech 2021, Graz, Austria.10.21437/Interspeech.2021-698 ↗
26. Image feature representation of the subband power distribution for robust sound event classification10.1109/TASL.2012.2226160 ↗
27. Snyder, D., Garcia-Romero, D., Sell, G., Povey, D., and Khudanpur, S. (2018, January 15). X-Vectors: Robust DNN Embeddings for Speaker Recognition. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Calgary, AB, Canada.10.1109/ICASSP.2018.8461375 ↗
28. A new variance-based approach for discriminative feature extraction in machine hearing classification using spectrogram features10.1016/j.dsp.2016.04.005 ↗
29. Variance Normalised Features for Language and Dialect Discrimination10.1007/s00034-020-01641-1 ↗
30. Jin, M., Song, Y., and McLoughlin, I. (2016, January 21–24). LID-senone Extraction via Deep Neural Networks for End-to-End Language Identification. Proceedings of the Odyssey, Bilbao, Spain.10.21437/Odyssey.2016-30 ↗
31. Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences10.1109/TASSP.1980.1163420 ↗
32. On an Extended Fisher Criterion for Feature Selection10.1109/TPAMI.1981.4767154 ↗
33. Oord, A.v.d., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K. (2016). WaveNet: A Generative Model for Raw Audio. arXiv.
34. Rafsanjani, M.A.H., Mawalim, C.O., Lestari, D.P., Sakti, S., and Unoki, M. (2024, January 3). Unsupervised Anomalous Sound Detection Using Timbral and Human Voice Disorder-Related Acoustic Features. Proceedings of the Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), Macao, China.10.1109/APSIPAASC63619.2025.10848563 ↗
35. Miao, X., McLoughlin, I., and Yan, Y. (2019, January 15). A New Time-Frequency Attention Mechanism for TDNN and CNN-LSTM-TDNN, with Application to Language Identification. Proceedings of the Interspeech, Graz, Austria.10.21437/Interspeech.2019-1256 ↗
36. Koutini, K., Schlüter, J., Eghbal-zadeh, H., and Widmer, G. (2022, January 18). Efficient Training of Audio Transformers with Patchout. Proceedings of the Interspeech, Incheon, Republic of Korea.10.21437/Interspeech.2022-227 ↗
37. Chen, K., Du, X., Zhu, B., Ma, Z., Berg-Kirkpatrick, T., and Dubnov, S. (2022, January 22). HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection. Proceedings of the ICASSP, Singapore.10.31219/osf.io/d264y ↗
38. Cai, P., Song, Y., Gu, Q., Jiang, N., Song, H., and McLoughlin, I. (2025, January 27). Detect Any Sound: Open-Vocabulary Sound Event Detection with Multi-Modal Queries. Proceedings of the 33rd ACM International Conference on Multimedia, Dublin, Ireland.10.1145/3746027.3755574 ↗
39. Chen, T., Yang, Y., Qiu, C., Fan, X., Guo, X., and Shangguan, L. (2024, January 3–7). Enabling Hands-Free Voice Assistant Activation on Earphones. Proceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services (MOBISYS), Tokyo, Japan.10.1145/3643832.3661890 ↗
40. Bregman, A.S. (1994). Auditory Scene Analysis: The Perceptual Organization of Sound, MIT Press.10.1121/1.408434 ↗
41. Sound Event Detection: A tutorial10.1109/MSP.2021.3090678 ↗
42. Zeng, X.M., Song, Y., Zhuo, Z., Zhou, Y., Li, Y.H., Xue, H., Dai, L.R., and McLoughlin, I. (2023, January 4). Joint Generative-Contrastive Representation Learning for Anomalous Sound Detection. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes, Greece.10.1109/ICASSP49357.2023.10095568 ↗
43. Miao, X., and McLoughlin, I. (2019). LSTM-TDNN with convolutional front-end for Dialect Identification in the 2019 Multi-Genre Broadcast Challenge. arXiv.
44. Jiang, Y., Song, Y., McLoughlin, I., Gao, Z., and Dai, L.R. (2019, January 15). An Effective Deep Embedding Learning Architecture for Speaker Verification. Proceedings of the Interspeech, Graz, Austria.10.21437/Interspeech.2019-1606 ↗
45. Sun, L., Du, J., Jiang, C., Zhang, X., He, S., Yin, B., and Lee, C.H. (2018, January 2). Speaker Diarization with Enhancing Speech for the First DIHARD Challenge. Proceedings of the Interspeech, Hyderabad, India.10.21437/Interspeech.2018-1742 ↗
46. Gao, Z., Song, Y., McLoughlin, I., Li, P., Jiang, Y., and Dai, L.R. (2019, January 15). Improving Aggregation and Loss Function for Better Embedding Learning in End-to-End Speaker Verification System. Proceedings of the Interspeech, Graz, Austria.10.21437/Interspeech.2019-1489 ↗
47. CNN-MoE Based Framework for Classification of Respiratory Anomalies and Lung Disease Detection10.1109/JBHI.2021.3064237 ↗
48. Lung Sound Classification Using Co-Tuning and Stochastic Normalization10.1109/TBME.2022.3156293 ↗
49. Milling, M., Pokorny, F.B., Bartl-Pokorny, K.D., and Schuller, B.W. (2022). Is Speech the New Blood? Recent Progress in AI-Based Disease Detection From Audio in a Nutshell. Front. Digit. Health, 4.10.3389/fdgth.2022.886615 ↗
50. Nonspeech7k dataset: Classification and analysis of human non-speech sound10.1049/sil2.12233 ↗
51. Kim, S.Y., Lee, H.M., Lim, C.Y., and Kim, H.W. (2025). Detection of Abnormal Symptoms Using Acoustic-Spectrogram-Based Deep Learning. Appl. Sci., 15.10.3390/app15094679 ↗
52. Music Deep Learning: Deep Learning Methods for Music Signal Processing—A Review of the State-of-the-Art10.1109/ACCESS.2023.3244620 ↗
53. Chen, R., Ghobakhlou, A., and Narayanan, A. (2024). Hierarchical Residual Attention Network for Musical Instrument Recognition Using Scaled Multi-Spectrogram. Appl. Sci., 14.10.20944/preprints202409.1632.v1 ↗
54. Self-Supervised Learning of Multi-Level Audio Representations for Music Segmentation10.1109/TASLP.2024.3379894 ↗
55. Thapa, N., and Lee, J. (2024). Dual-Path Beat Tracking: Combining Temporal Convolutional Networks and Transformers in Parallel. Appl. Sci., 14.10.3390/app142411777 ↗
56. Verma, P., and Berger, J. (2021). Audio transformers: Transformer architectures for large scale audio understanding. adieu convolutions. arXiv.
57. Bag-of-features methods for acoustic event detection and classification10.1109/TASLP.2017.2690574 ↗
58. Overlapping sound event recognition using local spectrogram features and the generalised hough transform10.1016/j.patrec.2013.02.015 ↗
59. Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networks10.1109/JSTSP.2018.2885636 ↗
60. Xia, W., and Koishida, K. (2019). Sound Event Detection in Multichannel Audio using Convolutional Time-Frequency-Channel Squeeze and Excitation. arXiv.10.21437/Interspeech.2019-1860 ↗
61. Alcázar, J.N., Zuccarello, P., and Cobos, M. (2020). Classification of Sound Scenes and Events in Real-World Scenarios with Deep Learning Techniques. [Ph.D. Thesis, Universitat de Valencia]. Available online: https://roderic.uv.es/rest/api/core/bitstreams/d50a3d5d-8481-4e33-8196-ff5af972771f/content.
62. Unsupervised Sound Separation Using Mixture Invariant Training
63. BirdNET: A deep learning solution for avian diversity monitoring10.1016/j.ecoinf.2021.101236 ↗
64. Separation of overlapping audio signals: A review on current trends and evolving approaches10.1016/j.sigpro.2024.109487 ↗
65. Sudo, Y., Itoyama, K., Nishida, K., and Nakadai, K. (2019, January 4). Environmental sound segmentation utilizing Mask U-Net. Proceedings of the 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Macau, China.10.1109/IROS40897.2019.8967954 ↗
66. Sudo, Y., Itoyama, K., Nishida, K., and Nakadai, K. (2020, January 12). Multi-channel Environmental sound segmentation. Proceedings of the IEEE/SICE International Symposium on System Integration (SII), Honolulu, HI, USA.10.1109/SII46433.2020.9025963 ↗
67. Baelde, M., Biernacki, C., and Greff, R. (2017, January 5). A mixture model-based real-time audio sources classification method. Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA.10.1109/ICASSP.2017.7952592 ↗
68. Hong, R., Cheng, W.H., Yamasaki, T., Wang, M., and Ngo, C.W. (2018). Mixup-Based Acoustic Scene Classification Using Multi-channel Convolutional Neural Network. Proceedings of the Advances in Multimedia Information Processing—PCM 2018, Springer International Publishing.
69. Phan, H., Maass, M., Mazur, R., and Mertins, A. (2015, January 29). Early event detection in audio streams. Proceedings of the 2015 IEEE International Conference on Multimedia and Expo (ICME), Torino, Italy.10.1109/ICME.2015.7177439 ↗
70. Zhao, X., Zhang, X., Zhao, C., Cho, J.H., Kaplan, L., and Jeong, D.H. (2023, January 4). Multi-Label Temporal Evidential Neural Networks for Early Event Detection. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes, Greece.10.1109/ICASSP49357.2023.10096305 ↗
71. Zhao, X., Zhang, X., Cheng, W., Yu, W., Chen, Y., Chen, H., and Chen, F. (2022, January 22). Seed: Sound event early detection via evidential uncertainty. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore.10.1109/ICASSP43922.2022.9746756 ↗
72. Mesaros, A., Heittola, T., and Virtanen, T. (2016, January 29). TUT database for acoustic scene classification and sound event detection. Proceedings of the 2016 24th European Signal Processing Conference (EUSIPCO), Budapest, Hungary.10.1109/EUSIPCO.2016.7760424 ↗
73. Detection and Classification of Acoustic Scenes and Events10.1109/TMM.2015.2428998 ↗
74. Çakir, E., and Virtanen, T. (2018, January 8). End-to-End Polyphonic Sound Event Detection Using Convolutional Recurrent Neural Networks with Learned Time-Frequency Representation Input. Proceedings of the 2018 International Joint Conference on Neural Networks (IJCNN), Rio de Janeiro, Brazil.10.1109/IJCNN.2018.8489470 ↗
75. Bittner, R., McFee, B., Salamon, J., Li, P., and Bello, J. (2017, January 23–27). Deep Salience Representations for F0 Estimation in Polyphonic Music. Proceedings of the 18th International Society for Music Information Retrieval Conference, Suzhou, China.
76. Leng, Y.R., Tran, H.D., Kitaoka, N., and Li, H. (2010, January 26). Selective gammatone filterbank feature for robust sound event recognition. Proceedings of the Interspeech, Makuhari, Chiba, Japan.10.21437/Interspeech.2010-617 ↗
77. Robust acoustic scene classification using a multi-spectrogram encoder-decoder framework10.1016/j.dsp.2020.102943 ↗
78. Wang, Y., Getreuer, P., Hughes, T., Lyon, R.F., and Saurous, R.A. (2017, January 5). Trainable frontend for robust and far-field keyword spotting. Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, LA, USA.10.1109/ICASSP.2017.7953242 ↗
79. Yin, H., Bai, J., Xiao, Y., Wang, H., Zheng, S., Chen, Y., Das, R.K., Deng, C., and Chen, J. (2025, January 6). Exploring Text-Queried Sound Event Detection with Audio Source Separation. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Hyderabad, India.10.1109/ICASSP49660.2025.10889789 ↗
80. Adavanne, S., and Virtanen, T. (2017). A Report on Sound Event Detection with Different Binaural Features, Available online: https://arxiv.org/abs/1710.02997.
81. Heittola, T., and Mesaros, A. (2017). DCASE 2017 Challenge Setup: Tasks, Datasets and Baseline System, Available online: https://inria.hal.science/hal-01627981/.
82. Lin, L., and Wang, X. (2019). Guided Learning Convolution System For Dcase 2019 Task 4, Institute of Computing Technology, Chinese Academy of Sciences. Technical Report.10.33682/53ed-z889 ↗
83. Gemmeke, J.F., Ellis, D.P.W., Freedman, D., Jansen, A., Lawrence, W., Moore, R.C., Plakal, M., and Ritter, M. (2017, January 5–9). Audio Set: An ontology and human-labeled dataset for audio events. Proceedings of the IEEE ICASSP 2017, New Orleans, LA, USA.10.1109/ICASSP.2017.7952261 ↗
84. Fonseca, E., Pons, J., Favory, X., Font, F., Bogdanov, D., Ferraro, A., Oramas, S., Porter, A., and Serra, X. (2017, January 23–27). Freesound Datasets: A platform for the creation of open audio datasets. Proceedings of the 18th International Society for Music Information Retrieval Conference (ISMIR), Suzhou, China.
85. Time–Frequency Feature Fusion for Noise Robust Audio Event Classification10.1007/s00034-019-01203-0 ↗
86. Ebbers, J., and Haeb-Umbach, R. (2022). Pre-Training and Self-Training for Sound Event Detection in Domestic Environments, Available online: https://ris.uni-paderborn.de/record/49113.
87. Turpault, N., Serizel, R., Parag Shah, A., and Salamon, J. (2019, January 25–26). Sound event detection in domestic environments with weakly labeled data and soundscape synthesis. Proceedings of the Workshop on Detection and Classification of Acoustic Scenes and Events, New York, NY, USA.10.33682/006b-jx26 ↗
88. Kim, J.W., Son, S.W., Song, Y., Kim, H.K., Song, I.H., and Lim, J.E. (2023). Semi-Supervised Learning-Based Sound Event Detection Using Frequency Dynamic Convolution With Large Kernel Attention For DCASE Challenge 2023 Task 4, Available online: https://arxiv.org/abs/2306.06461.
89. Schmid, F., Primus, P., Morocutti, T., Greif, J., and Widmer, G. (2024). Improving Audio Spectrogram Transformers For Sound Event Detection Through Multi-Stage Training. arXiv.10.1109/ICASSP49660.2025.10888942 ↗
90. Strong Labeling of Sound Events Using Crowdsourced Weak Labels and Annotator Competence Estimation10.1109/TASLP.2022.3233468 ↗
91. A survey of mechanical fault diagnosis based on audio signal analysis10.1016/j.measurement.2023.113294 ↗
92. Qurthobi, A., Maskeliūnas, R., and Damaševičius, R. (2022). Detection of Mechanical Failures in Industrial Machines Using Overlapping Acoustic Anomalies: A Systematic Literature Review. Sensors, 22.10.3390/s22103888 ↗
93. Zeng, X.M., Song, Y., McLoughlin, I., Liu, L., and Dai, L.R. (2023, January 20). Robust Prototype Learning for Anomalous Sound Detection. Proceedings of the Interspeech, Dublin, Ireland.10.21437/Interspeech.2023-1173 ↗
94. Koizumi, Y., Kawaguchi, Y., Imoto, K., Nakamura, T., Nikaido, Y., Tanabe, R., Purohit, H., Suefusa, K., Endo, T., and Yasuda, M. (2020, January 2–4). Description and Discussion on DCASE2020 Challenge Task2; Tokyo, Japan: Unsupervised Anomalous Sound Detection for Machine Condition Monitoring. Proceedings of the DCASE, Virtual.
95. Suefusa, K., Nishida, T., Purohit, H., Tanabe, R., Endo, T., and Kawaguchi, Y. (2020, January 4). Anomalous Sound Detection Based on Interpolation Deep Neural Network. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Virtual.10.1109/ICASSP40776.2020.9054344 ↗
96. Zhenhua, L., Jianqing, G., Kai, Y., and Jia, J. (2023). Predictive AutoEncoders Are Context-Aware Unsupervised Anomalous Sound Detectors. Proceedings of the Man-Machine Speech Communication, Springer Nature.10.1007/978-981-99-2401-1_9 ↗
97. Hendrycks, D., Mazeika, M., and Dietterich, T. (2019, January 6). Deep Anomaly Detection with Outlier Exposure. Proceedings of the ICLR, New Orleans, LA, USA.
98. Liu, Y., Guan, J., Zhu, Q., and Wang, W. (2022, January 22). Anomalous Sound Detection Using Spectral-Temporal Information Fusion. Proceedings of the ICASSP 2022—2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore.10.1109/ICASSP43922.2022.9747868 ↗
99. Han, B., Lv, Z., Jiang, A., Huang, W., Chen, Z., Deng, Y., Ding, J., Lu, C., Zhang, W.Q., and Fan, P. (2024, January 14). Exploring Large Scale Pre-Trained Models for Robust Machine Anomalous Sound Detection. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Seoul, Republic of Korea.10.1109/ICASSP48485.2024.10447183 ↗
100. Jiang, A., Han, B., Lv, Z., Deng, Y., Zhang, W.Q., Chen, X., Qian, Y., Liu, J., and Fan, P. (2024, January 1). AnoPatch: Towards Better Consistency in Machine Anomalous Sound Detection. Proceedings of the Interspeech, Kos, Greece.10.21437/Interspeech.2024-1761 ↗