A Review of Physical and Perceptual Feature Extraction Techniques for Speech, Music and Environmental Sounds
Source abstract
Abstract
Endowing machines with sensing capabilities similar to those of humans is a prevalent quest in engineering and computer science. In the pursuit of making computers sense their surroundings, a huge effort has been conducted to allow machines and computers to acquire, process, analyze and understand their environment in a human-like way. Focusing on the sense of hearing, the ability of computers to sense their acoustic environment as humans do goes by the name of machine hearing. To achieve this ambitious aim, the representation of the audio signal is of paramount importance. In this paper, we present an up-to-date review of the most relevant audio feature extraction techniques developed to analyze the most usual audio signals: speech, music and environmental sounds. Besides revisiting classic approaches for completeness, we include the latest advances in the field based on new domains of analysis together with novel bio-inspired proposals. These approaches are described following a taxonomy that organizes them according to their physical or perceptual basis, being subsequently divided depending on the domain of computation (time, frequency, wavelet, image-based, cepstral, or other domains). The description of the approaches is accompanied with recent examples of their application to machine hearing related problems.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Music and Audio Processing · Speech and Audio Processing · Animal Vocal Communication and Behavior
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
GTM - Grup de recerca en Tecnologies Mèdia, La Salle-Universitat Ramon Llull, Quatre Camins, 30, 08022 Barcelona, Spain
Funding
European Commission · Secretaria d\''Universitats i Recerca del Departament d\''Economia i Coneixement (Generalitat de Catalunya
References (100+)
Show source-provided references
1. Machine Hearing: An Emerging Field10.1109/MSP.2010.937498 ↗
2. Gerhard, D. (2003). Audio Signal Classification: History and Current Techniques, Department of Computer Science, University of Regina. Technical Report TR-CS 2003-07.
3. Temko, A. (2007). Acoustic Event Detection and Classification. [Ph.D. Thesis, Universitat Politècnica de Catalunya].
4. Dennis, J. (2014). Sound Event Recognition in Unstructured Environments Using Spectrogram Image Processing. [Ph.D. Thesis, School of Computer Engineering, Nanyang Technological University].
5. Robust speech detection in real acoustic backgrounds with perceptually motivated features10.1016/j.specom.2010.07.003 ↗
6. An overview of text-independent speaker recognition: From features to supervectors10.1016/j.specom.2009.08.009 ↗
7. Pieraccini, R. (2012). The Voice in the Machine. Building Computers That Understand Speech, MIT Press.10.7551/mitpress/9072.001.0001 ↗
8. Wang, A.L.C. (2003, January 26–30). An industrial-strength audio search algorithm. Proceedings of the 4th International Conference on Music Information Retrieval (ISMIR), Baltimore, MD, USA.
9. Wang, F., Wang, X., Shao, B., Li, T., and Ogihara, M. (2009, January 26–30). Tag Integrated Multi-Label Music Style Classification with Hypergraph. Proceedings of the 10th International Society for Music Information Retrieval Conference (ISMIR), Kobe, Japan.
10. Benetos, E., Kotti, M., and Kotropoulos, C. (2006, January 14–19). Musical Instrument Classification using Non-Negative Matrix Factorization Algorithms and Subset Feature Selection. Proceedings of the 2006 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Toulouse, France.
11. Liu, M., and Wan, C. (2001, January 24–28). Feature selection for automatic classification of musical instrument sounds. Proceedings of the ACM/IEEE Joint Conference on Digital Libraries (JCDL), Roanoke, VA, USA.10.1145/379437.379663 ↗
12. Automatic Mood Detection and Tracking of Music Audio Signals10.1109/TSA.2005.860344 ↗
13. A Survey of Audio-Based Music Classification and Annotation10.1109/TMM.2010.2098858 ↗
14. Environmental Sound Recognition With Time-Frequency Audio Features10.1109/TASL.2009.2017438 ↗
15. Valero, X., and Alías, F. (2012, January 27–31). Classification of audio scenes using Narrow-Band Autocorrelation features. Proceedings of the 20th European Signal Processing Conference (EUSIPCO), Bucharest, Romania.
16. Schafer, R.M. (1993). The Soundscape: Our Sonic Environment and the Tuning of the World, Inner Traditions/Bear & Co.
17. Features for content-based audio retrieval10.1016/S0065-2458(10)78003-7 ↗
18. Peltonen, V., Tuomi, J., Klapuri, A., Huopaniemi, J., and Sorsa, T. (2002, January 13–17). Computational Auditory Scene Recognition. Proceedings of the 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Orlando, FL, USA.10.1109/ICASSP.2002.1006149 ↗
19. Geiger, J., Schuller, B., and Rigoll, G. (2013, January 20–23). Large-scale audio feature extraction and SVM for acoustic scene classification. Proceedings of the 2013 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), New Paltz, NY, USA.10.1109/WASPAA.2013.6701857 ↗
20. Oppenheim, A.V., and Schafer, R.W. (1989). Discrete-Time Signal Processing, Prentice Hall.
21. Gygi, B. (2001). Factors in the Identification of Environmental Sounds. [Ph.D. Thesis, Indiana University].
22. Foote, J., and Uchihashi, S. (2001, January 22–25). The Beat Spectrum: A New Approach To Rhythm Analysis. Proceedings of the 2001 IEEE International Conference on Multimedia and Expo (ICME), Tokyo, Japan.10.1109/ICME.2001.1237863 ↗
23. Bellman, R. (2003). Dynamic Programming, Dover Publications.
24. Using One-Class SVMs and Wavelets for Audio Surveillance10.1109/TIFS.2008.2008216 ↗
25. Long-term storage capacity of reservoirs10.1061/TACEAT.0006518 ↗
26. Audio-based context recognition10.1109/TSA.2005.854103 ↗
27. Combination of autocorrelation-based features and projection measure technique for speaker identification10.1109/TSA.2005.848893 ↗
28. Musical genre classification of audio signals10.1109/TSA.2002.800560 ↗
29. A theory of primary sensations and spatial sensations measuring environmental noise10.1006/jsvi.2000.3272 ↗
30. Hierarchical Classification of Environmental Noise Sources by Considering the Acoustic Signature of Vehicle Pass-bys10.2478/v10168-012-0054-z ↗
31. An Overview on Perceptually Motivated Audio Indexing and Classification10.1109/JPROC.2013.2251591 ↗
32. Clemins, P.J., Trawicki, M.B., Adi, K., Tao, J., and Johnson, M.T. (2006, January 14–19). Generalized Perceptual Features for Vocalization Analysis Across Multiple Species. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toulouse, France.
33. Generalized perceptual linear prediction features for animal vocalization analysis10.1121/1.2203596 ↗
34. Peeters, G. (2004). A Large Set of Audio Features for Sound Description (Similarity And Classification) in the CUIDADO Project, IRCAM. Technical Report.
35. Sharan, R., and Moir, T. (2016). An overview of applications and advancements in automatic sound recognition. Neurocomputing.10.1016/j.neucom.2016.03.020 ↗
36. Gubka, R., and Kuba, M. (2013, January 29–31). A comparison of audio features for elementary sound based audio classification. Proceedings of the 2013 International Conference on Digital Technologies (DT), Zilina, Slovak Republic.10.1109/DT.2013.6566278 ↗
37. Boonmatham, P., Pongpinigpinyo, S., and Soonklang, T. (2012, January 3–5). A comparison of audio features of Thai Classical Music Instrument. Proceedings of the 7th International Conference on Computing and Convergence Technology (ICCCT), Seoul, South Korea.
38. A Comparison of Spectro-Temporal Representations of Audio Signals10.1109/TASL.2013.2283105 ↗
39. Spectral Analysis and Discrimination by Zero-crossings10.1109/PROC.1986.13663 ↗
40. Li, T., Ogihara, M., and Li, Q. (August, January 28). A Comparative Study on Content-based Music Genre Classification. Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Toronto, ON, Canada.
41. Aggregate features and ADABOOST for music classification10.1007/s10994-006-9019-7 ↗
42. Modeling timbre distance with temporal statistics from polyphonic music10.1109/TSA.2005.860352 ↗
43. Kobayashi, T., Hirose, K., and Nakamura, S. (2010, January 26–30). Noise robust voice activity detection using features extracted from the time-domain autocorrelation function. Proceedings of the 11th Annual Conference of the International Speech (InterSpeech), Makuhari, Japan.
44. El-Maleh, K., Klein, M., Petrucci, G., and Kabal, P. (2000, January 5–9). Speech/music discrimination for multimedia applications. Proceedings of the 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Istanbul, Turkey.
45. International Organization for Standardization (ISO)/International Organization for Standardization (IEC) Information technology—Multimedia content description interface. Available online: http://mpeg.chiariglione.org/standards/mpeg-7/audio.
46. Mitrović, D., Zeppelzauer, M., and Breiteneder, C. (2006, January 4–6). Discrimination and retrieval of animal sounds. Proceedings of the 12th International Multi-Media Modelling Conference, Beijing, China.
47. Muhammad, G., and Alghathbar, K. (2009, January 10–12). Environment Recognition from Audio Using MPEG-7 Features. Proceedings of the 4th International Conference on Embedded and Multimedia Computing, Jeju, Korea.10.1109/EM-COM.2009.5402978 ↗
48. Valero, X., and Alías, F. (2010, January 15–18). Applicability of MPEG-7 low level descriptors to environmental sound source recognition. Proceedings of the EAA EUROREGIO 2010, Ljubljana, Slovenia.
49. The measurement of the signal-to-noise ratio (SNR) in continuous speech10.1016/0167-6393(87)90066-5 ↗
50. Perception of aperiodicity in pathological voice10.1121/1.1858351 ↗
51. Farrús, M., Hernando, J., and Ejarque, P. (2007, January 27–31). Jitter and shimmer measurements for speaker recognition. Proceedings of the 8th Annual Conference of the International Speech Communication Association (InterSpeech), Antwerp, Belgium.10.21437/Interspeech.2007-147 ↗
52. Murthy, Y.V.S., and Koolagudi, S.G. (2015, January 3–6). Classification of vocal and non-vocal regions from audio songs using spectral features and pitch variations. Proceedings of the 28th Canadian Conference on Electrical and Computer Engineering (CCECE), Halifax, NS, Canada.
53. Jia, K., Pan, J.S., Zhao, Y., and Jain, L.C. (2013, January 16–18). Acoustic Features and Auditory Impressions of Death Growl and Screaming Voice. Proceedings of the Ninth International Conference on Intelligent Information Hiding and Multimedia Signal Processing (IIH-MSP), Beijing, China.
54. Jensen, K. (1999, January 13–15). Pitch independent prototyping of musical sounds. Proceedings of the IEEE 3rd Workshop on Multimedia Signal Processing, Copenhagen, Denmark.10.1109/MMSP.1999.793823 ↗
55. Toward semantic indexing and retrieval using hierarchical audio models10.1007/s00530-005-0183-6 ↗
56. Audio content analysis for online audiovisual data segmentation and classification10.1109/89.917689 ↗
57. Smith, D., Cheng, E., and Burnett, I.S. (2010, January 23–27). Musical Onset Detection using MPEG-7 Audio Descriptors. Proceedings of the 20th International Congress on Acoustics (ICA), Sydney, Australia.
58. Audio Content Classification Method Research Based on Two-step Strategy
59. Audio Feature Extraction and Analysis for Scene Segmentation and Classification10.1023/A:1008066223044 ↗
60. Jiang, H., Bai, J., Zhang, S., and Xu, B. (November, January 30). SVM-based audio scene classification. Proceedings of the 2005 IEEE International Conference on Natural Language Processing and Knowledge Engineering, Wuhan, China.
61. Scheirer, E.D., and Slaney, M. (1997, January 21–24). Construction and evaluation of a robust multifeature speech/music discriminator. Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Munich, Germany.
62. Lartillot, O., Eerola, T., Toiviainen, P., and Fornari, J. (2008, January 14–18). Multi-feature modeling of pulse clarity: Design, validation and optimization. Proceedings of the Ninth International Conference on Music Information Retrieval (ISMIR), Philadelphia, PA, USA.10.1007/978-3-540-78246-9_31 ↗
63. Using listener-based perceptual features as intermediate representations in music information retrieval10.1121/1.4892767 ↗
64. Georganas, N.D., and Popescu-Zeletin, R. (October, January 30). A robust audio classification and segmentation method. Proceedings of the 9th ACM International Conference on Multimedia, Ottawa, ON, Canada.
65. Foote, J. (August, January 30). Automatic Audio Segmentation using a Measure of Audio Novelty. Proceedings of the IEEE International Conference on Multimedia and Expo (ICME), New York, NY, USA.
66. Kurth, F., Gehrmann, T., and Müller, M. (2006, January 8–12). The Cyclic Beat Spectrum: Tempo-Related Audio Features for Time-Scale Invariant Audio Identification. Proceedings of the 7th International Conference on Music Information Retrieval (ISMIR), Victoria, BC, Canada.
67. Tempo and beat analysis of acoustic musical signals10.1121/1.421129 ↗
68. Khan, M.K.S., Al-Khatib, W.G., and Moinuddin, M. (2004, January 8–13). Automatic Classification of Speech and Music Using Neural Networks. Proceedings of the 2nd ACM International Workshop on Multimedia Databases, Washington, DC, USA.10.1145/1032604.1032620 ↗
69. Machine-learning Based Classification of Speech and Music10.1007/s00530-006-0034-0 ↗
70. Itakura, F. (1975, January 8–11). Line spectrum representation of linear predictor coefficients of speech signals. Proceedings of the 89th Meeting of the Acoustical Society of America, Austin, TX, USA.
71. Sarkar, A., and Sreenivas, T.V. (2005, January 4–8). Dynamic programming based segmentation approach to LSF matrix reconstruction. Proceedings of the 9th European Conference on Speech Communication and Technology (EuroSpeech), Lisbon, Portugal.10.21437/Interspeech.2005-188 ↗
72. Schroeder, M., and Atal, B. (1985, January 26–29). Code-excited linear prediction (CELP): High-quality speech at very low bit rates. Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Tampa, FL, USA.
73. Tsau, E., Kim, S.H., and Kuo, C.C.J. (July, January 30). Environmental sound recognition with CELP-based features. Proceedings of the 10th International Symposium on Signals, Circuits and Systems, Iasi, Romania.
74. Srinivasan, S., Petkovic, D., and Ponceleon, D. (November, January 30). Towards Robust Features for Classifying Audio in the CueVideo System. Proceedings of the Seventh ACM International Conference on Multimedia (Part 1), Orlando, FL, USA.
75. Farahani, G., Ahadi, S.M., and Homayounpoor, M.M. (2006, January 14–19). Use of Spectral Peaks in Autocorrelation and Group Delay Domains for Robust Speech Recognition. Proceedings of the 2006 IEEE International Conference on Acoustics Speech and Signal Processing, (ICASSP), Toulouse, France.
76. Audio classification based on MPEG-7 spectral basis representations10.1109/TCSVT.2004.826766 ↗
77. Automatic Music Genre Classification Based on Modulation Spectral Analysis of Spectral and Cepstral Features10.1109/TMM.2009.2017635 ↗
78. Dixon, S., Bainbridge, D., and Typke, R. (2007, January 23–27). Stereo Panning Features for Classifying Recording Production Style. Proceedings of the 8th International Conference on Music Information Retrieval (ISMIR), Vienna, Austria.
79. Stereo Panning Information for Music Information Retrieval Tasks
80. Significance of group delay functions in spectrum estimation10.1109/78.157227 ↗
81. Determination of instants of significant excitation in speech using group delay function10.1109/89.466662 ↗
82. Determination of Instants of Significant Excitation in Speech Using Hilbert Envelope and Group Delay Function10.1109/LSP.2007.896454 ↗
83. Beat tracking of musical performances using low-level audio features10.1109/TSA.2004.841053 ↗
84. Hegde, R.M., Murthy, H.A., and Rao, G.V.R. (2004, January 17–21). Application of the modified group delay function to speaker identification and discrimination. Proceedings of the 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing, (ICASSP), Montreal, QC, Canada.
85. Group delay functions and its applications in speech technology10.1007/s12046-011-0045-1 ↗
86. Automatic Music Mood Classification Based on Timbre and Modulation Features10.1109/TAFFC.2015.2427836 ↗
87. Hess, W. (1983). Pitch Determination of Speech Signals: Algorithms and Devices; Springer Series in Information Sciences, Springer-Verlag.10.1007/978-3-642-81926-1 ↗
88. Content-Based Classification, Search, and Retrieval of Audio10.1109/93.556537 ↗
89. Precise pitch profile feature extraction from musical audio for key detection10.1109/TMM.2006.870727 ↗
90. Noise robust voice activity detection based on periodic to aperiodic component ratio10.1016/j.specom.2009.08.003 ↗
91. Kristjansson, T.T., Deligne, S., and Olsen, P.A. (2005, January 4–8). Voicing features for robust speech detection. Proceedings of the 9th European Conference on Speech Communication and Technology (EuroSpeech), Lisbon, Portugal.10.21437/Interspeech.2005-186 ↗
92. Khao, P.C. (2012). Noise Robust Voice Activity Detection, Nanyang Technological University. Technical Report.
93. Srinivasan, S.H., and Kankanhalli, M.S. (2004, January 17–21). Harmonicity and dynamics-based features for audio. Proceedings of the 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Montreal, QC, Canada.
94. Dugelay, J., and Rose, K. (2001, January 3–5). Musical instrument timbres classification with spectral features. Proceedings of the Fourth IEEE Workshop on Multimedia Signal Processing (MMSP), Cannes, France.
95. Musical Instrument Timbres Classification with Spectral Features
96. A flexible framework for key audio effects detection and auditory context inference10.1109/TSA.2005.857575 ↗
97. Accurate short-term analysis of the fundamental frequency and the harmonics-to-noise ratio of a sampled sound
98. Lee, J.W., Kim, S., and Kang, H.G. (2014, January 4–9). Detecting pathological speech using contour modeling of harmonic-to-noise ratio. Proceedings of the 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Florence, Italy.10.1109/ICASSP.2014.6854749 ↗
99. Circularity in Judgments of Relative Pitch10.1121/1.1919362 ↗
100. Audio thumbnailing of popular music using chroma-based representations10.1109/TMM.2004.840597 ↗