Emotion Recognition Method for Call/Contact Centre Systems
Source abstract
Abstract
Nowadays, one of the important aspects of research on call/contact centre (CC) systems is how to automate their operations. Process automation is influenced by the continuous development in the implementation of virtual assistants. The effectiveness of virtual assistants depends on numerous factors. One of the most important is correctly recognizing the intent of clients conversing with the machine. Recognizing intentions is not an easy process, as often the client’s actual intentions can only be correctly identified after considering the client’s emotional state. When it comes to human–machine communication, the ability of a virtual assistant to recognize the client’s emotional state would greatly improve its effectiveness. This paper proposes a new method for recognizing interlocutors’ emotions dedicated directly to contact centre systems. The developed method provides opportunities to determine emotional states in text and voice channels. It provides opportunities to explore both the client’s and the agent’s emotional states. Information about agents’ emotions can be used to build their behavioural profiles, which is also applicable in contact centres. In addition, the paper explored the possibility of emotion assessment based on automatic transcriptions of recordings, which also positively affected emotion recognition performance in the voice channel. The research used actual conversations that took place during the operation of a large, commercial contact centre. The proposed solution makes it possible to recognize the emotions of customers contacting the hotline and agents handling these calls. Using this information in practical applications can increase the efficiency of agents’ work, efficiency of bots used in CC and increase customer satisfaction.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Technology and Human Factors in Education and Health · Advanced Scientific Research Methods · Information Systems and Technology Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Electrical Engineering, Automatics Control and Computer Science, Kielce University of Technology, 25-314 Kielce, Poland
Faculty of Mechatronics and Mechanical Engineering, Kielce University of Technology, 25-314 Kielce, Poland
Faculty of Management and Computer Modelling, Kielce University of Technology, 25-314 Kielce, Poland
Funding
European Union’s Smart Growth Operational Programme 2014–2022
References (76)
Show source-provided references
1. Jyoti, R., and Wardley, M. (2022, April 22). Unlocking the Transformative Power of AI for Contact Centers. IDC InfoBrief, October 2020. Available online: https://inthecloud.withgoogle.com/idc-infobrief/CCAI-IDC-Infobrief.html.
2. DMG Consulting LLC (2022). The State of Artificial Intelligence in the Contact Center, DMG Consulting LLC. Report.
3. Kask, S., Fitterer, R., and Anshelm, L. (2019). Augmenting Digital Customer Touchpoints: Best Practices for Transforming Customer Experience Through Conversational AI, University St. Gallen. Marketing Review.
4. Influence of the Contact Center Systems Development on Key Performance Indicators10.1109/ACCESS.2021.3066801 ↗
5. Google Cloud (2022, May 10). Natural Language API Basics. Available online: https://cloud.google.com/naturallanguage/docs/basics#sentiment_analysis.
6. Google Cloud (2022, August 01). Language Reference. Available online: https://cloud.google.com/dialogflow/es/docs/reference/language.
7. Performance analysis of an inbound call center with skills-based routing10.1007/s00291-004-0161-y ↗
8. Deep learning approaches for speech emotion recognition: State of the art and research challenges10.1007/s11042-020-09874-7 ↗
9. Roy, T., Marwala, T., and Chakraverty, S. (2020). A Survey of Classification Techniques in Speech Emotion Recognition. Mathematical Methods in Interdisciplinary Sciences, John Wiley & Sons Inc.10.1002/9781119585640.ch3 ↗
10. Databases, features and classifiers for speech emotion recognition: A review10.1007/s10772-018-9491-z ↗
11. Rubin, V.L., Stanton, J.M., and Liddy, E.D. (2022, August 10). Discerning Emotions in Texts, The AAAI Symposium on Exploring Attitude and Affect in Text AAAI-EAAI, Stanford, CA, 2004. Available online: https://www.aaai.org/Papers/Symposia/Spring/2004/SS-04-07/SS04-07-023.pdf.
12. Sathe, J.B., and Mali, M.P. (2017, January 5–6). A hybrid Sentiment Classification method using Neural Network and Fuzzy Logic. Proceedings of the 11th International Conference on Intelligent Systems and Control (ISCO), Coimbatore, India.10.1109/ISCO.2017.7855960 ↗
13. Sentiment analysis and the complex natural language10.1186/s40294-016-0016-9 ↗
14. Dragoni, M., Tettamanzi, A.G.B., and Pereira, C.D.C. (2014). A Fuzzy System for Concept-Level Sentiment Analysis. Semantic Web Evaluation Challenge, Springer.10.1007/978-3-319-12024-9_2 ↗
15. A method for improving bot effectiveness by recognising implicit customer intent in contact centre conversations10.1016/j.specom.2022.07.003 ↗
16. Ekman, P., and Friesen, W.V. (2003). Unmasking the Face: A Guide to Recognizing Emotions from Facial Clues, Malor Books.
17. Sentiment analysis and opinion mining
18. Agarwal, A., Xie, B., Vovsha, I., Rambow, O., and Passonneau, R. (2011, January 23). Sentiment analysis of Twitter data. Proceedings of the Workshop on Languages in Social Media, Portland, Orego.
19. Improving the Performance of Sentiment Analysis of Tweets Containing Fuzzy Sentiment Using the Feature Ensemble Model10.1109/ACCESS.2019.2963702 ↗
20. Deep Learning-Based Methods for Sentiment Analysis on Nepali COVID-19-Related Tweets10.1155/2021/2158184 ↗
21. Sentiment analysis in Facebook and its application to e-learning10.1016/j.chb.2013.05.024 ↗
22. Deep Convolution Neural Networks for Twitter Sentiment Analysis10.1109/ACCESS.2017.2776930 ↗
23. Enhancing Attention-Based LSTM With Position Context for Aspect-Level Sentiment Classification10.1109/ACCESS.2019.2893806 ↗
24. Wang, J., Yu, L.-C., Lai, K.R., and Zhang, X. (2016, January 7–12). Dimensional Sentiment Analysis Using a Regional CNN-LSTM Model. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Berlin, Germany.10.18653/v1/P16-2037 ↗
25. Twitter sentiment analysis using hybrid Spider Monkey optimization method10.1007/s12065-019-00334-2 ↗
26. A Pattern-Based Approach for Multi-Class Sentiment Analysis in Twitter10.1109/ACCESS.2017.2740982 ↗
27. Word clustering based on POS feature for efficient twitter sentiment analysis10.1186/s13673-018-0140-y ↗
28. Demszky, D., Movshovitz-Attias, D., Ko, J., Cowen, A., Nemade, G., and Ravi, S. (2020, January 6–8). GoEmotions: A Dataset of Fine-Grained Emotions. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.372 ↗
29. A BERT based dual-channel explainable text emotion recognition system10.1016/j.neunet.2022.03.017 ↗
30. Twitter Sentiment Analysis Based on Ordinal Regression10.1109/ACCESS.2019.2952127 ↗
31. Analysis of Emotionally Salient Aspects of Fundamental Frequency for Emotion Detection10.1109/TASL.2008.2009578 ↗
32. A comparative analysis of classifiers in emotion recognition through acoustic features10.1007/s10772-014-9239-3 ↗
33. Emotion recognition from speech: A review10.1007/s10772-011-9125-1 ↗
34. A survey of speech emotion recognition in natural environment10.1016/j.dsp.2020.102951 ↗
35. Noise at Workplaces in the Call Center10.2478/v10168-010-0024-2 ↗
36. Multiaspect Text Categorization Problem Solving: A Nearest Neighbours Classifier Based Approaches and Beyond
37. Document-level sentiment classification: An empirical comparison between SVM and ANN10.1016/j.eswa.2012.07.059 ↗
38. A quantum-inspired sentiment representation model for twitter sentiment analysis10.1007/s10489-019-01441-4 ↗
39. Pratama, B.Y., and Sarno, R. (2015, January 1). Personality classification based on Twitter text using Naive Bayes, KNN and SVM. Proceedings of the International Conference on Data and Software Engineering (ICoDSE), Yogyakarta, Indonesia.10.1109/ICODSE.2015.7436992 ↗
40. Saha, T., Patra, A., Saha, S., and Bhattacharyya, P. (2020, January 6–8). Towards Emotion-aided Multi-modal Dialogue Act Classification. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.402 ↗
41. Attention-based word embeddings using Artificial Bee Colony algorithm for aspect-level sentiment classification10.1016/j.ins.2020.09.038 ↗
42. Qayyum, A.B.A., Arefeen, A., and Shahnaz, C. (2019, January 28–30). Convolutional Neural Network (CNN) Based Speech-Emotion Recognition. Proceedings of the IEEE International Conference on Signal Processing, Information, Communication & Systems (SPICSCON), Dhaka, Bangladesh.
43. Zhou, J., Wang, G., Yang, Y., and Chen, P. (2006, January 17–19). Speech Emotion Recognition Based on Rough Set and SVM. Proceedings of the 5th IEEE International Conference on Cognitive Informatics, Beijing, China.10.1109/COGINF.2006.365676 ↗
44. Feraru, M., and Zbancioc, M. (2013, January 27–29). Speech emotion recognition for SROL database using weighted KNN algorithm. Proceedings of the International Conference on Electronics, Computers and Artificial Intelligence, Pitesti, Arkansas.10.1109/ECAI.2013.6636198 ↗
45. A Hierarchical Framework for Speech Emotion Recognition
46. Cho, J., Pappagari, R., Kulkarni, P., Villalba, J., Carmiel, Y., and Dehak, N. (2019). Deep Neural Networks for Emotion Recognition Combining Audio and Transcripts. arXiv.10.21437/Interspeech.2018-2466 ↗
47. Tripathi, S., Kumar, A., Ramesh, A., Singh, C., and Yenigalla, P. (2019). Deep learning based emotion recognition system using speech features and transcriptions. arXiv.
48. Kim, E., and Shin, J.W. (2019, January 12–17). DNN-based Emotion Recognition Based on Bottleneck Acoustic Features and Lexical Features. Proceedings of the ICASSP 2019—2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK.10.1109/ICASSP.2019.8683077 ↗
49. Li, J.-L., and Lee, C.-C. (2019, January 15–19). Attentive to Individual: A Multimodal Emotion Recognition Network with Personalized Attention Profile. Proceedings of the Interspeech 2019, Graz, Austria.10.21437/Interspeech.2019-2044 ↗
50. Santoso, J., Yamada, T., Makino, S., Ishizuka, K., and Hiramura, T. (September, January 30). Speech Emotion Recognition Based on Attention Weight Correction Using Word-Level Confidence Measure. Proceedings of the Interspeech 2021, Brno, Czechia.10.21437/Interspeech.2021-411 ↗
51. Survey on bimodal speech emotion recognition from acoustic and linguistic information fusion10.1016/j.specom.2022.03.002 ↗
52. NAWL (2022, June 12). Interactive Analysis of the NAWL Database, Available online: https://exp.lobi.nencki.gov.pl/nawl-analysis.
53. Call Transcription Methodology for Contact Center Systems10.1109/ACCESS.2021.3102502 ↗
54. Płaza, M., Trusz, S., Kęczkowska, J., Boksa, E., Sadowski, S., and Koruba, Z. (2022). Machine Learning Algorithms for Detection and Classifications of Emotions in Contact Center Applications. Sensors, 22.10.3390/s22145311 ↗
55. (2022, August 16). Scikit-Learn User Manual. Available online: https://scikit-learn.org/stable/modules/model_evaluation.html#.
56. Behera, B., Kumaravelan, G., and Kumar, P. (2019, January 18–20). Performance Evaluation of Deep Learning Algorithms in Biomedical Document Classification. Proceedings of the 11th International Conference on Advanced Computing (ICoAC), Chennai, India.10.1109/ICoAC48765.2019.246843 ↗
57. An insight into classification with imbalanced data: Empirical results and current trends on using data intrinsic characteristics10.1016/j.ins.2013.07.007 ↗
58. Classification of imbalanced data: A review10.1142/S0218001409007326 ↗
59. Chawla, N.V. (2009). Data mining for imbalanced datasets: An overview. Data Mining and Knowledge Discovery Handbook, Springer.10.1007/978-0-387-09823-4_45 ↗
60. Prusa, J., Khoshgoftaar, T.M., Dittman, D.J., and Napolitano, A. (2015, January 13–15). Using Random Undersampling to Alleviate Class Imbalance on Tweet Sentiment Data. Proceedings of the IEEE International Conference on Information Reuse and Integration, San Francisco, CA, USA.10.1109/IRI.2015.39 ↗
61. Imbalanced-learn: A Python toolbox to tackle the curse of imbalanced satasets in machine learning
62. The Imbalanced-Learn Developers (2022, June 01). Imbalanced-Learn Documentation. Available online: https://imbalanced-learn.org/.
63. An information-theoretic perspective of tf–idf measures10.1016/S0306-4573(02)00021-3 ↗
64. Scikit-learn: Machine Learning in Python
65. Scikitlearn (2022, May 11). Tf–idf Term Weighting. Available online: https://scikit-learn.org/stable/modules/feature_extraction.html#tfidf-term-weighting.
66. Gebre, B.G., Zampieri, M., Wittenburg, P., and Heskes, T. (2013, January 13). Improving Native Language Identification with TF-IDF Weighting. Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications, Association for Com-putational Linguistics, Atlanta, GA, USA.
67. Time-Frequency Analysis of EEG Signals for Human Emotion Detection10.1007/978-3-540-69139-6_68 ↗
68. Kong, J. (2008, January 27–30). A Study on Jitter, Shimmer and F0 of Mandarin Infant Voice by Developing an Applied Method of Voice Signal Processing. Proceedings of the Congress on Image and Signal Processing, Sanya, China.10.1109/CISP.2008.527 ↗
69. Korkmaz, O.E., and Atasoy, A. (2015, January 26–28). Emotion recognition from speech signal using mel-frequency cepstral coefficients. Proceedings of the 9th International Conference on Electrical and Electronics Engineering (ELECO), Bursa, Turkey.10.1109/ELECO.2015.7394435 ↗
70. Improved speech emotion recognition with Mel frequency magnitude coefficient10.1016/j.apacoust.2021.108046 ↗
71. Chamoli, A., Semwal, A., and Saikia, N. (2017, January 19–20). Detection of emotion in analysis of speech using linear predictive coding techniques (L.P.C). Proceedings of the International Conference on Inventive Systems and Control (ICISC), Coimbatore, India.10.1109/ICISC.2017.8068642 ↗
72. Basu, S., and Chakraborty, J. (2017, January 19–20). Aftabuddin Emotion recognition from speech using convolutional neural network with recurrent neural network architecture. Proceedings of the 2017 2nd International Conference on Communication and Electronics Systems (ICCES), Coimbatore, India.10.1109/CESYS.2017.8321292 ↗
73. Speech Emotion Recognition Using Fourier Parameters10.1109/TAFFC.2015.2392101 ↗
74. Aouani, H., and Ben Ayed, Y. (2018, January 21–24). Emotion recognition in speech using MFCC with SVM, DSVM and auto-encoder. Proceedings of the 2018 4th International Conference on Advanced Technologies for Signal and Image Processing (ATSIP), Sousse, Tunisia.10.1109/ATSIP.2018.8364518 ↗
75. Saste, S.T., and Jagdale, S.M. (2017, January 20–22). Emotion recognition from speech using MFCC and DWT for security system. Proceedings of the 2017 International Conference of Electronics, Communication and Aerospace Technology (ICECA), Coimbatore, India.10.1109/ICECA.2017.8203631 ↗
76. Nencki Affective Word List (NAWL): The cultural adaptation of the Berlin Affective Word List–Reloaded (BAWL-R) for Polish10.3758/s13428-014-0552-1 ↗