Comparison of Deep Learning Models and Various Text Pre-Processing Techniques for the Toxic Comments Classification
Source abstract
Abstract
The emergence of anti-social behaviour in online environments presents a serious issue in today’s society. Automatic detection and identification of such behaviour are becoming increasingly important. Modern machine learning and natural language processing methods can provide effective tools to detect different types of anti-social behaviour from the pieces of text. In this work, we present a comparison of various deep learning models used to identify the toxic comments in the Internet discussions. Our main goal was to explore the effect of the data preparation on the model performance. As we worked with the assumption that the use of traditional pre-processing methods may lead to the loss of characteristic traits, specific for toxic content, we compared several popular deep learning and transformer language models. We aimed to analyze the influence of different pre-processing techniques and text representations including standard TF-IDF, pre-trained word embeddings and also explored currently popular transformer models. Experiments were performed on the dataset from the Kaggle Toxic Comment Classification competition, and the best performing model was compared with the similar approaches using standard metrics used in data analysis.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Hate Speech and Cyberbullying Detection · Software Engineering Research · Advanced Malware Detection Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Cybernetics and Artificial Intelligence, Faculty of Electrical Engineering and Informatics, Technical University of Košice, 040 01 Kosice, Slovakia
Funding
Agentúra na Podporu Výskumu a Vývoja
References (66)
Show source-provided references
1. Cheng, J., Danescu-Niculescu-Mizil, C., and Leskovec, J. (2015, January 26–29). Antisocial behavior in online discussion communities. Proceedings of the 9th International Conference on Web and Social Media, University of, Oxford, Oxford, UK.
2. Elizabeth, B. (2013). Making People Behave: Anti-Social Behaviour, Politics and Policy, Willan Publishing. [2nd ed.].
3. Risch, J., and Krestel, R. (2020). Toxic Comment Detection in Online Discussions. Deep Learning-Based Approaches for Sentiment Analysis, Springer. Algorithms for Intelligent Systems.10.1007/978-981-15-1216-2_4 ↗
4. Dixon, L., Li, J., Sorensen, J., Thain, N., and Vasserman, L. (2018, January 2–3). Measuring and Mitigating Unintended Bias in Text Classification. Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, New Orleans, LA, USA.10.1145/3278721.3278729 ↗
5. Morzhov, S. (2020, January 20–24). Avoiding Unintended Bias in Toxicity Classification with Neural Networks. Proceedings of the Conference of Open Innovation Association, Yaroslavl, Russia.10.23919/FRUCT48808.2020.9087368 ↗
6. Mohammad, F. (2018). Is preprocessing of text really worth your time for online comment classification?. arXiv.
7. Role of Text Pre-processing in Twitter Sentiment Analysis10.1016/j.procs.2016.06.095 ↗
8. Shelar, A., and Huang, C.Y. (2018, January 13–15). Sentiment analysis of twitter data. Proceedings of the 2018 International Conference on Computational Science and Computational Intelligence, Las Vegas, NV, USA.10.1109/CSCI46756.2018.00252 ↗
9. Wang, S., and Manning, C.D. (2012, January 8–14). Baselines and bigrams: Simple, good sentiment and topic classification. Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, Jeju Island, Korea.
10. Techniques and applications for sentiment analysis10.1145/2436256.2436274 ↗
11. van Aken, B., Risch, J., Krestel, R., and Löser, A. (2018, January 31). Challenges for Toxic Comment Classification: An In-Depth Error Analysis. Proceedings of the 2nd Workshop on Abusive Language Online (ALW2), Brussels, Belgium.10.18653/v1/W18-5105 ↗
12. Waseem, Z., Davidson, T., Warmsley, D., and Weber, I. (August, January 30). Understanding Abuse: A Typology of Abusive Language Detection Subtasks. Proceedings of the First Workshop on Abusive Language Online, Vancouver, BC, Canada.
13. Sarnovský, M., Butka, P., Bednár, P., Babič, F., and Paralič, J. (2015, January 4–7). Analytical platform based on Jbowl library providing text-mining services in distributed environment. Proceedings of the Information and Communication Technology, Daejeon, Korea.10.1007/978-3-319-24315-3_32 ↗
14. Davidson, T., Warmsley, D., Macy, M., and Weber, I. (2017, January 15–18). Automated hate speech detection and the problem of offensive language. Proceedings of the 11th International Conference on Web and Social Media, Montréal, QC, Canada.10.1609/icwsm.v11i1.14955 ↗
15. Salminen, J., Almerekhi, H., Milenković, M., Jung, S.G., An, J., Kwak, H., and Jansen, B.J. (2018, January 25–28). Anatomy of online hate: Developing a taxonomy and machine learning models for identifying and classifying hate in online news media. Proceedings of the 12th International AAAI Conference on Web and Social Media, Stanford, CA, USA.10.1609/icwsm.v12i1.15028 ↗
16. Almerekhi, H., Jansen, B.J., Kwak, H., and Salminen, J. (2019, January 17–20). Detecting toxicity triggers in online discussions. Proceedings of the 30th ACM Conference on Hypertext and Social Media, Hof, Germany.10.1145/3342220.3344933 ↗
17. Fortuna, P., Soler, J., and Wanner, L. (2020, January 11–16). Toxic, Hateful, Offensive or Abusive? What Are We Really Classifying? An Empirical Analysis of Hate Speech Datasets. Proceedings of the 12th Language Resources and Evaluation Conference, Marseille, France.
18. Shtovba, S., Shtovba, O., and Petrychko, M. (2019, January 15–19). Detection of social network toxic comments with usage of syntactic dependencies in the sentences. Proceedings of the Second International Workshop on Computer Modeling and Intelligent Systems, Zaporizhzhia, Ukraine.10.32782/cmis/2353-25 ↗
19. Classification of online toxic comments using the logistic regression and neural networks models10.1063/1.5082126 ↗
20. Haralabopoulos, G., Anagnostopoulos, I., and McAuley, D. (2020). Ensemble deep learning for multilabel binary classification of user-generated content. Algorithms, 13.10.3390/a13040083 ↗
21. Fake Review Detection using Classification
22. Towards automatic filtering of fake reviews10.1016/j.neucom.2018.04.074 ↗
23. Ventirozos, F.K., Varlamis, I., and Tsatsaronis, G. (2018). Detecting aggressive behavior in discussion threads using text mining. Proceedings Computational Linguistics and Intelligent Text Processing, Springer.10.1007/978-3-319-77116-8_31 ↗
24. Chatzigeorgiou, A., Dondi, R., Herodotou, H., Kapoutsis, C., Manolopoulos, Y., Papadopoulos, G.A., and Sikora, F. (2020). Modelling of the Fake Posting Recognition in On-Line Media Using Machine Learning. SOFSEM 2020: Theory and Practice of Computer Science, Springer International Publishing.10.1007/978-3-030-38919-2 ↗
25. Machova, K., and Staronova, P. (2019, January 21–22). Selecting the Most Probable Author of Asocial Posting in Online Media. Proceedings of the 17th International Conference on Emerging eLearning Technologies and Applications (ICETA), Starý Smokovec, Slovakia.10.1109/ICETA48886.2019.9040096 ↗
26. Anindyati, L., Purwarianti, A., and Nursanti, A. (2019, January 20–22). Optimizing Deep Learning for Detection Cyberbullying Text in Indonesian Language. Proceedings of the 2019 International Conference on Advanced Informatics: Concepts, Theory, and Applications, Yogyakarta, Indonesia.10.1109/ICAICTA.2019.8904108 ↗
27. Al-Ajlan, M.A., and Ykhlef, M. (2018). Deep Learning Algorithm for Cyberbullying Detection. Int. J. Adv. Comput. Sci. Appl., 9.10.14569/IJACSA.2018.090927 ↗
28. Agrawal, S., and Awekar, A. (2018). Deep learning for detecting cyberbullying across multiple social media platforms. Advances in Information Retrieval, Springer. Lecture Notes in Computer Science.10.1007/978-3-319-76941-7_11 ↗
29. Ranasinghe, T., Zampieri, M., and Hettiarachchi, H. (2019). BRUMS at HASOC 2019: Deep Learning Models for Multilingual Hate Speech and Offensive Language Identificati on. FIRE (Working Notes), CEUR-WS.
30. Zimmerman, S., Fox, C., and Kruschwitz, U. (2019, January 7–12). Improving hate speech detection with deep learning ensembles. Proceedings of the LREC 2018—11th International Conference on Language Resources and Evaluation, Miyazaki, Japan.
31. Krešňáková, V.M., Sarnovský, M., and Butka, P. (2019, January 14–16). Deep learning methods for Fake News detection. Proceedings of the 2019 IEEE 19th International Symposium on Computational Intelligence and Informatics and 7th IEEE International Conference on Recent Achievements in Mechatronics, Automation, Computer Sciences and Robotics (CINTI-MACRo), Szeged, Hungary.10.1109/CINTI-MACRo49179.2019.9105317 ↗
32. Mestry, S., Singh, H., Chauhan, R., Bisht, V., and Tiwari, K. (2019, January 25–26). Automation in Social Networking Comments with the Help of Robust fastText and CNN. Proceedings of the 1st International Conference on Innovations in Information and Communication Technology, Chennai, India.10.1109/ICIICT1.2019.8741503 ↗
33. Anand, M., and Eswari, R. (2019, January 27–29). Classification of abusive comments in social media using deep learning. Proceedings of the 3rd International Conference on Computing Methodologies and Communication, Mettukadai, India.10.1109/ICCMC.2019.8819734 ↗
34. Srivastava, S., Khurana, P., and Tewari, V. (2018, January 25). Identifying Aggression and Toxicity in Comments using Capsule Network. Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018), Santa Fe, NM, USA.10.18653/v1/W19-3517 ↗
35. Georgakopoulos, S.V., Vrahatis, A.G., Tasoulis, S.K., and Plagianakos, V.P. (2018, January 9–12). Convolutional neural networks for toxic comment classification. Proceedings of the ACM International Conference Proceeding Series, Patras, Greece.10.1145/3200947.3208069 ↗
36. Ibrahim, M., Torki, M., and El-Makky, N. (2018, January 17–20). Imbalanced Toxic Comments Classification Using Data Augmentation and Deep Learning. Proceedings of the 17th IEEE International Conference on Machine Learning and Applications, Orlando, FL, USA.10.1109/ICMLA.2018.00141 ↗
37. Learning representations by back-propagating errors10.1038/323533a0 ↗
38. Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press.
39. Multilayer feedforward networks with a nonpolynomial activation function can approximate any function10.1016/S0893-6080(05)80131-5 ↗
40. Nair, V., and Hinton, G.E. (2010, January 21–24). Rectified linear units improve Restricted Boltzmann machines. Proceedings of the ICML 2010 27th International Conference on Machine Learning, Haifa, Israel.
41. Rumelhart, D.E., Hinton, G.E., and Williams, R.J. (1988). Neurocomputing: Foundations of Research, MIT Press. Chapter Learning Representations by Back-Propagating Errors.
42. Kingma, D., and Ba, J. (2015, January 7–9). Adam: A Method for Stochastic Optimization. Proceedings of the International Conference on Learning Representations, San Diego, CA, USA.
43. Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2012). Improving neural networks by preventing co-adaptation of feature detectors. arXiv.
44. Some methods of speeding up the convergence of iteration methods10.1016/0041-5553(64)90137-5 ↗
45. Dropout: A simple way to prevent neural networks from overfitting
46. Long short-term memory10.1162/neco.1997.9.8.1735 ↗
47. Graves, A. (2012). Supervised sequence labelling. Supervised Sequence Labelling with Recurrent Neural Networks, Springer.10.1007/978-3-642-24797-2 ↗
48. Transforming the LSTM Training Algorithm for Efficient FPGA-Based Adaptive Control of Nonlinear Dynamic Systems
49. Bidirectional recurrent neural networks10.1109/78.650093 ↗
50. Cho, K., Merrienboer, B., Gülçehre, Ç., Bougares, F., Schwenk, H., and Bengio, Y. (2014, January 25–29). Learning Phrase Representations using RNN. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.
51. Sun, C., Qiu, X., Xu, Y., and Huang, X. (2019, January 13). How to Fine-Tune BERT for Text Classification?. Proceedings of the Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Cham, Switzerland.10.1007/978-3-030-32381-3_16 ↗
52. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2–7). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the NAACL HLT 2019—2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MI, USA.
53. Attention is all you need
54. Sanh, V., Debut, L., Chaumond, J., and Wolf, T. (2020). DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv.
55. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv.
56. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., and Le, Q.V. (2020). XLNet: Generalized Autoregressive Pretraining for Language Understanding. arXiv.
57. Sammut, C., and Webb, G.I. (2010). TF–IDF. Encyclopedia of Machine Learning, Springer.10.1007/978-0-387-30164-8 ↗
58. Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv.
59. Pennington, J., Socher, R., and Manning, C.D. (2014, January 25–29). Glove: Global vectors for word representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1162 ↗
60. Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T. (2016). Enriching Word Vectors with Subword Information. arXiv.10.1162/tacl_a_00051 ↗
61. Asch, V.V. (2013). Macro-and Micro-Averaged Evaluation Measures, CLiPS.
62. Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., and Isard, M. (2016, January 2–4). Tensorflow: A system for large-scale machine learning. Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), Savannah, GA, USA.
63. Gulli, A., and Pal, S. (2017). Deep Learning with Keras, Packt Publishing Ltd.
64. Comparison of the predicted and observed secondary structure of T4 phage lysozyme10.1016/0005-2795(75)90109-9 ↗
65. Rastogi, C., Mofid, N., and Hsiao, F.I. (2020). Can We Achieve More with Less? Exploring Data Augmentation for Toxic Comment Classification. arXiv.
66. Saia, R., Corriga, A., Mulas, R., Reforgiato Recupero, D., and Carta, S. (2019, January 17–19). A Supervised Multi-Class Multi-Label Word Embeddings Approach for Toxic Comment Classification. Proceedings of the 11th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K 2019), Vienna, Austria.