BERT Models for Arabic Text Classification: A Systematic Review
Source abstract
Abstract
Bidirectional Encoder Representations from Transformers (BERT) has gained increasing attention from researchers and practitioners as it has proven to be an invaluable technique in natural languages processing. This is mainly due to its unique features, including its ability to predict words conditioned on both the left and the right context, and its ability to be pretrained using the plain text corpus that is enormously available on the web. As BERT gained more interest, more BERT models were introduced to support different languages, including Arabic. The current state of knowledge and practice in applying BERT models to Arabic text classification is limited. In an attempt to begin remedying this gap, this review synthesizes the different Arabic BERT models that have been applied to text classification. It investigates the differences between them and compares their performance. It also examines how effective they are compared to the original English BERT models. It concludes by offering insight into aspects that need further improvements and future work.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Sentiment Analysis and Opinion Mining · Text and Document Classification Technologies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Computing and Informatics, Saudi Electronic University, Jeddah 393453, Saudi Arabia
References (86)
Show source-provided references
1. Vijayan, V.K., Bindu, K., and Parameswaran, L. (2017, January 13–16). A comprehensive study of text classification algorithms. Proceedings of the 2017 International Conference on Advances in Computing, Communications and Informatics (ICACCI), Udupi, India.10.1109/ICACCI.2017.8125990 ↗
2. A survey on sentiment analysis challenges
3. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv.
4. A Primer in BERTology: What We Know About How BERT Works10.1162/tacl_a_00349 ↗
5. Zaib, M., Sheng, Q.Z., and Emma Zhang, W. (2016, January 1–5). A short survey of pretrained language models for conversational AI-a new age in NLP. Proceedings of the Australasian Computer Science Week Multiconference, Canberra, Australia.
6. Alshalan, R., and Al-Khalifa, H. (2020). A deep learning approach for automatic hate speech detection in the saudi twittersphere. Appl. Sci., 10.10.3390/app10238614 ↗
7. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V.J. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv.
8. Sanh, V., Debut, L., Chaumond, J., and Wolf, T.J. (2019). DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv.
9. Almuqren, L. (2021). Twitter Analysis to Predict the Satisfaction of Saudi Telecommunication Companies’ Customers. [Ph.D. Thesis, Durham University].
10. Investigating cross-lingual training for offensive language detection10.7717/peerj-cs.559 ↗
11. Antoun, W., Baly, F., and Hajj, H. (2020). Arabert: Transformer-based model for arabic language understanding. arXiv.
12. Abdul-Mageed, M., Elmadany, A., and Nagoudi, E.M.B. (2021, January 1–6). ARBERT & MARBERT: Deep bidirectional transformers for Arabic. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Online.10.18653/v1/2021.acl-long.551 ↗
13. A methodology for systematic mapping in environmental sciences10.1186/s13750-016-0059-6 ↗
14. PRISMA statement10.1097/EDE.0b013e3181fe7825 ↗
15. Gray literature: An important resource in systematic reviews10.1111/jebm.12266 ↗
16. Kitchenham, B., and Charters, S. (2007). Guidelines for Performing Systematic Literature Reviews in Software Engineering, EBSE.
17. Zhou, Y., Zhang, H., Huang, X., Yang, S., Babar, M.A., and Tang, H. (2015, January 27–29). Quality assessment of systematic reviews in software engineering: A tertiary study. Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering, Nanjing, China.10.1145/2745802.2745815 ↗
18. Challenges in approaching metasynthesis research10.1177/1049732306295879 ↗
19. Does ceasing exercise induce depressive symptoms? A systematic review of experimental trials including immunological and neurogenic markers10.1016/j.jad.2018.02.058 ↗
20. Alammary, A. (2019). Blended learning models for introductory programming courses: A systematic review. PLoS ONE, 14.10.1371/journal.pone.0221765 ↗
21. Bilal, S. (2021, January 26–28). A Linguistic System for Predicting Sentiment in Arabic Tweets. Proceedings of the 2021 3rd International Conference on Natural Language Processing (ICNLP), Beijing, China.10.1109/ICNLP52887.2021.00028 ↗
22. Surface and deep features ensemble for sentiment analysis of arabic tweets10.1109/ACCESS.2019.2924314 ↗
23. Pàmies Massip, M. (2021, December 19). Multilingual Identification of Offensive Content in Social Media. Available online: https://www.diva-portal.org/smash/get/diva2:1451543/FULLTEXT01.pdf.
24. Moudjari, L., Akli-Astouati, K., and Benamara, F. (2020, January 11–16). An Algerian corpus and an annotation platform for opinion and emotion analysis. Proceedings of the 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France.
25. Zero-Resource Multi-Dialectal Arabic Natural Language Understanding
26. Alshehri, A., Nagoudi, E.M.B., and Abdul-Mageed, M. (2020). Understanding and Detecting Dangerous Speech in Social Media, European Language Resource Association.
27. Abdul-Mageed, M., Zhang, C., Elmadany, A., and Ungar, L. (2020, January 8–12). Toward micro-dialect identification in diaglossic and code-switched environments. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, Online.10.18653/v1/2020.emnlp-main.472 ↗
28. AraCOVID19-MFH: Arabic COVID-19 Multi-label Fake News & Hate Speech Detection Dataset10.1016/j.procs.2021.05.086 ↗
29. An Experimental Study On Sentiment Classification Of Algerian Dialect Texts10.1016/j.procs.2020.09.111 ↗
30. Alsafari, S., Sadaoui, S., and Mouhoub, M. (2020, January 9–11). Deep learning ensembles for hate speech detection. Proceedings of the 2020 IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI), Baltimore, MD, USA.10.1109/ICTAI50040.2020.00087 ↗
31. Abdelali, A., Mubarak, H., Samih, Y., Hassan, S., and Darwish, K. (2021, January 19). QADI: Arabic dialect identification in the wild. Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine.
32. Hate and offensive speech detection on Arabic social media10.1016/j.osnem.2020.100096 ↗
33. Mubarak, H., Hassan, S., and Abdelali, A. (2021, January 19). Adult content detection on arabic twitter: Analysis and experiments. Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine.
34. Farha, I.A., and Magdy, W. (2021, January 19). Benchmarking transformer-based language models for Arabic sentiment and sarcasm detection. Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine.
35. Uyangodage, L., Ranasinghe, T., and Hettiarachchi, H. (2021, December 04). Transformers to fight the COVID-19 infodemic. Available online: https://arxiv.org/pdf/2104.12201.pdf.10.18653/v1/2021.nlp4if-1.20 ↗
36. Obied, Z., Solyman, A., Ullah, A., Fat’hAlalim, A., and Alsayed, A. (2021, January 26–28). BERT Multilingual and Capsule Network for Arabic Sentiment Analysis. Proceedings of the 2020 International Conference on Computer, Control, Electrical, and Electronics Engineering (ICCCEEE), Khartoum, Sudan.10.1109/ICCCEEE49695.2021.9429568 ↗
37. Mubarak, H., Rashed, A., Darwish, K., Samih, Y., and Abdelali, A. (2021, November 17). Arabic Offensive Language on Twitter: Analysis and Experiments. Available online: https://arxiv.org/pdf/2004.02192.pdf.
38. Safaya, A., Abdullatif, M., and Yuret, D. (2020, January 12–13). Kuisail at semeval-2020 task 12: Bert-cnn for offensive speech identification in social media. Proceedings of the Fourteenth Workshop on Semantic Evaluation, Barcelona, Spain.10.18653/v1/2020.semeval-1.271 ↗
39. El-Alami, F.-z., El Alaoui, S.O., and Nahnahi, N.E. (2021). Contextual semantic embeddings based on fine-tuned AraBERT model for Arabic text multi-class categorization. J. King Saud Univ. Comput. Inf. Sci.10.1016/j.jksuci.2021.02.005 ↗
40. Abdelali, A., Hassan, S., Mubarak, H., Darwish, K., and Samih, Y. (2021). Pre-training bert on arabic tweets: Practical considerations. arXiv.
41. Mansour, M., Tohamy, M., Ezzat, Z., and Torki, M. (2020, January 12). Arabic dialect identification using BERT fine-tuning. Proceedings of the Fifth Arabic Natural Language Processing Workshop, Barcelona, Spain.
42. Balaji, N.N.A., and Bharathi, B. (2020, January 12). Semi-supervised fine-grained approach for Arabic dialect detection task. Proceedings of the Fifth Arabic Natural Language Processing Workshop, Barcelona, Spain.
43. Abuzayed, A., and Al-Khalifa, H. (2021, January 19). Sarcasm and sentiment detection in Arabic tweets using BERT-based models and data augmentation. Proceedings of the sixth Arabic natural language processing workshop, Kyiv, Ukraine.
44. Saeed, H.H., Calders, T., and Kamiran, F. (2020, January 12). OSACT4 shared tasks: Ensembled stacked classification for offensive and hate speech in Arabic tweets. Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, Marseille, France.
45. Zhang, C., and Abdul-Mageed, M. (2019, January 1). No army, no navy: Bert semi-supervised learning of arabic dialects. Proceedings of the Fourth Arabic Natural Language Processing Workshop, Florence, Italy.10.18653/v1/W19-4637 ↗
46. Naski, M., Messaoudi, A., Haddad, H., BenHajhmida, M., Fourati, C., and Mabrouk, A.B.E. (2021, January 19). iCompass at Shared Task on Sarcasm and Sentiment Detection in Arabic. Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine.
47. Hassan, S., Samih, Y., Mubarak, H., and Abdelali, A. (2020, January 12–13). ALT at SemEval-2020 task 12: Arabic and English offensive language identification in social media. Proceedings of the Fourteenth Workshop on Semantic Evaluation, Barcelona, Spain.10.18653/v1/2020.semeval-1.249 ↗
48. Faraj, D., and Abdullah, M. (2021, January 19). Sarcasmdet at sarcasm detection task 2021 in arabic using arabert pretrained model. Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine.
49. Israeli, A., Nahum, Y., Fine, S., and Bar, K. (2021, January 19). The IDC System for Sentiment Classification and Sarcasm Detection in Arabic. Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine.
50. Aldjanabi, W., Dahou, A., Al-qaness, M.A., Abd Elaziz, M., Helmi, A.M., and Damaševičius, R. (2021). Arabic Offensive and Hate Speech Detection Using a Cross-Corpora Multi-Task Learning Model. Informatics, 8.10.3390/informatics8040069 ↗
51. Elgabry, H., Attia, S., Abdel-Rahman, A., Abdel-Ate, A., and Girgis, S. (2021, January 19). A contextual word embedding for Arabic sarcasm detection with random forests. Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine.
52. Alam, F., Shaar, S., Dalvi, F., Sajjad, H., Nikolov, A., Mubarak, H., Martino, G.D.S., Abdelali, A., Durrani, N., and Darwish, K. (2021, January 7–11). Fighting the COVID-19 infodemic: Modeling the perspective of journalists, fact-checkers, social media platforms, policy makers, and the society. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing., Punta Cana, Dominican Republic.10.18653/v1/2021.findings-emnlp.56 ↗
53. Al-Yahya, M., Al-Khalifa, H., Al-Baity, H., AlSaeed, D., and Essam, A. (2021). Arabic Fake News Detection: Comparative Study of Neural Networks and Transformer-Based Approaches. Complexity, 2021.10.1155/2021/5516945 ↗
54. Mulki, H., and Ghanem, B.J. (2021). Let-mi: An Arabic Levantine Twitter dataset for misogynistic language. arXiv.
55. Mubarak, H., Abdelali, A., Hassan, S., and Darwish, K. (2020, January 6–9). Spam detection on arabic twitter. Proceedings of the International Conference on Social Informatics, Pisa, Italy.10.1007/978-3-030-60975-7_18 ↗
56. Mubarak, H., and Hassan, S. (2021, January 19–20). Arcorona: Analyzing arabic tweets in the early days of coronavirus (COVID-19) pandemic. Proceedings of the 12th International Workshop on Health Text Mining and Information Analysis, Virtual Conference, Online.
57. El-Alami, F.-z., El Alaoui, S.O., and Nahnahi, N.E. (2021). A multilingual offensive language detection method based on transfer learning from transformer fine-tuning model. J. King Saud Univ. Comput. Inf. Sci.10.1016/j.jksuci.2021.07.013 ↗
58. Al-Twairesh, N. (2021). The Evolution of Language Models Applied to Emotion Analysis of Arabic Tweets. Information, 12.10.3390/info12020084 ↗
59. Husain, F., and Uzuner, O. (2021, January 19). Leveraging offensive language for sarcasm and sentiment detection in Arabic. Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine.
60. Wadhawan, A. (2021, January 19). Arabert and farasa segmentation based approach for sarcasm and sentiment detection in arabic tweets. Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine.
61. Bashmal, L., and AlZeer, D. (2021, January 19). ArSarcasm Shared Task: An Ensemble BERT Model for SarcasmDetection in Arabic Tweets. Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine.
62. Gaanoun, K., and Benelallam, I. (2021, January 19). Sarcasm and Sentiment Detection in Arabic language A Hybrid Approach Combining Embeddings and Rule-based Features. Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine.
63. Alharbi, A.I., and Lee, M. (2021, January 19). Multi-task learning using a combination of contextualised and static word embeddings for arabic sarcasm detection and sentiment analysis. Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine.
64. Abdel-Salam, R. (2021, January 19). Wanlp 2021 shared-task: Towards irony and sentiment detection in arabic tweets using multi-headed-lstm-cnn-gru and marbert. Proceedings of the Sixth Arabic Natural Language Processing Workshop, Kyiv, Ukraine.
65. Wu, S., and Dredze, M. (2020, January 9). Are all languages created equal in multilingual BERT?. Proceedings of the 5th Workshop on Representation Learning for NLP, Online.10.18653/v1/2020.repl4nlp-1.16 ↗
66. Abdaoui, A., Pradel, C., and Sigel, G. (2021, January 10). Load What You Need: Smaller Versions of Multilingual BERT. Proceedings of the SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, Online.10.18653/v1/2020.sustainlp-1.16 ↗
67. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. (2020, January 5–10). Unsupervised cross-lingual representation learning at scale. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.747 ↗
68. Lan, W., Chen, Y., Xu, W., and Ritter, A. (2020, January 16–20). An Empirical Study of Pre-trained Transformers for Arabic Information Extraction. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online.10.18653/v1/2020.emnlp-main.382 ↗
69. Safaya, A. (2022). Arabic-ALBERT. arXiv.
70. Deep learning--based text classification: A comprehensive review10.1145/3439726 ↗
71. Li, Q., Peng, H., Li, J., Xia, C., Yang, R., Sun, L., Yu, P.S., and He, L. (2021). A Survey on Text Classification: From Traditional to Deep Learning. ACM Trans. Intell. Syst. Technol., 37.
72. Virtanen, A., Kanerva, J., Ilo, R., Luoma, J., Luotolahti, J., Salakoski, T., Ginter, F., and Pyysalo, S. (2019). Multilingual is not enough: BERT for Finnish. arXiv.
73. Martin, L., Muller, B., Suárez, P.J.O., Dupont, Y., Romary, L., De La Clergerie, É.V., Seddah, D., and Sagot, B. (2020, January 5–10). CamemBERT: A Tasty French Language Model. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.645 ↗
74. Multilingual offensive language identification for low-resource languages
75. Jain, M., Mathew, M., and Jawahar, C. (2017, January 3–5). Unconstrained scene text and video text recognition for arabic script. Proceedings of the 2017 1st International Workshop on Arabic Script Analysis and Recognition (ASAR), Nancy, France.10.1109/ASAR.2017.8067754 ↗
76. Himdi, H., Weir, G., Assiri, F., and Al-Barhamtoshy, H. (2022). Arabic fake news detection based on textual analysis. Arab. J. Sci. Eng., 1–17.10.1007/s13369-021-06449-y ↗
77. Statista (2022, January 12). Leading Countries Based on Number of Twitter Users as of January 2022. Available online: https://www.statista.com/statistics/242606/number-of-active-twitter-users-in-selected-countries/.
78. Moores, B., and Mago, V. (2022). A Survey on Automated Sarcasm Detection on Twitter. arXiv.
79. Social emotion classification of short text via topic-level maximum entropy model10.1016/j.im.2016.04.005 ↗
80. BioBERT: A pre-trained biomedical language representation model for biomedical text mining10.1093/bioinformatics/btz682 ↗
81. Green AI10.1145/3381831 ↗
82. Sun, C., Shrivastava, A., Singh, S., and Gupta, A. (2017, January 22–29). Revisiting unreasonable effectiveness of data in deep learning era. Proceedings of the IEEE international conference on computer vision, Venice, Italy.10.1109/ICCV.2017.97 ↗
83. Al-Maimani, M.R., Al Naamany, A., and Bakar, A.Z.A. (2011, January 19–22). Arabic information retrieval: Techniques, tools and challenges. Proceedings of the 2011 IEEE GCC Conference and Exhibition (GCC), Dubai, United Arab Emirates.10.1109/IEEEGCC.2011.5752576 ↗
84. Named Entity Recognition in Chinese Medical Literature Using Pretraining Models
85. Khemakhem, I.T., Jamoussi, S., and Hamadou, A.B. (2013, January 8). Integrating morpho-syntactic features in English-Arabic statistical machine translation. Proceedings of the Second Workshop on Hybrid Approaches to Translation, Sofia, Bulgaria.
86. An analysis of Arabic-English translation: Problems and prospects10.7575/aiac.alls.v.10n.1p.58 ↗