Towards AI-Generated Essay Classification Using Numerical Text Representation
Source abstract
Abstract
The detection of essays written by AI compared to those authored by students is increasingly becoming a significant issue in educational settings. This research examines various numerical text representation techniques to improve the classification of these essays. Utilizing a diverse dataset, we undertook several preprocessing steps, including data cleaning, tokenization, and lemmatization. Our system analyzes different text representation methods such as Bag of Words, TF-IDF, and fastText embeddings in conjunction with multiple classifiers. Our experiments showed that TF-IDF weights paired with logistic regression reached the highest accuracy of 99.82%. Methods like Bag of Words, TF-IDF, and fastText embeddings achieved accuracies exceeding 96.50% across all tested classifiers. Sentence embeddings, including MiniLM and distilBERT, yielded accuracies from 93.78% to 96.63%, indicating room for further refinement. Conversely, pre-trained fastText embeddings showed reduced performance, with a lowest accuracy of 89.88% in logistic regression. Remarkably, the XGBoost classifier delivered the highest minimum accuracy of 96.24%. Specificity and precision were above 99% for most methods, showcasing high capability in differentiating between student-created and AI-generated texts. This study underscores the vital role of choosing dataset-specific text representations to boost classification accuracy.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Natural Language Processing Techniques · Topic Modeling · Advanced Text Analysis Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Machine Learning, University of Economics in Katowice, 1 Maja 50, 40-287 Katowice, Poland
Łukasiewicz Research Network–Institute of Innovative Technologies EMAG, Leopolda 31, 40-189 Katowice, Poland
References (36)
Show source-provided references
1. Tang, R., Chuang, Y.-N., and Hu, X. (2023). The Science of Detecting LLM-Generated Texts. arXiv.
2. Tauginienė, L., Gaižauskaitė, I., Glendinning, I., Kravjar, J., Ojstersek, M., Robeiro, L., Odineca, T., Marino, F., Cosentino, M., and Sivasubramaniam, S. (2024, March 06). Glossary for Academic Integrity. ENAI Report (Revised Version), October 2018. Available online: https://www.academicintegrity.eu/wp/wp-content/uploads/2023/02/EN-Glossary_revised_final_24.02.23.pdf.
3. ENAI Recommendations on the ethical use of Artificial Intelligence in Education10.1007/s40979-023-00133-4 ↗
4. Feature-based detection of automated language models: Tackling GPT-2, GPT-3 and Grover10.7717/peerj-cs.443 ↗
5. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv.
6. Ippolito, D., Duckworth, D., Callison-Burch, C., and Eck, D. (2020, January 5–10). Automatic Detection of Generated Text is Easiest when Humans are Fooled. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.164 ↗
7. Sarzaeim, P., Doshi, A.M., and Mahmoud, Q.H. (2023, January 17–19). A Framework for Detecting AI-Generated Text in Research Publications. Proceedings of the International Conference on Advanced Technologies, Istanbul, Turkey.10.58190/icat.2023.28 ↗
8. Somesha, M., and Alwyn, R.P. (2022). Classification of Phishing Email Using Word Embedding and Machine Learning Techniques. J. Cyber Secur. Mobil., 279–320.
9. Kwon, H., and Lee, J. (2024). Detecting Textual Backdoor Attacks via Class Difference for Text Classification System. IEICE Trans. Inf. Syst., 2023EDP7160.10.1587/transinf.2023EDP7160 ↗
10. Detecting Textual Adversarial Examples through Text Modification on Text Classification Systems10.1007/s10489-022-03313-w ↗
11. Inżynieria języka dla systemu Thetos
12. Vetulani, Z. (2004). Komunikacja CzłOwieka z Maszyną. Komputerowe Modelowanie Kompetencji jęZykowej, Akademicka Oficyna Wydawnicza EXIT.
13. Advances in natural language processing10.1126/science.aaa8685 ↗
14. Płóciennik, I., and Podlawska, D. (2011). Słownik Wiedzy o Języku, Wydawnictwo Szkolne PWN.
15. Chollet, F. (2018). Deep Learning with Python, Manning Publications Co.
16. A Survey on Text Pre-Processing & Feature Extraction Techniques in Natural Language Processing
17. Silva, S., Pereira, R., and Ribeiro, R. (2018, January 13–16). Machine learning in incident categorization automation. Proceedings of the IEEE 13th Iberian Conference on Information Systems and Technologies (CISTI), Caceres, Spain.10.23919/CISTI.2018.8399244 ↗
18. Wang, D. (2020). Semantic Representation and Inference for NLP. [Ph.D. Thesis, University of Copenhagen].
19. Mikolov, T., Sutskever, I., Chen, K., Corrado, G., and Dean, J. (2013, January 5–10). Distributed representations of words and phrases and their compositionality. Proceedings of the 26th International Conference on Neural Information Processing Systems, Lake Tahoe, NV, USA.
20. Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., and Mikolov, T. (2016). FastText.zip: Compressing text classification models. arXiv.
21. D’Sa, A.G., Illina, I., and Fohr, D. (2020, January 6–8). BERT and fastText Embeddings for Automatic Detection of Toxic Speech. Proceedings of the International Multi-Conference on: “Organization of Knowledge and Advanced Technologies” (OCTA), Tunis, Tunisia.10.1109/OCTA49274.2020.9151853 ↗
22. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 10–15). BERT: Pretraining of Deep Bidirectional Transformers for Language Understanding. Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, DC, USA.
23. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., and Polosukhin, I. (2017). Attention is all you need. arXiv.
24. A Survey on Evaluation of Large Language Models
25. GPT-3: Its nature, scope, limits, and consequences10.1007/s11023-020-09548-1 ↗
26. OpenAI (2023). GPT-4 Technical Report. arXiv.
27. Shyam, P., Sastry, G., and Askell, A. (2020). Language models are few-shot learners. arXiv.
28. Ziegler, D.M., Stiennon, N., Wu, J., Brown, T.B., Radford, A., Amodei, D., and Irving, G. (2019). Fine-tuning language models from human preferences. arXiv.
29. Naveed, H., Khan, A.U., Qiu, S., Saqib, M., Anwar, S., Usman, M., and Mian, A. (2023). A Comprehensive Overview of Large Language Models. arXiv.
30. Exploring the limits of transfer learning with a unified text-to-text transformer
31. Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., and Raffel, C. (2021, January 6–11). mt5: A massively multilingual pre-trained text-to-text transformer. Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online.10.18653/v1/2021.naacl-main.41 ↗
32. Press, O., Smith, N., and Lewis, M. (2022, January 25–29). Train short, test long: Attention with Linear Biases Enables Input Length Extrapolation. Proceedings of the International Conference on Learning Representations, Online.
33. Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y. (2021). Roformer: Enhanced transformer with rotary position embedding. arXiv.
34. (2024, April 17). SBERT.net Sentence-Transformers. Available online: https://www.sbert.net/docs/pretrained_models.html.
35. Sanh, V., Debut, L., Chaumond, J., and Wolf, T. (2019). DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv.
36. (2024, February 15). DAIGT V2 Training Set. Available online: https://www.kaggle.com/datasets/thedrcat/daigt-v2-train-dataset.