One-Class Learning for AI-Generated Essay Detection
Source abstract
Abstract
Detection of AI-generated content is a crucially important task considering the increasing attention towards AI tools, such as ChatGPT, and the raised concerns with regard to academic integrity. Existing text classification approaches, including neural-network-based and feature-based methods, are mostly tailored for English data, and they are typically limited to a supervised learning setting. Although one-class learning methods are more suitable for classification tasks, their effectiveness in essay detection is still unknown. In this paper, this gap is explored by adopting linguistic features and one-class learning models for AI-generated essay detection. Detection performance of different models is assessed in different settings, where positively labeled data, i.e., AI-generated essays, are unavailable for model training. Results with two datasets containing essays in L2 English and L2 Spanish show that it is feasible to accurately detect AI-generated essays. The analysis reveals which models and which sets of linguistic features are more powerful than others in the detection task.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Text Readability and Simplification · Natural Language Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science, American University, Washington, DC 20016, USA
Department of Linguistics, University of Pittsburgh, Pittsburgh, PA 15260, USA
References (53)
Show source-provided references
1. Sallam, M. (2023). ChatGPT utility in healthcare education, research, and practice: Systematic review on the promising perspectives and valid concerns. Healthcare, 11.10.3390/healthcare11060887 ↗
2. Chatting about ChatGPT: How may AI and GPT impact academia and libraries?10.1108/LHTN-01-2023-0009 ↗
3. A conversation on artificial intelligence, chatbots, and plagiarism in higher education10.1007/s12195-022-00754-8 ↗
4. The World Wide Web for academic purposes: Old study skills for new?10.1016/S0889-4906(00)00035-1 ↗
5. Writing in an electronic age: A case study of L2 composing processes10.1016/j.jeap.2010.10.002 ↗
6. Crothers, E., Japkowicz, N., and Viktor, H. (2022). Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods. arXiv.
7. Bostrom, N., and Yudkowsky, E. (2018). Artificial Intelligence Safety and Security, Chapman and Hall/CRC.10.1201/9781351251389-4 ↗
8. Social media-based COVID-19 sentiment classification model using Bi-LSTM10.1016/j.eswa.2022.118710 ↗
9. Bidirectional LSTM with self-attention mechanism and multi-channel features for sentiment classification10.1016/j.neucom.2020.01.006 ↗
10. Kumari, R., Ashok, N., Ghosal, T., and Ekbal, A. (2021, January 18–22). A multitask learning approach for fake news detection: Novelty, emotion, and sentiment lend a helping hand. Proceedings of the 2021 International Joint Conference on Neural Networks (IJCNN), Shenzhen, China.10.1109/IJCNN52387.2021.9534218 ↗
11. Damasceno, L.P., Shafer, A., Japkowicz, N., Cavalcante, C.C., and Boukouvalas, Z. (2022, January 10–12). Efficient Multivariate Data Fusion for Misinformation Detection During High Impact Events. Proceedings of the Discovery Science: 25th International Conference, DS 2022, Montpellier, France.10.1007/978-3-031-18840-4_19 ↗
12. Jing, Q., Yao, D., Fan, X., Wang, B., Tan, H., Bu, X., and Bi, J. (2021, January 18–22). TRANSFAKE: Multi-task Transformer for Multimodal Enhanced Fake News Detection. Proceedings of the 2021 International Joint Conference on Neural Networks (IJCNN), Shenzhen, China.10.1109/IJCNN52387.2021.9533433 ↗
13. Han, H., Ke, Z., Nie, X., Dai, L., and Slamu, W. (2023). Multimodal Fusion with Dual-Attention Based on Textual Double-Embedding Networks for Rumor Detection. Appl. Sci., 13.10.3390/app13084886 ↗
14. Prasad, N., Saha, S., and Bhattacharyya, P. (2021, January 18–22). A Multimodal Classification of Noisy Hate Speech using Character Level Embedding and Attention. Proceedings of the 2021 International Joint Conference on Neural Networks (IJCNN), Shenzhen, China.10.1109/IJCNN52387.2021.9533371 ↗
15. Alghamdi, J., Lin, Y., and Luo, S. (2023). Does Context Matter? Effective Deep Learning Approaches to Curb Fake News Dissemination on Social Media. Appl. Sci., 13.10.3390/app13053345 ↗
16. Allouch, M., Mansbach, N., Azaria, A., and Azoulay, R. (2023). Utilizing Machine Learning for Detecting Harmful Situations by Audio and Text. Appl. Sci., 13.10.3390/app13063927 ↗
17. Rubin, V.L., Conroy, N., Chen, Y., and Cornwell, S. (2016, January 17). Fake news or truth? Using satirical cues to detect potentially misleading news. Proceedings of the Second Workshop on Computational Approaches to Deception Detection, San Diego, CA, USA.10.18653/v1/W16-0802 ↗
18. Feng, L., Jansche, M., Huenerfauth, M., and Elhadad, N. (2010, January 23–27). A comparison of features for automatic readability assessment. Proceedings of the 23rd International Conference on Computational Linguistics (COLING 2010), Beijing, China.
19. Argamon-Engelson, S., Koppel, M., and Avneri, G. (1998, January 26–27). Style-based text categorization: What newspaper am I reading. Proceedings of the AAAI Workshop on Text Categorization, Madison, WI, USA.
20. Automatically categorizing written texts by author gender10.1093/llc/17.4.401 ↗
21. Pérez-Rosas, V., Kleinberg, B., Lefevre, A., and Mihalcea, R. (2017). Automatic detection of fake news. arXiv.
22. Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. (2019, January 6–9). The Curious Case of Neural Text Degeneration. Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA.
23. Ippolito, D., Duckworth, D., Callison-Burch, C., and Eck, D. (2020, January 5–10). Automatic Detection of Generated Text is Easiest when Humans are Fooled. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.164 ↗
24. Feature-based detection of automated language models: Tackling GPT-2, GPT-3 and Grover10.7717/peerj-cs.443 ↗
25. Gehrmann, S., Harvard, S., Strobelt, H., and Rush, A.M. (August, January 28). GLTR: Statistical Detection and Visualization of Generated Text. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, ACL 2019, Florence, Italy.10.18653/v1/P19-3019 ↗
26. Text readability and intuitive simplification: A comparison of readability formulas
27. Corizzo, R., and Leal-Arenas, S. (2023, January 18–23). A Deep Fusion Model for Human vs. Machine-Generated Essay Classification. Proceedings of the 2023 International Joint Conference on Neural Networks (IJCNN), Broadbeach, Australia.10.1109/IJCNN54540.2023.10191322 ↗
28. Rewicki, F., Denzler, J., and Niebling, J. (2023). Is It Worth It? Comparing Six Deep and Classical Methods for Unsupervised Anomaly Detection in Time Series. Appl. Sci., 13.10.3390/app13031778 ↗
29. Ryan, S., Corizzo, R., Kiringa, I., and Japkowicz, N. (2019, January 16–19). Pattern and anomaly localization in complex and dynamic data. Proceedings of the 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA), Boca Raton, FL, USA.10.1109/ICMLA.2019.00285 ↗
30. Lian, Y., Geng, Y., and Tian, T. (2023). Anomaly Detection Method for Multivariate Time Series Data of Oil and Gas Stations Based on Digital Twin and MTAD-GAN. Appl. Sci., 13.10.3390/app13031891 ↗
31. Spatially-aware autoencoders for detecting contextual anomalies in geo-distributed data10.1007/978-3-030-88942-5_36 ↗
32. Herskind Sejr, J., Christiansen, T., Dvinge, N., Hougesen, D., Schneider-Kamp, P., and Zimek, A. (2021). Outlier detection with explanations on music streaming data: A case study with danmark music group ltd. Appl. Sci., 11.10.3390/app11052270 ↗
33. Faber, K., Corizzo, R., Sniezynski, B., and Japkowicz, N. (2022, January 13–16). Active Lifelong Anomaly Detection with Experience Replay. Proceedings of the 2022 IEEE 9th International Conference on Data Science and Advanced Analytics (DSAA), Shenzhen, China.10.1109/DSAA54385.2022.10032405 ↗
34. One-class ensembles for rare genomic sequences identification10.1007/978-3-030-61527-7_23 ↗
35. Baly, R., Karadzhov, G., Alexandrov, D., Glass, J., and Nakov, P. (2018). Predicting factuality of reporting and bias of news media sources. arXiv.10.18653/v1/D18-1389 ↗
36. Horne, B.D., and Adali, S. (2017, January 15–18). This just in: Fake news packs a lot in title, uses simpler, repetitive content in text body, more similar to satire than real news. Proceedings of the Eleventh International AAAI Conference on Web and Social Media, Montreal, QC, Canada.10.1609/icwsm.v11i1.14976 ↗
37. Hube, C., and Fetahu, B. Proceedings of the Companion Proceedings of the Web Conference, Lyon, France, 23–27 April 2018.
38. The case for latent variable vs deep learning methods in misinformation detection: An application to covid-1910.1007/978-3-030-88942-5_33 ↗
39. Wang, W., Yu, Y., and Sheng, J. (2006, January 8–11). Image retrieval by emotional semantics: A study of emotional space and feature extraction. Proceedings of the 2006 IEEE International Conference on Systems, Man and Cybernetics, Taipei, Taiwan.
40. The spread of true and false news online10.1126/science.aap9559 ↗
41. A comprehensive study on lexicon based approaches for sentiment analysis10.51983/ajcst-2019.8.S2.2037 ↗
42. Voutilainen, A. (2003). The Oxford Handbook of Computational Linguistics, Oxford University Press.
43. Schölkopf, B., Williamson, R.C., Smola, A.J., Shawe-Taylor, J., and Platt, J.C. (2000). Advances in Neural Information Processing Systems, MIT Press.
44. Breunig, M.M., Kriegel, H.P., Ng, R.T., and Sander, J. (2000, January 15–18). LOF: Identifying density-based local outliers. Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data, Dallas, TX, USA.10.1145/342009.335388 ↗
45. Liu, F.T., Ting, K.M., and Zhou, Z.H. (2008, January 15–19). Isolation forest. Proceedings of the 2008 Eighth IEEE International Conference on Data Mining, Pisa, Italy.10.1109/ICDM.2008.17 ↗
46. Kriegel, H., Schubert, M., and Zimek, A. (2008, January 24–27). Angle-based outlier detection in high-dimensional data. Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Las Vegas, NV, USA.10.1145/1401890.1401946 ↗
47. Pham, N., and Pagh, R. (2012, January 12–16). A near-linear time approximation algorithm for angle-based outlier detection in high-dimensional data. Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Beijing, China.10.1145/2339530.2339669 ↗
48. Goldstein, M., and Score, A.D.H.b.O. (2012, January 24–27). A fast Unsupervised Anomaly Detection Algorithm. Proceedings of the KI-2012: Poster and Demo Track, 35th German Conference on Artificial Intelligence, Saarbrücken, Germany.
49. Linguistic feature based learning model for fake news detection and classification10.1016/j.eswa.2020.114171 ↗
50. Zhu, T. (2022). From Textual Experiments to Experimental Texts: Expressive Repetition in “Artificial Intelligence Literature”. arXiv.
51. Language transfer
52. Haspelmath, M., and Michaelis, S.M. (2017, January 27–29). Analytic and synthetic. Proceedings of the Language Variation-European Perspectives VI: Selected Papers from the Eighth International Conference on Language Variation in Europe (ICLaVE 8), Leipzig, Germany.10.1075/silv.19.01has ↗
53. Filippova, K. (2010, January 23–27). Multi-sentence compression: Finding shortest paths in word graphs. Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), Beijing, China.