Be Sure to Use the Same Writing Style: Applying Authorship Verification on Large-Language-Model-Generated Texts
Source abstract
Abstract
Recently, there have been significant advances and wide-scale use of generative AI in natural language generation. Models such as OpenAI’s GPT3 and Meta’s LLaMA are widely used in chatbots, to summarize documents, and to generate creative content. These advances raise concerns about abuses of these models, especially in social media settings, such as large-scale generation of disinformation, manipulation campaigns that use AI-generated content, and personalized scams. We used stylometry (the analysis of style in natural language text) to analyze the style of AI-generated text. Specifically, we applied an existing authorship verification (AV) model that can predict if two documents are written by the same author on texts generated by GPT2, GPT3, ChatGPT and LLaMA. Our AV model was trained only on human-written text and was effectively used in social media settings to analyze cases of abuse. We generated texts by providing the language models with fanfiction snippets and prompting them to complete the rest of it in the same writing style as the original snippet. We then applied the AV model across the texts generated by the language models and the human written texts to analyze the similarity of the writing styles between these texts. We found that texts generated with GPT2 had the highest similarity to the human texts. Texts generated by GPT3 and ChatGPT were very different from the human snippet, and were similar to each other. LLaMA-generated texts had some similarity to the original snippet but also has similarities with other LLaMA-generated texts and texts from other models. We then conducted a feature analysis to identify the features that drive these similarity scores. This analysis helped us answer questions like which features distinguish the language style of language models and humans, which features are different across different models, and how these linguistic features change over different language model versions. The dataset and the source code used in this analysis have been made public to allow for further analysis of new language models.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Authorship Attribution and Profiling · Natural Language Processing Techniques · Topic Modeling
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science and Engineering, Tandon School of Engineering, New York University, Brooklyn, NY 11201, USA
Funding
U.S. National Science Foundation · New York University
References (55)
Show source-provided references
1. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2025, February 24). Language Models are Unsupervised Multitask Learners. OpenAI, Available online: https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf.
2. Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., and Askell, A. (2020). Language Models are Few-Shot Learners. arXiv.
3. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., and Azhar, F. (2023). LLaMA: Open and Efficient Foundation Language Models. arXiv.
4. Herbold, S., Hautli-Janisz, A., Heuer, U., Kikteva, Z., and Trautsch, A. (2023). AI, write an essay for me: A large-scale comparison of human-written versus ChatGPT-generated essays. arXiv.10.1038/s41598-023-45644-9 ↗
5. Moslem, Y., Haque, R., Kelleher, J.D., and Way, A. (2023). Adaptive machine translation with large language models. arXiv.
6. Clinical text summarization: Adapting large language models can outperform human experts
7. Role of chat gpt in public health10.1007/s10439-023-03172-7 ↗
8. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H.P.d.O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., and Brockman, G. (2021). Evaluating large language models trained on code. arXiv.
9. Defending against neural fake news
10. Sedova, K., McNeill, C., Johnson, A., Joshi, A., and Wulkan, I. (2021). AI and the Future of Disinformation Campaigns. Cent. Secur. Emerg. Technol.
11. The history of digital spam10.1145/3299768 ↗
12. TikTok (2025, February 24). New Labels for Disclosing AI-Generated Content. Available online: https://newsroom.tiktok.com/en-us/new-labels-for-disclosing-ai-generated-content.
13. Clegg, N. (2025, February 24). Labeling AI-Generated Images on Facebook, Instagram and Threads. Available online: https://about.fb.com/news/2024/02/labeling-ai-generated-images-on-facebook-instagram-and-threads/.
14. Guo, B., Zhang, X., Wang, Z., Jiang, M., Nie, J., Ding, Y., Yue, J., and Wu, Y. (2023). How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection. arXiv.
15. Uchendu, A., Le, T., Shu, K., and Lee, D. (2020, January 16–20). Authorship Attribution for Neural Text Generation. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online.10.18653/v1/2020.emnlp-main.673 ↗
16. Solaiman, I., Brundage, M., Clark, J., Askell, A., Herbert-Voss, A., Wu, J., Radford, A., Krueger, G., Kim, J.W., and Kreps, S. (2019). Release strategies and the social impacts of language models. arXiv.
17. Liu, Y., Zhang, Z., Zhang, W., Yue, S., Zhao, X., Cheng, X., Zhang, Y., and Hu, H. (2023). ArguGPT: Evaluating, understanding and identifying argumentative essays generated by GPT models. arXiv.
18. Surveying Stylometry Techniques and Applications10.1145/3132039 ↗
19. Authorship Analysis Studies: A Survey
20. Halvani, O., Winter, C., and Graner, L. (2019, January 26–29). Assessing the Applicability of Authorship Verification Methods. Proceedings of the 14th International Conference on Availability, Reliability and Security, ARES ’19, Canterbury, UK.10.1145/3339252.3340508 ↗
21. Koppel, M., and Schler, J. (2004, January 4–8). Authorship Verification as a One-Class Classification Problem. Proceedings of the Twenty-First International Conference on Machine Learning, ICML ’04, Banff, AB, Canada.10.1145/1015330.1015448 ↗
22. Weerasinghe, J., Singh, R., and Greenstadt, R. (2021, January 21–24). Feature Vector Difference based Authorship Verification for Open-World Settings—Notebook for PAN at CLEF 2021. Proceedings of the CLEF 2021 Labs and Workshops, Notebook Papers, Bucharest, Romania.
23. Singh, R., Weerasinghe, J., and Greenstadt, R. (2021, January 21–24). Writing Style Change Detection on Multi-Author Documents—Notebook for PAN at CLEF 2021. Proceedings of the CLEF 2021 Labs and Workshops, Notebook Papers, Bucharest, Romania.
24. Using Authorship Verification to Mitigate Abuse in Online Communities10.1609/icwsm.v16i1.19359 ↗
25. Bischoff, S., Deckers, N., Schliebs, M., Thies, B., Hagen, M., Stamatatos, E., Stein, B., and Potthast, M. (2020). The Importance of Suppressing Domain Style in Authorship Analysis. arXiv.
26. Arampatzis, A., Kanoulas, E., Tsikrika, T., Vrochidis, S., Joho, H., Lioma, C., Eickhoff, C., Névéol, A., Cappellato, L., and Ferro, N. (2020, January 22–25). Overview of PAN 2020: Authorship Verification, Celebrity Profiling, Profiling Fake News Spreaders on Twitter, and Style Change Detection. Proceedings of the 11th International Conference of the CLEF Association (CLEF 2020), Thessaloniki, Greece.
27. Crestani, F., Braschler, M., Savoy, J., Rauber, A., Müller, H., Losada, D., Heinatz, G., Cappellato, L., and Ferro, N. (2019, January 9–12). Overview of PAN 2019: Author Profiling, Celebrity Profiling, Cross-domain Authorship Attribution and Style Change Detection. Proceedings of the 10th International Conference of the CLEF Association (CLEF 2019), Lugano, Switzerland.
28. Bellot, P., Trabelsi, C., Mothe, J., Murtagh, F., Nie, J., Soulier, L., SanJuan, E., Cappellato, L., and Ferro, N. (2018, January 10–14). Overview of PAN 2018: Author Identification, Author Profiling, and Author Obfuscation. Proceedings of the Experimental IR Meets Multilinguality, Multimodality, and Interaction, 9th International Conference of the CLEF Initiative (CLEF 2018), Avignon, France.10.1007/978-3-319-98932-7 ↗
29. Jones, G., Lawless, S., Gonzalo, J., Kelly, L., Goeuriot, L., Mandl, T., Cappellato, L., and Ferro, N. (2017, January 11–14). Overview of PAN 2017: Author Identification, Author Profiling, and Author Obfuscation. Proceedings of the Experimental IR Meets Multilinguality, Multimodality, and Interaction, 8th International Conference of the CLEF Initiative (CLEF 2017), Dublin, Ireland.10.1007/978-3-319-65813-1 ↗
30. Fuhr, N., Quaresma, P., Larsen, B., Gonçalves, T., Balog, K., Macdonald, C., Cappellato, L., and Ferro, N. (2016, January 5–8). Overview of PAN 2016—New Challenges for Authorship Analysis: Cross-genre Profiling, Clustering, Diarization, and Obfuscation. Proceedings of the Experimental IR Meets Multilinguality, Multimodality, and Interaction, 7th International Conference of the CLEF Initiative (CLEF 2016), Évora, Portugal.10.1007/978-3-319-44564-9 ↗
31. Candan, K., Ionescu, B., Goeuriot, L., Larsen, B., Müller, H., Joly, A., Maistro, M., Piroi, F., Faggioli, G., and Ferro, N. (2021, January 21–24). Overview of PAN 2021: Authorship Verification, Profiling Hate Speech Spreaders on Twitter, and Style Change Detection. Proceedings of the 12th International Conference of the CLEF Association (CLEF 2021), Virtual Event.
32. Barrón-Cedeño, A., Da San Martino, G., Esposti, M.D., Sebastiani, F., Macdonald, C., Pasi, G., Hanbury, A., Potthast, M., Faggioli, G., and Ferro, N. (2022, January 5–8). Overview of PAN 2022: Authorship Verification, Profiling Irony and Stereotype Spreaders, and Style Change Detection. Proceedings of the Experimental IR Meets Multilinguality, Multimodality, and Interaction, 13th International Conference of the CLEF Association (CLEF 2022), Bologna, Italy. Lecture Notes in Computer Science.10.1007/978-3-031-13643-6 ↗
33. Overview of PAN 2023: Authorship Verification, Multi-Author Writing Style Analysis, Profiling Cryptocurrency Influencers, and Trigger Detection10.1007/978-3-031-42448-9_29 ↗
34. Goeuriot, L., Mulhem, P., Quénot, G., Schwab, D., Nunzio, G.M.D., Soulier, L., Galuscakova, P., Herrera, A.G.S., Faggioli, G., and Ferro, N. (2024, January 9–12). Overview of PAN 2024: Multi-Author Writing Style Analysis, Multilingual Text Detoxification, Oppositional Thinking Analysis, and Generative AI Authorship Verification. Proceedings of the Experimental IR Meets Multilinguality, Multimodality, and Interaction, 15th International Conference of the CLEF Association (CLEF 2024), Grenoble, France. Lecture Notes in Computer Science.10.1007/978-3-031-71908-0 ↗
35. Johansson, I.R. (2025, February 24). A Tale of Two Texts, a Robot, and Authorship: A Comparison Between a Human-Written and a ChatGPT-Generated Text (Dissertation). Available online: https://urn.kb.se/resolve?urn=urn:nbn:se:mau:diva-60674.
36. Faggioli, G., Ferro, N., Galuščáková, P., and Herrera, A.G.S. (2024, January 9–12). Overview of the “Voight-Kampff” Generative AI Authorship Verification Task at PAN and ELOQUENT 2024. Proceedings of the Working Notes of CLEF 2024–Conference and Labs of the Evaluation Forum, Grenoble, France. CEUR Workshop Proceedings.
37. Keskar, N.S., McCann, B., Varshney, L.R., Xiong, C., and Socher, R. (2019). CTRL: A Conditional Transformer Language Model for Controllable Generation. arXiv.
38. Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. (2025, February 24). Improving Language Understanding by Generative Pre-Training. OpenAI, Available online: https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf.
39. Lample, G., and Conneau, A. (2019). Crosslingual language model pretraining. arXiv.
40. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R.R., and Le, Q.V. (2019, January 8–14). XLNet: Generalized autoregressive pretraining for language understanding. Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada.
41. Dathathri, S., Madotto, A., Lan, J., Hung, J., Frank, E., Molino, P., Yosinski, J., and Liu, R. (2019). Plug and play language models: A simple approach to controlled text generation. arXiv.
42. Ng, N., Yee, K., Baevski, A., Ott, M., Auli, M., and Edunov, S. (2019). Facebook Fair’s WMT19 news translation task submission. arXiv.10.18653/v1/W19-5333 ↗
43. The psychological meaning of words: LIWC and computerized text analysis methods10.1177/0261927X09351676 ↗
44. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv.
45. Distinguishing academic science writing from humans or ChatGPT with over 99% accuracy using off-the-shelf machine learning tools10.1016/j.xcrp.2023.101426 ↗
46. Mitrović, S., Andreoletti, D., and Ayoub, O. (2023). Chatgpt or human? detect and explain. explaining decisions of machine learning model for detecting short chatgpt-generated text. arXiv.
47. Are You Robert or RoBERTa? Deceiving Online Authorship Attribution Models Using Neural Text Generators10.1609/icwsm.v16i1.19304 ↗
48. Weerasinghe, J., and Greenstadt, R. (2020, January 22–25). Feature Vector Difference based Neural Network and Logistic Regression Models for Authorship Verification—Notebook for PAN at CLEF 2020. Proceedings of the CLEF 2020 Labs and Workshops, Notebook Papers, Thessaloniki, Greece.
49. A survey of modern authorship attribution methods10.1002/asi.21001 ↗
50. Writeprints: A Stylometric Approach to Identity-level Identification and Similarity Detection in Cyberspace
51. Wolf, T. (2020). Transformers: State-of-the-Art Natural Language Processing. arXiv.10.18653/v1/2020.emnlp-demos.6 ↗
52. Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (2017). A Unified Approach to Interpreting Model Predictions. Advances in Neural Information Processing Systems 30, Curran Associates, Inc.
53. OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., and Altman, S. (2024). GPT-4 Technical Report. arXiv.
54. Team, G., Anil, R., Borgeaud, S., Alayrac, J.B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A.M., Hauth, A., and Millican, K. (2024). Gemini: A Family of Highly Capable Multimodal Models. arXiv.
55. DeepSeek-AI, Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., and Zhang, C. (2024). DeepSeek-V3 Technical Report. arXiv.