Temporal Dynamics of Harmful Speech in Chatbot–User Dialogues: A Comparative Study of LLM and Chit-Chat Systems
Source abstract
Abstract
Harmful language in conversational AI poses distinct safety and governance challenges, as Large Language Model (LLM) chatbots interact in private, one-to-one settings. Understanding the types of harm and their temporal concentration is crucial for responsible deployment and time-aware moderation. This study investigates the types and diurnal dynamics of harmful speech, comparing patterns between play-oriented chit-chat and task-oriented LLM services.We analyze two large-scale, real-world English corpora: a chit-chat service (SimSimi; 8.7 M utterances) and an LLM service (WildChat; 610 K utterances). Using the Perspective API for multi-label classification (Toxicity, Profanity, Insult, Identity Attack, Threat), we estimate the incidence of harm categories and compare their distribution across five dayparts. Our analysis shows that harmful speech is significantly more prevalent in the chit-chat context than in the LLM service. Across both platforms, Toxicity and Profanity are the dominant categories. Temporally, harmful speech concentrates most frequently during the dawn daypart. We contribute an empirical baseline on how harm varies by chatbot modality and time of day, offering practical guidance for designing dynamic, platform-specific moderation policies.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Hate Speech and Cyberbullying Detection · AI in Service Interactions · Misinformation and Its Impacts
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science and Engineering, Gyeongsang National University, 501 Jinju-daero, Jinju-si 52828, Gyeongsangnam-do, Republic of Korea
Department of Mechanical Convergence Engineering, Gyeongsang National University, 501 Jinju-daero, Jinju-si 52828, Gyeongsangnam-do, Republic of Korea
Funding
Gyeongsang National University
References (49)
Show source-provided references
1. Si, W.M., Backes, M., Blackburn, J., De Cristofaro, E., Stringhini, G., Zannettou, S., and Zhang, Y. (2022, January 7–11). Why So Toxic? Measuring and Triggering Toxic Behavior in Open-Domain Chatbots. Proceedings of the CCS ’22: 2022 ACM SIGSAC Conference on Computer and Communications Security, Los Angeles, CA, USA.10.1145/3548606.3560599 ↗
2. Chin, H., Molefi, L.W., and Yi, M.Y. (2020, January 25–30). Empathy Is All You Need: How a Conversational Agent Should Respond to Verbal Abuse. Proceedings of the CHI ’20: 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA.10.1145/3313831.3376461 ↗
3. Song, H., Hong, J., Jung, C., Chin, H., Shin, M., Choi, Y., Choi, J., and Cha, M. (2024, January 20–25). Detecting offensive language in an open chatbot platform. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), Torino, Italy.
4. Bender, E.M., Gebru, T., McMillan-Major, A., and Shmitchell, S. (2021, January 3–10). On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. Proceedings of the FAccT ’21: 2021 ACM Conference on Fairness, Accountability, and Transparency, Virtual.10.1145/3442188.3445922 ↗
5. AI models collapse when trained on recursively generated data10.1038/s41586-024-07566-y ↗
6. A survey on automatic detection of hate speech in text10.1145/3232676 ↗
7. A systematic review of hate speech automatic detection using natural language processing10.1016/j.neucom.2023.126232 ↗
8. The potential of chatbots for emotional support and promoting mental well-being in different cultures: Mixed methods study10.2196/51712 ↗
9. Lin, Z., Wang, Z., Tong, Y., Wang, Y., Guo, Y., Wang, Y., and Shang, J. (2023). Toxicchat: Unveiling hidden challenges of toxicity detection in real-world user-ai conversation. arXiv.10.18653/v1/2023.findings-emnlp.311 ↗
10. Recent advances in deep learning based dialogue systems: A systematic survey10.1007/s10462-022-10248-8 ↗
11. Roller, S., Boureau, Y.L., Weston, J., Bordes, A., Dinan, E., Fan, A., Gunning, D., Ju, D., Li, M., and Poff, S. (2020). Open-domain conversational agents: Current progress, open problems, and future directions. arXiv.
12. (2025, December 04). Replika CEO Eugenia Kuyda Launches Wabi. Business Insider. Available online: https://www.businessinsider.com/replika-ceo-eugenia-kuyda-launch-wabi-2025-10.
13. The New York Times (2025, December 04). Character.ai Abandons Making AI Models after $2.7bn Google Deal. 2024. Reports 20 Million Monthly Active Users. Available online: https://www.ft.com/content/f2a9b5d4-05fe-4134-b4fe-c24727b85bba.
14. SimSimi Inc. (2025, December 04). SimSimi Official Website. Available online: https://simsimi.kr/.
15. (2025, December 04). How People Use Claude for Support, Advice, and Companionship. Available online: https://www.anthropic.com/news/how-people-use-claude-for-support-advice-and-companionship.
16. Chatterji, A., Cunningham, T., Deming, D.J., Hitzig, Z., Ong, C., Shan, C.Y., and Wadman, K. (2025). How People Use Chatgpt, National Bureau of Economic Research. Technical Report.10.3386/w34255 ↗
17. Cohn, T., He, Y., and Liu, Y. (2020). RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models. Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16–20 November 2020, Association for Computational Linguistics.
18. Shen, X., Chen, Z., Backes, M., Shen, Y., and Zhang, Y. (2024, January 14–18). “Do Anything Now”: Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models. Proceedings of the CCS ’24: 2024 on ACM SIGSAC Conference on Computer and Communications Security, Salt Lake City, UT, USA.10.1145/3658644.3670388 ↗
19. The Security of Using Large Language Models: A Survey with Emphasis on ChatGPT10.1109/JAS.2024.124983 ↗
20. Kim, Y., and Kim, J.H. (2021). The impact of ethical issues on public understanding of artificial intelligence. Proceedings of the 23rd HCI International Conference, HCII 2021, Virtual Event, 24–29 July 2021, Springer.10.1007/978-3-030-78642-7_67 ↗
21. Bouamor, H., Pino, J., and Bali, K. (2023). Toxicity in chatgpt: Analyzing persona-assigned language models. Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, 6–10 December 2023, Association for Computational Linguistics.
22. Weeks, C., Cheruvu, A., Abdullah, S.M., Kanchi, S., Yao, D., and Viswanath, B. (2023, January 4–8). A first look at toxicity injection attacks on open-domain chatbots. Proceedings of the 39th Annual Computer Security Applications Conference, Austin, TX, USA.10.1145/3627106.3627122 ↗
23. Diurnal and seasonal mood vary with work, sleep, and daylength across diverse cultures10.1126/science.1202775 ↗
24. Grinberg, N., Naaman, M., Shaw, B., and Lotan, G. (2013, January 8–11). Extracting diurnal patterns of real world activity from social media. Proceedings of the International AAAI Conference on Web and Social Media, Cambridge, MA, USA.
25. Sleep deprivation lowers inhibition and enhances impulsivity to negative stimuli10.1016/j.bbr.2010.09.020 ↗
26. The online disinhibition effect10.1089/1094931041291295 ↗
27. Active night-time tweeting is associated with meaningfully lower mental wellbeing in a UK birth cohort study10.1038/s41598-025-14745-y ↗
28. Social media use and adolescent sleep patterns: Cross-sectional findings from the UK millennium cohort study10.1136/bmjopen-2019-031161 ↗
29. Kumar, D., Hancock, J., Thomas, K., and Durumeric, Z. (May, January 30). Understanding the behaviors of toxic accounts on reddit. Proceedings of the ACM Web Conference 2023, Austin, TX, USA.10.1145/3543507.3583522 ↗
30. Zhao, W., Ren, X., Hessel, J., Cardie, C., Choi, Y., and Deng, Y. (2024). Wildchat: 1m chatgpt interaction logs in the wild. arXiv.
31. Chiruzzo, L., Ritter, A., and Wang, L. (2025). On the Role of Speech Data in Reducing Toxicity Detection Bias. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Albuquerque, NM, USA, 29 April–4 May 2025, Association for Computational Linguistics.
32. IBM (2025, December 04). Context Window (Think Blog Topic Page). Available online: https://www.ibm.com/think/topics/context-window.
33. Shin, M., Chin, H., Song, H., Choi, Y., Choi, J., and Cha, M. (2024). Context-Aware Offensive Language Detection in Human-Chatbot Conversations. Proceedings of the 2024 IEEE International Conference on Big Data and Smart Computing (BigComp), Bangkok, Thailand, 18–21 February 2024, IEEE.10.1109/BigComp60711.2024.00049 ↗
34. Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tur, D., Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., and Zhou, Y. (2021). Bot-Adversarial Dialogue for Safe Conversational Agents. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online, 6–11 June 2021, Association for Computational Linguistics.
35. Greenberg, A. (2025, December 04). Now Anyone Can Deploy Google’s Troll-Fighting AI. WIRED (Security). Available online: https://www.wired.com/2017/02/googles-troll-fighting-ai-now-belongs-world/.
36. Lima, L.H.Q., Pagano, A.S., and da Silva, A.P.C. (2024, January 14–15). Toxic Content Detection in online social networks: A new dataset from Brazilian Reddit Communities. Proceedings of the 16th International Conference on Computational Processing of Portuguese—Volume 1, Santiago de Compostela, Spain.
37. Saveski, M., Roy, B., and Roy, D. (2021, January 19–23). The structure of toxic conversations on Twitter. Proceedings of the Web Conference 2021, Ljubljana, Slovenia.10.1145/3442381.3449861 ↗
38. Toxicity on Social Media During the 2022 Mpox Public Health Emergency: Quantitative Study of Topical and Network Dynamics10.2196/52997 ↗
39. Shen, Z., and Paik, I. (2025). Temporal Modeling of Social Media for Depression Forecasting: Deep Learning Approaches with Pretrained Embeddings. Appl. Sci., 15.10.3390/app152011274 ↗
40. Andrews, S., Ellis, D.A., Shaw, H., and Piwek, L. (2015). Beyond self-report: Tools to compare estimated and real-world smartphone use. PLoS ONE, 10.10.1371/journal.pone.0139004 ↗
41. The pulse of mood online: Unveiling emotional reactions in a dynamic social media landscape10.1145/3728313 ↗
42. User-chatbot conversations during the COVID-19 pandemic: Study based on topic modeling and sentiment analysis10.2196/40922 ↗
43. Exploring the influence of user characteristics on verbal aggression towards social chatbots10.1080/0144929X.2024.2362957 ↗
44. The future of artificial intelligence and robotics in the retail and service sector: Sketching the field of consumer-robot-experiences10.1007/s11747-023-00948-0 ↗
45. The impact of AI identity disclosure on consumer unethical behavior: A social judgment perspective10.1016/j.jretconser.2023.103606 ↗
46. Anonymous social media—Understanding the content and context of Yik Yak10.1016/j.chb.2015.11.043 ↗
47. The effects of group centrality and accountability on conformity to cyber aggressive norms: Two messaging app experiments10.1016/j.chb.2021.106754 ↗
48. Bogdan, A., Dospinescu, N., and Dospinescu, O. (2025). Beyond Credibility: Understanding the Mediators Between Electronic Word-of-Mouth and Purchase Intention. arXiv.
49. Finding information diffusion’s seed nodes in online social networks using a special degree centrality10.1007/s42979-024-02683-x ↗