Social Botomics: A Systematic Ensemble ML Approach for Explainable and Multi-Class Bot Detection
Source abstract
Abstract
OSN platforms are under attack by intruders born and raised within their own ecosystems. These attacks have multiple scopes from mild critiques to violent offences targeting individual or community rights and opinions. Negative publicity on microblogging platforms, such as Twitter, is due to the infamous Twitter bots which highly impact posts’ circulation and virality. A wide and ongoing research effort has been devoted to develop appropriate countermeasures against emerging “armies of bots”. However, the battle against bots is still intense and unfortunately, it seems to lean on the bot-side. Since, in an effort to win any war, it is critical to know your enemy, this work aims to demystify, reveal, and widen inherent characteristics of Twitter bots such that multiple types of bots are recognized and spotted early. More specifically in this work we: (i) extensively analyze the importance and the type of data and features used to generate ML models for bot classification, (ii) address the open problem of multi-class bot detection, identifying new types of bots, and share two new datasets towards this objective, (iii) provide new individual ML models for binary and multi-class bot classification and (iv) utilize explainable methods and provide comprehensive visualizations to clearly demonstrate interpretable results. Finally, we utilize all of the above in an effort to improve the so called Bot-Detective online service. Our experiments demonstrate high accuracy, explainability and scalability, comparable with the state of the art, despite multi-class classification challenges.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Spam and Phishing Detection · Network Security and Intrusion Detection · Advanced Malware Detection Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Informatics, Aristotle University of Thessaloniki, 54124 Thessaloniki, Greece
Funding
Horizon 2020 Framework Programme · Operational Program Competitiveness, Entrepreneurship and Innovation - RESEARCH | CREATE | INNOVATE
References (68)
Show source-provided references
1. Science, new media, and the public10.1126/science.1232329 ↗
2. Stieglitz, S., Brachten, F., Ross, B., and Jung, A.K. (2017). Do social bots dream of electric sheep? A categorisation of social media bot accounts. arXiv.
3. The next-generation bots interfering with the US election10.1038/d41586-020-03034-5 ↗
4. Lee, K., Eoff, B., and Caverlee, J. (2011, January 17–21). Seven months with the devils: A long-term study of content polluters on twitter. Proceedings of the International AAAI Conference on Web and Social Media, Barcelona, Spain.
5. Arming the public with artificial intelligence to counter social bots10.1002/hbe2.115 ↗
6. Cresci, S., Di Pietro, R., Petrocchi, M., Spognardi, A., and Tesconi, M. (2017, January 3–7). The paradigm-shift of social spambots: Evidence, theories, and tools for the arms race. Proceedings of the WWW ’17: 26th International World Wide Web Conference, Perth, Australia.10.1145/3041021.3055135 ↗
7. Bessi, A., Coletto, M., Davidescu, G.A., Scala, A., Caldarelli, G., and Quattrociocchi, W. (2015). Science vs conspiracy: Collective narratives in the age of misinformation. PLoS ONE, 10.10.1371/journal.pone.0118093 ↗
8. Yang, K.C., Varol, O., Hui, P.M., and Menczer, F. (2020, January 7–12). Scalable and generalizable social bot detection through data selection. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i01.5460 ↗
9. Sayyadiharikandeh, M., Varol, O., Yang, K.C., Flammini, A., and Menczer, F. (2020, January 19–23). Detection of novel social bots by ensembles of specialized classifiers. Proceedings of the 29th ACM International Conference on Information & Knowledge Management, Virtual Event.10.1145/3340531.3412698 ↗
10. Fame for sale: Efficient detection of fake Twitter followers10.1016/j.dss.2015.09.003 ↗
11. Ratkiewicz, J., Conover, M., Meiss, M., Gonçalves, B., Flammini, A., and Menczer, F. (2011, January 17–21). Detecting and tracking political abuse in social media. Proceedings of the International AAAI Conference on Web and Social Media, Barcelona, Spain.
12. Bessi, A., and Ferrara, E. (2016). Social bots distort the 2016 US Presidential election online discussion. First Monday, 21.10.5210/fm.v21i11.7090 ↗
13. Detecting automation of twitter accounts: Are you a human, bot, or cyborg?10.1109/TDSC.2012.75 ↗
14. Weaponized health communication: Twitter bots and Russian trolls amplify the vaccine debate10.2105/AJPH.2018.304567 ↗
15. Chatzakou, D., Kourtellis, N., Blackburn, J., De Cristofaro, E., Stringhini, G., and Vakali, A. (2017, January 25–28). Mean birds: Detecting aggression and bullying on twitter. Proceedings of the 2017 ACM on Web Science Conference, Troy, NY, USA.10.1145/3091478.3091487 ↗
16. Marlow, T., Miller, S., and Roberts, J.T. (2021, October 13). Twitter Discourses on Climate Change: Exploring Topics and the Presence of Bots. Available online: https://osf.io/preprints/socarxiv/h6ktm/.10.31235/osf.io/h6ktm ↗
17. Young, V.A. (2021, October 01). Nearly Half of the Twitter Accounts Discussing ’Reopening America’ May Be Bots. Available online: https://www.cmu.edu/news/stories/archives/2020/may/twitter-bot-campaign.html.
18. Cresci, S., Lillo, F., Regoli, D., Tardelli, S., and Tesconi, M. (2018, January 25–28). $FAKE: Evidence of spam and bot activity in stock microblogs on Twitter. Proceedings of the International AAAI Conference on Web and Social Media, Stanford, CA, USA.10.1609/icwsm.v12i1.15073 ↗
19. Cashtag piggybacking: Uncovering spam and bot activity in stock microblogs on Twitter10.1145/3313184 ↗
20. Savage, S., Monroy-Hernandez, A., and Höllerer, T. (March, January 26). Botivist: Calling volunteers to action using online bots. Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, San Francisco, CA, USA.
21. Luckerson, V. (2021, October 01). Can Twitter Solve Its Big, Bad Bot Problem?. Available online: https://www.theringer.com/tech/2018/3/8/17093982/twitter-bot-problem.
22. A decade of social bot detection10.1145/3409116 ↗
23. (2021, October 01). Twitter Co-Founder Jack Dorsey Answers Twitter Questions from Twitter|Tech Support|WIRED. Available online: https://youtu.be/de8wRd2TQQU?t=99.
24. Conger, K. (2021, October 01). Twitter, in Widening Crackdown, Removes over 70,000 QAnon Accounts. Available online: https://www.nytimes.com/2021/01/11/technology/twitter-removes-70000-qanon-accounts.html.
25. Craig Timberg, E.D. (2021, October 01). Twitter Is Sweeping Out Fake Accounts Like Never before, Putting User Growth at Risk. Available online: https://www.washingtonpost.com/technology/2018/07/06/twitter-is-sweeping-out-fake-accounts-like-never-before-putting-user-growth-risk/.
26. Hutchinson, A. (2021, October 01). Twitter Says That Its Getting Better at Detecting and Removing Bots, Outlines Common Misinterpretations. Available online: https://www.socialmediatoday.com/news/twitter-says-that-its-getting-better-at-detecting-and-removing-bots-outlin/578272/.
27. Tsvetkova, M., García-Gavilanes, R., Floridi, L., and Yasseri, T. (2017). Even good bots fight: The case of Wikipedia. PLoS ONE, 12.10.1371/journal.pone.0171774 ↗
28. Varol, O., Ferrara, E., Davis, C., Menczer, F., and Flammini, A. (2017, January 15–18). Online human-bot interactions: Detection, estimation, and characterization. Proceedings of the International AAAI Conference on Web and Social Media, Montreal, QC, Canada.10.1609/icwsm.v11i1.14871 ↗
29. Bots increase exposure to negative and inflammatory content in online social systems10.1073/pnas.1803470115 ↗
30. Fake news on Twitter during the 2016 US presidential election10.1126/science.aau2706 ↗
31. Assessing the risks of ‘infodemics’ in response to COVID-19 epidemics10.1038/s41562-020-00994-6 ↗
32. Charting the landscape of online cryptocurrency manipulation10.1109/ACCESS.2020.3003370 ↗
33. The rise of social bots10.1145/2818717 ↗
34. Yardi, S., Romero, D., and Schoenebeck, G. (2010). Detecting spam in a twitter network. First Monday, 15.10.5210/fm.v15i1.2793 ↗
35. Lee, K., Caverlee, J., and Webb, S. (2010, January 19–23). Uncovering social spammers: Social honeypots+ machine learning. Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, Geneva, Switzerland.10.1145/1835449.1835522 ↗
36. Yang, C., Harkreader, R.C., and Gu, G. (2011). Die free or live hard? Empirical evaluation and new design for fighting evolving twitter spammers. International Workshop on Recent Advances in Intrusion Detection, Springer.10.1007/978-3-642-23644-0_17 ↗
37. Lin, P.C., and Huang, P.M. (2013, January 27–30). A study of effective features for detecting long-surviving Twitter spam accounts. Proceedings of the 2013 15th International Conference on Advanced Communications Technology (ICACT), PyeongChang, Korea.
38. Mccord, M., and Chuah, M. (2011). Spam detection on twitter using traditional classifiers. International Conference on Autonomic and Trusted Computing, Springer.10.1007/978-3-642-23496-5_13 ↗
39. Empirical evaluation and new design for fighting evolving twitter spammers10.1109/TIFS.2013.2267732 ↗
40. Cai, C., Li, L., and Zengi, D. (2017, January 22–24). Behavior enhanced deep bot detection in social media. Proceedings of the 2017 IEEE International Conference on Intelligence and Security Informatics (ISI), Beijing, China.10.1109/ISI.2017.8004887 ↗
41. A hybrid approach for detecting automated spammers in twitter10.1109/TIFS.2018.2825958 ↗
42. Davis, C.A., Varol, O., Ferrara, E., Flammini, A., and Menczer, F. (2016, January 11–15). Botornot: A system to evaluate social bots. Proceedings of the 25th International Conference Companion on World Wide Web, Montréal, QC, Canada.10.1145/2872518.2889302 ↗
43. Social bots: Human-like by means of human control?10.1089/big.2017.0044 ↗
44. The spread of low-credibility content by social bots10.1038/s41467-018-06930-7 ↗
45. Screening out social bots interference: Are there any silver bullets?10.1109/MCOM.2019.1800520 ↗
46. Cresci, S., Petrocchi, M., Spognardi, A., and Tognazzi, S. (July, January 30). Better safe than sorry: An adversarial approach to improve social bot detection. Proceedings of the 10th ACM Conference on Web Science, Boston, MA, USA.
47. Using improved conditional generative adversarial networks to detect social bots on Twitter10.1109/ACCESS.2020.2975630 ↗
48. Profiling online social behaviors for compromised account detection10.1109/TIFS.2015.2482465 ↗
49. Emergent properties, models, and laws of behavioral similarities within groups of Twitter users10.1016/j.comcom.2019.10.019 ↗
50. Chavoshi, N., Hamooni, H., and Mueen, A. (2016, January 12–15). Debot: Twitter Bot Detection via Warped Correlation. Proceedings of the 2016 IEEE 16th International Conference on Data Mining (ICDM), Barcelona, Spain.10.1109/ICDM.2016.0096 ↗
51. Mazza, M., Cresci, S., Avvenuti, M., Quattrociocchi, W., and Tesconi, M. (July, January 30). Rtbust: Exploiting temporal patterns for botnet detection on twitter. Proceedings of the 10th ACM Conference on Web Science, Boston, MA, USA.
52. Catching synchronized behaviors in large networks: A graph mining approach10.1145/2746403 ↗
53. Inferring lockstep behavior from connectivity pattern in large graphs10.1007/s10115-015-0883-y ↗
54. Hooi, B., Song, H.A., Beutel, A., Shah, N., Shin, K., and Faloutsos, C. (2016, January 13–17). Fraudar: Bounding graph fraud in the face of camouflage. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.10.1145/2939672.2939747 ↗
55. Grimme, C., Assenmacher, D., and Adam, L. (2018). Changing perspectives: Is it sufficient to detect social bots?. International Conference on Social Computing and Social Media, Springer.10.1007/978-3-319-91521-0_32 ↗
56. Echeverr a, J., De Cristofaro, E., Kourtellis, N., Leontiadis, I., Stringhini, G., and Zhou, S. (2018, January 3–7). LOBO: Evaluation of generalization deficiencies in Twitter bot classifiers. Proceedings of the 34th Annual Computer Security Applications Conference, San Juan, PR, USA.10.1145/3274694.3274738 ↗
57. Samper-Escalante, L.D., Loyola-González, O., Monroy, R., and Medina-Pérez, M.A. (2021). Bot Datasets on Twitter: Analysis and Challenges. Appl. Sci., 11.10.3390/app11094105 ↗
58. Kouvela, M., Dimitriadis, I., and Vakali, A. (, January 2–4). Bot-Detective: An explainable Twitter bot detection service with crowdsourcing functionalities. Proceedings of the 12th International Conference on Management of Digital EcoSystems, Abu Dhabi, United Arab Emirates.
59. Contrast pattern-based classification for bot detection on twitter10.1109/ACCESS.2019.2904220 ↗
60. Rauchfleisch, A., and Kaiser, J. (2020). The False positive problem of automatic bot detection in social science research. PLoS ONE, 15.10.2139/ssrn.3565233 ↗
61. Social fingerprinting: Detection of spambot groups through DNA-inspired behavioral modeling
62. Gilani, Z., Farahbakhsh, R., Tyson, G., Wang, L., and Crowcroft, J. (August, January 31). Of bots and humans (on twitter). Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2017, Sydney, Australia.10.1145/3110025.3110090 ↗
63. He, H., Bai, Y., Garcia, E.A., and Li, S. (2008, January 1–8). ADASYN: Adaptive synthetic sampling approach for imbalanced learning. Proceedings of the 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), Hong Kong, China.
64. Batista, G.E., Bazzan, A.L., and Monard, M.C. (2003, January 3–5). Balancing Training Data for Automated Annotation of Keywords: A Case Study. Proceedings of the II Brazilian Workshop on Bioinformatics, Macaé, Brazil.
65. A study of the behavior of several methods for balancing machine learning training data10.1145/1007730.1007735 ↗
66. Ribeiro, M.T., Singh, S., and Guestrin, C. (2016, January 13–17). “Why should i trust you?” Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.10.1145/2939672.2939778 ↗
67. Visani, G., Bagli, E., and Chesani, F. (2020). OptiLIME: Optimized LIME Explanations for Diagnostic Computer Algorithms. arXiv.
68. Alvarez-Melis, D., and Jaakkola, T.S. (2018). On the robustness of interpretability methods. arXiv.