A Multimodal Deep Learning Model Using Text, Image, and Code Data for Improving Issue Classification Tasks
Source abstract
Abstract
Issue reports are valuable resources for the continuous maintenance and improvement of software. Managing issue reports requires a significant effort from developers. To address this problem, many researchers have proposed automated techniques for classifying issue reports. However, those techniques fall short of yielding reasonable classification accuracy. We notice that those techniques rely on text-based unimodal models. In this paper, we propose a novel multimodal model-based classification technique to use heterogeneous information in issue reports for issue classification. The proposed technique combines information from text, images, and code of issue reports. To evaluate the proposed technique, we conduct experiments with four different projects. The experiments compare the performance of the proposed technique with text-based unimodal models. Our experimental results show that the proposed technique achieves a 5.07% to 14.12% higher F1-score than the text-based unimodal models. Our findings demonstrate that utilizing heterogeneous data of issue reports helps improve the performance of issue classification.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Software Engineering Research · Advanced Malware Detection Techniques · Software Engineering Techniques and Practices
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of AI Convergence Engineering, Gyeongsang National University, 501 Jinjudaero, Jinju-si 52828, Gyeongsangnam-do, Republic of Korea
Department of Aerospace and Software Engineering, Gyeongsang National University, 501 Jinjudaero, Jinju-si 52828, Gyeongsangnam-do, Republic of Korea
Funding
Ministry of Education and National Research Foundation of Korea · Ministry of Education
References (46)
Show source-provided references
1. Automated classification of software issue reports using machine learning techniques: An empirical study10.1007/s11334-017-0294-1 ↗
2. Fan, Q., Yu, Y., Yin, G., Wang, T., and Wang, H. (2017, January 9–10). Where is the road for issue reports classification based on text mining?. Proceedings of the 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), Toronto, ON, Canada.10.1109/ESEM.2017.19 ↗
3. Automated classification of issue reports from a software issue tracker
4. Panichella, A. (September, January 31). A systematic comparison of search algorithms for topic modelling—A study on duplicate bug report identification. Proceedings of the Search-Based Software Engineering: 11th International Symposium, SSBSE 2019, Tallinn, Estonia. Proceedings 11.10.1007/978-3-030-27455-9_2 ↗
5. Lu, M., and Liang, P. (2017, January 15–16). Automatic classification of non-functional requirements from augmented app user reviews. Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering, Karlskrona Sweden.10.1145/3084226.3084241 ↗
6. Zhu, Y., Pan, M., Pei, Y., and Zhang, T. (2019). A bug or a suggestion? an automatic way to label issues. arXiv.
7. Kallis, R., Di Sorbo, A., Canfora, G., and Panichella, S. (October, January 29). Ticket tagger: Machine learning driven issue classification. Proceedings of the 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME), Cleveland, OH, USA.10.1109/ICSME.2019.00070 ↗
8. Kim, M., Kim, Y., and Lee, E. (2022, January 22–24). Deep learning-based production and test bug report classification using source files. Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings, Pittsburgh, PA, USA.10.1145/3510454.3528646 ↗
9. Zhifang, L., Kun, W., Qi, Z., Shengzong, L., Yan, Z., and Jianbiao, H. (2022). Classification of open source software bug report based on transfer learning. Expert Syst., e13184.10.1111/exsy.13184 ↗
10. Classifying issue reports according to feature descriptions in a user manual based on a deep learning model10.1016/j.infsof.2021.106743 ↗
11. Antoniol, G., Ayari, K., Di Penta, M., Khomh, F., and Guéhéneuc, Y.G. (2008, January 27–30). Is it a bug or an enhancement? A text-based approach to classify change requests. Proceedings of the 2008 Conference of the Center for Advanced Studies on Collaborative Research: Meeting of Minds, Richmond Hill, ON, Canada.10.1145/1463788.1463819 ↗
12. Herzig, K., Just, S., and Zeller, A. (2013, January 18–26). It’s not a bug, it’s a feature: How misclassification impacts bug prediction. Proceedings of the 2013 35th International Conference on Software Engineering (ICSE), San Francisco, CA, USA.10.1109/ICSE.2013.6606585 ↗
13. Combining text mining and data mining for bug report classification10.1002/smr.1770 ↗
14. Network-clustered multi-modal bug localization10.1109/TSE.2018.2810892 ↗
15. Chakraborty, S., and Ray, B. (2021, January 15–19). On multi-modal learning of editing source code. Proceedings of the 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), Melbourne, Australia.10.1109/ASE51524.2021.9678559 ↗
16. Zhang, W., Zhao, J., and Wang, S. (2021, January 14–17). SusTriage: Sustainable Bug Triage with Multi-modal Ensemble Learning. Proceedings of the IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, Melbourne, Australia.10.1145/3498851.3498993 ↗
17. Summaira, J., Li, X., Shoib, A.M., Li, S., and Abdul, J. (2021). Recent advances and trends in multimodal deep learning: A review. arXiv.
18. Yu, L., Poirson, P., Yang, S., Berg, A.C., and Berg, T.L. (2016, January 11–14). Modeling context in referring expressions. Proceedings of the Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands. Proceedings, Part II 14.10.1007/978-3-319-46475-6_5 ↗
19. Plummer, B.A., Wang, L., Cervantes, C.M., Caicedo, J.C., Hockenmaier, J., and Lazebnik, S. (2015, January 7–13). Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.303 ↗
20. Johnson, J., Karpathy, A., and Fei-Fei, L. (2016, January 27–30). Densecap: Fully convolutional localization networks for dense captioning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.494 ↗
21. Xu, J., Mei, T., Yao, T., and Rui, Y. (2016, January 27–30). Msr-vtt: A large video description dataset for bridging video and language. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.571 ↗
22. You, Q., Jin, H., Wang, Z., Fang, C., and Luo, J. (2016, January 27–30). Image captioning with semantic attention. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.503 ↗
23. Yan, X., Yang, J., Sohn, K., and Lee, H. (2016, January 11–14). Attribute2image: Conditional image generation from visual attributes. Proceedings of the Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands. Proceedings, Part IV 14.10.1007/978-3-319-46493-0_47 ↗
24. Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., and Lee, H. (2016, January 19–24). Generative adversarial text to image synthesis. Proceedings of the International Conference on Machine Learning, New York City, NY, USA.
25. Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., and He, X. (2018, January 18–23). Attngan: Fine-grained text to image generation with attentional generative adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00143 ↗
26. Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., and Parikh, D. (2015, January 7–13). Vqa: Visual question answering. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.279 ↗
27. Ben-Younes, H., Cadene, R., Cord, M., and Thome, N. (2017, January 22–29). Mutan: Multimodal tucker fusion for visual question answering. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.285 ↗
28. Fukui, A., Park, D.H., Yang, D., Rohrbach, A., Darrell, T., and Rohrbach, M. (2016). Multimodal compact bilinear pooling for visual question answering and visual grounding. arXiv.10.18653/v1/D16-1044 ↗
29. Kim, J.H., On, K.W., Lim, W., Kim, J., Ha, J.W., and Zhang, B.T. (2016). Hadamard product for low-rank bilinear pooling. arXiv.
30. Liang, W., Jiang, Y., and Liu, Z. (2021). GraghVQA: Language-guided graph neural networks for graph-based visual question answering. arXiv.10.18653/v1/2021.maiworkshop-1.12 ↗
31. Multi-modal Deep Learning Approach for Flood Detection
32. Kim, E., and McCoy, K.F. (2018, January 22–24). Multimodal deep learning using images and text for information graphic classification. Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility, Galway, Ireland.10.1145/3234695.3236357 ↗
33. Audebert, N., Herold, C., Slimani, K., and Vidal, C. (2019, January 16–20). Multimodal deep networks for text and image-based document classification. Proceedings of the Machine Learning and Knowledge Discovery in Databases: International Workshops of ECML PKDD 2019, Würzburg, Germany. Proceedings, Part I.10.1007/978-3-030-43823-4_35 ↗
34. CB-Fake: A multimodal deep learning framework for automatic fake news detection using capsule neural network and BERT10.1007/s11042-021-11782-3 ↗
35. Multimodal deep learning for music genre classification
36. Fusion of facial expressions and EEG for multimodal emotion recognition10.1155/2017/2107451 ↗
37. Tripathi, S., Tripathi, S., and Beigi, H. (2018). Multi-modal emotion recognition on iemocap dataset using deep learning. arXiv.
38. IEMOCAP: Interactive emotional dyadic motion capture database10.1007/s10579-008-9076-6 ↗
39. Sarica, S., and Luo, J. (2021). Stopwords in technical language processing. PLoS ONE, 16.10.1371/journal.pone.0254937 ↗
40. Multimodal intelligence: Representation learning, information fusion, and applications10.1109/JSTSP.2020.2987728 ↗
41. Imbalance class problems in data mining: A review
42. Data imbalance in classification: Experimental evaluation10.1016/j.ins.2019.11.004 ↗
43. Arcuri, A., and Briand, L. (2011, January 21–28). A practical guide for using statistical tests to assess randomized algorithms in software engineering. Proceedings of the 33rd International Conference on Software Engineering, Honolulu, HI, USA.10.1145/1985793.1985795 ↗
44. Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., and Jiang, D. (2020). Codebert: A pre-trained model for programming and natural languages. arXiv.10.18653/v1/2020.findings-emnlp.139 ↗
45. Yu, Z., Yu, J., Fan, J., and Tao, D. (2017, January 22–29). Multi-modal factorized bilinear pooling with co-attention learning for visual question answering. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.202 ↗
46. Jiang, S., Hu, J., Magee, C.L., and Luo, J. (2022). Deep learning for technical document classification. IEEE Trans. Eng. Manag., 1–17.10.1109/TEM.2022.3152216 ↗