AbstractTrace: The Use of Execution Traces to Cluster, Classify, Prioritize, and Optimize a Bloated Test Suite
Source abstract
Abstract
Due to the incremental and iterative nature of the software testing process, a test suite may become bloated with redundant, overlapping, and similar test cases. This paper aims to optimize a bloated test suite by employing an execution trace that encodes runtime events into a sequence of characters forming a string. A dataset of strings, each of which represents the code coverage and execution behavior of a test case, is analyzed to identify similarities between test cases. This facilitates the de-bloating process by providing a formal mechanism to identify, remove, and reduce extra test cases without compromising software quality. This form of analysis allows for the clustering and classification of test cases based on their code coverage and similarity score. This paper explores three levels of execution traces and evaluates different techniques to measure their similarities. Test cases with the same code coverage should generate the exact string representation of runtime events. Various string similarity metrics are assessed to find the similarity score, which is used to classify, detect, and rank test cases accordingly. Additionally, this paper demonstrates the validity of the approach with two case studies. The first shows how to classify the execution behavior of various test cases, which can provide insight into each test case’s internal behavior. The second shows how to identify similar test cases based on their code coverage.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Software Engineering Research · Software Testing and Debugging Techniques · Software Reliability and Analysis Research
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Software Engineering, Faculty of Computer and Information Technology, Jordan University of Science and Technology (JUST), Irbid 22110, Jordan
Department of Computer Science and Engineering, New Mexico Institute of Mining and Technology, Socorro, NM 87801, USA
Funding
Jordan University of Science and Technology (JUST)
References (80)
Show source-provided references
1. Spillner, A., and Linz, T. (2021). Software Testing Foundations: A Study Guide for the Certified Tester Exam-Foundation Level-ISTQB® Compliant, Dpunkt Verlag.
2. Single and multi-objective test cases prioritization for self-driving cars in virtual environments10.1145/3533818 ↗
3. Design of Software Reliability Growth Model for Improving Accuracy in the Software Development Life Cycle (SDLC)
4. Iterative design and testing within the software development life cycle10.1023/A:1018528506161 ↗
5. Towards reliable online just-in-time software defect prediction10.1109/TSE.2022.3175789 ↗
6. Buckley, I.A., and Fernandez, E.B. (2023). Dependability Patterns: A Survey. Computers, 12.10.3390/computers12100214 ↗
7. A survey on different approaches for software test case prioritization
8. Relation-based test case prioritization for regression testing10.1016/j.jss.2020.110539 ↗
9. Runtime Monitoring of Software Execution Trace: Method and Tools10.1109/ACCESS.2020.3003087 ↗
10. Al-Sharif, Z.A., Abdalrahman, W.F., and Jeffery, C.L. (2021, January 24–26). Encoding Test Cases using Execution Traces. Proceedings of the 2021 12th International Conference on Information and Communication Systems (ICICS), Valencia, Spain.10.1109/ICICS52457.2021.9464569 ↗
11. Test case selection and prioritization using machine learning: A systematic literature review10.1007/s10664-021-10066-6 ↗
12. Almufareh, M.F., and Humayun, M. (2023). Improving the Safety and Security of Software Systems by Mediating SAP Verification. Appl. Sci., 13.10.3390/app13010647 ↗
13. Krichen, M. (2023). A survey on formal verification and validation techniques for internet of things. Appl. Sci., 13.10.3390/app13148122 ↗
14. Exploring better black-Box test case prioritization via log analysis
15. Regression test prioritization leveraging source code similarity with tree kernels10.1002/smr.2653 ↗
16. The Alamo execution monitor architecture10.1016/S1571-0661(05)80658-9 ↗
17. Jeffery, C., Mohamed, S., Al Gharaibeh, J., Pereda, R., and Parlett, R. (2024, November 19). Programming with Unicon, Available online: http://unicon.org/book/ub.pdf.
18. Jeffery, C.L. (2012). Program Monitoring and Visualization: An Exploratory Approach, Springer Science & Business Media.
19. Jeffery, C.L. (2024, February 26). Unicon: An Extended Dialect of Icon. Available online: http://unicon.org/.
20. Rahmani, A., Min, J.L., and Maspupah, A. (2020, January 18–19). An evaluation of code coverage adequacy in automatic testing using control flow graph visualization. Proceedings of the 2020 IEEE 10th Symposium on Computer Applications & Industrial Electronics (ISCAIE), Penang, Malaysia.10.1109/ISCAIE47305.2020.9108838 ↗
21. Hemmati, H. (2015, January 3–5). How effective are code coverage criteria?. Proceedings of the 2015 IEEE International Conference on Software Quality, Reliability and Security, Vancouver, BC, Canada.10.1109/QRS.2015.30 ↗
22. Jung, P., Kang, S., and Lee, J. (2022). Reducing redundant test executions in software product line testing—A case study. Electronics, 11.10.3390/electronics11071165 ↗
23. Reducing redundancy of test cases generation using code smell detection and refactoring
24. Ai4vis: Survey on artificial intelligence approaches for data visualization
25. Alawneh, L., and Hamou-Lhadj, A. (2009, January 10–12). Execution traces: A new domain that requires the creation of a standard metamodel. Proceedings of the Advances in Software Engineering: International Conference on Advanced Software Engineering and Its Applications, ASEA 2009 Held as Part of the Future Generation Information Technology Conference, FGIT 2009, Jeju Island, Republic of Korea. Proceedings.10.1007/978-3-642-10619-4_31 ↗
26. Jeffery, C.L. (2001, January 1). Goal-directed object-oriented programming in Unicon. Proceedings of the 2001 ACM Symposium on Applied Computing (SAC ’01), Las Vegas, NV, USA.10.1145/372202.372347 ↗
27. Griswold, R.E. (2024, February 26). The Icon Programming Language. Available online: https://www2.cs.arizona.edu/icon/.
28. Jeffery, C., Auguston, M., and Underwood, S. (2002, January 7–11). Towards fully automatic execution monitoring. Proceedings of the International Workshop on Radical Innovations of Software and Systems Engineering in the Future, Venice, Italy.
29. Auguston, M., Jeffery, C., and Underwood, S. (2002, January 23–27). A framework for automatic debugging. Proceedings of the Proceedings 17th IEEE International Conference on Automated Software Engineering, Edinburgh, UK.10.1109/ASE.2002.1115015 ↗
30. Al-Sharif, Z., and Jeffery, C. (2009, January 8–12). An extensible source-level debugger. Proceedings of the 2009 ACM Symposium on Applied Computing (SAC ’09), Honolulu, HI, USA.10.1145/1529282.1529397 ↗
31. Advanced and efficient execution trace management for executable domain-specific modeling languages10.1007/s10270-017-0598-5 ↗
32. Firmware code instrumentation technology for internet of things-based services10.1007/s11280-020-00838-3 ↗
33. NET/C# instrumentation for search-based software testing10.1007/s11219-023-09645-1 ↗
34. Armor: Protecting Software Against Hardware Tracing Techniques10.1109/TIFS.2024.3372816 ↗
35. Lopes, N.P. (2023, January 25–26). Torchy: A Tracing JIT Compiler for Pytorch. Proceedings of the 32nd ACM SIGPLAN International Conference on Compiler Construction (CC ’23), Montréal, QC, Canada.10.1145/3578360.3580266 ↗
36. Pekarek, D., and Mössenböck, H. (2020, January 4–6). Trcview: Interactive architecture agnostic execution trace analysis. Proceedings of the 17th International Conference on Managed Programming Languages and Runtimes (MPLR ’20), Virtual, UK.10.1145/3426182.3426190 ↗
37. NOD4J: Near-omniscient debugging tool for Java using size-limited execution trace10.1016/j.scico.2021.102630 ↗
38. Advanced Instrumentation Techniques in Java Applications
39. Mastrangelo, L., and Hauswirth, M. (2014, January 23–26). JNIF: Java native instrumentation framework. Proceedings of the 2014 International Conference on Principles and Practices of Programming on the Java Platform: Virtual Machines, Languages, and Tools (PPPJ ’14), Cracow, Poland.10.1145/2647508.2647516 ↗
40. Binder, W., Hulaas, J., and Moret, P. (2007, January 5–7). Advanced Java bytecode instrumentation. Proceedings of the 5th International Symposium on Principles and Practice of Programming in Java (PPPJ07), Lisboa, Portugal.10.1145/1294325.1294344 ↗
41. Discovering similarity inclusion dependencies10.1145/3588929 ↗
42. Combining Approximate String Matching Algorithms and Term Frequency In The Detection of Plagiarism
43. A normalized Levenshtein distance metric10.1109/TPAMI.2007.1078 ↗
44. Comparison of jaccard, dice, cosine similarity coefficient to find best fitness value for web retrieved documents using genetic algorithm
45. Hidayat, W., Utami, E., and Hartanto, A.D. (2020, January 24–25). Effect of Stemming Nazief & Adriani on the Ratcliff/Obershelp algorithm in identifying level of similarity between slang and formal words. Proceedings of the 2020 3rd International Conference on Information and Communications Technology (ICOIACT), Yogyakarta, Indonesia.10.1109/ICOIACT50329.2020.9331973 ↗
46. Comparative analysis of Software Development Life Cycle models (SDLC)
47. Bierig, R., Brown, S., Galván, E., and Timoney, J. (2021). Essentials of Software Testing, Cambridge University Press.10.1017/9781108974073 ↗
48. Test case prioritization approaches in regression testing: A systematic literature review10.1016/j.infsof.2017.08.014 ↗
49. Be more familiar with our enemies and pave the way forward: A review of the roles bugs played in software failures10.1016/j.jss.2017.06.069 ↗
50. Survey on software testing practices10.1049/iet-sen.2011.0066 ↗
51. Chen, Y., Li, P., Xu, J., Guo, S., Zhou, R., Zhang, Y., Wei, T., and Lu, L. (2020, January 18–21). Savior: Towards bug-driven hybrid testing. Proceedings of the 2020 IEEE Symposium on Security and Privacy (SP), San Francisco, CA, USA.10.1109/SP40000.2020.00002 ↗
52. Perera, A., Turhan, B., Aleti, A., and Böhme, M. (2021). How good does a Defect Predictor need to be to guide Search-Based Software Testing?. arXiv.10.1145/3324884.3416612 ↗
53. Shimari, K., Ishio, T., Kanda, T., and Inoue, K. (October, January 29). Near-omniscient debugging for java using size-limited execution trace. Proceedings of the 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME), Cleveland, OH, USA.10.1109/ICSME.2019.00068 ↗
54. Bertolino, A. (2007, January 23–25). Software testing research: Achievements, challenges, dreams. Proceedings of the Future of Software Engineering (FOSE’07), Minneapolis, MN, USA.10.1109/FOSE.2007.25 ↗
55. Hynninen, T., Kasurinen, J., Knutas, A., and Taipale, O. (2018, January 21–25). Software testing: Survey of the industry practices. Proceedings of the 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), Opatija, Croatia.10.23919/MIPRO.2018.8400261 ↗
56. Labuschagne, A., Inozemtseva, L., and Holmes, R. (2017, January 4–8). Measuring the cost of regression testing in practice: A study of Java projects using continuous integration. Proceedings of the Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE’17), Paderborn, Germany.10.1145/3106237.3106288 ↗
57. Ahamed, S. (2010). Studying the feasibility and importance of software testing: An Analysis. arXiv.
58. An overview of regression testing
59. Regression testing minimization, selection and prioritization: A survey10.1002/stvr.430 ↗
60. Regression testing in an industrial environment10.1145/274946.274960 ↗
61. Test case reduction techniques-survey
62. AL-Sharif, Z., and Jeffery, C. (2009, January 27–29). An Agent-Oriented Source-Level Debugger on Top of a Monitoring Framework. Proceedings of the 2009 Sixth International Conference on Information Technology: New Generations, Las Vegas, NV, USA.10.1109/ITNG.2009.305 ↗
63. Al-Sharif, Z.A., Jeffery, C.L., and Said, M.H. (2014, January 3–6). Debugging with dynamic temporal assertions. Proceedings of the 2014 IEEE International Symposium on Software Reliability Engineering Workshops, Naples, Italy.10.1109/ISSREW.2014.60 ↗
64. Al-Sharif, Z., and Jeffery, C.L. (2009, January 1–3). Language Support for Event-based Debugging. Proceedings of the 21st International Conference on Software Engineering and Knowledge Engineering (SEKE 2009), Boston, MA, USA.
65. Babaei, M., Bagherzadeh, M., and Dingel, J. (2020, January 16–23). Efficient reordering and replay of execution traces of distributed reactive systems in the context of model-driven development. Proceedings of the 23rd ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS ’20), Virtual Event.10.1145/3365438.3410939 ↗
66. Utting, M., Legeard, B., Dadeau, F., Tamagnan, F., and Bouquet, F. (2020, January 3–6). Identifying and generating missing tests using machine learning on execution traces. Proceedings of the 2020 IEEE International Conference On Artificial Intelligence Testing (AITest), Oxford, UK.10.1109/AITEST49225.2020.00020 ↗
67. Test case prioritization using bat algorithm10.2174/2213275912666190226154344 ↗
68. Test case prioritization to examine software for fault detection using PCA extraction and K-means clustering with ranking10.1007/s00500-020-05517-z ↗
69. Discrete and combinatorial gravitational search algorithms for test case prioritization and minimization
70. Messaoudi, S., Shin, D., Panichella, A., Bianculli, D., and Briand, L.C. (2021, January 11–17). Log-based slicing for system-level test cases. Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA ’21), Virtual Event.10.1145/3460319.3464824 ↗
71. Test case redundancy detection and removal using code coverage analysis
72. The application of machine learning in test case prioritization—A review
73. Test Case Reduction Using Data Mining Classifier Techniques10.17706/jsw.11.7.656-663 ↗
74. Enhancing test case reduction by k-means algorithm and elbow method
75. A survey on test case reduction techniques
76. LSTM-based deep learning for spatial–temporal software testing10.1007/s10619-020-07291-1 ↗
77. Zhou, J., Li, F., Dong, J., Zhang, H., and Hao, D. (2020, January 12–15). Cost-effective testing of a deep learning model through input reduction. Proceedings of the 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE), Coimbra, Portugal.10.1109/ISSRE5003.2020.00035 ↗
78. Machine learning applied to software testing: A systematic mapping study10.1109/TR.2019.2892517 ↗
79. Soto-Valero, C., Durieux, T., and Baudry, B. (2021, January 23–28). A longitudinal analysis of bloated java dependencies. Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), Athens, Greece.10.1145/3468264.3468589 ↗
80. Nagy, S., Nguyen-Tuong, A., Hiser, J.D., Davidson, J.W., and Hicks, M. (2021, January 15–19). Same Coverage, Less Bloat: Accelerating Binary-only Fuzzing with Coverage-preserving Coverage-guided Tracing. Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security (CCS ’21), Virtual Event.10.1145/3460120.3484787 ↗