Performance Analysis of Maximal Risk Evaluation Formulas for Spectrum-Based Fault Localization
Source abstract
Abstract
The effectiveness analysis of risk evaluation formulas has become a significant research area in spectrum-based fault localization (SBFL). The risk evaluation formula is designed and widely used to evaluate the likelihood of a program spectrum to be faulty. There are numerous empirical and theoretical studies to investigate and compare the performance between sixty risk evaluation formulas. According to previous research, these sixty risk evaluation formulas together form a partially ordered set. Among them, nine formulas are maximal. These nine formulas can further be grouped into five maximal risk evaluation formula groups so that formulas in the same group have the same performance. Moreover, previous research showed that we cannot theoretically compare formulas across these five maximal formula groups. However, experimental data “suggests” that a maximal formula in one group could outperform another one (from a different group) more frequently, though not always. This inspired us to further investigate the performance between any two maximal formulas in different maximal formula groups. Our approach involves two major steps. First, we propose a new condition to compare between two different maximal formulas. Based on this new condition, we present five different scenarios under which a formula performs better than another. This is different from the condition suggested by the previous theoretical study. We performed an empirical study to compare different maximal formulas using our condition. Our results showed that among two maximal risk evaluation formulas, it is feasible to identify one that can outperform the others more frequently.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Software Testing and Debugging Techniques · Software Reliability and Analysis Research · Software Engineering Research
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Engineering, Northwestern Polytechnical University, Xi’an 710072, China
Department of Computer Science and Software Engineering, Swinburne University of Technology, Hawthorn, VIC 3122, Australia
School of Information Science, Zhejiang Sci-Tech University, Hangzhou 310018, China
Funding
National Natural Science Foundation of China
References (40)
Show source-provided references
1. An empirical investigation of program spectra10.1145/277633.277647 ↗
2. Jones, J.A., and Harrold, M.J. (2005, January 7–11). Empirical evaluation of the tarantula automatic fault-localization technique. Proceedings of the 20th IEEE/ACM international Conference on Automated Software Engineering, Long Beach, CA, USA.10.1145/1101908.1101949 ↗
3. A survey on software fault localization10.1109/TSE.2016.2521368 ↗
4. Abreu, R., Zoeteweij, P., and Van Gemund, A.J. (2007, January 10–14). On the accuracy of spectrum-based fault localization. Proceedings of the Testing: Academic and Industrial Conference Practice and Research Techniques- MUTATION (TAICPART-MUTATION 2007), Windsor, UK.10.1109/TAICPART.2007.4344104 ↗
5. Chen, Y., Probert, R.L., and Sims, D.P. (October, January 30). Specification-based regression test selection with risk analysis. Proceedings of the 2002 Conference of the Centre for Advanced Studies on Collaborative Research, Toronto, ON, Canada.
6. Jones, J.A., Harrold, M.J., and Stasko, J.T. (2001, January 13–14). Visualization for fault localization. Proceedings of the ICSE 2001 Workshop on Software Visualization, Toronto, ON, Canada.
7. Abreu, R., Zoeteweij, P., and Van Gemund, A.J. (2006, January 18–20). An evaluation of similarity coefficients for software fault localization. Proceedings of the 2006 12th Pacific Rim International Symposium on Dependable Computing (PRDC’06), Riverside, CA, USA.10.1109/PRDC.2006.18 ↗
8. A practical evaluation of spectrum-based fault localization10.1016/j.jss.2009.06.035 ↗
9. Wong, W.E., Qi, Y., Zhao, L., and Cai, K.Y. (2007, January 24–27). Effective fault localization using code coverage. Proceedings of the 31st Annual International Computer Software and Applications Conference (COMPSAC 2007), Beijing, China.10.1109/COMPSAC.2007.109 ↗
10. Pearson, S., Campos, J., Just, R., Fraser, G., Abreu, R., Ernst, M.D., Pang, D., and Keller, B. (2017, January 20–28). Evaluating and improving fault localization. Proceedings of the 39th International Conference on Software Engineering, Buenos Aires, Argentina.10.1109/ICSE.2017.62 ↗
11. Xie, X. (2012). On the Analysis of Spectrum-Based Fault Localization. [Ph.D. Thesis, Swinburne University of Technology].
12. Lee, H.J., Naish, L., and Ramamohanarao, K. (2009, January 8–11). Study of the relationship of bug consistency with respect to performance of spectra metrics. Proceedings of the 2009 2nd IEEE International Conference on Computer Science and Information Technology, Beijing, China.
13. A model for spectra-based software diagnosis10.1145/2000791.2000795 ↗
14. A theoretical analysis of the risk evaluation formulas for spectrum-based fault localization10.1145/2522920.2522924 ↗
15. Chen, T.Y., Xie, X., Kuo, F.C., and Xu, B. (2015, January 1–5). A revisit of a theoretical analysis on spectrum-based fault localization. Proceedings of the 2015 IEEE 39th Annual Computer Software and Applications Conference, Taichung, Taiwan.10.1109/COMPSAC.2015.196 ↗
16. Xie, X., Kuo, F.C., Chen, T.Y., Yoo, S., and Harman, M. (2013). Provably optimal and human-competitive results in sbse for spectrum based fault localisation. International Symposium on Search Based Software Engineering, Springer.10.1007/978-3-642-39742-4_17 ↗
17. Human competitiveness of genetic programming in spectrum-based fault localisation: Theoretical and empirical analysis10.1145/3078840 ↗
18. No pot of gold at the end of program spectrum rainbow: Greatest risk evaluation formula does not exist
19. Smart debugging software architectural design in SDL10.1016/j.jss.2004.06.026 ↗
20. An empirical investigation of the relationship between spectra differences and regression faults10.1002/1099-1689(200009)10:3<171::AID-STVR209>3.0.CO;2-J ↗
21. Agrawal, H., Horgan, J.R., London, S., and Wong, W.E. (1995, January 24–27). Fault localization using execution slices and dataflow tests. Proceedings of the Sixth International Symposium on Software Reliability Engineering, ISSRE’95, Toulouse, France.
22. Effective program debugging based on execution slices and inter-block data dependency10.1016/j.jss.2005.06.045 ↗
23. Jones, J.A., Harrold, M.J., and Stasko, J. (2002, January 25). Visualization of test information to assist fault localization. Proceedings of the 24th International Conference on Software Engineering, ICSE 2002, Orlando, FL, USA.10.1145/581396.581397 ↗
24. Chen, M.Y., Kiciman, E., Fratkin, E., Fox, A., and Brewer, E. (2002, January 23–26). Pinpoint: Problem determination in large, dynamic internet services. Proceedings of the International Conference on Dependable Systems and Networks, Washington, DC, USA.
25. A family of code coverage-based heuristics for effective fault localization10.1016/j.jss.2009.09.037 ↗
26. Wong, E., Wei, T., Qi, Y., and Zhao, L. (2008, January 9–11). A crosstab-based statistical method for effective fault localization. Proceedings of the 2008 1st International Conference on Software Testing, Verification, and Validation, Lillehammer, Norway.10.1109/ICST.2008.65 ↗
27. Yoo, S. (2012). Evolving human competitive spectra-based fault localisation techniques. International Symposium on Search Based Software Engineering, Springer.10.1007/978-3-642-33119-0_18 ↗
28. On habitat and association of species of anopheline larvae in south-eastern Madras
29. (2019, April 15). SLOCCOUNT. Available online: http://www.dwheeler.com/sloccount/sloccount.html.
30. Slice-based statistical fault localization10.1016/j.jss.2013.08.031 ↗
31. Exploring the usefulness of unlabelled test cases in software fault localization10.1016/j.jss.2017.07.027 ↗
32. Robustness of spectrum-based fault localisation in environments with labelling perturbations10.1016/j.jss.2018.09.091 ↗
33. (2019, April 15). SIR. Available online: https://sir.csc.ncsu.edu/portal/index.ph.
34. Xie, X., Wong, W.E., Chen, T.Y., and Xu, B. (2011, January 13–14). Spectrum-based fault localization: Testing oracles are no longer mandatory. Proceedings of the 2011 11th International Conference on Quality Software, Madrid, Spain.10.1109/QSIC.2011.20 ↗
35. Metamorphic slice: An application in spectrum-based fault localization10.1016/j.infsof.2012.08.008 ↗
36. Jiang, B., Zhang, Z., Tse, T., and Chen, T.Y. (2009, January 20–24). How well do test case prioritization techniques support statistical fault localization. Proceedings of the 2009 33rd Annual IEEE International Computer Software and Applications Conference, Seattle, WA, USA.10.1109/COMPSAC.2009.23 ↗
37. Jiang, B., and Chan, W. (2010, January 14–15). On the integration of test adequacy, test case prioritization, and statistical fault localization. Proceedings of the 2010 10th International Conference on Quality Software, Zhangjiajie, China.10.1109/QSIC.2010.64 ↗
38. Jiang, B., Chan, W., and Tse, T. (2011, January 13–14). On practical adequate test suites for integrated test case prioritization and fault localization. Proceedings of the 2011 11th International Conference on Quality Software, Madrid, Spain.10.1109/QSIC.2011.37 ↗
39. Fault localization prioritization: Comparing information-theoretic and coverage-based approaches10.1145/2491509.2491513 ↗
40. Yu, Y., Jones, J., and Harrold, M.J. (2008, January 10–18). An empirical study of the effects of test-suite reduction on fault localization. Proceedings of the 2008 ACM/IEEE 30th International Conference on Software Engineering, Leipzig, Germany.