A Hybrid Approach to Software Testing Efficiency: Stacked Ensembles and Deep Q-Learning for Test Case Prioritization and Ranking

Test case prioritization and ranking play a crucial role in software testing by improving fault detection efficiency and ensuring software reliability. While prioritization selects the most relevant test cases for optimal coverage, ranking further refines their execution order to detect critical faults earlier. This study investigates machine learning techniques to enhance both prioritization and ranking, contributing to more effective and efficient testing processes. We first employ advanced feature engineering alongside ensemble models, including Gradient Boosted, Support Vector Machines, Random Forests, and Naive Bayes classifiers to optimize test case prioritization, achieving an accuracy score of and significantly improving the Average Percentage of Fault Detection (APFD). Subsequently, we introduce a deep Q-learning framework combined with a Genetic Algorithm (GA) to refine test case ranking within priority levels. This approach achieves a rank accuracy of , demonstrating robust performance despite the increasing computational demands of specialized variation operators. Our findings highlight the effectiveness of stacked ensemble learning and reinforcement learning in optimizing test case prioritization and ranking. This integrated approach improves testing efficiency, reduces late-stage defects, and improves overall software stability. The study provides valuable information for AI-driven testing frameworks, paving the way for more intelligent and adaptive software quality assurance methodologies.