Risk-Based Test Case Prioritization Using Large Language Models in Regression Testing

Abstract

Regression testing is critical to ensuring software quality after performing code modifications. However, complete test execution on complex and robust test suites can be infeasible due to time and resource constraints. Therefore, test case prioritization (TCP) strategies aim to organize test cases to increase fault detection rates early during test execution. This study proposes a risk-based test case prioritization approach that leverages large language models (LLMs) to estimate the fault-proneness of individual methods to guide the prior- itization process. An LLM is fine-tuned to predict the risk score of each function based on several software metrics, which is used to perform static analysis of test cases to determine an overall risk ranking. The prioritized test suites are evaluated using established metrics, including Fault Detection Rate (FDR) and Average Percentage of Faults Detected (APFD). The evaluation of this approach is compared against baseline techniques such as coverage-based and randomized prioritization. The results of this experiment, conducted on open-source Java projects, determined that the risk-based LLM prioritization approach outperforms traditional TCP methods in early fault detection, highlighting the potential of including LLMs in regression testing workflows.