A Benchmark Infrastructure for Fault Characterization in Combinatorial Testing

Efficient Automated Fault Localization for Combinatorial Testing

Combinatorial testing has shown to be a promising technique for detecting software faults in the past. Most of the research in this area has been about finding ways to generate fewer test cases while still covering all combinations of input parameters up to a certain testing strength. But this reduction in test cases also has some disadvantages in practice. After the execution of a combinatorial test suite, the detected failures still need to be mapped to errors in the source code. This can be hard, because every failed test case may contain many possible causes. The process to automatically localize the real failure inducing combinations from a combinatorial test suite is called fault characterization. Over the last years, there have been several developments in this area. New algorithms, which follow different approaches in how to find the real causes, have been presented, and some superficial comparisons between them have been done. Fault characterization algorithms usually make some assumptions about the underlying system and, therefore, are often not broadly applicable. Because there is no comprehensive evaluation and comparison of current fault characterization algorithms, it is hard to choose what to use in practice. This thesis now presents the design of a benchmark infrastructure, which can be used for such an evaluation. It is implemented in the form of a prototype application written in Java. Finally, this infrastructure is used to evaluate and compare some selected fault characterization algorithms.