A Framework for Cost-aware Combinatorial Testing

Combinatorial testing is a technique to detect faults caused by interactions of several parameters. Most algorithms that generate test suites for combinatorial testing are developed with the intent to minimize the number of test cases. Due to this, the generated suites are relatively small, e. g., compared to exhaustive testing. Having small test suites is a general approach to reduce execution costs. But this approach has limitations as it stems from the assumption that all test cases have the same cost. In a real life testing setting, this is not always the case. Not only execution time can impact tests cost, there are other aspects, such as tests reuse, tests order, parameter-value configurations costs, etc. Considering those aspects is the active combinatorial testing research area. Different approaches that generate tests with real life costs awareness are developed. They are usually designed for specific types of costs and many of them are not available to end users. This is why software where those approaches can be integrated is a necessity. Having that software allows to automate process of cost-aware tests generations and provide a generation mechanism tailored towards user’s needs. The concept of such software was designed. As a proof of concept, the software was implemented and several cost-aware generation approaches were integrated in it. The concept and implementation were evaluated and future research directions were identified.