Towards a fault-detection benchmark for evaluating software product line testing approaches

Software Product Lines (SPLs) are families of related software systems distinguished by the set of features each one provides. The commonly large number of variants that can be derived from an SPL poses a unique set of challenges, because it is not feasible to test all the individual variants. Over the last few years many approaches for SPL testing have been devised. They usually select a set of variants to test based on some covering criterion. A problem when evaluating these testing approaches is properly comparing them to one another. Even though some benchmarks have been proposed, they focus on covering criteria and do not consider fault data in their analysis. Considering the dire lack of publicly available fault data, in this paper we present the first results of our ongoing project to introduce simulated faults into SPLs along with using evolutionary techniques for synthesizing unit test cases for SPL examples.