Automated test reuse for highly configurable software

Dealing with highly configurable systems is generally very complex. Researchers and practitioners have conceived hundreds of different analysis techniques to deal with different aspects of configurable systems. One large focal point is the testing of configurable software. This is challenging due to the large number of possible configurations. Moreover, tests themselves are rarely configurable and instead built for specific configurations. However, existing tests need to be adapted to run on a different configuration. In this paper, we report on an experiment about automatically reusing existing tests in configurable systems. We used manually developed tests for specific configurations of three configurable systems and investigated how changing the configuration affects the tests. Subsequently, we employed an approach for automated reuse to generate new test variants (by reusing from existing ones) for combinations of previous configurations and compared their results to the ones from existing tests. Our results showed that we could directly reuse some tests for different configurations. Nonetheless, our automatically generated test variants generally yielded better results. Our generated tests had a higher or equal success rate to the existing tests in most cases. Even in the cases the success rate was equal, our generated tests generally had higher code coverage.