Automating test reuse for highly configurable software: An Experiment

Dealing with highly configurable systems is generally very complex. Hundreds of different analysis techniques have been conceived to deal with different aspects of configurable systems. One large focal point is the testing of configurable software. This is challenging due to the large number of possible configurations and because tests themselves are rarely configurable and instead built for specific configurations. Existing tests can usually not be reused on other configurations. Therefore, tests need to be adapted for the specific configuration they are supposed to test. In this paper we report on an experiment about reusing tests in a configurable system. We used manually developed tests for specific configurations of Bugzilla and investigated which of them could be reused for other configurations. Moreover, we automatically generated new test variants (by automatically reusing from existing ones) for combinations of previous configurations. Our results showed that we can directly reuse some tests for configurations which they were not intended for. Nonetheless, our automatically generated test variants generally yielded better results. When applying original tests to new configurations we found an average success rate for the tests of 81,84%. In contrast, our generated test variants achieved an average success rate of 98,72%. This is an increase of 16,88%.