Designing a Test Model for a Configurable System: An Exploratory Study of Preprocessor Directives and Feature Toggles

Testing is important in software development, but it has high cost. Thus, techniques to reduce the cost of software testing have been proposed. Model-based testing, one of such techniques, focuses on automatizing the generation of test cases. In the context of highly configurable systems, model-based testing must capture the system behavior and also encode the variability that exists among the variants. Previous research has shown promising results in applying model-based testing to configurable systems. Test models that encode variability into them directly improve the reasoning for faults from interactions. However, there is no study about the use of different variability mechanisms to encode variability in test models. In this paper, we investigate advantages and drawbacks of test model designs exploring the use of two variability mechanisms, namely preprocessor directives and feature toggles. The results are discussed in regard to run-time reasoning and re-configuration, alongside with metrics about complexity and maintainability. With this work, we contribute to the testing activity of highly configurable systems by providing engineers insights of comparing two well-known and widely used variability mechanisms, which can support informed decisions when choosing for which mechanisms to use for model-based testing.