A Case Study on Testing, Commissioning, and Operation of Very-Large-Scale Software Systems

An increasing number of software systems today are very-large-scale software systems (VLSS) with system-of-systems (SoS) architectures. Due to their heterogeneity and complexity VLSS are difficult to understand and analyze, which results in various challenges for development and evolution. Existing software engineering processes, methods, and tools do not sufficiently address the characteristics of VLSS. Also, there are only a few empirical studies on software engineering for VLSS. We report on results of an exploratory case study involving engineers and technical project managers of an industrial automation VLSS for metallurgical plants. The paper provides empirical evidence on how VLSS are tested, commissioned, and operated in practice. The paper discusses practical challenges and reports industrial requirements regarding process and tool support. In particular, software processes and tools need to provide general guidance at the VLSS level as well as specific methods and tools for systems that are part of the VLSS. Processes and tools need to support multi-disciplinary engineering across system boundaries. Furthermore, managing variability and evolution is success-critical in VLSS verification and validation.