Are CISQ Reliability Measures Practical? A Research Perspective (Best Paper Award)

The Object Management Group (OMG), which is driven by industry, proposes an operational standard for measuring reliability by providing specifications for 29 reliability measures. The goal of this article is to systematically assess whether (1) the provided measurement specifications are suitable to be implemented in a static testing tool used in practice and (2) the measures are suitable for capturing reliability issues in software. Therefore, we implemented the CISQ measures (CISQ is the consortium responsible for driving the quality related topics of the OMG) for Java in our quality measurement tool after defining assumptions resulting from the language independent and at some point imprecise specifications. In format of a case study, the CISQ-based measurement tool has then been applied on several versions of the open source project HSQLDB. The results show that CISQ measures properties that are vital for fulfilling reliability requirements. In the course of the case study, the engineers of HSQLDB fixed a number of issues identified by our tooling as they were considered to be critical. While a number of rule violations are considered to be still problematic, they could not be fixed since the engineers did not have the code ownership. In these cases, they proposed improvement suggestions to the responsible teams.