Implementing the Perturbation Approach for Reliability Assessment: A Case Study in the Context of Flight Delay Prediction

Organizations employ prediction models as a foundation for decision-making. A prediction model learned from training data is often only evaluated using global quality indicators, e.g., accuracy and precision. These global indicators, however, do not provide guidance regarding the reliability of the prediction for a specific input case. In this paper, we instantiate a generic reference process for implementing reliability assessment methods for specific input cases on the real-world use case of flight delay prediction. We specifically implement the perturbation approach to reliability assessment for this use case and then describe the steps that were taken to train the prediction model, with an emphasis on the activities required to implement the perturbation approach. The perturbation approach consists of slightly altering feature values for an individual input case, e.g., within the margins of error of a sensed value, and observe whether the prediction of the model changes, which would render the prediction unreliable. The implementation of the perturbation approach requires decisions and documentations along the various stages of the data mining process. A generic tool can be used to document and perform reliability assessment using the perturbation approach.