Automated Data Quality Monitoring

Most existing methodologies agree that the assessment of data quality (DQ) is a cyclic process, which has to be carried out continuously. Nevertheless, the majority of DQ tools allow the evaluation of data sources only at specific points in time, and the automation and scheduling is therefore in the responsibility of the user. In contrast, automated DQ monitoring allows the evaluation of applied DQ improvements as well as the comparability between different system states. The reproducibility of DQ assessments is also an important topic for the scientific community in order to review algorithms that improve the DQ of an information system. We are developing a tool for DQ monitoring and our research covers the investigation of suitable DQ metrics for continuous monitoring as well as the development of a standardized approach to storing DQ assessment results over time. In addition, statistical methods to analyze and visualize the resulting time series data are selected and applied.