Risk Assessment in AI System Engineering: Experiences and Lessons Learned from a Practitioner’s Perspective

Unlike the controlled conditions of AI system engineering laboratories, where adversarial vulnerabilities under specific threat models can be examined in isolation, in practical environments, such vulnerabilities are commonly intertwined with additional risks, including data or concept drift. In this paper, we explore the potential risks associated with the development and deployment of machine learning (ML) systems in real-world applications. We discuss secure ML engineering practices, their benefits, and their drawbacks and evaluate them based on their effectiveness in real-life use cases. Our study aims to provide a foundation for risk analysis and decision-making in practical ML applications where performance and security threats are highly intertwined.