Supporting High-Level to Low-Level Requirements Coverage Reviewing with Large Language Models

Description

Refining high-level requirements into low-level ones is a common task, especially in safety-critical systems engineering. The objective is to describe every important aspect of the high-level requirement in a low-level requirement, ensuring a complete and correct im- plementation of the system’s features. To this end, standards and regulations for safety-critical systems require reviewing the cover- age of high-level requirements by all its low-level requirements to ensure no missing aspects. The challenge of supporting automatic reviews for requirements coverage originates from the distinct levels of abstraction between high-level and low-level requirements, their reliance on natural language, and the often different vocabulary used. The rise of Large Language Models (LLMs), trained on extensive text corpora and ca- pable of contextualizing both high-level and low-level requirements, opens new avenues for addressing this challenge. This paper presents an initial study to explore the performance of LLMs in assessing requirements coverage. We employed GPT-3.5 and GPT-4 to analyze requirements from five publicly accessible data sets, determining their ability to detect if low-level require- ments sufficiently address the corresponding high-level require- ment. Our findings reveal that GPT-3.5, utilizing a zero-shot prompt- ing strategy augmented with the prompt of explaining, correctly identifies complete coverage in four out of five evaluation data sets. Additionally, it exhibits an impressive 99.7% recall rate in ac- curately identifying instances where coverage is incomplete due to removing a single low-level requirement across our entire set of evaluation data.

Period	16 Apr 2024
Event title	21st International Conference on Mining Software Repositories (MSR ’24), April , 2024, Lisbon, Portugal.
Event type	Conference
Location	PortugalShow on map