Breaking Barriers in HW-to-SW Stack Verification

The ever-increasing complexity of embedded Hardware (HW)/Software (SW)
systems poses manifold design as well as verification challenges when developing these systems. One way to cope with this demand is the so-called shiftleft strategy. The idea is to shift SW development of the otherwise sequential
HW and SW development to the left to concurrently develop both parts. To enable this strategy, so-called Virtual Prototypes (VPs) are heavily utilized, allowing early design space exploration and execution of the SW. While this approach has shown to be beneficial for HW/SW co-design, the verification still lacks this holistic system view. With the increasing complexity of the full HW-to-SW stack verification, generating reference outputs for tests becomes increasingly unfeasible, leading to the well-known oracle problem. Additionally, the analysis of the full HW-to-SW stack is mostly considered with isolated HW and isolated SW analysis.
This thesis aims to overcome the barriers in HW-to-SW stack verification,
specifically those posed by the oracle problem and isolated analysis. The first
part of this thesis utilizes Metamorphic Testing (MT) to address the oracle problem.
By employing MT, the need for golden reference outputs is eliminated,
effectively resolving the oracle problem. This is done by comparing the relations
between inputs and outputs of multiple executions, rather than comparing
the outputs to expected reference outputs. This MT methodology is applied
to embedded graphics libraries. Leveraging VPs, an automated test framework
is implemented. While the test execution is fully automated, deriving the relations still is a manual task. To solve this, Large Language Models (LLMs) are
utilized to generate the relations. Overall, numerous previously unknown bugs
have been identified with this approach. In addition, a high structural coverage
was achieved.
The second part of this thesis aims to break the isolated analysis barriers in
HW-to-SW stack verification by perceiving the HW and SW as one holistic
system. A new methodology based on the Host-to-SW memory hierarchy and
dynamic runtime instrumentation is derived to holistically trace the system.
Based on this, a unified HW/SW coverage is formalized, and a new paradigm
called relation coverage is introduced. The experiments done with this new
paradigm show that relation coverage is able to uncover previously unknown
bugs that are otherwise hard to identify. Next, to determine the root causes of
the identified bugs, the intricate HW/SW interactions of such systems are visualized by introducing Function Lifetime Diagram (FLD). In an extensive evaluation, the root causes of the bugs, as identified by relation coverage, have
been found. An additional evaluation of the HW/SW interactions of a Neural
Network (NN) for handwritten number recognition further shows the applicability in more complex embedded systems. To further support the design of such complex systems, the methodology is extended to support the analysis of
Domain Specific Architectures (DSAs). This enables more sophisticated HW/SW
partitioning choices for optimal system performance.