Evaluating LLMs on Formal Models Coursework

Description

The use of Large Language Models is on the rise in educational
settings such as creating teaching material, helping with
assignments or solving exercises. At the same time these new
possibilities bring new challenges in regards to the understanding
of the capabilities of LLMs. To offer a glimpse into the current
capabilities of LLMs in regards to solving coursework, we developed
a set of questions from our Bachelor's course Formal Models, which
allows us to rate and compare the generated answers of LLMs. We
developed an evaluation template to rate the responses, which allows
us to not only compare the end result of given answers, but also
intermediate steps and understanding of given tasks. The evaluations
are presented on an interactive website, which allows us to offer
new insights and enables a faster response to newly released models.

Period	02 Aug 2025
Event title	ThEdu'25
Event type	Workshop
Location	Stuttgart, GermanyShow on map
Degree of Recognition	International