Monitoring automatically gained difficulty rankings with mathematics educational theories and experts

Automatically difficulty-ranked tasks would benefit technology-enhanced learning in mathematics, opening adaptive testing for a broader audience. How to achieve this goal in a resource-saving way and guarantee high-ranking quality? This paper follows a community approach for calibration based on the Elo-Rating-System and seeks an instrument to monitor gained task difficulty rankings automatically. Thus, rankings of 18 Algebra-tasks, elaborated following Bloom’s Revised Taxonomy, Webb’s DOK Framework, and Smith & Stein’s LCD, are compared to 5 expert rankings and contrasted to empirical solution frequencies from 64 students in grades 11 and 12. A mixed methods approach will guide the decision for a monitoring instrument for the automatic calibration process implemented in an open test- and trainings-platform based on the GeoGebra classroom containing final exam topics, providing formative assessment and sustaining bridge courses in the STEM fields.