טכניון מכון טכנולוגי לישראל
הטכניון מכון טכנולוגי לישראל - בית הספר ללימודי מוסמכים  
M.Sc Thesis
M.Sc StudentGeiger Omer
SubjectAlgorithmic Exam Generation
DepartmentDepartment of Computer Science
Supervisor Professor Shaul Markovitch
Full Thesis textFull thesis text - English Version


Abstract

Given a class of students, and a pool of questions in the domain of study, what subset will constitute a “good” exam?  Millions of educators are dealing with this difficult problem worldwide, yet the task of composing exams is still performed manually. In this work we present a novel algorithmic framework for exam composition. Our main formulation requires two input components: a student population represented by a distribution over a set of overlay models, each consisting of a set of mastered abilities, or actions; and a target model ordering that, given any two student models, defines which should be graded higher. To determine the performance of a student model on a potential question, we test whether it satisfies a disjunctive action landmark, i.e., whether its abilities are sufficient to follow at least one solution path. Based on these, we present a novel utility function for evaluating exams. An exam is highly evaluated if it is expected to order the student population with high correlation to the target order. In an alternative formulation we devised, the target ordering is replaced with a target grade mapping indicating the desired grade for each student. In this case, good exams are those for which the expected grades are close to those specified by the target mapping. The merit of our algorithmic framework is exemplified with real auto-generated questions in two domains: middle-school algebra and trigonometric equations.