M.Sc Thesis

M.Sc StudentMaien Dor-David
SubjectBetter Prediction of Mutation Score
DepartmentDepartment of Computer Science
Supervisor ASSOCIATE PROF. Joseph Gil
Full Thesis textFull thesis text - English Version


Mutation score is widely accepted to be a reliable measurement for the

effectiveness of software tests. Recent studies, however, show that mutation analysis is extremely costly and hard to use in practice. The research community invested significant efforts to reduce these costs; however, no practical and real-time solutions were proposed so far.

We present a novel direct prediction model of mutation score using neural networks. Relying solely on static code features that do not require generation of mutants or execution of the tests, we predict mutation score with an accuracy better than a quintile. When we include statement coverage as a feature, our accuracy rises to about a decile. Using a similar approach, we also gain a 4% improvement in the state-of-the-art results for binary test effectiveness prediction. Intending to propose a practical approach, we introduce an intuitive, easy-to-calculate set of features based on a simple analysis of the Code's AST. Our new set of features is superior to previously studied sets of features in its prediction performance, and it also significantly reduces the engineering trade-offs involved in the extraction process.

We publish the largest dataset of test-class level mutation score and static code features to date, for future research.

Finally, we discuss how our approach could be integrated into real-world systems, IDEs, CI tools, and testing frameworks and the implication of our findings for future studies in the field.