טכניון מכון טכנולוגי לישראל
הטכניון מכון טכנולוגי לישראל - בית הספר ללימודי מוסמכים  
M.Sc Thesis
M.Sc StudentShay Hummel
SubjectEvaluating the Effectiveness of Search Engines Using
Minimal Relevance Judgments and Relevance Models
DepartmentDepartment of Industrial Engineering and Management
Supervisor Full Professor Kurland Oren
Full Thesis textFull thesis text - English Version


Abstract

Information retrieval is the task of retrieving data relevant to an information need from a collection of information sources. The evaluation of retrieval effectiveness focuses mainly on whether the retrieved data satisfies the information need. The evaluation process requires a significant amount of resources such as: a collection of information sources, a set of queries representing information needs, qualified judges, and standard measures considering all data in hand. All of these require a lot of time to produce. The increasing size of data collections has made the evaluation process highly challenging. The evaluation of automated search engines has been a subject of interest for many researchers for many years. Research in the area of evaluation of search engines focuses mainly on how to quantify user satisfaction with the retrieved data using minimal effort.


We present a novel approach to evaluating the effectiveness of search engines using minimal relevance judgments. Our approach is based on constructing relevance models using relevant documents. Discrepancy (or similarity) between a given ranked documents list and a reference list, constructed by using the relevance model, serves as an estimate for the quality of the original ranked list. Using experiments performed with TREC data we show that using our approach can reduce the number of required judgments for ranking the effectiveness of the systems by 99\%, while maintaining a level of up to 90\% ranking accuracy. We compare our results to those of state-of-the-art approaches and show that our method performs better than these approaches when using minimal relevance judgments.