Evaluation in Multimedia Recommender Systems: A Practical Guide

With the widespread availability of media technologies, such as real-time streaming, new IoT devices and smartphones, multimedia data are extensively increased and the big multimedia data are rapidly spreaded over various social networks. Thus, different multimedia recommender systems have been emerging to help users select the useful multimedia objects. However, due to distinct features of multimedia objects, it is difficult to conduct a proper evaluation for the multimedia recommender systems, and the evaluation from the general recommender systems might not be totally adopted to evaluate them. In this paper, we therefore review and analyze the evaluation criteria that are used in the previous multimedia recommender system papers. Based on the review, we propose a set of the practical advices to lead practitioners and researchers to perform evaluations for multimedia recommender systems.