Comparing Offline and Online Evaluation Results of Recommender Systems
Authors
Kordík, P.; Řehořek, T.; Bíža, O.; Bartyzal, R.; Podsztavek, O.; Povalyev, I.P.
Year
2018
Published
REVEAL RecSyS 2018 workshop proceedings. New York: ACM, 2018.
Type
Proceedings paper
Departments
Annotation
Recommender systems are usually trained and evaluated on historical data. Offline evaluation is, however, tricky and offline performance can be an inaccurate predictor of the online performance measured in production due to several reasons. In this paper, we experiment with two offline evaluation strategies and show that even a reasonable and popular strategy can produce results that are not just biased, but also in direct conflict with the true performance obtained in the online evaluation. We investigate offline policy evaluation techniques adapted from reinforcement learning and explain why such techniques fail to produce an unbiased estimate of the online performance in the “watch next” scenario of a large-scale movie recommender system. Finally, we introduce a new evaluation technique based on Jaccard Index and show that it correlates with the online performance.
Reducing Cold Start Problems in Educational Recommender Systems
Authors
Year
2016
Published
2016 International Joint Conference on Neural Networks (IJCNN). San Francisco: American Institute of Physics and Magnetic Society of the IEEE, 2016. p. 3143-3149. ISSN 2161-4407. ISBN 978-1-5090-0620-5.
Type
Proceedings paper
Annotation
Educational data can help us to personalise university information systems. In this paper, we show how educational data can be used to improve the performance of interaction-based recommender systems. Educational data is transformed to student profiles helping to prevent cold start problems when recommending projects to students with few user interactions. Our results show that our hybrid interaction based recommender boosted by educational profiles significantly outperforms bestseller recommendation, which is a mainstream recommendation method for cold start users.
A Soft Computing Approach to Knowledge Flow Synthesis and Optimization
Authors
Year
2013
Published
Soft Computing Models in Industrial and Environmental Applications. Heidelberg: Springer, 2013. pp. 23-32. ISSN 2194-5357. ISBN 978-3-642-32921-0.
Type
Proceedings paper
Departments
Annotation
In the areas of Data Mining (DM) and Knowledge Discovery (KD), large
variety of algorithms has been developed in the past decades, and the research is still
ongoing. Data mining expertise is usually needed to deploy the algorithms available.
Specifically, a process of interconnected actions referred to as knowledge flow (KF)
needs to be assembled when the algorithms are to be applied to given data. In this
paper, we propose an innovative evolutionary approach to automated KF synthesis
and optimization. We demonstrate the evolutionary KF synthesis on the problem
of classifier construction. Both preprocessing and machine learning actions are selected
and configured by means of evolution to produce a model that fits very well
for a given dataset.
Using Interactive Evolution for Exploratory Data Analysis
Authors
Year
2011
Published
Proceedings of the 6th International Scientific and Technical Conference (CSIT'2011). Lviv: Lviv Polytechnic National University, 2011. pp. 131-135. ISBN 978-966-2191-04-2.
Type
Proceedings paper
Departments
Annotation
Multivariate data are difficult to explore. The most popular linear projection techniques mapping data to 2-dimensional space often fail to reveal the patterns of interest. Non-linear mapping techniques are both slow and inefficient. In this paper, we propose a heuristic that allows users to adjust the parameters of mapping techniques just by stating their preferences iteratively. The preliminary results on real-world dataset demonstrate the power of our approach.