Knowledge Delta based improvement and continuous evaluation of retrieval engines – Kodicare

In Kodicare, we generically call such difference “knowledge delta”. Evaluation of several environments, knowing their knowledge deltas, leads to measuring and qualifying “results deltas”. Online systems require continuous evaluation with a stable and meaningful environment that guarantees the reproducibility and explainability of systems results. A controlled environment quantifying both “knowledge deltas” and “result deltas” will support such continuous evaluation, and enable the provision of explanations for system engineers through the analysis of related changes in the two “deltas”. The theoretical results will be confronted to real cases defined by a French company that deploys a web search engine (Qwant).

Modeling and experimentation of the result deltas.

Modeling and experimentation of the knowledge deltas.
Scaling up the proposal to be able to handle the Qwant case.

Published articles:
P. Mulhem, G. Gonzalez Saez, A. Mannion, D. Schwab, and J. Frej. LIG-Health at Adhoc and Spoken IR Consumer Health Search: expandingqueries using UMLS and FastText. InCLEF 2020, Thessaloniki (on line),Greece, Sept. 2020.
L. Goeuriot, H. Suominen, L. Kelly, A. Miranda-Escalada, M. Krallinger,Z. Liu, G. Pasi, G. G. Saez, M. Viviani, and C. Xu. Overview of the clefehealth evaluation lab 2020. InInternational Conference of the Cross-Language Evaluation Forum for European Languages, pages 255–271. Springer, 2020
L. Goeuriot, H. Suominen, L. Kelly, L. A. Alemany, N. Brew-Sam, V. Cotik,D. Filippo, G. G. Sáez, F. Luque, P. Mulhem, G. Pasi, R. Roller, S. Senevi-ratne, J. Vivaldi, M. Viviani, and C. Xu. CLEF ehealth evaluation lab2021. In D. Hiemstra, M. Moens, J. Mothe, R. Perego, M. Potthast, andF. Sebastiani, editors,Advances in Information Retrieval - 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28 - April1, 2021, Proceedings, Part II, volume 12657 of Lecture Notes in ComputerScience, pages 593–600. Springer, 2020
N. Gonzalez Sáez, L. Goeuriot, and P. Mulhem. Addressing different evaluation environments for information retrieval through pivot systems. In A. Doucet and A. Chifu, editors,COnférence en Recherche d’Informationset Applications - CORIA 2021, French Information Retrieval Conference,Grenoble, France, April 15, 2021. ARIA, 2021
G. Gonzalez-Saez, P. Mulhem, and L. Goeuriot. Towards the evaluationof information retrieval systems on evolving datasets with pivot systems.InCLEF 2021, Lecture Notes in Computer Science, pages Accepted, bePublished. Springer, 2021

Evaluating search systems requires setting up an environment: select a paradigm, metrics, a dataset, etc. The choice of an environment is rarely motivated objectively, and the impact of its variations (choosing a dataset against another, altering one) is rarely measured. Such objectivity comes from a quantifiable understanding of the differences between datasets, documents, or test queries. In Kodicare, we generically call such difference “knowledge delta”. Evaluation of several environments, knowing their knowledge deltas, leads to measuring and qualifying “results deltas”. Online systems require continuous evaluation with a stable and meaningful environment; which guarantees the reproducibility and explainability of systems results. The environment and result deltas will be able to support such continuous evaluation, and to provide explanations. The theoretical results will be confronted to real cases defined by a French company that deploys a web search engine (Qwant).

Scientific and technical challenges:
To our knowledge, no such framework dedicated to real continuous evaluation of information retrieval systems exist, due to the numerous parameters that must be handled. The deltas proposed by Kodicare are then a sensible way to tackle this problem. Continuous evaluation is only possible with real cases, which are often difficult to define without the help of web search companies. The large implication of Qwant will help the project define usable proposals.



Expected results:
• The innovative theoretical solution explored by this project is to define in a common framework “knowledge delta” and “result deltas”, and quantify them such that: results are comparable over the time (like a regression test) and the search engine adapts to changes in user behaviour and information need.
• A transversal focus of the project will explore the transparency that such a continuous evaluation system must support.
• New evaluation paradigm with a potential impact on many connected communities