Living Lab Evaluation for Life and Social Sciences Search Platforms -- LiLAS at CLEF 2021
Meta-evaluation studies of system performances in controlled offline evaluation campaigns, like TREC and CLEF, show a need for innovation in evaluating IR-systems. The field of academic search is no exception to this. This might be related to the fact that relevance in academic search is multilayered and therefore the aspect of user-centric evaluation is becoming more and more important. The Living Labs for Academic Search (LiLAS) lab aims to strengthen the concept of user-centric living labs for the domain of academic search by allowing participants to evaluate their retrieval approaches in two real-world academic search systems from the life sciences and the social sciences. To this end, we provide participants with metadata on the systems' content as well as candidate lists with the task to rank the most relevant candidate to the top. Using the STELLA-infrastructure, we allow participants to easily integrate their approaches into the real-world systems and provide the possibility to compare different approaches at the same time.
Code (0)
등록된 구현이 없습니다.
Tasks
RetrievalSimilar Papers 제목 키워드 기반
Overview of LiLAS 2021 -- Living Labs for Academic Search
The Living Labs for Academic Search (LiLAS) lab aims to strengthen the concept of user-centric living labs for academic search. The methodological gap between real-world and lab-based evaluation should be bridged by allo…
RetrievalA bibliometric analysis and scoping study to identify English-language perspectives on slums
Slums, informal settlements, and deprived areas are urban regions characterized by poverty. According to the United Nations, over one billion people reside in these areas, and this number is projected to increase. Additi…
Knowledge Graphs for the Life Sciences: Recent Developments, Challenges and Opportunities
The term life sciences refers to the disciplines that study living organisms and life processes, and include chemistry, biology, medicine, and a range of other related disciplines. Research efforts in life sciences are h…
Knowledge GraphsManagementscientific discoveryHybrid Life: Integrating Biological, Artificial, and Cognitive Systems
Artificial life is a research field studying what processes and properties define life, based on a multidisciplinary approach spanning the physical, natural and computational sciences. Artificial life aims to foster a co…
Artificial LifeTowards interactive evaluations for interaction harms in human-AI systems
Current AI evaluation paradigms that rely on static, model-only tests fail to capture harms that emerge through sustained human-AI interaction. As interactive AI systems, such as AI companions, proliferate in daily life,…
Ethics