Proceedings of the 2013 workshop on Living labs for information retrieval evaluation
Bibliographic record
Abstract
It is our great pleasure to welcome you to the Workshop on Living Labs for Information Retrieval Evaluation -- LL'13, held at CIKM 2013 in San Francisco, on November 1, 2013. In the past few years the information retrieval (IR) community has been exploring ways to move further away from the Cranfield style evaluation paradigm, and make evaluations more realistic (more centered on real users, their needs and behaviours). As part of this drive, living labs, which involve and integrate users in the research process, have been proposed. Living labs would offer huge benefits to the community, such as: availability of, potentially larger, cohorts of real users and their behaviours; cross-comparability across research centres; and greater knowledge transfer between industry and academia, when industry partners are involved. The need for this methodology is further amplified by the increased reliance of IR approaches on proprietary data; living labs are a way to bridge the data divide between academia and industry. Progress towards realising actual living labs has nevertheless been limited. There are many challenges to be overcome before the benefits associated with living labs for IR can be realised, including challenges associated with living labs architecture and design, hosting, maintenance, security, privacy, participant recruiting, and scenarios and tasks for use development. This workshop brings together, for the first time, people interested in progressing the living labs for IR evaluation methodology. Our aim is to work together to identify natural use cases, barriers to success, and share opinions on ways and means of addressing them. The call for papers attracted 7 submissions, all of which were found acceptable by the program committee. These include 2 short papers, 2 position papers, and 3 demonstrators. In addition, the workshop programme features an invited talk by Jan Pedersen (Microsoft Bing). The workshop is intended to be highly interactive to encourage group discussion and active collaboration among attendees; multiple breakout sessions are scheduled throughout the day. A final discussion session wraps up the event with the objective to identify and formulate specific action items for future research and development.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.181 | 0.178 |
| Meta-epidemiology (narrow) | 0.003 | 0.002 |
| Meta-epidemiology (broad) | 0.003 | 0.003 |
| Bibliometrics | 0.003 | 0.002 |
| Science and technology studies | 0.004 | 0.006 |
| Scholarly communication | 0.018 | 0.019 |
| Open science | 0.008 | 0.016 |
| Research integrity | 0.006 | 0.011 |
| Insufficient payload (model declined to judge) | 0.058 | 0.022 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".