Learning Context-aware Term Importance for Query Performance Prediction
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
Ad hoc retrieval, a cornerstone task in Information Retrieval (IR) , aims to rank documents in response to a user’s query, often without prior knowledge of the user’s specific information need. While transformer-based neural rankers have achieved state-of-the-art performance in ad hoc retrieval, their effectiveness varies significantly across queries. Certain queries—commonly referred to as hard queries —remain particularly challenging, highlighting critical gaps in retrieval models. Identifying these hard queries is essential for improving retrieval systems, motivating the task of Query Performance Prediction (QPP) , which aims to estimate the effectiveness of a query without requiring access to relevance judgments. In this article, we propose Context-aware Query Performance Prediction ( CA-QPP ) , a novel post-retrieval QPP method, which builds on the foundations of perturbation-based QPP methods that hypothesize a relationship between query sensitivity to small perturbations and query retrieval effectiveness. Building on this foundation, our approach exposes the given query to perturbations by constructing two query variations: an effective variation emphasizing terms that enhance retrieval and an ineffective variation accentuating terms that hinder it. By contrasting the retrieval outcomes of these variations using a cross-encoder model, CA-QPP captures the interplay of term contributions and predicts the performance for the given query. We evaluate CA-QPP on the widely used MS MARCO datasets and their associated query sets, including TREC DL 2019 , TREC DL 2020 , DL-Hard , TREC DL 2021 , and TREC DL 2022 , which feature extensive human-labeled relevance judgments. Our experiments demonstrate that CA-QPP consistently outperforms traditional and neural-based QPP baselines across standard correlation metrics, including Pearson’s \(\rho\) , Kendall’s \(\tau\) , and Spearman’s \(\rho\) . Through a detailed case study, we further illustrate the mechanics of CA-QPP and provide empirical evidence for its ability to model the contextual impact of individual query terms, making it a robust framework for query performance prediction.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.000 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.001 | 0.001 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it