{"id":"W4400528870","doi":"10.1145/3626772.3657992","title":"LLM4Eval: Large Language Model for Evaluation in IR","year":2024,"lang":"en","type":"article","venue":"","topic":"Topic Modeling","field":"Computer Science","cited_by":19,"is_retracted":false,"has_abstract":true,"ca_institutions":"Microsoft (Canada); University of Waterloo","funders":"Engineering and Physical Sciences Research Council; Vrije Universiteit Amsterdam; Universiteit van Amsterdam","keywords":"Computer science; Programming language; Natural language processing","routes":{"ca_aff":true,"ca_fund":false,"ca_venue":false,"about_ca":false,"invisible_to_affiliation_only":false},"retraction":null,"screen":null,"direct_labels":[],"prediction":{"model_version":"codex-gemma-dda1882f352a","candidate_categories":[],"consensus_categories":[],"category_scores_codex":[0.0008046131,0.0000457082,0.00004876927,0.0000803166,0.00001614145,0.00008528848,0.0002113367,0.00002772573,0.00002042665],"category_scores_gemma":[0.00003309386,0.00003945611,0.00002784947,0.0001258281,0.000001970374,0.0002682492,0.00006510258,0.00004450347,0.00002350155],"about_ca_system_candidate":false,"about_ca_system_consensus":false,"about_ca_system_score_codex":0.0000503306,"about_ca_system_score_gemma":0.00007967529,"about_ca_topic_candidate":false,"about_ca_topic_consensus":false,"about_ca_topic_score_codex":0.00001660998,"about_ca_topic_score_gemma":0.0001036257,"domain_scores_codex":[0.9993374,0.00001718345,0.0001055764,0.0002281503,0.0001689621,0.0001426957],"domain_scores_gemma":[0.9996809,0.00004090216,0.000006854765,0.0002258332,0.0000264826,0.00001904754],"domain_codex":null,"domain_gemma":null,"domain_candidate":null,"domain_consensus":null,"study_design_codex":"theoretical_or_conceptual","study_design_gemma":"simulation_or_modeling","study_design_scores_codex":[0.000001977279,0.00003453412,0.00004564663,0.00006314702,0.000006395544,0.000005233102,0.008394921,0.08709473,0.0008123877,0.701455,0.001653305,0.2004327],"study_design_scores_gemma":[0.0002017297,0.000005035674,0.0000222323,0.00001283259,0.000002102655,7.385244e-7,0.00003462153,0.9799245,0.000183099,0.01925362,0.000307501,0.00005197119],"study_design_candidate":"simulation_or_modeling","study_design_consensus":null,"genre_codex":"methods","genre_gemma":"empirical","genre_scores_codex":[0.0244868,0.0003427362,0.9709436,0.0008156312,0.0001939564,0.0002258825,0.000001591996,0.000169845,0.002820011],"genre_scores_gemma":[0.8963992,0.000001515001,0.1017267,0.0002291331,0.00003871108,0.00005281468,0.000002383671,0.00000473208,0.001544791],"genre_candidate":"methods","genre_consensus":null,"teacher_disagreement_score":0.8928298,"threshold_uncertainty_score":0.1608973,"prediction_status":"machine_predicted_unvalidated"},"machine_scores":{"provisional":true,"baseline":true,"maturity_gate_passed":false,"score_opus":0.05657543100605471,"score_gpt":0.3574910327463317,"score_spread":0.300915601740277,"validation_status":"score_only:v0-immature-baseline","note":"Baseline scores from an immature model (maturity gate not passed). Scores rank; they never assert a category."}}