{"id":"W2333065964","doi":"10.1145/2854946.2854993","title":"Are Secondary Assessors Uncertain When They Disagree About Relevance Judgements?","year":2016,"lang":"en","type":"article","venue":"","topic":"Mobile Crowdsensing and Crowdsourcing","field":"Computer Science","cited_by":0,"is_retracted":false,"has_abstract":true,"ca_institutions":"University of Waterloo","funders":"Natural Sciences and Engineering Research Council of Canada; King Saud bin Abdulaziz University for Health Science; University of Waterloo; King Abdulaziz University; King Saud University","keywords":"Judgement; Relevance (law); Certainty; Adjudication; Test (biology); Information retrieval; Psychology; Computer science; Mathematics; Political science; Law","routes":{"ca_aff":true,"ca_fund":true,"ca_venue":false,"about_ca":false,"invisible_to_affiliation_only":false},"retraction":null,"screen":null,"direct_labels":[],"prediction":{"model_version":"metacan-v3-hybrid-931329e0061c","candidate_categories":["metaresearch"],"consensus_categories":[],"category_scores_codex":[0.1449988,0.0008617284,0.001870606,0.004600396,0.00462656,0.008039905,0.002001352,0.003356465,0.004643031],"category_scores_gemma":[0.5371016,0.001082279,0.001624145,0.002286983,0.005422073,0.006186425,0.005165434,0.004060423,0.002349453],"about_ca_system_candidate":false,"about_ca_system_consensus":false,"about_ca_system_score_codex":0.002461042,"about_ca_system_score_gemma":0.003170052,"about_ca_topic_candidate":false,"about_ca_topic_consensus":false,"about_ca_topic_score_codex":0.00486823,"about_ca_topic_score_gemma":0.003643885,"domain_scores_codex":[0.8084918,0.1293729,0.01252459,0.01081563,0.03468648,0.004108615],"domain_scores_gemma":[0.4141744,0.4102823,0.06402071,0.03863192,0.0661133,0.006777421],"domain_codex":null,"domain_gemma":"methods","domain_candidate":"methods","domain_consensus":null,"study_design_codex":"observational","study_design_gemma":"observational","study_design_scores_codex":[0.003867022,0.0004108939,0.4326863,0.001698728,0.001188864,0.002012535,0.2218124,0.001749595,0.01148459,0.01496356,0.014718,0.2934075],"study_design_scores_gemma":[0.001084108,0.002263095,0.4629936,0.003987514,0.001318963,0.006865479,0.2085513,0.03127046,0.02186422,0.150036,0.1079915,0.001773745],"study_design_candidate":"observational","study_design_consensus":"observational","genre_codex":"empirical","genre_gemma":"empirical","genre_scores_codex":[0.853907,0.003131151,0.08670229,0.01701245,0.0007746265,0.0009203059,0.0007202332,0.0006899527,0.03614196],"genre_scores_gemma":[0.9862015,0.0002795888,0.009931546,0.001840034,0.0002190275,0.0003243271,0.0001728126,0.00006758788,0.0009636284],"genre_candidate":"empirical","genre_consensus":"empirical","teacher_disagreement_score":0.8550012,"threshold_uncertainty_score":0.766836,"prediction_status":"machine_predicted_unvalidated"},"machine_scores":{"provisional":true,"baseline":true,"maturity_gate_passed":false,"score_opus":0.02126795997712938,"score_gpt":0.2489718051851484,"score_spread":0.227703845208019,"validation_status":"score_only:v0-immature-baseline","note":"Baseline scores from an immature model (maturity gate not passed). Scores rank; they never assert a category."}}