{"id":"W2105897558","doi":"10.1162/coli.2009.35.4.35402","title":"From Annotator Agreement to Noise Models","year":2009,"lang":"en","type":"article","venue":"Computational Linguistics","topic":"Rough Sets and Fuzzy Logic","field":"Computer Science","cited_by":77,"is_retracted":false,"has_abstract":true,"ca_institutions":"Kellogg's (Canada)","funders":"","keywords":"Benchmarking; Computer science; Noise (video); Agreement; Data set; Quality (philosophy); Set (abstract data type); Resource (disambiguation); Data quality; Data mining; Data science; Econometrics; Artificial intelligence; Mathematics; Linguistics; Physics","routes":{"ca_aff":true,"ca_fund":false,"ca_venue":false,"about_ca":false,"invisible_to_affiliation_only":false},"retraction":null,"screen":null,"direct_labels":[],"prediction":{"model_version":"metacan-v3-hybrid-931329e0061c","candidate_categories":["metaresearch"],"consensus_categories":[],"category_scores_codex":[0.2712898,0.002643172,0.004258384,0.008287196,0.00466699,0.01524129,0.006899028,0.006341117,0.003291137],"category_scores_gemma":[0.5429263,0.002324083,0.002472288,0.007787501,0.01856312,0.02144884,0.02013682,0.008231635,0.001719489],"about_ca_system_candidate":false,"about_ca_system_consensus":false,"about_ca_system_score_codex":0.007126923,"about_ca_system_score_gemma":0.004966308,"about_ca_topic_candidate":false,"about_ca_topic_consensus":false,"about_ca_topic_score_codex":0.005219304,"about_ca_topic_score_gemma":0.00432706,"domain_scores_codex":[0.6317576,0.2636082,0.01116808,0.04028907,0.04927701,0.00390013],"domain_scores_gemma":[0.3804966,0.5186472,0.02126362,0.05124028,0.02664683,0.001705453],"domain_codex":null,"domain_gemma":null,"domain_candidate":null,"domain_consensus":null,"study_design_codex":"theoretical_or_conceptual","study_design_gemma":"theoretical_or_conceptual","study_design_scores_codex":[0.0009949803,0.0001383852,0.02893059,0.001629628,0.001818451,0.0005920044,0.01359471,0.08688071,0.002239024,0.7240312,0.01169966,0.1274507],"study_design_scores_gemma":[0.00006509005,0.00009787746,0.0038932,0.0004339998,0.0001880239,0.0002606566,0.001074972,0.1218932,0.002256125,0.8602432,0.009403256,0.0001904436],"study_design_candidate":"theoretical_or_conceptual","study_design_consensus":"theoretical_or_conceptual","genre_codex":"methods","genre_gemma":"methods","genre_scores_codex":[0.0221137,0.001952593,0.9578827,0.005809915,0.0004335112,0.0002223523,0.0005374651,0.0004714281,0.01057642],"genre_scores_gemma":[0.7322374,0.001337781,0.2534929,0.00307462,0.0008621493,0.001374946,0.001553218,0.001210021,0.004857035],"genre_candidate":"methods","genre_consensus":"methods","teacher_disagreement_score":0.2712898,"threshold_uncertainty_score":0.8986294,"prediction_status":"machine_predicted_unvalidated"},"machine_scores":{"provisional":true,"baseline":true,"maturity_gate_passed":false,"score_opus":0.02859109390663595,"score_gpt":0.2738249099219155,"score_spread":0.2452338160152796,"validation_status":"score_only:v0-immature-baseline","note":"Baseline scores from an immature model (maturity gate not passed). Scores rank; they never assert a category."}}