{"id":"W2105897558","doi":"10.1162/coli.2009.35.4.35402","title":"From Annotator Agreement to Noise Models","year":2009,"lang":"en","type":"article","venue":"Computational Linguistics","topic":"Rough Sets and Fuzzy Logic","field":"Computer Science","cited_by":77,"is_retracted":false,"has_abstract":true,"ca_institutions":"Kellogg's (Canada)","funders":"","keywords":"Benchmarking; Computer science; Noise (video); Agreement; Data set; Quality (philosophy); Set (abstract data type); Resource (disambiguation); Data quality; Data mining; Data science; Econometrics; Artificial intelligence; Mathematics; Linguistics; Physics","routes":{"ca_aff":true,"ca_fund":false,"ca_venue":false,"about_ca":false,"invisible_to_affiliation_only":false},"retraction":null,"screen":null,"direct_labels":[],"prediction":{"model_version":"codex-gemma-dda1882f352a","candidate_categories":[],"consensus_categories":[],"category_scores_codex":[0.00008613839,0.0001228149,0.0001283765,0.00006792277,0.0001054261,0.0001519835,0.0005380135,0.00003482736,0.000009374538],"category_scores_gemma":[0.0002092068,0.0001183416,0.00004273839,0.000210982,0.00001240429,0.00004868642,0.000107752,0.00007384265,0.0001868959],"about_ca_system_candidate":false,"about_ca_system_consensus":false,"about_ca_system_score_codex":0.00004258307,"about_ca_system_score_gemma":0.00007465041,"about_ca_topic_candidate":false,"about_ca_topic_consensus":false,"about_ca_topic_score_codex":0.00001622923,"about_ca_topic_score_gemma":6.454408e-7,"domain_scores_codex":[0.998859,0.00001966469,0.0002326232,0.0003303178,0.0003601741,0.0001982074],"domain_scores_gemma":[0.9990359,0.0001148479,0.0000549163,0.0002769359,0.000353943,0.000163478],"domain_codex":null,"domain_gemma":null,"domain_candidate":null,"domain_consensus":null,"study_design_codex":"simulation_or_modeling","study_design_gemma":"simulation_or_modeling","study_design_scores_codex":[0.000004885535,0.00008206347,0.00002095205,0.00000128802,0.000007538333,0.0000255658,0.0005491913,0.5370138,0.000005843559,0.4370436,0.00817441,0.01707087],"study_design_scores_gemma":[0.000107417,0.00006518699,0.001641256,0.000005818647,0.000002594358,6.481425e-7,0.000003760897,0.5630154,0.000006514489,0.4270432,0.007999836,0.0001083693],"study_design_candidate":"simulation_or_modeling","study_design_consensus":"simulation_or_modeling","genre_codex":"methods","genre_gemma":"empirical","genre_scores_codex":[0.001496219,0.00006396708,0.9809409,0.001201159,0.0008395578,0.0001150369,0.00003174173,0.0001080678,0.01520328],"genre_scores_gemma":[0.5452018,0.000001004245,0.451488,0.002675169,0.0005741731,0.000001805182,0.00002782246,0.000002849207,0.00002739685],"genre_candidate":"methods","genre_consensus":null,"teacher_disagreement_score":0.5437056,"threshold_uncertainty_score":0.4825832,"prediction_status":"machine_predicted_unvalidated"},"machine_scores":{"provisional":true,"baseline":true,"maturity_gate_passed":false,"score_opus":0.02859109390663595,"score_gpt":0.2738249099219155,"score_spread":0.2452338160152796,"validation_status":"score_only:v0-immature-baseline","note":"Baseline scores from an immature model (maturity gate not passed). Scores rank; they never assert a category."}}