{"id":"W2955229766","doi":"10.1145/3331184.3331356","title":"Quantifying Bias and Variance of System Rankings","year":2019,"lang":"en","type":"article","venue":"","topic":"Bayesian Modeling and Causal Inference","field":"Computer Science","cited_by":6,"is_retracted":false,"has_abstract":true,"ca_institutions":"University of Waterloo","funders":"","keywords":"Variance (accounting); Pooling; Statistics; Estimator; Conflation; Mathematics; Bias of an estimator; Rank (graph theory); Econometrics; Computer science; Minimum-variance unbiased estimator; Artificial intelligence; Combinatorics","routes":{"ca_aff":true,"ca_fund":false,"ca_venue":false,"about_ca":false,"invisible_to_affiliation_only":false},"retraction":null,"screen":null,"direct_labels":[],"prediction":{"model_version":"metacan-v3-hybrid-931329e0061c","candidate_categories":[],"consensus_categories":[],"category_scores_codex":[0.03154519,0.00101659,0.001619865,0.005361697,0.0007214332,0.003438833,0.00102872,0.001016076,0.001183438],"category_scores_gemma":[0.2016818,0.0005733589,0.0008104221,0.004629068,0.002321001,0.003713389,0.002937298,0.001432391,0.0004450451],"about_ca_system_candidate":false,"about_ca_system_consensus":false,"about_ca_system_score_codex":0.00139917,"about_ca_system_score_gemma":0.001637053,"about_ca_topic_candidate":false,"about_ca_topic_consensus":false,"about_ca_topic_score_codex":0.0025825,"about_ca_topic_score_gemma":0.002821951,"domain_scores_codex":[0.972109,0.01163171,0.001709359,0.004368097,0.008962151,0.001219662],"domain_scores_gemma":[0.8336659,0.1284177,0.01009557,0.01694449,0.01010316,0.0007731976],"domain_codex":null,"domain_gemma":null,"domain_candidate":null,"domain_consensus":null,"study_design_codex":"design_other","study_design_gemma":"simulation_or_modeling","study_design_scores_codex":[0.0008447291,0.0002031438,0.3210342,0.001119492,0.003325537,0.0003567582,0.003124224,0.1625749,0.01459076,0.1593513,0.006732033,0.326743],"study_design_scores_gemma":[0.00008174418,0.0003635903,0.1105436,0.0002657485,0.0004666582,0.00049499,0.00062954,0.4935232,0.01713553,0.3694268,0.00678496,0.0002836678],"study_design_candidate":"simulation_or_modeling","study_design_consensus":null,"genre_codex":"methods","genre_gemma":"methods","genre_scores_codex":[0.2410752,0.001516434,0.7483545,0.0008832198,0.0001769664,0.0001397722,0.001049025,0.001060009,0.00574484],"genre_scores_gemma":[0.9379815,0.0003100519,0.05953103,0.0002011642,0.000161002,0.0001522231,0.000818899,0.0002291877,0.0006149019],"genre_candidate":"methods","genre_consensus":"methods","teacher_disagreement_score":0.03154519,"threshold_uncertainty_score":0.1668289,"prediction_status":"machine_predicted_unvalidated"},"machine_scores":{"provisional":true,"baseline":true,"maturity_gate_passed":false,"score_opus":0.06170573545308153,"score_gpt":0.2608923500520292,"score_spread":0.1991866145989477,"validation_status":"score_only:v0-immature-baseline","note":"Baseline scores from an immature model (maturity gate not passed). Scores rank; they never assert a category."}}