{"id":"W2468693985","doi":"10.1145/2899415.2899473","title":"Benchmarking Introductory Programming Exams","year":2016,"lang":"en","type":"article","venue":"","topic":"Teaching and Learning Programming","field":"Computer Science","cited_by":7,"is_retracted":false,"has_abstract":true,"ca_institutions":"University of Toronto","funders":"","keywords":"Benchmark (surveying); Benchmarking; Computer science; Mathematics education; Psychology; Management","routes":{"ca_aff":true,"ca_fund":false,"ca_venue":false,"about_ca":false,"invisible_to_affiliation_only":false},"retraction":null,"screen":null,"direct_labels":[],"prediction":{"model_version":"codex-gemma-dda1882f352a","candidate_categories":[],"consensus_categories":[],"category_scores_codex":[0.0006693082,0.0001073672,0.00009547911,0.00008389619,0.0001627421,0.0001832524,0.0006038433,0.00003893058,0.00004437027],"category_scores_gemma":[0.0001125125,0.0000650752,0.00004989075,0.0001895173,0.00004011765,0.000432104,0.0002047534,0.0001231876,0.0001181992],"about_ca_system_candidate":false,"about_ca_system_consensus":false,"about_ca_system_score_codex":0.00003909629,"about_ca_system_score_gemma":0.00002849796,"about_ca_topic_candidate":false,"about_ca_topic_consensus":false,"about_ca_topic_score_codex":0.00002812086,"about_ca_topic_score_gemma":0.000003805819,"domain_scores_codex":[0.9987861,0.00009000399,0.0001475181,0.0004053587,0.0002016885,0.0003693093],"domain_scores_gemma":[0.9992176,0.0001271082,0.00005502568,0.0004850222,0.00002649858,0.00008870139],"domain_codex":null,"domain_gemma":null,"domain_candidate":null,"domain_consensus":null,"study_design_codex":"design_other","study_design_gemma":"not_applicable","study_design_scores_codex":[6.141656e-7,0.00001607065,0.002175533,0.000002657968,0.000005198657,0.000003850732,0.0003649608,0.000001832992,0.0004604011,0.02318464,0.0003436694,0.9734406],"study_design_scores_gemma":[0.0002277733,0.0001053758,0.003427686,0.00004979411,0.000002563766,0.00002444048,0.00002977803,0.0003632317,0.0006833712,0.000231277,0.9946103,0.0002443854],"study_design_candidate":"not_applicable","study_design_consensus":null,"genre_codex":"methods","genre_gemma":"empirical","genre_scores_codex":[0.0130623,0.00007314638,0.9818665,0.002108424,0.0005915418,0.00007488883,4.23923e-8,0.001020167,0.00120301],"genre_scores_gemma":[0.6609383,0.000002314044,0.336964,0.00008808529,0.0002915437,0.00000940136,2.206757e-7,0.00000767661,0.001698561],"genre_candidate":"methods","genre_consensus":null,"teacher_disagreement_score":0.9942666,"threshold_uncertainty_score":0.265369,"prediction_status":"machine_predicted_unvalidated"},"machine_scores":{"provisional":true,"baseline":true,"maturity_gate_passed":false,"score_opus":0.01227547792997245,"score_gpt":0.2325014494734201,"score_spread":0.2202259715434476,"validation_status":"score_only:v0-immature-baseline","note":"Baseline scores from an immature model (maturity gate not passed). Scores rank; they never assert a category."}}