{"id":"W6912907843","doi":"10.5281/zenodo.6475255","title":"DaCapo Benchmarks Evaluation Snapshot 29a657f","year":2021,"lang":"en","type":"other","venue":"Zenodo (CERN European Organization for Nuclear Research)","topic":"","field":"","cited_by":1,"is_retracted":false,"has_abstract":true,"ca_institutions":"Discovery Air (Canada)","funders":"","keywords":"Snapshot (computer storage); Suite; Commit; Benchmark (surveying); Data integrity","routes":{"ca_aff":true,"ca_fund":false,"ca_venue":false,"about_ca":false,"invisible_to_affiliation_only":false},"retraction":null,"screen":null,"direct_labels":[{"model":"gemma","categories":[],"domain":null,"study_design":"not_applicable","genre":"dataset","about_ca_system":false,"about_ca_topic":false,"confidence":"high","status":"direct model label, unvalidated"},{"model":"gpt","categories":["insufficient_payload"],"domain":null,"study_design":"not_applicable","genre":"other","about_ca_system":false,"about_ca_topic":false,"confidence":"low","status":"direct model label, unvalidated"}],"prediction":{"model_version":"metacan-v3-hybrid-931329e0061c","candidate_categories":[],"consensus_categories":[],"category_scores_codex":[0.003458961,0.002322551,0.0009614634,0.003110847,0.001253336,0.003019887,0.003815311,0.001116014,0.1099639],"category_scores_gemma":[0.009692842,0.001081137,0.0009878767,0.00379884,0.0005508612,0.00332934,0.002341968,0.003287925,0.1056625],"about_ca_system_candidate":false,"about_ca_system_consensus":false,"about_ca_system_score_codex":0.001998097,"about_ca_system_score_gemma":0.00290078,"about_ca_topic_candidate":false,"about_ca_topic_consensus":false,"about_ca_topic_score_codex":0.02061405,"about_ca_topic_score_gemma":0.02166292,"domain_scores_codex":[0.9951669,0.0007808222,0.0002675321,0.0005035446,0.002684882,0.0005963741],"domain_scores_gemma":[0.994493,0.0008190143,0.0001366552,0.001089983,0.002897144,0.0005640755],"domain_codex":null,"domain_gemma":null,"domain_candidate":null,"domain_consensus":null,"study_design_codex":"not_applicable","study_design_gemma":"not_applicable","study_design_scores_codex":[0.0001523406,0.00005036601,0.000230239,0.0002479873,0.00001530218,0.00002612618,0.00002073368,0.0007333749,0.0008055638,0.001253648,0.9841262,0.01233795],"study_design_scores_gemma":[0.0002203997,0.0001189699,0.001971193,0.0001713549,0.00003295722,0.0001347052,0.00006924347,0.007486205,0.006228499,0.003156915,0.980334,0.0000755451],"study_design_candidate":"not_applicable","study_design_consensus":"not_applicable","genre_codex":"dataset","genre_gemma":"dataset","genre_scores_codex":[0.009735615,0.003279146,0.02114859,0.002098466,0.002015196,0.0006873906,0.3558525,0.3057492,0.299434],"genre_scores_gemma":[0.02630871,0.0008043073,0.0182916,0.000900997,0.0002156698,0.0006470808,0.8392489,0.06417493,0.04940787],"genre_candidate":"dataset","genre_consensus":"dataset","teacher_disagreement_score":0.1099639,"threshold_uncertainty_score":0.367866,"prediction_status":"machine_predicted_unvalidated"},"machine_scores":{"provisional":true,"baseline":true,"maturity_gate_passed":false,"score_opus":0.06143326660752452,"score_gpt":0.2927058871414575,"score_spread":0.2312726205339329,"validation_status":"score_only:v0-immature-baseline","note":"Baseline scores from an immature model (maturity gate not passed). Scores rank; they never assert a category."}}