{"id":"W4403648112","doi":"10.1145/3691621.3694939","title":"A Preliminary Study of Multilingual Code Language Models for Code Generation Task Using Translated Benchmarks","year":2024,"lang":"en","type":"preprint","venue":"","topic":"Natural Language Processing Techniques","field":"Computer Science","cited_by":2,"is_retracted":false,"has_abstract":true,"ca_institutions":"University of British Columbia, Okanagan Campus; Kelowna General Hospital; University of British Columbia","funders":"","keywords":"Computer science; Code (set theory); Task (project management); Programming language; Code generation; Natural language processing; Operating system; Engineering","routes":{"ca_aff":true,"ca_fund":false,"ca_venue":false,"about_ca":false,"invisible_to_affiliation_only":false},"retraction":null,"screen":null,"direct_labels":[],"prediction":{"model_version":"metacan-v3-hybrid-931329e0061c","candidate_categories":[],"consensus_categories":[],"category_scores_codex":[0.008542735,0.002117818,0.0009678099,0.001670981,0.001068437,0.002228117,0.002398446,0.001287179,0.004852888],"category_scores_gemma":[0.059388,0.0007028728,0.0009002361,0.001800435,0.00105118,0.003828171,0.002556732,0.002910555,0.002650711],"about_ca_system_candidate":false,"about_ca_system_consensus":false,"about_ca_system_score_codex":0.001426753,"about_ca_system_score_gemma":0.002266196,"about_ca_topic_candidate":false,"about_ca_topic_consensus":false,"about_ca_topic_score_codex":0.01078523,"about_ca_topic_score_gemma":0.01411172,"domain_scores_codex":[0.9899055,0.005978625,0.0008092917,0.001756977,0.001159343,0.0003903127],"domain_scores_gemma":[0.9598457,0.02226872,0.001305062,0.006244203,0.009351403,0.0009848878],"domain_codex":null,"domain_gemma":null,"domain_candidate":null,"domain_consensus":null,"study_design_codex":"design_other","study_design_gemma":"simulation_or_modeling","study_design_scores_codex":[0.005433159,0.006142241,0.05078329,0.004888415,0.0007268645,0.001784518,0.008593065,0.1441999,0.05530956,0.009824319,0.05499592,0.6573188],"study_design_scores_gemma":[0.001445262,0.005813355,0.03072337,0.0006138467,0.0003741908,0.001043599,0.004783357,0.8026832,0.09706699,0.01063395,0.04443023,0.0003887977],"study_design_candidate":"simulation_or_modeling","study_design_consensus":null,"genre_codex":"empirical","genre_gemma":"empirical","genre_scores_codex":[0.8497536,0.001484773,0.1117136,0.001124954,0.0006095191,0.001521672,0.006707777,0.01543907,0.01164505],"genre_scores_gemma":[0.8073357,0.0003337307,0.1603363,0.0006140468,0.0001136885,0.001624123,0.02353656,0.0024138,0.003692111],"genre_candidate":"empirical","genre_consensus":"empirical","teacher_disagreement_score":0.01078523,"threshold_uncertainty_score":0.04517883,"prediction_status":"machine_predicted_unvalidated"},"machine_scores":{"provisional":true,"baseline":true,"maturity_gate_passed":false,"score_opus":0.05547638290687799,"score_gpt":0.3526947562477853,"score_spread":0.2972183733409073,"validation_status":"score_only:v0-immature-baseline","note":"Baseline scores from an immature model (maturity gate not passed). Scores rank; they never assert a category."}}