Replication package for the paper: "A Study on the Pythonic Functional Constructs' Understandability"

Replication Package for "A Study on the Pythonic Functional Constructs' Understandability" to appear at ICSE 2024 Authors: Cyrine Zid, Fiorella Zampetti, Giuliano Antoniol, Massimiliano Di penta Article Preprint: https://mdipenta.github.io/files/ICSE24_funcExperiment.pdf Artifacts: https://doi.org/10.5281/zenodo.8191782 License: GPL V3.0 This package contains folders and files with code and data used in the study described in the paper. In the following, we first provide all fields required for the submission, and then report a detailed description of all repository folders. Artifact Description Purpose The artifact is about a controlled experiment aimed at investigating the extent to which Pythonic functional constructs have an impact on source code understandability. The artifact archive contains: The material to allow replicating the study (see Section Experimental-Material) Raw quantitative results, working datasets, and scripts to replicate the statistical analyses reported in the paper. Specifically, the executable part of the replication package reproduces figures and tables of the quantitative analysis (RQ1 and RQ2) of the paper starting from the working datasets. Spreadsheets used for the qualitative analysis (RQ3). We apply for the following badges: Available and reusable: because we provide all the material that can be used to replicate the experiment, but also to perform the statistical analyses and the qualitative analyses (spreadsheets, in this case) Provenance Paper preprint link: https://mdipenta.github.io/files/ICSE24_funcExperiment.pdf Artifacts: https://doi.org/10.5281/zenodo.8191782 Data Results have been obtained by conducting the controlled experiment involving Prolificworkers as participants. Data collection and processing followed a protocol approved by the University ethical board. Note that all data enclosed in the artifact is completely anonymized and does not contain sensible information. Further details about the provided dataset can be found in the Section Results' directory and files Setup and Usage (for executable artifacts): See the Section Scripts to reproduce the results, and instructions for running them Experiment-Material/ Contains the material used for the experiment, and, specifically, the following subdirectories: Google-Forms/ Contains (as PDF documents) the questionnaires submitted to the ten experimental groups. Task-Sources/ Contains, for each experimental group (G-1...G-10), the sources used to produce the Google Forms, and, specifically: - The cover letter (Letter.docx). - A directory for each experimental task (Lambda 1, Lambda 2, Comp 1, Comp 2, MRF 1, MRF 2, Lambda Comparison, Comp Comparison, MRF Comparison). Each directory contains: (i) the exercise text (in both Word and .txt format), the source code snippet, and its .png image to be used in the form. Note: the "Comparison" tasks do not have any exercise as the purpose is always the same, i.e., to compare the (perceived) understandability of the snippets and return the results of the comparison. Code-Examples-Table1/ Contains the source code snippets used as objects of the study (the same you can find under "Task-Sources/"), named as reported in Table 1. Results' directory and files raw-responses/ Contains, as spreadsheets, the raw responses provided by the study participants through Google forms. raw-results-RQ1/ Contains the raw results for RQ1. Specifically, the directory contains a subdirectory for each group (G1-G10). Each subdirectory contains: - For each user (named using their Prolific IDs, a directory containing, for each question (Q1-Q6) the produced python code (Qn.py) its output (QnR.txt) and its StdErr output (QnErr.txt). - "expected-outputs/": A directory containing the expected outputs for each task (Qn.txt). working-results/RQ1-RQ2-files-for-statistical-analysis/ Contains three .csv files used as input for conducting the statistical analysis and drawing the graphs for addressing the first two research questions of the study. Specifically: ConstructUsage.csv contains the declared frequency usage of the three functional constructs object of the study. This file is used to draw Figure 4. The file contains an entry for each participant, reporting the (text-coded) frequency of construct usage for Comprehension, Lambda, and MRF. RQ1.csv contains the collected data used for the mixed-effect logistic regression relating the use of functional constructs with the correctness of the change task, as well as the logistic regression relating the use of map/reduce/filter functions with the correctness of the change task. The csv file contains an entry for each answer provided by each subject, and features the following columns: Group: experimental group to which the participant is assigned User: user ID Time: task time in seconds Approvals: number of approvals on previous tasks performed on Prolific Student: whether the participant declared themselves as a student Section: section of the questionnaire (lambda, comp, or mrf) Construct: specific construct being presented (same as "Section" for lambda and comp, for mrf it says whether it is a map, reduce, or filter) Question: question id, from Q1 to Q6, indicate the ordering of the question MainFactor: main factor treatment for the given question - "f" for functional, "p" for procedural counterpart Outcome: TRUE if the task was correctly performed, FALSE otherwise Complexity: cyclomatic complexity of the construct (empty for mrf) UsageFrequency: usage frequency of the given construct RQ1Paired-RQ2.csv contains the collected data used for the ordinal logistic regression of the relationship between the perceived ease of understanding of the functional constructs and (i) participants' usage frequency, and (ii) constructs' complexity (except for map/reduce/filter). The file features a row for each participant, and the columns are the following: Group: experimental group to which the participant is assigned User: user ID Time: task time in seconds Approvals: number of approvals on previous tasks performed on Prolific Student: whether the participant declared themselves as a student LambdaF: result for the change task related to a lambda construct LambdaP: result for the change task related to the procedural counterpart of a lambda construct CompF: result for the change task related to a comprehension construct CompP: result for the change task related to the procedural counterpart of a comprehension construct MrfF: result for the change task related to an MRF construct MrfP: result for the change task related to the procedural counterpart of a MRF construct LambdaComp: perceived understandability level for the comparison task (RQ2) between a lambda and its procedural counterpart CompComp: perceived understandability level for the comparison task (RQ2) between a comprehension and its procedural counterpart MrfComp: perceived understandability level for the comparison task (RQ2) between a MRF and its procedural counterpart LambdaCompCplx: cyclomatic complexity of the lambda construct involved in the comparison task (RQ2) CompCompCplx: cyclomatic complexity of the comprehension construct involved in the comparison task (RQ2) MrfCompType: type of MRF construct (map, reduce, or filter) used in the comparison task (RQ2) LambdaUsageFrequency: self-declared usage frequency on lambda constructs CompUsageFrequency: self-declared usage frequency on comprehension constructs MrfUsageFrequency: self-declared usage frequency on MRF constructs LambdaComparisonAssessment: outcome of the manual assessment of the answer to the "check question" required for the lambda comparison ("yes" means valid, "no" means wrong, "moderatechatgpt" and "extremechatgpt" are the results of GPTZero) CompComparisonAssessment: as above, but for comprehension MrfComparisonAssessment: as above, but for MRF working-results/inter-rater-RQ3-files/ This directory contains four .csv files used as input for computing the inter-rater agreement for the manual labeling used for addressing RQ3. Specifically, you will find one file for each functional construct, i.e., comprehension.csv, lambda.csv, and mrf.csv, and a different file used for highlighting the reasons why participants prefer to use the procedural paradigm, i.e., procedural.csv. working-results/RQ2ManualValidation.csv This file contains the results of the manual validation being done to sanitize the answers provided by our participants used for addressing RQ2. Specifically, we coded the behaviour description using four different levels: (i) correct ("yes"), (ii) somewhat correct ("partial"), (iii) wrong ("no"), and (iv) automatically generated. The file features a row for each participant, and the columns are the following: ID: ID we used to refer the participant in the paper's qualitative analysis Group: experimental group to which the participant is assigned ProlificID: user ID Comparison for lambda construct description: answer provided by the user for the lambda comparison task Final Classification: our assessment of the lambda comparison answer Comparison for comprehension description: answer provided by the user for the comprehension comparison task Final Classification: our assessment of the comprehension comparison answer Comparison for MRF description: answer provided by the user for the MRF comparison task Final Classification: our assessment of the MRF comparison answer working-results/RQ3ManualValidation.xlsx This file contains the results of the open coding applied to address our third research question. Specifically, you will find four sheets, one for each functional construct and one for the procedural paradigm. Each sheet reports the provided answers together with the categories assigned to them. Each sheet contains the following columns: ID: ID we used to refer the participant in the paper's qualitative analysi

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.