Proceedings of the 5th Workshop on Evaluation and Usability of Programming Languages and Tools
Bibliographic record
Abstract
It is our great pleasure to welcome you to the 5th Workshop on Evaluation and Usability of Programming Languages and Tools -- PLATEAU'14. This year's workshop continues its tradition of being the forum for presentation of research results and experience reports on methods, metrics, and techniques for evaluating the usability of languages and language tools. PLATEAU gives researchers and practitioners a unique opportunity to share their perspectives with others interested in the various aspects of evaluation of mprogramming languages and tools. Putting together PLATEAU 2014 was a team effort. We first thank the authors for providing the content of the program. We are grateful to the program committee who worked very hard in reviewing papers and providing feedback for authors. Finally, we thank the hosting organization SPLASH, sponsors: ACM and SIGPLAN, conference supporters, and the SPLASH Workshop Co-Chairs Stephanie Balzer and Du Li. Workshop Overview Programming languages exist to enable programmers to develop software effectively. But how efficiently programmers can write software depends on the usability of the languages and tools that they develop with. The aim of this workshop is to discuss methods, metrics and techniques for evaluating the usability of languages and language tools. The supposed benefits of such languages and tools cover a large space, including making programs easier to read, write, and maintain; allowing programmers to write more flexible and powerful programs; and restricting programs to make them more safe and secure. We plan to gather the intersection of researchers in the programming language, programming tool, and human-computer interaction communities to share their research and discuss the future of evaluation and usability of programming languages and tools. We are also interested in the input of other members of the programming research community working on related areas, such as refactoring, design patterns, program analysis, program comprehension, software visualization, end-user programming, and other programming language paradigms. Main Themes and Goals Following on from the four previous workshops at SPLASH, this workshop aims to bring together practitioners and researchers interested in discussing usability and evaluation of programming languages and tools with respect to language design and related areas. We will consider: empirical studies of programming languages; methodologies and philosophies behind language and tool evaluation; software design metrics and their relations to the underlying language; user studies of language features and software engineering tools; visual techniques for understanding programming languages; critical comparisons of programming paradigms, such as object-oriented vs. functional; and tools to support evaluating programming languages. We have two goals: Develop and sustain a research community that shares ideas and collaborates on research related to the evaluation and usability of languages and tools. Encourage the languages and tools communities to think more critically about how usability affects the design and adoption of languages and tools. Workshop Progam For the workshop program we solicited three kinds of papers: research papers, position papers, and hypothesis papers. Research papers can describe work-in-progress or recently completed work on the themes and goals of the workshop or related topics. Position papers can report on experience, question accepted wisdom, raise challenging open problems, or propose speculative new approaches. Hypothesis papers will identify and collect the unsubstantiated beliefs of the research community or software industry. The hypotheses can be collected from mailing lists, blog posts, paper introductions, developer forums, or interviews. Authors were encouraged to document the source or sources of each hypothesis and to discuss the impact of the hypotheses on research or practice. Research papers were limited to 10 pages in length and both position and hypothesis papers limited to 2 pages in length. The call for papers attracted submissions from the following countries: Australia, Austria, Canada, Chile, China, Germany, New Zealand, and the United States. The program committee reviewed and accepted the following papers: Track Reviewed Accepted Rate Research Full Papers 8 5 63% Position / Hypothesis Papers 6 5 83%
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.061 | 0.081 |
| Meta-epidemiology (narrow) | 0.002 | 0.002 |
| Meta-epidemiology (broad) | 0.002 | 0.002 |
| Bibliometrics | 0.004 | 0.002 |
| Science and technology studies | 0.003 | 0.004 |
| Scholarly communication | 0.017 | 0.013 |
| Open science | 0.004 | 0.010 |
| Research integrity | 0.004 | 0.006 |
| Insufficient payload (model declined to judge) | 0.048 | 0.016 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".