Bibliographic record
Abstract
Experimental research has been an important research method in the Information Systems (IS) discipline.Recently, we have seen an expansion in the types of experiments conducted beyond traditional laboratory (lab) and field experiments.These new types of experiments, which leverage the online environment, provide new opportunities as well as new challenges for IS researchers.This diversity also creates the need for authors and reviewers to understand the respective strengths and limitations of various types of experimental research, and not mechanically apply the lens of their favorite type of experiment.The purpose of this editorial is to highlight the reasons that have propelled new types of experiments, categorize these along a set of dimensions, discuss their strengths and weaknesses, and highlight some new issues that emerge with these new opportunities for research.Our objective is not to be exhaustive in terms of the various types of experiments but to highlight opportunities and challenges that emerge for online variants that are more prominently seen in IS research.We, therefore, constrain our focus to lab, field, and natural experiments and their online variants.1 Changing Landscape of Experiments in IS ResearchExperiments have been a major research method in IS research since the origins of the field.We have recently seen a stronger interest in experiments, especially those occurring online.This can be attributed to the Internet providing two sets of opportunities: (1) a field setting for experimentation as a prominent locus of economic transactions and social interactions (for field and natural experiments), and (b) opportunities to recruit larger subject pools more efficiently and reach more diverse samples with reduced administrative and financial costs (for lab experiments) (Hergueux and Jacquemet 2015).Online transactions and interactions have created both the need and the opportunity for online field experiments to understand the various types of social and economic activities in which people engage online.The availability of persistent trace data for these online transactions and inter-1 Harrison and List (2004) use six criteria to define the "field" context of an experiment: "the nature of the subject pool, the nature of the information that the subjects bring to the task, the nature of the commodity, the nature of the task or trading rules applied, the nature of the stakes, and the nature of the environment that the subject operates in" (p.1012).Based on these characteristics, they classify experiments into four categories: a traditional lab experiment, an artefactual field experiment (lab experiment but with subjects that are representative of the population), a framed field experiment, and a natural field experiment.Their first two categories correspond to lab experiments, whereas the third and fourth categories correspond to field and natural experiments, respectively.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.016 | 0.152 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.002 | 0.002 |
| Bibliometrics | 0.003 | 0.002 |
| Science and technology studies | 0.004 | 0.002 |
| Scholarly communication | 0.009 | 0.007 |
| Open science | 0.005 | 0.003 |
| Research integrity | 0.012 | 0.012 |
| Insufficient payload (model declined to judge) | 0.133 | 0.073 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".