Bibliographic record
Abstract
Experimental research has been an important research method in the Information Systems (IS) discipline.Recently, we have seen an expansion in the types of experiments conducted beyond traditional laboratory (lab) and field experiments.These new types of experiments, which leverage the online environment, provide new opportunities as well as new challenges for IS researchers.This diversity also creates the need for authors and reviewers to understand the respective strengths and limitations of various types of experimental research, and not mechanically apply the lens of their favorite type of experiment.The purpose of this editorial is to highlight the reasons that have propelled new types of experiments, categorize these along a set of dimensions, discuss their strengths and weaknesses, and highlight some new issues that emerge with these new opportunities for research.Our objective is not to be exhaustive in terms of the various types of experiments but to highlight opportunities and challenges that emerge for online variants that are more prominently seen in IS research.We, therefore, constrain our focus to lab, field, and natural experiments and their online variants. 1 Changing Landscape of Experiments in IS ResearchExperiments have been a major research method in IS research since the origins of the field.We have recently seen a stronger interest in experiments, especially those occurring online.This can be attributed to the Internet providing two sets of opportunities: (1) a field setting for experimentation as a prominent locus of economic transactions and social interactions (for field and natural experiments), and (b) opportunities to recruit larger subject pools more efficiently and reach more diverse samples with reduced administrative and financial costs (for lab experiments) (Hergueux and Jacquemet 2015).Online transactions and interactions have created both the need and the opportunity for online field experiments to understand the various types of social and economic activities in which people engage online.The availability of persistent trace data for these online transactions and inter-1 Harrison and List (2004) use six criteria to define the "field" context of an experiment: "the nature of the subject pool, the nature of the information that the subjects bring to the task, the nature of the commodity, the nature of the task or trading rules applied, the nature of the stakes, and the nature of the environment that the subject operates in" (p.1012).Based on these characteristics, they classify experiments into four categories: a traditional lab experiment, an artefactual field experiment (lab experiment but with subjects that are representative of the population), a framed field experiment, and a natural field experiment.Their first two categories correspond to lab experiments, whereas the third and fourth categories correspond to field and natural experiments, respectively.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.000 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".