Emerging problems of data quality in citizen science

Why is this work in the frame?

A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.

Full frame distilled prediction

Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.

Candidate categories

Metaresearch, Open science

Consensus categories

Metaresearch

Domain

Candidate signal: noneConsensus signal: none

Study design

Candidate signal: Not applicableConsensus signal: Not applicable

Genre

Candidate signal: EditorialConsensus signal: Editorial

Teacher disagreement score

0.054

Threshold uncertainty score

1.000

Validation status

machine_predicted_unvalidated · codex-gemma-dda1882f352a

Codex and Gemma teacher scores by category

Machine scores (provisional)

The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.

Teacher spread

0.183 · how far apart the two teachers sit on this one work

Validation status

score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it

Abstract

The role of citizen science in research and natural resource monitoring and management is increasing, as evidenced by the growing number of peer-reviewed publications (including a special section in this journal) and calls for involving citizens in monitoring and governance (through, for example, “participatory research” [Danielsen et al. 2014] and “participatory monitoring” [Kennett et al. 2015]). Citizen science projects can be targeted to a specific research question (and thus involve very specific data-collection protocols) or can be more open-ended (giving rise to a need to collect data for which the uses may be unknown or changing) (Wiersma 2010). Advances in online content production and sharing technologies (i.e., Web 2.0), mobile computing, and sensor-equipped devices have contributed to a dramatic rise in online citizen science projects, in which citizens contribute sightings (e.g., eBird [Sullivan et al. 2009]), transcribe data (e.g., Old Weather [Eveleigh et al. 2013]), or classify phenomena (e.g., Galaxy Zoo [Hopkin 2007]). It is these online projects, also referred to as crowdsourcing (Franzoni & Sauermann 2014), which have been the focus of our research and that inform the opinions presented here. Galaxy Zoo exemplifies an initiative that began as a targeted project in which citizens were engaged in the relatively simple task of classifying images of galaxies as one of 3 shapes (Hopkin 2007). The goal was to distribute a large workload among a large number of people. Citizen participation grew quickly, which led project sponsors to create an online forum to accommodate the large volume of comments and questions. Through this forum, a number of unanticipated categories of celestial bodies arose, including 2 from Dutch school teacher Hanny Van Arkel, who noted the “green peas” phenomena (Cardamone et al. 2009) and a new body that became known as “Hanny's Voorwerp” (Lintott et al. 2009). The Galaxy Zoo story provides an example of the different dimensions of data quality in citizen science. The researchers anticipated a small, fixed set of categories of galaxy shapes and designed the data-collection interface accordingly. One dimension of data quality (Lewandowski & Specht 2015) is data accuracy; others include data completeness and timeliness. (For a complete discussion of the many dimensions of data quality, see Wang and Strong [1996]). In the case of Galaxy Zoo, data accuracy is measured as the proportion of images correctly classified by galaxy shape. Had it not been for the attentiveness of one person who went beyond the task of classifying galaxies into predetermined categories and was able to communicate this to the researchers via the online forum, what turned out to be important new phenomena might have gone undiscovered. Failure to discover these phenomena would have affected the data-quality dimension of completeness because not all celestial bodies in the images would have been cataloged. Thus, the data quality would be diminished. Lewandowski and Specht (2015) describe 4 dimensions of data quality in their broad review of biology-themed citizen science: data accuracy and precision; sufficient sample size; and standardized sampling procedures (including sufficient spatial and temporal representation). These dimensions are congruent with good scientific practice and thus suggest that the criteria used to measure the quality of citizens’ data should fit the standards of professional science. In this sense, citizen science amounts to asking citizens to fill in the blanks in a story written by scientists. Although it is helpful for citizen scientists to adhere to standards of scientific practice, the process of doing science includes more than simply collecting and processing data. As Stevens et al. (2014:21) admitted: “Often …participants might be viewed as sensors or data collectors, but they're rarely invited to decide what data to collect or to contribute to the data analysis or interpretation, even though they … might have valuable insights,” a view echoed in a recent Nature commentary by Kennett et al. (2015). The online forum created by the Galaxy Zoo project manifests a design decision that allowed for participants to provide valuable new insights and contribute beyond simply classifying images. Because discoveries resulted from one individual going beyond the assigned task, an open question is how many discoveries went undetected because other participants failed to notice particular features (given the prescribed task) or noticed but failed to post on the site's forum. Based on examples such as this one, we argue that data quality in citizen science is much more than data accuracy. Because citizens generally lack formal scientific training, they view problems and issues in light of their own knowledge and interests, creating fertile ground for discoveries. This perspective – that citizen scientists view problems differently than scientists – means that the quality of data should be defined as more than simply consistency with data collected under scientific protocols. Quality of data also includes the extent to which the design of a specific project facilitates citizens’ abilities to spot something interesting, unexpected, or novel. Rather than seeing inexperience and lack of formal scientific training as threats to data quality, we suggest these characteristics improve data quality, provided that researchers are able to understand how to take full advantage of them. It is also important to consider that citizen scientists are not a homogenous group. He and Wiggins (2015) characterize citizen scientists as members of “communities.” Such communities are thought to be at the opposite end of the spectrum of the larger “crowd” that is referred to in discussions on crowd sourcing. This community is a subset of the public with specific interests, whereas the crowd usually refers to a broader citizenry. These community members may have some training and expertise; thus, we consider them “expert amateurs” (Van Arkel is an example; she self-identifies as an amateur astronomer.) and thus not representative of the full suite of potential participants in online citizen science projects. This broader view of data quality in citizen science is consistent with prevailing conceptualizations within the information-based fields of computer science, information systems, and philosophy of information (Wang & Strong 1996; Floridi 2012). Research in these areas treats data quality as a multidimensional construct (e.g., Wang and Strong [1996] identify hundreds of dimensions). Consequently, caution is warranted in emphasizing a particular dimension of data quality in citizen science projects; trade-offs in different dimensions of data quality are inevitable (Pipino et al. 2002; Scannapieco et al. 2005; Batini & Scannapieca 2006). Recent empirical evidence shows that data completeness (because most people are excluded) may be compromised to increase data accuracy (if the task can only be completed by a member of the community or an expert amateur). Analysis of participation patterns and data collected by citizens suggests that one reason that accuracy in the identification of objects does not differ between experts and citizen scientists (Crall et al. 2011; Jordan et al. 2012; Nagy et al. 2012) is often because citizen scientists in these projects already have expertise and a high level of interest in the topic. For example, to participate in eBird, one must already have facility with, or at least interest in, bird identification; many dedicated birders have as good (or better) field identification skills than professional ornithologists. Many citizen science projects, therefore, may actually inhibit widespread participation because of the requirement to provide data at a level that matches the expertise of the project sponsor, thus resulting in a trade-off in 2 dimensions of data quality (Parsons et al. 2011). We contend that to truly engage a broad array of citizenry in science, projects should be designed to be as inclusive as possible, rather than limited to expert amateurs. Our proposed solution is to consider how project design influences quality. We have shown through laboratory and field experiments that data contributed through a flexible approach that allows nonexperts to provide descriptions of the observed organism (e.g., plant and animal) can have higher classification accuracy (Lukyanenko et al. 2014a) and greater numbers of observations reported (Lukyanenko et al. 2014b) than data contributed through traditional approaches to citizen science that require citizens to report observations using predetermined categories (e.g., species). We have explored alternative approaches to citizen science data collection in hopes of minimizing what seems to be an inevitable trade-off between data-quality dimensions. Specifically, we propose a flexible, instance-based approach to data collection that allows a contributor to classify data at the level at which they feel competent, rather than requiring participants to meet scientific standards that only expert amateurs may be capable of (Lukyanenko et al. 2014a). We contend that in trying to hold amateurs to scientific standards, researchers not only ask nonexperts to perform often unrealistic tasks, but also risk missing the opportunity to fully engage with people in the core objective of discovery. The emerging problem of quality in citizen science is, therefore, writing a story in which citizens contribute to the plot. Many of the ideas in this paper were formulated while R.L. was a PhD candidate and funded by an NSERC Canada Graduate Scholarship. We are grateful to our colleagues, especially R. Sieber and G. Wachinger, for insightful discussions and to A. Wals and one anonymous reviewer for comments that helped improve this manuscript.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.

The record

Venue

Conservation Biology

Topic

Data Quality and Management

Field

Decision Sciences

Canadian institutions

Memorial University of NewfoundlandUniversity of Saskatchewan

Funders

Natural Sciences and Engineering Research Council of Canada

Keywords

Citizen scienceCrowdsourcingCitizen journalismOpen scienceData scienceOpen dataResource (disambiguation)World Wide WebComputer sciencePolitical sciencePublic relations

Has abstract in OpenAlex

yes

API

/api/v1/works/W2284343953