MétaCan
Menu
Back to cohort
Record W6925515292 · doi:10.17605/osf.io/yk35t

Face Puzzle Study 4 Preregistration

2022· other· en· W6925515292 on OpenAlex

Why this work is in the frame

A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.

aboutThe title or abstract carries a Canadian signal from the geographic lexicon.
no affNo Canadian affiliation: this work is invisible to an affiliation-only frame.
No Canadian affiliation. An affiliation-only frame, the usual design, would never have seen this work. It is one of the works that make the case for inverting the frame.

Bibliographic record

VenueOpen Science Framework · 2022
Typeother
Languageen
FieldMedicine
TopicReproductive Biology and Fertility
Canadian institutionsnot available
Fundersnot available
KeywordsFacial expressionNeurotypicalEmotion classificationBoredomEveryday lifeSet (abstract data type)Affective scienceFacial Action Coding SystemStimulus (psychology)Jealousy

Abstract

fetched live from OpenAlex

Identifying a person’s emotions through their facial expressions is necessary in helping us navigate social interactions. Some individuals, such as those with autism spectrum disorder (ASD), have difficulties in accurately identifying emotions from faces. These difficulties affect everyday interactions and contribute to the diagnostic phenotype. Facial emotion recognition (FER) tasks have been developed in an attempt to measure facial expression recognition abilities in both neurotypical and psychiatric populations in order to assess and quantify potential impairments. These tasks are also useful in retesting participants undergoing social skills or other training in order to track any improvements. FER tests to date often suffer from two limitations. First, stimulus sets used in established FER tasks are often limited to basic emotions (happiness, sadness, anger, disgust, fear, and surprise; plus neutral expressions as baseline), and add no or only few complex emotions (e.g., jealousy and boredom (Montagne et al., 2007)). Using only basic emotions to quantify FER abilities could potentially result in reduced external validity of the tasks by not accounting for the wide array of more complex emotions encountered in everyday life. Second, FER tasks often use static images of emotional faces, which do not capture naturalistic and dynamic aspects facial emotion recognition. Previously, the Face Puzzle tasks addressed these limitations by utilizing dynamic video stimuli featuring 15 actors portraying a wider variety of emotions to more closely approximate real life facial emotion recognition. (Kliemann et al., 2013). This stimulus set consisted of 25 videos of emotional facial expressions, with 5 basic (angry, happy, disgusted, fearful, surprised) and 20 complex emotions (interested, amused, aggrieved, troubled, jealous, enthusiastic, apologetic, disappointed, relieved, expectant, bored, compassionate, contemptuous, pardoning, embarrassed, wistful, furious, content, confident, doubtful), for a total of 11 positive and 14 negative emotions. In an initial validation study the Face Puzzle tasks showed good internal consistency, consistent external validity and sensitivity to impaired FER in adult individuals with ASD. Originally, the stimuli and task were designed in German, leaving it an open question whether intended emotion expressions and respective labels are valid in the English language, and thus whether the task is valid for use in English as well. The overall aim of this project is thus to validate the stimulus set and task design for the English language. In Study 1 of this project, believability, valence and arousal of video stimuli were rated and a new set of validated video stimuli was established (see Study 1 preregistration for details on the process; resulting emotion items are compassionate, bored, wistful, surprised, relieved, envious, furious, worried, enthusiastic, expectant, disgusted, angry, happy, forgiving, doubtful, content, embarrassed, disappointed, interested, fearful, confident, apologetic, contemptuous, amused, and touched). In Study 2, we determined construct validity of the items combined into the new Face Puzzle explicit task. The outcome of the procedure (see Study 2 for details) fell short of the original aim of a Cronbach’s alpha of 0.7 (Tavakol & Dennick, 2011) with a value of 0.683. It is possible that at least two factors might be relevant to evaluate this result. First, task performance in the Face puzzle task may be influenced by verbal intelligence and/or education levels. Second, we did not measure other emotional face processing or other social cognitive functioning tasks in online subjects or compared it to atypical social cognitive populations, such as autism, making it challenging to evaluate performance and external validity on the Face Puzzle task. To address these issues, we will conduct Study 3 (see details on preregistration for Study 3) and 4 as follows: In order to assess whether these results are reliable, we will conduct an item analysis to determine the construct validity of the explicit task using a set of non-mTurk participants and including a sample with reported difficulties in social cognition in general, and facial emotion recognition in specific (Lozier et al., 2014). Internal consistency will then be measured by calculating Cronbach’s alpha. We expect to find a high internal consistency (> 0.7) when conducting the task with non-mTurk participants (Hypothesis 1). In this study, we will also assess the English version of the Face Puzzle explicit task’s sensitivity to atypical social cognition. To this end, we will compare performance (accuracy and reaction times) of individuals with ASD to a matched (age, gender, IQ) neurotypical control group. We expect to find group effects in accuracy, reaction times, and composite measures (reaction time over % accuracy) as follows: Accuracy: We expect higher accuracies on the Face Puzzle explicit task for the NT than ASD participants (Hypothesis 2a). Reaction times: We expect faster reaction times on the Face Puzzle explicit task for NT than ASD participants (Hypothesis 2b). Composite measures: We expect lower composite measures of reaction time and accuracy (reaction time over % accuracy for correct items to yield accuracy-adjusted response times) for NT than ASD participants (Hypothesis 2c). We will also assess relation of Face Puzzle performance to other social cognitive tasks: Here, we expect that performance on the Face Puzzle explicit task to positively correlate with performance on the Reading the Mind in the Eyes Test (RMET; Baron-Cohen et al., 2001a; Hypothesis 3a), the Penn Emotion Recognition Test (ER-40; Kohler et al., 2003; Hypothesis 3b), and the Bell Lysaker Emotion Recognition Task (BLERT; Bell et al. 1997; Hypothesis 3c). Regarding the relationship with intellectual functioning, we expect performance on the Face Puzzle explicit task to either show no relation to the verbal subscale of the Kaufman Brief Intelligence Test, 2nd edition (KBIT-2; Kaufman & Kaufman, 2004; Hypothesis 4a) or a weak positive correlation (Hypothesis 4b). Regarding the relationship with autistic traits, we expect performance on the Face Puzzle explicit task to show a negative correlation with the Autism Quotient (AQ; Baron-Cohen et al., 2001b; Hypothesis 5a) for both ASD and NT groups, measured per group. We also expect a negative correlation with the Ritvo Autism Asperger’s Diagnostic Scale (RAADS-R; Ritvo et al., 2011; Hypothesis 5b) for the ASD group. Regarding the relationship with alexithymia, we expect performance on the Face Puzzle explicit task to show a negative correlation with performance on the Toronto Alexithymia Scale (TAS; Bagby et al., 1986; Hypothesis 6) across both groups.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.

Direct model labels (unvalidated)

Per-model category and study-design labels from the labeling rounds. They are machine output, unvalidated, and the disagreement between models ships as data. No study design here is MEDLINE-validated yet.

Model armCategoriesStudy designConfidence
gemmaMetaresearch
Domain: Reproducibility · Genre: Protocol
About the Canadian research system: no · About a Canadian topic: no
Not applicablehigh
gptOpen science
Domain: not available · Genre: Protocol
About the Canadian research system: no · About a Canadian topic: no
Other designhigh
models splitAgreement compares identical category sets and study designs across arms.

Full frame distilled prediction

Teacher imitation

Not calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.

metaresearch head score (Codex)0.002
metaresearch head score (Gemma)0.001
Version: codex-gemma-dda1882f352aValidation status: machine_predicted_unvalidated
Candidate categoriesInsufficient payload (model declined to judge)
Consensus categoriesnone
DomainCandidate signal: none · Consensus signal: none
Study designCandidate signal: Not applicable · Consensus signal: Not applicable
GenreCandidate signal: Other · Consensus signal: Other
Teacher disagreement score0.224
Threshold uncertainty score0.981

Codex and Gemma teacher scores by category

CategoryCodexGemma
Metaresearch0.0020.001
Meta-epidemiology (narrow)0.0000.000
Meta-epidemiology (broad)0.0000.000
Bibliometrics0.0000.001
Science and technology studies0.0000.001
Scholarly communication0.0000.000
Open science0.0020.001
Research integrity0.0000.001
Insufficient payload (model declined to judge)0.0200.000

Machine scores (provisional)

The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.

Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.

Opus teacher head0.051
GPT teacher head0.394
Teacher spread0.343 · how far apart the two teachers sit on this one work
Validation statusscore_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it