AN EXPERIMENTAL APPROACH TO THE CONSTRUCTION OF BINARY DECISION CLASSES FROM CARD SORT DATA
Bibliographic record
Abstract
This thesis presents work done towards understanding the data collected from a card sorting study of facial photographs. In that study, 25 participants sorted 356 photos (178 Caucasian and 178 First Nations) into piles based on similarity. Photos placed in the same pile are deemed to be similar, photos in different piles are deemed to be dissimilar. Looking to establish binary decision classes is reasonable because the underlying question that participants answered was “Are these photos similar or not?”. There may also be more than two decision classes to describe all the behaviours. For example, an initial split into decision classes may be thought of as “doing something” and “not doing something”. The latter could be split into two, and the whole process repeated. Differences amongst the sorting behaviours of participants are evident, but the reason for these differences is difficult to determine. An early hypothesis was that perceived race was being used a criterion for some participants but not for others. An analysis that looked at the ratio of Caucasian and First Nations photos in each pile was used determine a pair of decision classes from which accurate classifiers could be built. Open questions from that earlier work include the basis for participants making those decisions and whether the behaviour supported by a small amount of carefully chosen data would be supported by all the data. There are several million possible decision class pairs that could be used to split those 25 participants into 2 groups. This work applies a knowledge discovery approach to find other candidate decision classes for this data, for which accurate classifiers can also be built. Each participant made a relatively small number of direct comparisons and a large number of indirect comparisons to determine whether each pair of photos (63,190 in all) was similar or dissimilar. For each pair, a binary vector was used to record the judgement of each participant (0 if the participant thought the pair was similar, 1 if dissimilar). These vectors were used as the basis for the present study. Each pair of photos can be said to have a certain power to discern between participants. If all participants gave the same judgement for a pair, the pair has no power to discern between participants. Conversely, a pair which 12 or 13 participants had rated similar (or dissimilar) has the highest power to discern between participants, because for this pair there will be the most disagreement when considering pairs of participants. This work focuses on those pairs with maximum discernibility. To generalize earlier work on a heuristic for evaluating candidate decision classes, it is hypothesized that a t-test could be used to give a better indication about the suitability of a decision class pair. To this end, some experimental analysis of the card sorting study data was undertaken and the rough set attribute reduction methodology was used to evaluate the findings from the computational experiment. ii
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.000 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.001 |
| Open science | 0.003 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".