How far can you rely on a concept test: the generalizability of testing over occasions
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
In practice, product managers have to assume consumer evaluations of concepts generalise from the time (and research environment) of concept testing to the time (and market environment) of market introduction. However, little is known about the temporal stability or generalisability of the results of concept testing over occasions. Rarely have concept-testing studies incorporated testing of the same concepts on the same respondents on more than one occasion. This research investigates the importance of occasions as a source of error variance in estimates of the generalisability of concept test scores for both minor and major innovations within the context of Generalisability theory. The study collected concept evaluations of ten innovations from members of an online panel on three occasions, approximately a month apart. The results show that the three-way interaction among subjects, concepts and occasions is a substantial contributor to variation in concept testing of both major and minor innovations, with the contribution for major innovations even more substantial than for minor innovations. Moreover, failure to recognize occasions as an explicit source of variance in the generalisability analyses will lead managers to overestimate the generalisability of their decision studies. However, the impact of neglecting occasions varies by purpose of measurement and associated object of measurement. This research provides insight about how well concept testing can generalise over occasions. Concept test evaluations provided on an initial exposure are more favourable than will be received on any later occasions, and apparent differences in consumer evaluations of a particular concept in an initial test do not provide a generalisable basis for identifying which consumers will respond most favourably to it on a later occasion. For concept testing to be used for targeting or segmentation, more occasions will need to be sampled.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.015 | 0.050 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.001 | 0.001 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.003 | 0.000 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.001 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it