Meta-informational Cue Inconsistency and Judgment of Information Accuracy: Spotlight on Intelligence Analysis
Bibliographic record
Abstract
Meta-information is information about information that can be used as cues to guide judgments and decisions. Three types of meta-information that are routinely used in intelligence analysis are source reliability, information credibility and classification level. The first two cues are intended to speak to information quality (in particular, the probability that the information is accurate), but classification level is not (it is intended as information about information sensitivity). Two experiments involving professional intelligence analysts (N = 25 and 27, respectively) manipulated meta-information in a 6 (source reliability) x 6 (information credibility) x 2 (classification) repeated-measures design. Ten additional items were retested to measure interindividual reliability (82 trials in total). Analysts judged the probability of information accuracy based on its meta-informational profile. In both experiments, the judged probability of information accuracy was sensitive to ordinal position on the scale and directionality. That is, the directionality of linguistic terms used to anchor the levels of the two scales led analysts to group the first three levels of each scale in a positive group and the fourth and fifth levels in a negative group, with the directionally neutral “cannot be judged” options falling between these groups. Critically, as reliability and credibility cue inconsistency increased, there was a corresponding decrease in intra-analyst reliability, inter-analyst agreement, and effective cue utilization in judgment. In Experiment 2, there was also evidence of a secrecy bias among analysts in which “TOP SECRET” intelligence was judged more likely to be accurate than identical intelligence marked “OFFICIAL.”
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.002 | 0.001 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.075 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".