Avoiding Confirmation Bias in a Safety Management System (SMS)
Bibliographic record
Abstract
Confidence in a safety management system (SMS) could be undermined by a tendency to search for, interpret, choose, or recall information in a way that supports a favorable view of how safety risk is managed in an organization. This tendency, sometimes called "confirmation bias", might be unintentional. Otherwise, it might be the result of a misguided loyalty to an organization, a reluctance to "open a can of worms" or even an intention to mislead or deceive. A temptation to rely on Generative AI as a substitute for critical thinking in the development and maintenance of a SMS might also be a source of confirmation bias. This paper proposes the use of Eliminative Argumentation (EA) to avoid confirmation bias in a SMS. Our approach is an adaptation of an innovative method for safety assurance developed by researchers at Carnegie Mellon University. At the heart of the method, confidence can be increased by identifying potential doubts, and then eliminating these doubts where it is reasonable to do so. A hierarchically structured argument for confidence in the SMS can be explicitly documented using a simple notation that mirrors the structure of the SMS. Following the structure described in Federal Aviation Administration (FAA) Order 8000.369C, the main branches of this argument cover the four components of a SMS, namely the Safety Policy, Safety Risk Management (SRM), Safety Assurance, and Safety Promotion. In turn, each of these main branches are further split into subbranches. For example, one of the sub-branches of the SMR branch of the argument addresses the need to track identified hazards and monitor implemented safety risk controls/ mitigations to ensure that they achieve their intended safety performance targets. To this level, the argument is an instance of a template that can be re-used for other instances of a SMS based on FAA Order 8000.369C. Lower levels of this tree-shaped argument are tailored to the specific details of how the SMS is implemented and maintained. For example, there might be doubts, also known as "defeaters", in the argument at this level that challenge the process used by the organization to decide that particular hazards are adequately controlled. Such defeaters are often expressed in the form of "What if ...?" questions. For example, "what if a decision that an identified hazard is mitigated is made by an unqualified person?". Such defeaters are often eliminated by supplementary details - for example, details about the minimum qualifications of personnel in a particular role. Using appropriate tool support, the structured argument for confidence in the SMS can be linked to Key Performance Indicators (KPI) to measure the effectiveness of the SMS. For example, one such KPI measures how many days on average it takes to analyze and resolve safety issues raised by personnel. If and when this measured value exceeds a defined threshold, it will be flagged as a problem for the SMS. Among other influences on a SMS, this paper will also take account of the impact of relying on Generative AI. In summary, the first contribution of this paper is showing how trust in a SMS can be increased by means of Eliminative Argumentation, especially as a means of avoiding confirmation bias. The second contribution of this paper is showing how KPIs can be used to further increase confidence in a SMS.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.032 | 0.083 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.003 | 0.001 |
| Science and technology studies | 0.003 | 0.011 |
| Scholarly communication | 0.005 | 0.011 |
| Open science | 0.002 | 0.007 |
| Research integrity | 0.004 | 0.004 |
| Insufficient payload (model declined to judge) | 0.003 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".