Measuring Perceived Risk of Pitfalls Associated with Systems Engineering Tradeoff Analyses
Bibliographic record
Abstract
The U.S. Department of Defense (DoD) has recently revised the defense acquisition system to address suspected root causes of unwanted acquisition outcomes. One of the major changes in the revised acquisition system is an increased emphasis on systems engineering trade-offs made between capability requirements and lifecycle costs early in the acquisition process (Cilli, Parnell, Cloutier, & Zigh, 2015). Given that systems engineering trade-off analyses will play a pivotal role in future defense acquisition efforts, this paper takes an in-depth look at the state of systems engineering trade-off analysis capability through a review of relevant literature and a survey of systems engineering professionals and military operations research professionals involved in defense acquisition. The survey was developed to measure the perceived level of difficulty associated with compliance to the revised defense acquisition system mandate for early systems engineering trade-off analyses and to measure perceived likelihood and impact of potential pitfalls within systems engineering trade-off studies. The survey instrument was designed using Survey Monkey and was deployed through a link posted on several groups within LinkedIn, a professional social media site, and was also sent directly via email to those with known experience in this research area. Although increased systems engineering activity early in the life cycle is a compelling change for DoD, the findings of the literature review and the survey of practitioners both indicate that there is much to be done in order to position the systems engineering community for success so that the improved defense acquisition outcomes as envisioned by the architects of 2015 DoDI 5000.02 can be realized.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.037 | 0.230 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.000 | 0.001 |
| Bibliometrics | 0.004 | 0.002 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.003 | 0.004 |
| Open science | 0.001 | 0.003 |
| Research integrity | 0.001 | 0.002 |
| Insufficient payload (model declined to judge) | 0.002 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".