A multi-criteria decision analysis framework for evaluating deep learning models in healthcare research
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
Selecting the appropriate deep learning (DL) model for healthcare research poses a significant challenge due to the diversity of evaluation criteria and the complex nature of health-related tasks, where a single metric like accuracy is often insufficient. Motivated by the need for a structured, multi-criteria approach, this study proposes a Multi-Criteria Decision Analysis (MCDA) framework using the Analytic Hierarchy Process (AHP). Our primary contribution is the development of a comprehensive decision-making framework that integrates multiple evaluation criteria, such as accuracy, sensitivity, specificity, and computational complexity, alongside empirical data from existing literature to systematically compare DL models. The framework was validated through a use case involving the selection of the best DL model for diagnosing COVID-19 using X-ray images, where we compared eight popular models, including ResNet34, SqueezeNet, and AlexNet, and it was also evaluated through comparative scenarios using traditional methods, including weighted sum, weighted average, and accuracy-based evaluation. Quantitative results show that SqueezeNet achieved the highest score in the AHP framework (88.64), while ResNet34 performed best in traditional methods such as weighted sum (588.49) and accuracy ranking (98.33%). A sensitivity analysis further demonstrated the impact of varying criteria weights, showing how changes in the importance of accuracy and precision, influenced model ranking. These findings highlight the flexibility and robustness of the AHP framework in addressing the complexities of model selection in healthcare research. The implications of this work suggest that a structured, data-driven evaluation approach can provide more nuanced and reliable insights compared to traditional methods like single-metric evaluations, ultimately supporting more informed decision-making in healthcare applications. • Introduce a multi-criteria framework for evaluating deep learning in healthcare. • Identify key evaluation criteria for deep learning through a literature review. • Balance performance and complexity in deep learning model selection. • Validate the framework with a case study on diagnosing COVID-19 using deep learning. • Compare the framework to other methods, like weighted average, to show effectiveness.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.021 | 0.011 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.005 | 0.008 |
| Science and technology studies | 0.001 | 0.000 |
| Scholarly communication | 0.002 | 0.001 |
| Open science | 0.002 | 0.001 |
| Research integrity | 0.000 | 0.003 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it