Low-Cost Fine-Tuning of Data-Efficient Image Transformers on Knee X-Ray Imaging for Osteoarthritis Detection
Bibliographic record
Abstract
Introduction: Technological advancements in artificial intelligence within the field of medicine, specifically skeletal pathology, have witnessed exponential growth in recent years. Researchers have trained deep learning models on radiographs to improve the detection of diseases. There is precedence for low data training on musculoskeletal imaging tasks, and “specialized” medical imaging-related tasks in general have exhibited high performance on low amounts of data. Thus, the Data-efficient Image Transformer (DeiT) model has potential to surpass conventional convolutional models in detecting musculoskeletal diseases due to its ability to extract relevant features with scarce data. Methods: This study utilizes a DeiT model pre-trained on ImageNet 2012. The model was fine-tuned on labelled knee X-rays of patients with and without osteoarthritis using the PyTorch library. Fine-tuning and testing were done on a Google Colab notebook using a T4 GPU. A hyperparameter sweep cycling through different dropout values, optimizers, and image input sizes was tested. Results were recorded using multiple accuracy metrics. Results: The DeiT-B 384 model with unspecified dropout had the highest scores out of all the model variations. The DeiT’s composite performance in diagnosing knee osteoarthritis exceeded that of convolutional models. Fine-tuning resulted in accuracies within the standard established by current literature for the 384 model, but not for the 224 model. This suggests that larger input models trained on lower quality images performed better than smaller input models trained on higher quality images. Implications: Fine-tuning can be an alternative to training from scratch for medical imaging. Future studies should expand their scope by taking additional patient details into account to increase diagnostic accuracy and provide local and individualized patient care. The absence of these variables in our model’s training potentially limited its accuracy. Current real-time diagnostic errors are less than even the best performing computer vision models, so accuracy must significantly improve before there is incentive to adopt these methods.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.008 | 0.003 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.002 | 0.003 |
| Science and technology studies | 0.001 | 0.005 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.002 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".