Interobserver and Human–Artificial Intelligence Concordance in Differentiating Between Invasive and In Situ Melanoma
Bibliographic record
Abstract
Background Machine learning algorithms including convolutional neural networks (CNNs) have recently made significant advances in research settings. Even though several algorithms nowadays are targeted directly to the consumer market, their implementation in clinical practice is still pending. Most melanomas are easy to recognize even without the aid of dermoscopy. Nonetheless, it is often more challenging to discriminate between invasive melanoma and melanoma in situ (MIS) in a preoperative setting even with the assistance of dermoscopy. Although several dermoscopic features suggestive of MIS and invasive melanomas have been presented, their usefulness in a larger setting is limited by how well physicians agree on their presence or absence. Objective The overarching aims of this research project are to identify useful dermoscopic features to help dermatologists predict melanoma thickness and to develop CNNs that can assist dermatologists in the preoperative assessment of melanoma thickness. The ultimate aim is to develop algorithms that can strengthen patient care, improve clinical decision-making, and be used in routine health care. Methods We have included dermoscopic images as well as clinical close-up images of invasive melanomas and MIS from our department during the time period of January 1, 2016, to December 31, 2020. Using this image material, we have trained, validated, and tested two separated CNNs based on dermoscopic and clinical close-up images. We have also invited dermatologists to review the test sets and, for a subset of the dermoscopic images, asked them to specify the presence of prespecified dermoscopic features. Subsequently, we compared CNN outputs to the combined dermatologists’ output for all lesions and assessed the interobserver agreement for several dermoscopic features. Results The CNN developed using dermoscopic images performed on par with the invited dermatologists whereas the CNN using clinical close-up images was outperformed by the group of dermatologists. Two dermoscopic features (atypical blue-white structures and shiny white lines) both displayed a moderate to substantial interobserver agreement and were both indicative of invasive melanomas >1.0 mm. Conclusions CNNs used to differentiate between invasive melanomas and MIS might be an example of a clinically relevant machine learning application, but they need further refinement and evaluation in prospective clinical trials. Only a few dermoscopic features are helpful in distinguishing melanoma thickness. Conflicts of Interest None declared.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.000 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".