On the proper use of structural similarity for the robust evaluation of medical image synthesis models
Notice bibliographique
Résumé
PURPOSE: To propose good practices for using the structural similarity metric (SSIM) and reporting its value. SSIM is one of the most popular image quality metrics in use in the medical image synthesis community because of its alleged superiority over voxel-by-voxel measurements like the average error or the peak signal noise ratio (PSNR). It has seen massive adoption since its introduction, but its limitations are often overlooked. Notably, SSIM is designed to work on a strictly positive intensity scale, which is generally not the case in medical imaging. Common intensity scales such as the Houndsfield units (HU) contain negative numbers, and they can also be introduced by image normalization techniques such as the z-normalization. METHODS: We created a series of experiments to quantify the impact of negative values in the SSIM computation. Specifically, we trained a three-dimensional (3D) U-Net to synthesize T2-weighted MRI from T1-weighted MRI using the BRATS 2018 dataset. SSIM was computed on the synthetic images with a shifted dynamic range. Next, to evaluate the suitability of SSIM as a loss function on images with negative values, it was used as a loss function to synthesize z-normalized images. Finally, the difference between two-dimensional (2D) SSIM and 3D SSIM was investigated using multiple 2D U-Nets trained on different planes of the images. RESULTS: The impact of the misuse of the SSIM was quantified; it was established that it introduces a large downward bias in the computed SSIM. It also introduces a small random error that can change the relative ranking of models. The exact values for this bias and error depend on the quality and the intensity histogram of the synthetic images. Although small, the reported error is significant considering the small SSIM difference between state-of-the-art models. It was shown therefore that SSIM cannot be used as a loss function when images contain negative values due to major errors in the gradient calculation, resulting in under-performing models. 2D SSIM was also found to be overestimated in 2D image synthesis models when computed along the plane of synthesis, due to the discontinuities between slices that is typical of 2D synthesis methods. CONCLUSION: Various types of misuse of the SSIM were identified, and their impact was quantified. Based on the findings, this paper proposes good practices when using SSIM, such as reporting the average over the volume of the image containing tissue and appropriately defining the dynamic range.
Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.
Comment cette classification a été obtenuedéplier
Prédiction machine sur la base complète
Imitation des enseignantsNi prévalence calibrée, ni vérité terrain. Validation humaine à venir. Le volet Gemma est une étiquette directe du modèle pour chaque travail de la base, lue sur la notice réduite au titre. Le volet Codex est un classifieur appris des 10 348 étiquettes directes de Codex et calibré sur les taux pondérés de l'échantillon; les champs sans appui suffisant ne portent aucun appel Codex. Le mode candidate est l'union des deux volets; le consensus est leur intersection. Ces sorties portent le statut machine_predicted_unvalidated et ne sont pas des étiquettes humaines.
Scores du classifieur distillé par catégorie (deux têtes)
| Catégorie | Codex | Gemma |
|---|---|---|
| Métarecherche | 0,012 | 0,062 |
| Méta-épidémiologie (sens strict) | 0,002 | 0,000 |
| Méta-épidémiologie (sens large) | 0,001 | 0,001 |
| Bibliométrie | 0,002 | 0,001 |
| Études des sciences et des technologies | 0,001 | 0,002 |
| Communication savante | 0,002 | 0,003 |
| Science ouverte | 0,002 | 0,003 |
| Intégrité de la recherche | 0,002 | 0,002 |
| Charge utile insuffisante (le modèle a refusé de juger) | 0,002 | 0,001 |
Scores machine (provisoires)
Les deux têtes enseignantes du modèle étudiant, lues sur ce travail. Un score ordonne la base pour la relecture; il n'affirme jamais une catégorie, et le statut de validation accompagne chaque rangée tel quel.
Scores de référence d'un modèle non mature (critères de maturité non atteints, 7 itérations). Un score ordonne; il n'affirme jamais une catégorie.
score_only:v0-immature-baseline · tel quel depuis la passe de notation : score_only signifie que le nombre peut ordonner les travaux, et qu'aucune étiquette de catégorie n'en découleClassification
machine, non validéePrédiction automatique; un appel candidat d’une seule source (Gemma direct ou Codex distillé), pas un consensus.
Le détail, modèle par modèle et score par score, se trouve en fin de page sous « Comment cette classification a été obtenue ».