Expert- and Nonexpert-friendly Framework for Deep Learning Image Segmentation Demonstrates Successes Across Applications in vEM, CryoEM, MicroCT and Fluorescence Microscopy

Convolutional neural networks (CNNs) have demonstrated tremendous capacity for object classification and labeling, image denoising, and image segmentation tasks in demanding biological applications. Numerous CNN solutions have been reported over the past decade for these applications, but they are frequently specialized for a particular imaging modality, and they are often nontrivial to deploy by nonexperts. Further, there persists a widespread misconception that deep learning networks require enormous volumes of annotated training data to reach levels of high accuracy and high robustness. Here we present an update on Segmentation Wizard [1], a framework that encapsulates and organizes training data with associated CNN models for training. Segmentation Wizard exists as a feature in the commercially available deep learning enabled Dragonfly image analysis software [2], licensed at no cost for non-commercial use in most territories. We report two novel enhancements to Segmentation Wizard, image intensity calibration and access to general-purpose pre-trained models. We show here how inexperienced users can prepare and maintain ground truth segmentations and train multiple models simultaneously to high performance and robustness, often with less than 5MPix of curated ground truth. We demonstrate example successes in stained tissue volume EM (vEM) (both FIBSEM and SBFSEM), unstained cellular cryoEM (cryoET and cryoFIBSEM), x-ray microCT, and fluorescence microscopy. Segmentation Wizard is installed by default with the standard installer of Dragonfly for both Windows and Linux, negating any requirements for configuration expertise. Updated Dragonfly installers are provided one to three times every year, delivering feature updates, but also support for the latest generation of CUDA-enabled GPUs. The initial release of deep learning tools in 2018 was built on TensorFlow 1.6. Frequent software releases have shipped to track with TensorFlow updates; the current production release is on TensorFlow 2.4, and the next update will be linked against 2.11. One of the primary benefits of Segmentation Wizard is to provide a user-friendly experience to nonexperts. Tuning of hyperparameters and data augmentation settings are available by experts, who may also import Keras models or manually design and revise novel model architectures. But nonexpert users are encouraged to accept default settings which are well-optimized for most cases. After users have prepared frames of labeled data to be taken as ground truth, the framework lets users train and compare multiple CNN models. The variety of models includes multiple U-Net derivatives as well as DenseNet architectures and others. The models are classified as 2D, 2.5D, or 3D, to indicate that they consider only one slice of grayscale image data at a time, multiple slices, or multiple slices with 3D convolutional kernels. Through the use of general-purpose pretrained models and aggressive data augmentation, models require only a small volume of training data, and training new models is rapid and not very computationally expensive. In cases where users have followed the intensity calibration protocol, model training and inference happens on calibrated intensities rather than raw image intensities, but intensity calibration is always reversible. Model training can be monitored in real time, with the conclusion of training each epoch, the user is presented with a visualization of the model prediction for that epoch; this is greatly preferred to monitoring only the numerical loss function or waiting until training is concluded to visualization the model’s predictive performance. Finally, we note that Segmentation Wizard boosts productivity by letting users make intermediate predictions which can easily be corrected and inserted into the training volume for further training. Application to stained vEM tissues is useful for labeling image data coming from FIBSEM and SBFSEM experiments. We showcase here the work on human retinal pigment epithelium and adjoining neural tissue [3]. These samples are often large volumes (200GB image datasets) spanning many lateral microns of tissue. The segmentation we present shows that segmentation can resolve and label more than 10 classes of cellular ultrastructure (Fig 1) with high fidelity. Left: 2D rendering of 2D U-Net segmentation of SBFSEM of human retinal pigment epithelium and adjacent photoreceptor outer segments. Right: 3D rendering of 2D U-Net segmentation of cryo-PFIBSEM of a Chlamydomonas cell (EMPIAR 11275). Previous success in cryoEM data has been demonstrated and described in general [4] and with training data augmented by digital phantoms [5]. We show now this success extends to cryo-PFIBSEM (Fig 1) as seen on data generously shared publicly (EMPIAR-11275) [6]. It is notable because this 13-class model was trained on only five slices of the dataset and was segmented by a nonexpert referencing a cartoon diagram of Chlamydomonas cellular ultrastructure. This automated segmentation of this sample was achieved with fewer than three days of investigator time. This trained model can be applied to additional datasets of the same type without any additional training. With continuous refinement since it was first described in 2018, Segmentation Wizard is now a mature, user-friendly, and intuitive platform for training deep learning models for any biological imaging modality. Both novice and advanced users will find it useful and easy to use with multiple published tutorials for various use cases. Recent enhancements further drive high productivity for automated segmentation across a wide application space for imaging scientists working in biological systems.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.