Computer‐aided diagnostics in digital pathology

Image Cytometry has old roots. Already in the 1930s Torbjörn Casperson measured UV light absorption in cell nuclei and showed that malignant nuclei absorbed more light than normal ones 1. Staining methods later made it possible to do these kinds of measurements also in visible light. When digital computers became available there was interest in using them for developing methods of analyzing microscopy images to obtain quantitative, objective data. Already in 1965 one of the pioneers, Judith Prewitt wrote: “Thus available biochemical techniques make it possible to prepare biological materials so that morphological integrity is preserved, key constituents are stained stoichiometrically, and the specimens are favorably dispersed for effective imaging one by one. Scanning microscopes now have the requisite sensitivity, resolution, and stability to sample such objects and make photometric measurements over a wide range of magnifications and wavelengths within the visible and near-visible spectrum. Furthermore, modern large capacity, high speed data facilities at last provide the ability to manipulate the hitherto unmanageable quantities of optical information contained within all but the simplest images” 2. Over the coming years many projects demonstrated that quantitative data from cells and tissues could be used to extract quantitative features and use these to create malignancy grades that correlated well with patient outcome. An example from our own early work is shown in Ref. 3. But the impact on routine clinical pathology has been very limited. It is likely that a main reason for the limited use of image cytometry in clinical routine was the need to digitize the microscopy images. Pathologists were used to working with microscopes, not with scanners or cameras and computer software. This is changing now as whole slide scanners are making digital images routinely available in rapidly increasing numbers of pathology departments. Will this have an impact also on applications of image analysis in routine clinical pathology? What does the current digitization revolution in routine pathology mean to the prospect of getting computer-aided pathological diagnosis and grading into routine use? For decades methods on how to use quantitative methods in pathology have been published often with promising results but they have had very little impact on routine pathology workflows. One of the main obstacles preventing widespread adoption has been the need to digitize the slides. Currently digital pathology is making rapid progress mainly for facilitating visual analysis remotely, getting second opinion, consulting sub-specialists and achieving efficient archival storage. Will this paradigm shift in pathology also have a major impact on the use of computer-aided diagnosis in pathology? Although we received an excellent collection of submissions the original timeframe we anticipated had to be extended. This could be partly due to the increased focus and interest in this area. As we were putting together this special issue, other journals also launched special issues on similar topics and there is an increasing number of meetings and activities (several conferences, workshops and hackathons) competing for the interests and time of the relatively small number of researchers in the field. Another contributing factor may be that the fact that we finally are seeing increased use of cytometry and image analysis in digital pathology has led to the establishment and growth of a number of companies working hard to develop and launch products. So a large fraction of the research and development of the field is now company-based. In contrast to academic researchers their first priority is not to write papers about their progress. Several of the authors of the papers we received do have links to and cooperation with companies and one article is directly contributed by a company-based research group. There is a surprisingly strong focus on breast cancer five out of the seven papers are dealing with that disease. The papers are dealing with quantification of very different aspects of the cancer and they together very nicely illustrate how image analysis can be used to facilitate and improve several aspects of the diagnostic process. Since breast cancer is one of the most common cancers this is interesting in itself. But most of the aspects that have been studied are also as relevant for other cancers so these papers should also be of interest to readers who are primarily interested in other types of cancer. Systems for image analysis of pathology slides can either be designed to do the analysis fully automatically on the whole slide image or interactively on regions of interest selected by the pathologist. The former choice has the advantage that the processing can be done as soon as the slide has been scanned and without anyone waiting while the processing is taking place. But it makes it necessary to have full automation adjusting, for example, to different staining intensities and setting the processing parameters automatically. The interactive option allows input of critical parameters and direct monitoring and possibly also tuning of the analysis process. But the demands for processing speed will be high since the pathologist will be waiting for the results while doing the diagnostic work. We have examples of both approaches among the studies reported in this issue. Valkonen et al. (this issue, page 555) describe a system for finding cancerous tissue in metastases in whole slide H&E stained sections. They have used a classical statistical approach with a random forest classifier and applied it to data from a publicly available challenge data set called Camyleon16. The so called challenges, where a well characterized set of images representing a diagnostic problem are made available together with ground truth data and together with another dataset without ground truth, has become rather popular recently. The idea is that results of an algorithms performance on the second dataset can be submitted and objectively evaluated and compared to other approaches thus making it possible to objectively compare performances on the same material. This has a potential to significantly boost progress in the field and has already done so for several applications (https://grand-challenge.org/Home/). The challenge datasets can also be used for developing and comparing algorithms outside the formally organized competitions. That is what has been done in the study by Valkonen et al. (this issue, page 555). They show good accuracy in detecting metastatic tissue versus normal tissue with an area under the curve, AUC, of around 90%. Importantly they also got rather good results, an AUC of around 85%, when they trained the algorithms on data from one lab and tested it on data from another lab using a different scanner. Such robustness tests are very important in making algorithms routinely useful. The processing time was around 90 min for a whole slide image, acceptable for a background process that fully automatically can produce a heat-map of where in the specimen metastases are likely located. This can save time for the subsequent visual verification as well as in the case where no hotspots are detected so that the pathologist can sign it off after a quick check. Romo-Bucheli et al. (this issue, page 566) has also used challenge datasets, AMIDA for development and testing and MITOS2012 for additional testing. Their article deals with detection of mitoses in ER positive breast cancers relating those to the genetic profile as determined by the Oncotype DX gene expression test. They have used the recently very popular and successful deep learning convolutional neural network, DCNN, approach. That approach has led to drastically improved results in many difficult image analysis tasks including detection of mitoses. Valkonen et al. (this issue, page 555) also make the comment in their article that DCNN-based methods recently has shown impressive results and they suggest that their approach could be augmented by a DCNN. The DCNN implementation used by Romo-Bucheli et al. (this issue, page 566) is almost two orders of magnitude faster than previous published work on mitoses detection using DCNN. This is important since their approach requires around a second per high resolution field while earlier methods need around a minute and a sample may have >500 fields. Their results indicate that mitoses counting may be an alternative to gene expression tests in determining patient treatment. Both these papers use the old, well established H&E stain. But this stain is not really optimal for image analysis 4. The color contrast between different tissue components is not as good as it could and should be. Ideally the introduction of digital pathology should also lead to the introduction of better staining methods. Future will show if that will happen or if the community will accept the lower performance in favor of not having to change the established staining procedure. An alternative way to mitosis counting for estimating cell proliferation in tumors is to immunohistochemically stain for the Ki-67 protein and count the relative number of positive cells. This counting should be done in a “hot spot,” the part of the tumor that has the highest fraction of positive cells and this makes the method time consuming and prone to significant variations between individual assessments. There have over the years been numerous papers trying to automate this count. In the article by Lindberg et al. (this issue, page 574) they use TMA:s with adjacent sections stained with PCK and Ki-67 for detecting hotspots and doing counts within these. The PCK layer is used for finding tumor cells and Ki-67 for determining if they are positive or negative. They show good correlation with tedious manual counting provided the hot spots are large enough to cover at least 400 cells. Even though the method has been developed on TMA it can potentially be used for whole tissue sections but they do not provide any data on time performance. The fact that they use two adjacent sections for the analysis makes it necessary to register the images of these to each other. For that purpose they use software for so called virtual double staining from the Visiopharm company. Immunohistochemical staining can also provide information about estrogen, ER and progesterone, PR receptors. Also in this case the manual scoring is tedious and error prone. In the article presented by Trahearn et al. (this issue, page 585) they present an algorithm for scoring these receptors based on binning of pixels into intensity groups. They also use algorithms for registering adjacent sections with H&E, ER and PR stains to each other. The H&E section is used for manually outlining where the estimation should be done prior to the automated analysis. The registration makes it easy to see to what extent ER and PR are colocalized. The analysis is done on 20 randomly selected 20× fields and the processing only takes a couple of seconds for a whole case. A large part of the article deals with discussing the discrepancies between pathologist scores and machine scores pointing at the importance of quality control of staining and imaging procedures. The article by Paulik et al. (this issue, page 595) covers both topics discussed in the previously mentioned two papers in that they quantify Ki-67, ER and PR in the same approach. Interestingly they also cover fluorescence DAPI positive slides stained for HER2-FISH in the same processing pipeline as the immunohistochemically stained slides. The article has a strong focus on computational throughput. It is mainly based on well-established algorithms from the literature but demonstrates that they could be implemented in a robust and computationally efficient way. The average processing time for a whole slide with around a billion pixels is 8 min. Another important aspect when quantitative methods are to be used in routine work is that they are robust against unavoidable staining variations between different batches or even labs. This aspect is discussed and they show promising results. The article is also interesting in that the authors are associated to a vendor, 3DHISTECH, so not all commercial actors are uninterested in writing papers about their results. The only exception from the focus on breast cancer among our cancer related contributions is the article by Abas et al. (this issue, page 609) They study computer-assisted quantification of CD31 T cells in follicular lymphoma and compare it to scores given by a group of experienced pathologists. The pathologist do both tedious individual cellular markings and conventional estimation, “eyeballing.” The computer quantification agrees better to the cellular markings than the eyeballing results do. It also shows less deviations from the average pathologist readings than most of the pathologists. Although based on a very limited material this study demonstrates the potential of image analysis in a typical quantification task. Modern genomic technology offers great potential in characterizing an individual patient and the cancer that need to be treated leading to the concept of personalized medicine. The Oncotype DX gene expression test discussed in the article by Romo-Bucheli et al. (this issue, page 566) is an example of an assay that already finds clinical use. But these tests can only determine gene expression in bulk, the information about the exact location of the gene expressions in the tissue is lost. The ER, PR, and Ki-67 assays give information about the distribution of individual proteins that are known to have great importance in the diagnosis and treatment planning of a patient. But there are many more genes that code for proteins that influence tumor development. One way of studying that is through chromosome karyotyping. M-FISH is a staining technique for classifying individual chromosomes and detecting aberrations. But analyzing such images is tedious. Available classification algorithms are not sufficiently accurate and robust to use clinically. Wang et al. (this issue, page 622) describe how a patch-based tensor decomposition algorithm can achieve significantly better results than the conventional pixel-wise approaches. The article is very mathematical presenting recently developed tensor-based analysis algorithms. And the results seem very promising. The M-FISH method uses five fluorescent stains that attach to specific DNA sequences so that each chromosome is coded with a unique combination of colors and thus can be identified. This concept can also be used to identify the distribution of proteins in tissue. The spectral resolution and signal to noise limitations of fluorescent microscopy limits the number of specific sequences that can be detected to around five but through clever staining and de-staining tricks and through application of gene amplification technology such as RCA and advanced image processing it is possible to map hundreds of proteins to their spatial locations in tissue. This offers the perspective of creating patient specific gene expression maps that can be the basis for future personalized medicine that takes the highly heterogeneous character of cancer into account. There is a very recent overview article that describes the current state of the art in that exciting field 5. So in summary the papers in this special issue point to the fact that quantitative methods have a potential of being put into use in the digital pathology systems of today supplying objective, quantitative information that can supplement the visual diagnostic work by pathologists. The methods need to be validated on larger materials and standardization issues need to be given great attention so that the results can be trusted irrespective of where the samples are prepared. The recent progress in DCNN has been demonstrated as very useful in the papers presented here and more impact on the cytometry field can be expected in the near future. In addition to giving more objective and quantitative data for assessments that can be done visually we are beginning to see new analysis methods that go beyond what we can see by our eyes. Digital pathology will revolutionize pathology in the same way as digitization has revolutionized most other sectors of society. But image cytometry applied to digital pathology has an even greater potential of revolutionizing not only pathology but medical diagnosis as a whole.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.