Experimental Design and Statistical Methods for Improved Hit Detection in High-Throughput Screening

Identification of active compounds in high-throughput screening (HTS) contexts can be substantially improved by applying classical experimental design and statistical inference principles to all phases of HTS studies. The authors present both experimental and simulated data to illustrate how true-positive rates can be maximized without increasing false-positive rates by the following analytical process. First, the use of robust data preprocessing methods reduces unwanted variation by removing row, column, and plate biases. Second, replicate measurements allow estimation of the magnitude of the remaining random error and the use of formal statistical models to benchmark putative hits relative to what is expected by chance. Receiver Operating Characteristic (ROC) analyses revealed superior power for data preprocessed by a trimmed-mean polish method combined with the RVM t-test, particularly for small- to moderate-sized biological hits. Identification of active compounds in high-throughput screening (HTS) contexts can be substantially improved by applying classical experimental design and statistical inference principles to all phases of HTS studies. The authors present both experimental and simulated data to illustrate how true-positive rates can be maximized without increasing false-positive rates by the following analytical process. First, the use of robust data preprocessing methods reduces unwanted variation by removing row, column, and plate biases. Second, replicate measurements allow estimation of the magnitude of the remaining random error and the use of formal statistical models to benchmark putative hits relative to what is expected by chance. Receiver Operating Characteristic (ROC) analyses revealed superior power for data preprocessed by a trimmed-mean polish method combined with the RVM t-test, particularly for small- to moderate-sized biological hits. IDENTIFICATION OF ACTIVE COMPOUNDS IN high-throughput screening (HTS) contexts can be substantially improved by applying classical experimental design and statistical inference principles to all phases of HTS studies. Good experimental design at the data acquisition phase serves 2 broad purposes: (1) improves internal validity by reducing the possibility that observed effects have been caused by confounding factors and (2) minimizes unwanted variation in activity measurements stemming from human, biological, and equipment sources. Statistical methods at the data preprocessing (normalization) phase can further reduce unwanted variation, which cannot be controlled procedurally. At the inference phase, the magnitude of the remaining random error, inherent in any biological system, can be estimated by replicate measurements and taken into consideration when deciding which of the putative hits are sufficiently reliable to warrant follow-up. The information from the random error observed in a particular screen can also be used to estimate anticipated false-negative rates for future similar studies. Although the advantages of statistical procedures for HTS were described more than a decade ago,1Lutz MW, Menius JA, Laskody RG, Domanico PL, Goetz AS, Saussy DL, Rimele T: Statistical considerations in high throughput screening [Online]. Retrieved from http://www.netsci.org/Science/Screening/feature05.htmlGoogle Scholar statistical treatment of HTS data is only now becoming more common as researchers search for ways to reduce false positives and false negatives. Various methods proposed to characterize the quality of screens2Gunter B Brideau C Pikounis B Liaw A Statistical and graphical methods for quality control determination of high-throughput screening data.J Biomol Screen. 2003; 8: 624-633Crossref PubMed Scopus (52) Google Scholar, 3Zhang JH Chung TD Oldenburg KR A simple statistical parameter for use in evaluation and validation of high throughput screening assays.J Biomol Screen. 1999; 4: 67-73Crossref PubMed Scopus (5352) Google Scholar, 4Zhang XD A pair of new statistical parameters for quality control in RNA interference high-throughput screening assays.Genomics. 2007; 89: 552-561Crossref PubMed Scopus (139) Google Scholar include: estimating of false-positive and false-negative rates as part of assay quality assessments,5Sui YX Wu ZJ Alternative statistical parameter for high-throughput screening assay quality assessment.J Biomol Screen. 2007; 12: 229-234Crossref PubMed Scopus (88) Google Scholar removing bias within and across plates,6Brideau C Gunter B Pikounis B Liaw A Improved statistical methods for hit selection in high-throughput screening.J Biomol Screen. 2003; 8: 634-647Crossref PubMed Scopus (260) Google Scholar, 7Kevorkov D Makarenkov V Statistical analysis of systematic errors in high-throughput screening.J Biomol Screen. 2005; 10: 557-567Crossref PubMed Scopus (46) Google Scholar, 8Wu ZJ Liu DM Sui YX Quantitative assessment of hit detection and confirmation in single and duplicate high-throughput screenings.J Biomol Screen. 2008; 13: 159-167Crossref PubMed Scopus (40) Google Scholar improving hit/nonhit ranking,9Zhang XD Ferrer M Espeseth AS Marine SD Stec EM Crackower MA et al.The use of strictly standardized mean difference for hit selection in primary RNA interference high-throughput screening experiments.J Biomol Screen. 2007; 12: 497-509Crossref PubMed Scopus (73) Google Scholar conceptualizing assay validation within a statistical framework,10Coma I Clark L Diez E Harper G Herranz J Hofmann G et al.Process validation and screen reproducibility in high-throughput screening.J Biomol Screen. 2009; 14: 66-76Crossref PubMed Scopus (34) Google Scholar and obtaining random error estimates for use in statistical tests to identify hits.11Buxser S Vroegop S Calculating the probability of detection for inhibitors in enzymatic or binding reactions in high-throughput screening.Anal Biochem. 2005; 340: 1-13Crossref PubMed Scopus (16) Google Scholar,12Malo N Hanley JA Cerquozzi S Pelletier J Nadon R Statistical practice in high-throughput screening data analysis.Nat Biotechnol. 2006; 24: 167-175Crossref PubMed Scopus (531) Google Scholar There remains a need, however, for elucidation of experimental design and data analysis principles tailored to HTS applications. We extend our previous arguments12Malo N Hanley JA Cerquozzi S Pelletier J Nadon R Statistical practice in high-throughput screening data analysis.Nat Biotechnol. 2006; 24: 167-175Crossref PubMed Scopus (531) Google Scholar that popular methods for bias correction and inference are deficient and offer alternatives. We propose an analytical protocol that combines randomization, replication, and efficient statistical methods, allowing examination of model assumptions, calculation of p-values to benchmark putative hits, control of false-positive rates at levels specified by the experimenter, and an increase in true-positive rates for HTS applications. Specifically, we use 4 data sets to compare various approaches for addressing unwanted variation in HTS measurements: experimental data obtained initially without randomization procedures (as is the case with most screening studies), a repeat of the experiment but with randomization procedures, a control experiment, and a dilution series. We counter the fundamental misunderstanding among many researchers that high correlations between replicate plates are desirable; we demonstrate why this typically reflects bias in the measurements rather than good biological reproducibility. We also show that normalization based on a trimmed-mean polish (TP) provides the desirable statistical characteristics of bias correction and measurement independence and performs better than Z-scores or B-scores. Finally, we show that TP scores, when combined with the RVM t-test approach, provide variance and p-value distributions that agree with theoretical expectations. Some 1120 chemical compounds were tested to determine if they correct the trafficking defect of the phenylalanine deletion mutant form of cystic fibrosis transmembrane conductance regulator (CFTR) protein ΔF508 (see supplementary information methods online). Fourteen 96-well plates were run in duplicate. Including incubation time, the screen was run in 4 days. Plates were processed in sets of 5, followed immediately by a duplicate set processed in the same sequence. Compounds that correct the mutant protein trafficking defect are detected by an increase in fluorescence (arbitrary units)—large measured values are more likely to be regarded as biologically valid hits. This screen was the same as experiment A except for 2 aspects: processing order was randomized for all steps in the protocol, and replicates were obtained in 3 independent runs (i.e., blocks). An inactive compound from experiment A was tested in all of the 80 middle wells of six 96-well plates. Plate processing order was randomized for all steps. This experiment uses a different assay and target than the cystic fibrosis screen. A known protein synthesis inhibitor was arrayed within each of 6 replicated plates in 10 concentrations (0.0098, 0.0195, 0.039, 0.078, 0.1563, 0.2344, 0.3125, 0.4687, 0.625, and 1.25 µM). Four replicates of each of the 10 concentrations and 24 negative controls (DMSO) were randomly located in the 64 middle wells of 96-well plates. Positive controls (anisomycin at 50 µM) and negative controls (DMSO) were placed in alternating wells on the 1st, 2nd, 11th, and 12th columns. Firefly and renilla luciferase activity measurements were obtained for each well; low measured values corresponded to hits. To circumvent the unrealistically high proportion (40/64) of true hits within each plate, we generated random samples from the data to mimic hit proportions that might be expected from a standard primary screen. Removing potential row and column biases with the TP score normalization method was deemed inappropriate for these data because differences among the rows and columns reflected biological differences in addition to any potential biases. Let i = 1,...., I rows; j = 1,...., J columns; and p = 1,...., P plates. Accordingly, the data were normalized as follows: xijp–x˜pMADp(1) where xijp is the compound measurement corresponding to the well located in row i, column j, and plate p; x̃p and MADp are, respectively, the median and the median absolute deviation of all measurements within plate p. For each of 100 simulation runs, we randomly sampled (with replacement) 1120 normalized measurements from the empirical data set (14 plates × 80 values per plate). Some 1064 “nonhits” were sampled from the 144 negative control measurements (6 plates × 24 values per plate). Four consecutive concentrations were chosen. For each concentration, 14 hits were sampled from the 24 concentration-specific measurements (6 plates × 4 replicate values per plate), yielding a rate of true hits of 5% within each simulation run. We repeated this simulation for 3 different sets of concentrations (i.e., the 4 highest, the 4 lowest, and the 4 in the middle). Hits were identified according to various statistical criteria, and false-positive/false-negative rates were calculated (see Inferential Statistics section below). We compared 3 non-control-based normalization methods. First, the Z-score method: Zijp=xijp–x¯psp(2) where xijp is the compound measurement corresponding to the well located in row i, column j, and plate p; x̃p and sp are, respectively, the mean and the standard deviation of all measurements within plate p. Second, for the B-score, the residual (rijp) of the measurement for row i and column j on the pth plate is obtained by fitting a 2-way median polish13Tukey JW Exploratory Data Analysis. Addison-Wesley, Reading, MA1977Google Scholar and is defined below as rijp=yijp–y⌣ijp=yijp–(μ⌣p+R⌣ip+C⌣jp).(3) The residual is defined as the difference between the observed result (yijp) and the fitted value ( ŷijp ), defined as the estimated average of the plate (µ̂p) + estimated systematic measurement offset for row i on plate p (R̂ip) + estimated systematic measurement column offset for column j on plate p (Ĉjp). The median polish is an iterative algorithm that alternates row and column operations. Considering the rows first, for each row, the row median is subtracted from every element in that row. For each column, the median of the revised numbers is then subtracted from every element in that column. This continues until all medians are 0 or reach some predefined minimal difference from 0. For each plate p, MADp is the median absolute deviation of all residuals within the plate (rijp). The B-score, without the smoothing function,6Brideau C Gunter B Pikounis B Liaw A Improved statistical methods for hit selection in high-throughput screening.J Biomol Screen. 2003; 8: 634-647Crossref PubMed Scopus (260) Google Scholar is calculated as follows: Bijp=rijpMADp.(4) Third, the TP(10) score method: TP(10)ijp=rijp(10)MADp,(5) where rijp(10) are the residuals obtained by a 2-way polish13Tukey JW Exploratory Data Analysis. Addison-Wesley, Reading, MA1977Google Scholar using the S-Plus (TIBCO Spotfire, Somerville, MA) 2-way function with trim = 0.10. All 3 methods rescale measurements so that they are comparable across plates; in addition, the B-score and the TP score correct for row and column effects and are resistant to outliers.6Brideau C Gunter B Pikounis B Liaw A Improved statistical methods for hit selection in high-throughput screening.J Biomol Screen. 2003; 8: 634-647Crossref PubMed Scopus (260) Google Scholar The p-values to decide which compounds should be deemed as hits were defined using statistical tests on K replicates. For each compound measurement, a standard 1-sample t-test with K – 1 degrees of freedom was calculated as t=x¯k–constantsk1/K(6) where x̃k and sk are the arithmetic mean and the standard deviation, respectively, of the K replicated normalized measurements; the constant was taken to be zero. The ratio is then evaluated against a t-distribution with K – 1 degrees of freedom for estimation of associated p-values. Because of cost and time issues, the number of replicates is usually very small. As such, this test relies on imprecise estimates of variance and has corresponding low sensitivity (high false-negative The RVM 1-sample t-test provides a between the low sensitivity of the 1-sample t-test and the common error of a 1-sample are to an with parameters a and as DM and analysis of with high throughput biological assay PubMed Scopus Google A random variance model for detection of in 2003; PubMed Scopus Google We estimated the a and parameters by fitting the to an according to the method described by and A random variance model for detection of in 2003; PubMed Scopus Google The RVM t-test is calculated as where and where x̃k and are the arithmetic mean and the respectively, of the K replicated and the parameters a and are estimated from the data from all compounds by fitting an to the a t-distribution with K – 1 + degrees of that variance ( is estimated by a average of the and an estimate of the error variance the error distributions of different with to – and N Hanley JA Cerquozzi S Pelletier J Nadon R Statistical practice in high-throughput screening data analysis.Nat Biotechnol. 2006; 24: 167-175Crossref PubMed Scopus (531) Google Scholar This to an increase of degrees of freedom the standard 1 and of column effects of data for 2 A and the of measurements and hits, the of the measured values be a however, that the of the replicate set in the screen 2 of systematic error between replicate and differences in the of column bias across replicate sets (see for column for the 2 The distributions of the randomized screen data are more in with the 3 of systematic error variation in the measurements that cannot be controlled be by normalization of the data (see section for more on the 2 that trimmed-mean polish the and column effects observed in in to the popular Z-score normalization method (see were obtained for the row effects The advantages of TP(10) are further in 3 by analysis of data in which the same compound was tested in every well in the same across all plates measurement As such, in the of systematic the same random was expected for all wells of every measured values should be with in the same on plates and should show within the of however, that the data were across the of biases. that the TP(10) the the expected between plates for Z-scores to the data because they are the across all 6 replicate plates in show correlations among independent measurements for the The at 1 that wells in to each each are = correlations between each well and the well A was observed that repeated at every the correlations until the middle row and then a at the well in the row of the column the of = was observed for which to immediately wells across Although Z-scores some of correction TP(10) the correction reducing the at the various to Because Z-scores any row and column they can data of various if the true biological are Although correct these they can data into numbers of false V P D A N Nadon R An efficient method for the detection and of systematic error in high-throughput 2007; PubMed Scopus Google Scholar with these previous both the Z-score and the B-score data data for the experiment C data TP(10) scores, by generated data with only a of replicates is that formal statistical models can be used to benchmark putative hits relative to what is expected by the statistical our of the of our model as to the experiment A and to the randomized experiment B data (see for a of the As with our previous N Hanley JA Cerquozzi S Pelletier J Nadon R Statistical practice in high-throughput screening data analysis.Nat Biotechnol. 2006; 24: 167-175Crossref PubMed Scopus (531) Google Scholar the distributions of to the constant variance by the Z-score to data across compounds or across replicates for each compound for both the and randomized that the number of observed hits with the test likely reflects an high false-positive rate The the expected the RVM model for both the and the randomized The relative of for the screen and the p-value the number of replicates used for the the standard and the RVM 1-sample to the p-value distributions within the p-value for the which the use of the tests the randomized screen data the expected p-value distributions for both statistical with power for the RVM test this the standard t-test from a of degrees of freedom to the number of replicates. of the 2 is the but the are different because the activity measurements are by different estimates of the standard the 2 A and statistical hits were expected in both and we p-value distributions as a of of from a number of different A compound be in the the experiment, and have the to the fluorescence of the on the or to the cystic fibrosis transmembrane regulator to the of the and the binding from the For the biological of the however, the in the activity measurements that to high TP(10) score values in Accordingly, is to estimate p-values for hit with the that effects in the be how the effects might among replicates the validity of obtained from statistical tests based on as the are to however, when are replicates. method to circumvent this in the is to any of the replicate than the be The is that are more detected because are many according to a known the RVM model used The is that compounds with replicate should have The can be used as the probability to if are any is to estimate the number of (i.e., the observed by a and the estimated parameters of the that a predefined false rate with an with K – 1 and degrees of freedom as K is the number of For the randomized screen we of using the rate A to false B Scopus Google Scholar with a of 0.10. Finally, of p-values to be within the For 5% of the compounds are expected to have p-values by chance. For the randomized of the p-values were that hits are present of the compounds were identified as hits using a = values also have been used to reduce the false-negative rate (with the increase in false We a dilution experiment dilution in in which various concentrations of an active compound were randomly well on a 96-well 6 Receiver Operating Characteristic (ROC) that compare the of 3 statistical tests based on random samples generated from the The RVM t-test the false at false-positive 6 also that false were by increasing the number of for hits. We provide experimental design principles and statistical methods to hit detection in illustrate that sensitivity and of can be by (1) robust preprocessing of data that row and column effects and (2) statistical methods based on replicates and that from compounds to variance in and reproducibility that can be by an are for the false-negative rate among to moderate-sized biological hits. For of variation, we randomization and of processing steps to provide the to valid of activity levels by the effects of potential as processing To reduce differences across and increase we the to what can and what Statistics for and (see also and the randomized for we defined a as a replicated and we randomized plate processing order within each run. Various Scholar of and preprocessed data allow assessment of measurement further statistical We data distributions to against measurement of plate and can a of bias that can be by robust preprocessing methods as the TP can provide for measurement of replicate plates can biases in the is in all of these methods to compare the observed data to measurement expectations. et N Hanley JA Cerquozzi S Pelletier J Nadon R Statistical practice in high-throughput screening data analysis.Nat Biotechnol. 2006; 24: 167-175Crossref PubMed Scopus (531) Google Scholar we in of non-control-based normalization methods and the B-score C Gunter B Pikounis B Liaw A Improved statistical methods for hit selection in high-throughput screening.J Biomol Screen. 2003; 8: 634-647Crossref PubMed Scopus (260) Google Scholar We have more however, that the B-score can false positives because data B-score V P D A N Nadon R An efficient method for the detection and of systematic error in high-throughput 2007; PubMed Scopus Google Scholar We propose the trimmed-mean polish as an to the B-score, which has good and superior with this we used a trim value of trim values should be used if more than true hits are expected within some columns or rows a of the value used by the B-score the for hit can be based on p-values the of the probability of what is expected by The RVM we used is particularly well for HTS with replicates. As in the of HTS researchers have to use duplicate measurements as by screening as the however, offer advantages measurements can be detected and the statistical analyses are to sensitivity and measurements and the RVM be when the primary of a is to determine effects The in statistical power by these methods in however, when is in to effects when screening with the of with biological For the replicate provides when are small. For the 5% false-positive for a hit with a 1-sample t-test with 2 replicates is the for 3 replicates is are observed for 4 and replicates of and degrees of freedom can be with the RVM t-test, which as a for N Hanley JA Cerquozzi S Pelletier J Nadon R Statistical practice in high-throughput screening data analysis.Nat Biotechnol. 2006; 24: 167-175Crossref PubMed Scopus (531) Google DM and analysis of with high throughput biological assay PubMed Scopus Google A random variance model for detection of in 2003; PubMed Scopus Google G models and empirical methods for in 3 PubMed Scopus Google Scholar present and for statistical inference relative to the more The typically number of replicates the standard statistical practice of obtaining the very number of present in high-throughput can be for empirical approaches as the RVM that in addition to these analytical researchers to randomization as an part of the screening process. on the of statistical and design methods be by in We for the randomized and the screen This was by the and to the and et for for this is on the of at with

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.