Yes, Size Does Matter!

We will use the paper by Hunstad et al1 to address an oversight commonly made by investigators and frequently overlooked by the reviewers. The issue we would like to address pertains to the sample size of a trial. The authors labeled their study as a multicenter randomized study. Although there may be other issues with the design of this randomized controlled trial (RCT), we will not address them in this Hub. The optimal methodology of an RCT design is provided in the references for interested readers.2,3 So let’s return to the sample size calculation, which is the main objective of this Hub. The authors performed their study to compare a new type of liposuction (tissue liquefaction liposuction, TLL) to the traditional standard (suction-assisted liposuction, SAL). In their study design, the authors compared treatments with TTL on one side of each study subject to SAL on the other side. By the way, we like this type of side-to-side comparison, because each study patient serves as their own control, which, of course, is a pretty good match! In this study, the authors treated 31 female patients. Is 31 the right number? How do you know? More importantly, why is a chosen number the right − or wrong − number? When we perform a study such as the one performed by Hunstad and his coauthors,1 we cannot examine the whole population to see if TLL is superior to SAL. (Even if we could study the whole population, there would be no more patients left to treat!) To determine whether TTL (novel technology) is better than SAL (standard technology), we select a small sample from the target population (all those eligible for liposuction) and perform our clinical experiment. The number of patients selected is – yes, hold on to your seats − the sample size. If we find some positive results, we declare that this novel technology is beneficial in our sample and, by inference, to the larger population. This interrelationship between the population and sample are shown in Figure 1. Making a statistical inference. The interrelationship between a population and the study sample is shown here. Sampling is the process of selecting a group from the population that is representative of the population. Inclusion and exclusion criteria are key in this process. For the given characteristic or outcome being assessed, there is a mean for the population (true mean, or μ) and a mean for the sample (sample mean, or X). Likewise, there is a standard deviation for the population (σ) and for the sample (SD) in terms of the characteristic or outcome being assessed. Making a statistical inference. The interrelationship between a population and the study sample is shown here. Sampling is the process of selecting a group from the population that is representative of the population. Inclusion and exclusion criteria are key in this process. For the given characteristic or outcome being assessed, there is a mean for the population (true mean, or μ) and a mean for the sample (sample mean, or X). Likewise, there is a standard deviation for the population (σ) and for the sample (SD) in terms of the characteristic or outcome being assessed. How do we determine whether this inference that we’re making is true? The process of drawing conclusions valid for the population from data from the sample group is called statistical inference. Choosing the correct number of patients to include in a study has everything to do with statistical inference. To make statistical inference, there are two approaches: (A) hypothesis testing; and (B) estimation-point estimate and interval estimation (confidence interval). Let’s look at hypothesis testing first. The question the authors want to answer is “In patients undergoing liposuction, does TTL result in a lower total score of for bruising, swelling, tenderness, and incision appearance in comparison to SAL at 30 days post-surgery?” We want to know that the answer to this question is real. Hypothesis testing confirms or refutes the hypothesis that the study results did not occur by chance but rather occurred due to an effect or an association between the intervention and primary outcome. A study is usually designed around a predetermined null hypothesis (H0) and alternate hypothesis (H1). Let’s see how this works in the present study, focusing on bruising, swelling, tenderness, and incision appearance. The null hypothesis would state that “There is no difference between bruising with TTL and bruising with SAL.” The alternative hypothesis would be the opposite, or “There is a difference between bruising with TTL and bruising with SAL.” We can express this mathematically, where the hypotheses would look like following: The basic format of the hypotheses above represents a two-sided equivalence study. It is two-sided because the alternative hypothesis states that the mean difference can be either higher or lower than the anticipated difference between interventions (ie, that the bruising could be more with the TTL, or it could be more with the SAL). It is an equivalence study as the null hypothesis states that there is no difference between TTL and SAL, thus are equivalent to one another. It is important to consider these hypotheses before designing the study as they will determine the type of study (superiority, equivalence, or noninferiority),4 as well as whether the study is going to be a one- or two-sided study. These pieces of information will determine which sample size formula is the best to use for that particular study. OK, so we know how to write our null hypothesis and alternative hypothesis now, but we need to test them. In hypothesis testing, we talk about type I and type II errors. Type I errors (denoted by the Greek letter α) tell us about the probability of rejecting the null hypothesis when it is true (rejection error). Another way of expressing this is something like stating there is an association between TTL and total bruising score, when in fact there is none. Type II errors (denoted by the Greek letter β) tell us about the probability of accepting the null hypothesis when it is false (acceptance error). This is akin stating there is no association between TTL and total bruising score, when in fact there is one. By convention, the probabilities of α and β are fixed a priori so that we know how much each of these may play a role in our hypothesis testing. The commonly accepted values are 0.05 (or 5%) for α . . . hey wait a minute . . . 0.05 sounds kind of familiar. The commonly accepted value for β is 0.2 (or 20%). The power of the study is represented by 1-β, which is equal to 0.8 (or 80%). Once the statistical analyses are completed, the predefined significance level (P-value) is then used to determine whether or not the null hypothesis should be rejected and the alternative hypothesis is accepted. In general, this is done by comparing the P-value given by the statistical test to the predefined P-value. If the P-value is ≤ α, then the null is rejected and it can be concluded that there is likely an effect. Aha! So that’s why we say an outcome having a P-value of less than or equal to 0.05 is significant – it means that the outcome is likely true and reflects the outcome in the general population. How likely is that it is true? 95%, because α was set at less than or equal to 0.05 (5%), so the chance of a type I error is a possibility of 0.05 or 5%). Alternatively, if the P-value is > α, then we do not reject the null hypothesis and it can be concluded that there is likely no effect. (Or similar to above, it is also possible that a type II error exists with a probability of 20% of being wrong, because β is 0.20) (Table 1). Hypothesis Testing Hypothesis Testing As mentioned above, the sample size calculation requires predetermined information such as the type of study (superiority, equivalence, or noninferiority), which will determine whether or not the study will be a one- or two-sided study. In addition to this information, the primary outcome is another piece of information required to calculate the sample size. In other words, you design the study to have the right number of patients to answer the most important question (the primary outcome). In the present article by Hunstad et al,1 we do not know for certain what this primary outcome is. In the Methods section of the paper, we are told this is “efficiency” of device for surgeons (ie, strokes/time unit). In the Statistical Methods subsection, however, we are told that the primary endpoint was the total score of bruising, swelling, tenderness, and incision appearance. If you don’t define the primary outcome, you can’t set up the study correctly. Authors therefore need to be very explicit in defining the primary outcome. The next important measure we need is the minimal clinical important difference (MCID) or Δ between TTL and SAL. We also call this the effect size. If the primary outcome was the total score (bruising, etc.), the authors should have told us what difference (Δ) in score between TTL and SAL would be considered clinically important. In other words, we need to know what difference in total score our colleagues would consider important to be willing to adopt the TTL (novel technology) and abandon SAL (standard technology). Some of our colleagues may want to switch to the new technology with only 2%, difference in bruising score, some 5%, and still some others may require a 50% difference. You will shortly see how this variation in preference causes problems for our sample size. As you can see from the formula, it is easy for investigators to manipulate the sample size by inserting in the formula a “convenient Δ or MCID.” They do this when they are unable to recruit enough patients for their study. As you can see from the formula above if the Δ is small the n (sample size) increases. This means you need more patients for your study. If the Δ on the other hand is made larger then you do not need as many patients. So…readers be aware of this sleight of hand! Investigator should always support the choice of Δ by referencing their sources such as the literature, the consensus of experts (or still better the consensus of our patients who supposedly will benefit from the new technology!). Again, the exact equation necessary for a given study is determined by all the factors listed above. This is an equation that could have been used by Hunstad et al1 to calculate a proper sample size. It should be noted that the above formula will change if proportions are used. We encourage readers to be familiar with the calculation of sample size and the power of their study and be able to appraise an article’s validity on this issue.6 The authors reported a combined mean total score reduction in bruising, swelling, treatment site tenderness, and incision appearance of 2.09, which is a 12.5% difference in mean total score (TLL, 11.4; SAL, 13.1). Since they don’t say, we don’t know what Hunstad et al1 consider as a MCID in their study. Because this was not reported, it is difficult to determine what exactly that 2.09 difference in mean total score actually means. When investigators use a validated scale (instrument, patient reported outcome) in a clinical experiment, they also report the MCID for that particular scale. This MCID is entered into the sample size calculation formula. If the recruitment achieved their pre-determined sample size, they did not have any drop outs and the P-value was < 0.05, then their conclusions would likely be valid. You may be wondering, “what’s the big deal?” What if you, as the surgeon, are most concerned with bruising and a significant difference in a bruising score might compel you to begin to use the alternative treatment (in this case, the TLL device)? How much of a difference would you want to see? If we plug the typical values for α (0.05) and β (0.20) into the equation, and work backwards from the authors sample size of 31, we see that with an 80% power, they would be able to detect an effect size (Δ) of about 53% reduction in bruising (bolded in Table 2). In other words, the authors would not have been able detect such a small effect size of 12.5% (or 16%, which is what they said they observed) difference between treatments using their sample size of 31. The sample size of 31 actually only yields a power of approximately 14%. If other amounts of bruising reduction are indicated, you’d need a different sample size to show it, as in the table below. The Relationship Between the MCID, Power, and Sample Size Based on Total Mean Score of Bruising, Swelling, Tenderness, and Incision Appearance *As the article being evaluated used paired data, these sample size estimates would refer to the total number of PAIRS required. The Relationship Between the MCID, Power, and Sample Size Based on Total Mean Score of Bruising, Swelling, Tenderness, and Incision Appearance *As the article being evaluated used paired data, these sample size estimates would refer to the total number of PAIRS required. Not calculating the sample size based on the primary outcome and a “legitimate” MCID will limit the value of the conclusion being made, including both type I and type II errors. As the study found an effect, we know that a type I error (finding an effect when there is no effect) could have occurred in the Hunstad et al study. So now you can see the importance of sample size and the related MCID in determining the impact of a study. A study outcome can be statistically significant but clinically irrelevant, as authors reported a significant 16% (actually 12.5%) difference between TLL and SAL but did not have a sample size big enough to show that difference. Evaluating the sample size, the MCID and whether the appropriate statistical analysis methods were used is how we can be sure that a study is both valid and relevant. If you are performing a study, once you have an idea of the necessary sample size required, it is then imperative to adjust for some unavoidable factors in a clinical study such as low recruitment rates, loss to follow up and dropout rates. This can be done by using a common rate from other clinical studies and inflating the calculated sample size by that number (ie, 20% or 30%). The calculation of the sample size requires some careful thought and this should be done before the study begins. To determine your required sample size, the MCID needs to be determined beforehand by reviewing the literature. If you cannot find it there, then you can obtain a consensus from the experts (in this case the liposuction experts or still better a consensus of liposuction patients who will be benefitting from the TTL). Most importantly, the sample size needs guidance from a biostatistician before the project starts. The authors have no conflict of interests to disclose related to the content of this article. The authors received no financial support for the research, authorship, and publication of this article.

Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.