Pilot studies can be used to try to predict an appropriate sample size for the large-scale project and / or to improve various aspects of the study design. Studies are often time-consuming and expensive to carry out, so it is crucial that researchers have confidence in the key steps they will take when conducting this type of study to avoid wasting time and resources. Therefore, a pilot study should answer a simple question: "Can the study be carried out on a large scale in the way that has been planned or do some components need to be modified?" Reports from pilot studies should be of high quality to allow readers to correctly interpret the results and implications.
What are the main reasons for conducting a pilot study?
Pilot studies are carried out to assess the feasibility of some crucial components of the large-scale study. These can normally be divided into 4 main aspects:
Process: where the feasibility of the key steps in the main study is evaluated (for example, recruitment rate, retention levels and eligibility criteria).
Resources: assessing the time and resource issues that may occur during the main study (for example, how long will the main study take to complete; whether the use of any equipment will be feasible or the evaluation form (s) selected for the main study studying are the best possible)
Management: problems with data management and with the team involved in the study (for example, if there were problems with collecting all the data needed for future analyzes; if the data collected is highly variable and if the data from different institutions can analyzed together).
Reasons for not conducting a pilot study
Researchers should not simply label a study a "pilot study" in the hope of justifying a small sample size. Pilot studies should always have their objectives linked to feasibility and should inform researchers about the best way to carry out the future large-scale project.
How to interpret a pilot study
Readers should interpret the pilot studies with care. Here are some key things to consider when evaluating a pilot study:
The objectives of pilot studies should always be linked to feasibility and the crucial component to be tested should always be indicated.
The method section should present the success criteria. Sample sizes may vary in the pilot studies (different articles present different sample size calculations) but the population of the pilot study, from which the sample is formed, must be the same as that of the main study. However, participants in the pilot study should not be entered into the large-scale study. This is because participants can change their behavior later if they had previously been involved in the research.
The pilot study may or may not be a randomized trial (depending on the nature of the study). If the researchers randomize the sample in the pilot study, it is important that the randomization process remains the same in the large-scale project. If the authors decide to test the feasibility of randomization through a pilot study, different types of randomization procedures could be used.
Pilot Study Results
In addition to the methods section, the results of the pilot studies should be read carefully. Although pilot studies often present results related to the efficacy of interventions, these results should be interpreted as 'potential efficacy'. Results from pilot studies should always focus on feasibility, rather than statistical significance. However, the results of pilot studies should nevertheless be provided with measures of variability (such as confidence intervals), particularly as the sample size of these studies is usually relatively small and this could lead to biased results. After an interpretation of the results, the pilot studies should conclude with one of the following:
(1) the main study is not feasible;
(2) the main study is feasible, with changes in the protocol;
(3) the main study is feasible with no changes to the protocol O
(4) the main study is feasible with close follow-up.
Any recommended changes to the protocol should be clearly described.
The future study
A pilot study should provide information on whether a large-scale study is feasible and list recommended amendments to the future study design. You may be able to think of other feasibility questions relevant to your specific intervention, population, or design. When designing a pilot study, it is important to establish clear quantitative benchmarks for the feasibility measures by which you will assess successful or unsatisfactory viability (for example, a benchmark for assessing adherence rates might be that at least 70 percent of participants from each branch will attend at least 8 of the 12 scheduled group sessions).
These benchmarks should be relevant to the specific treatment conditions and the study population, and will therefore vary from study to study. While the use of a randomized design is not always necessary for pilot studies, having a comparison group can provide a more realistic review of recruitment rates, randomisation procedures, implementation of interventions, procedures to maintain blind evaluations and the possibility of evaluating differential dropout rates.
Measures of feasibility are likely to vary between 'open label' designs, where participants know what they are signing up for, versus a randomized design where they will be assigned to a group. In addition to providing important feasibility data as described above, pilot studies also provide an opportunity for study teams to develop good clinical practices to improve the rigor and reproducibility of their research. This includes the development of informed consent documentation and procedures, data collection tools, regulatory reporting procedures, and monitoring procedures.
Objectives of the pilot studies
The goal of pilot studies is not to test hypotheses; therefore inferential statistics should not be proposed. Therefore, it is not necessary to provide power analysis for the proposed sample size of your pilot study. Instead, the sample size of the proposed pilot study should be based on practical considerations including the flow of participants, budget constraints, and the number of participants required to reasonably assess the feasibility objectives. This testing of the methods and procedures that will be used in a larger study is the fundamental basis that we wish to support, to pave the way for the larger scale efficacy trial. As part of this process, researchers can also spend time refining their intervention through iterative development and then testing the feasibility of their final approach.
Rather than focus on feasibility and acceptability, too often the proposed pilot studies focus on inappropriate results, such as determining "preliminary efficacy". The most common misuses of pilot studies include:
Try to evaluate the safety / tolerability of a treatment,
Seeking to provide a preliminary test of the research hypothesis, and
Estimation of effect sizes for larger-scale study power calculations.
Why can't pilot studies be used to assess safety and tolerability?
Researchers often propose to examine the "preliminary safety" of an intervention within a pilot study; however, due to the small sample sizes typically involved in pilot work, they cannot provide useful safety information except in extreme cases where death occurs or repeated serious adverse events arise. For most of the interventions proposed by the researchers, the potential safety concerns are minimal / rare and therefore unlikely to be detected in a small pilot study. If any safety concerns are detected, group specific rates with 95 percent confidence intervals for adverse events should be reported. However, if no safety concerns are demonstrated in the pilot study, the researchers cannot conclude that the intervention is safe.
Why can't pilot studies provide a "preliminary test" of the research hypothesis?
We routinely see specific targets for pilot feasibility studies that propose to assess the "preliminary efficacy" of intervention A for condition X. However, there are two main reasons why pilot studies cannot be used for this purpose. First, at the time a pilot study is conducted, there is a limited state of knowledge about the best methods to implement the intervention in the study patient population.
Therefore, conclusions about whether the intervention "works" are premature because you do not yet know if you implemented it correctly. Second, due to the smaller sample sizes used in the pilot studies, they do not have the power to answer questions about efficacy. Therefore, any estimated effect size cannot be interpreted: it is not known whether the "preliminary test" has returned a true result, a false positive result, or a false negative result.
Why can't pilot studies estimate effect sizes for larger-scale power calculations?
Since any effect size estimated from a pilot study is unstable, it does not provide a useful estimate for power calculations. If the effect size estimated from the pilot study was really too large (i.e. a false positive result or type I error), the power calculations for the subsequent trial would indicate fewer participants than is actually needed to detect a clinically significant effect. eventually resulting in a negative test.
On the other hand, if the effect size estimated from the pilot study was really too small (i.e. a false negative result or type II error), the subsequent trial may not even be carried out due to assumptions of that the intervention does not work. If the subsequent trial was designed, the power calculations would indicate a much larger number of participants than is actually needed to detect an effect, which may reduce the chances of funding (too expensive) or, if funded, expose a unnecessary number of participants to the intervention arms.
Thabane L, Ma J, Chu R, et al. A tutorial on pilot studies: what, why and how? BMC Med Res Methodol. 2010; 10: 1.
Cocks K and Torgerson DJ. Sample Size Calculations for Randomized Pilot Trials: A Confidence Interval approach. Journal of Clinical Epidemiology. 2013.
Lancaster GA, Dodd S, Williamson PR. Design and analysis of pilot studies: recommendations for good practice. J Eval Clin Pract. 2004; 10 (2): 307-12.
Moore et al. Recommendations for Planning Pilot Studies in Clinical and Translational Research. Clin Transl Sci. 2011 October ; 4(5): 332–337.