Editorial |
Good practice in statistical reporting for Family Practice
Institute of Health Sciences Education, Queen Mary, University of London, 2 Newark Street, London E1 2AT, UK
Email: s.eldridge{at}qmul.ac.uk.
Good research depends not only on good ideas. The research must also be conducted, analysed and reported well. At Family Practice, we aim to publish high-quality reports of high-quality research. For publications based on quantitative studies, one important element of good reporting is reporting statistical aspects of the study correctly. In this editorial, we discuss a number of important factors that lead to a report which we consider to be statistically sound.
Guidance for reporting quantitative studies has mostly focused on randomized controlled trials, for example, in the CONSORT (Consolidated Standards of Reporting Trials) statement,1 and more recently an extended statement for cluster randomized trials.2 Randomized controlled trials published in Family Practice must follow the guidance for statistical reporting contained in CONSORT. The observations in this editorial, however, apply just as much to the reporting of other quantitative studies such as case–control, cohort and cross-sectional studies as they do to trials.
One important factor which contributes to a statistically sound report is the appropriate treatment of relevant estimates, effect sizes and P-values. Statistically sound studies focus more on reporting estimates of interest and their associated confidence intervals, than on P-values. This is not a new observation; statisticians have been making it for years. The reasons are detailed in a 1986 paper by Martin Gardner and Doug Altman.3 In essence, effect sizes and confidence intervals give much more information than P-values, and more relevant information in that they describe the likely effect of whatever the study focuses on and uncertainty in that effect. Concentration on P-values tends to give too much weight to whether a result falls below or above an arbitrary (albeit well thought out) cut-off; when in practice, similar analyses of the same outcome and same size with P-values of 0.049 and 0.051 are likely to have very similar results.
A second factor which contributes to statistically sound reports, in some ways related to the first, is the treatment of clinical significance. A sound report comments on the clinical as well as on the statistical significance of results. Papers emphasizing the importance of reporting clinical importance have mostly focused on randomized trials,4 but the principle applies to other quantitative studies. Clinically important findings can be defined as those which would result in a change in patient management or policy. Thus, clinical importance is a relevant information for the reader. In addition, over-reliance on statistical significance can lead to misinterpretation of a statistically significant result as being clinically important; if the sample size for a study is very large, very small effect sizes can be statistically significant but may not suggest a need for a change in patient management or policy. Conversely, if a study is very small, effect sizes may be potentially clinically important, but may not reach statistical significance.
This leads neatly to a third and related factor. Statistically sound reports include some explanation for sample size. In a randomized controlled trial, this is usually in the form of a formal sample size calculation.1 The importance of such a calculation is that it specifies the primary end-point (safeguarding against changes of outcome to enhance trial findings), highlights any difficulties in achieving required sample size and, if reported correctly, gives an explicit indication of the investigators' judgement about clinical importance. A formal sample size calculation can be presented for other types of quantitative study but this may not always be appropriate. For example, if the study is based on a survey of all those using a pain clinic, then the sample is of all users, and a formal calculation is unnecessary, although an explanation of the sample size is still useful. In addition, an indication of the authors' views about the clinical significance of the findings is essential in these cases, as the statistical significance of results can be heavily dependent on the actual sample size, in the case of the example given here, the number of users of the pain clinic.
Fundamental to sound statistical reporting is the appropriate analysis of results. Analyses can be inappropriate for many reasons. Here we highlight two features of good practice in analysis which are particularly relevant to papers published in Family Practice. The first feature is appropriately accounting for the structure of the data. In recent years, there has been growing recognition of the importance of accounting for clustering in the analysis of cluster randomized trials, which are becoming increasingly common in family practice.5 Data clustering can also occur in other circumstances. For example, in a retrospective study looking at episodes of care in the previous year, some individuals may have multiple episodes. Since we may expect that episodes for one person are more alike than episodes for two different people, it is a mistake to analyse episodes as if each episode was independent of every other episode. Account should be taken in the analysis of the clustering of episodes within people. Most standard statistical packages now contain procedures for undertaking such analyses, although a statistician may be needed to operate them!
The second feature of good practice in analysis is the appropriate treatment of subgroups. Several papers by Matthews and Altman in 1996 clearly outline the issues with these types of analysis, where the sample is split into several groups and the original analysis conducted separately on each.6–8 The problem with subgroup analyses is that different effects in different subgroups may arise by chance. It is important to test to see if these differences have arisen by chance. The statistical term for finding different effects in different subgroups is interaction. Thus, it is important to test for the significance of any interaction. This is a different procedure from the usual replication of the original analysis in several subgroups. Matthews and Altman8 describe a method of testing for the significance of interaction, although in practice, this is nowadays more commonly done by fitting an extra term in the original analysis conducted on the whole sample. Such a test is essential to ensure sound statistical reporting and interpretation.
The factors listed here are not unconnected, as we have already suggested. They are also neither new, as the reference list testifies, nor exhaustive. Nevertheless, they appear to us to be important in ensuring good statistical practice in reporting papers in Family Practice.
Declaration
Funding: None.
Ethical approval: Not applicable.
Conflicts of interest: None declared.
Notes
Eldridge S. Good practice in statistical reporting for Family Practice. Family Practice 2007; 24: 93–94.
References
1 Altman DG, Schulz KF, Moher D, et al. (2001) The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med 134:8663–694.
2 Campbell MK, Elbourne DR, Altman DG. (2004) CONSORT statement: extension to cluster randomised trials. Br Med J 328:7441702–708.
3 Gardner MJ and Altman DG. (1986) Confidence intervals rather than P values: estimation rather than hypothesis testing. Br Med J (Clin Res Ed) 292:6522746–750.[Medline]
4 Chan KB, Man-Son-Hing M, Molnar FJ, Laupacis A. (2001) How well is the clinical importance of study results reported? An assessment of randomized controlled trials. Can Med Assoc J 165:91197–1202.
5 Campbell MK, Mollison J, Steen N, Grimshaw JM, Eccles M. (2000) Analysis of cluster randomized trials in primary care: a practical approach. Fam Pract 17:2192–196.
6 Altman DG and Matthews JN. (1996) Statistics notes. Interaction 1: feterogeneity of effects. Br Med J 313:7055486.
7 Matthews JN and Altman DG. (1996) Statistics notes. Interaction 2: compare effect sizes not P values. Br Med J 313:7060808.
8 Matthews JN and Altman DG. (1996) Interaction 3: how to examine heterogeneity. Br Med J 313:7061862.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  J.-K. Soler, I. Okkes, M. Wood, and H. Lamberts The coming of age of ICPC: celebrating the 21st birthday of the International Classification of Primary Care Fam. Pract., August 1, 2008; 25(4): 312 - 317. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||