Family Practice Advance Access published online on September 1, 2008
Family Practice, doi:10.1093/fampra/cmn051
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Spectrum bias—why clinicians need to be cautious when applying diagnostic test studies
Health Methodology Group, University of Manchester, Block 3, First Floor, University Place, Oxford Road, Manchester M13 9PL, UK
Email: Brian.Willis{at}manchester.ac.uk
Received 24 September 2007; Revised 28 June 2008; Accepted 29 July 2008.
 |
Abstract |
|---|
 Top Abstract IntroductionWhat is spectrum bias?The causesClinical case 1Clinical case 2DiscussionDeclarationAppendixReferences |
|---|
When applying study results to their practice, the clinician is constrained by a number of factors, perhaps none more important than spectrum bias, which describes the effect a change in patient case mix may have on the performance of a test. Although the literature contains notable examples of spectrum bias, the emphasis has been to demonstrate its existence and its implications on study design rather than how it affects the clinician. Here a definition is proposed before considering it from a GP's perspective. As a patient's probability of disease is in part determined by the test's result, having reliable estimates of a test's performance is imperative to making good decisions on patient management. Knowing how the test performs on a patient usually means knowing its performance within a particular subgroup. Unfortunately, studies tend to report weighted average estimates of performance across broad populations. Such estimates may be inaccurate at an individual level and at a population level with the overall performance of the test in practice varying significantly from the average estimate reported, owing to differing case mixes. To avert such problems, investigators should design studies to evaluate tests over all relevant subgroups, and where this is not possible, to be explicit about the case mix in the study sample. Furthermore, GPs should endeavour to know both individual patients and practice populations as a whole in terms of demographics and co-morbidities before applying study results to their patients.
Keywords. Diagnostic tests, decision science, epidemiology, post-graduate education, sensitivity and specificity, likelihood ratio.
|
Introduction |
|---|
|
TopAbstractIntroductionWhat is spectrum bias?The causesClinical case 1Clinical case 2DiscussionDeclarationAppendixReferences |
|---|
Since it was first suggested as the ‘new paradigm for medical practice’,1 evidence-based medicine has become one of the daily realities to the practicing clinician. Unfortunately, its effective application is fraught with pitfalls,2 none more so than when using it to assist diagnosis.
In one editorial, it was remarked that the presence of spectrum bias, verification bias and a lack of blinding between the test and the reference standard consistently blighted studies on diagnostic tests to the extent that clinicians should be cautious before deciding to apply the results of a diagnostic study to their own practice.3
This article will concentrate on one of these biases in particular, namely spectrum bias. Not only does it represent a major constraint to using diagnostic tests in clinical practice, it is, perhaps, one of the least appreciated amongst clinicians. Following a brief explanation and discussion of the causes, it is illustrated by two clinical cases before discussing the implications for GPs.
|
What is spectrum bias? |
|---|
|
TopAbstractIntroductionWhat is spectrum bias?The causesClinical case 1Clinical case 2DiscussionDeclarationAppendixReferences |
|---|
In clinical practice, a diagnosis is often formulated by applying a sequence of tests which start the moment the patient enters the consulting room. Whether the test is asking a specific question or organizing a blood screen of renal function, the diagnostic process involves gathering information and modifying the probability of a diagnosis in the light of that information. This process is formalized in Bayes’ theorem4 and is schematically represented below (Fig. 1, also see Appendix).
|
The likelihood ratio represents a measurement of performance of the test and is derived from two other measures, the sensitivity and specificity.5 These latter two measures are dependent upon the threshold for a test positive, resulting in a trade-off between them––a low threshold results in a high sensitivity and low specificity, a relationship which may reverse as the threshold is increased.6
For a given threshold, it is often assumed that these performance measures remain constant between different settings giving rise to the relationship that the predictive value (post-test probability) of a test changes with the prevalence.5 However, the sensitivity, specificity and the likelihood ratio7 may vary between different patient subgroups even when the test threshold remains constant, and this lies at the heart of the concept of spectrum bias.
Although it has long been recognized,8 somewhat surprisingly spectrum bias has yet to receive an agreed formal definition. Essentially, it has two elements.
- (i) The performance of the test changes when applied to different patient subgroups.
- (ii) There is a bias which results from this change of performance.
- (ii) There is a bias which results from this change of performance.
The first condition may be illustrated by considering the influence that co-morbidities or medications taken may have on the probability of a false-positive or false-negative result. For example, in a 60-year-old man who feels unwell with pallor, the absence of chest pain is more likely to be a false negative for predicting myocardial ischaemia if he also suffers diabetes, than if he does not.9
Unlike the first condition, the necessity of the second is open to debate as some believe that spectrum bias is not really a bias and should be described as a spectrum effect, part of the normal variation expected in a test's performance.10
Yet, studies which provide estimates of a test's performance are often only unbiased within the original study population, and when they are applied to other populations such estimates may be biased. Unfortunately, thinking of spectrum bias in such a way would make it widespread and offer little pragmatic benefit.
A more practical approach is to consider whether the change in the estimate of a test's performance changes the probability of particular diagnoses (Fig. 1) sufficiently to impact upon decisions in the diagnostic process.7 Thus, an error in the probability of a diagnosis may lead to anyone of the following; the correct diagnosis still appearing more likely than other diagnoses, another ‘incorrect’ diagnosis appearing more likely or more than one diagnosis appearing equally likely. To the reasonable clinician, the latter two will lead to errors in clinical judgement such as instigating unnecessary investigations, treatments or according the wrong diagnosis or prognosis.
Extending on previous work,7 this approach forms the basis of the definition proposed here––spectrum bias is the error in clinical judgment which may result when a test with known performance characteristics at a known threshold in a known population is applied to a different population with the expectation that it maintains the same performance characteristics.
Here the performance characteristics are the sensitivity, specificity and the likelihood ratio of the test but do not include the positive and negative predictive values. A test's performance should not be confused with its accuracy which is defined as the proportion of correct test results (true positives and true negatives) in all those tested.11
|
The causes |
|---|
|
TopAbstractIntroductionWhat is spectrum bias?The causesClinical case 1Clinical case 2DiscussionDeclarationAppendixReferences |
|---|
The causes of spectrum bias were originally recognized by Ransonhoff and Feinstein8 and essentially reduce to three main factors.12 It is the change of one or more of these factors between settings which gives rise to spectrum bias.
Case mix of patients with disease
The test may be more sensitive to those with more severe disease, an effect seen in case-controlled studies which often report higher test accuracies13–16 than cohort studies by having a study population consisting of the sickest of the sick and the fittest of the fit.13
Case mix of patients without disease
The relative proportions of co-morbidities which are either similar or important differential diagnoses to the target disorder can affect the test's performance.
These first two factors are also affected by increased complexity of the case mix, most often seen in specialist settings, where there are higher proportions of patients in which the usual tests have failed to offer a diagnosis at a primary care level.12
Prevalence of disease
The performance characteristics of the test are often considered independent of the prevalence of disease.11,17 When the diagnosis of disease is clear-cut and the reference standard is perfect this is undoubtedly the case.18 However, in practice, disease status is often part of a continuum and a cut-off point is imposed to separate the diseased from the non-diseased.
Measurement error in the reference standard means that misclassification is most likely with individuals who have a disease status close to the cut off point.18 The effect of this is that the performance characteristics vary significantly with disease prevalence,18 and this explains part of the effect seen in empirical studies.8,19
Investigators have confirmed how changing one or more of these factors affects a test's performance, the urinary dipstick in detecting urinary tract infections,20 the rapid antigen test in detecting streptococcal pharyngitis21–23 and the magnetic resonance imaging (MRI) in detecting multiple sclerosis,24 all have their sensitivity and specificity affected by one or more of the above factors.
|
Clinical case 1 |
|---|
|
TopAbstractIntroductionWhat is spectrum bias?The causesClinical case 1Clinical case 2DiscussionDeclarationAppendixReferences |
|---|
In her surgery, Dr Jones sees a 12-year-old boy who is complaining of a sore throat. He has three additional features with a lack of cough, a history of fever and tonsillar exudate. She is aware that group A beta haemolytic streptococcus (GABHS) may cause such symptoms and wonders how likely this is based on these features.
Centor's original article derived probabilities that GABHS was the cause of a sore throat based on the number of additional features present.25 The study population was those with a sore throat attending a US emergency department. The prevalence of GABHS in this group was 17% but in those with three additional features the average probability for GABHS was 30.1–34.1%.25
However, in a 12 year old, the prevalence of GABHS with these features may be higher. McIsaac et al.26 considered the effect of age in addition to the clinical features identified by Centor et al.25 and found in children with similar features to Dr Jones’ patient the probability of GABHS was 68% [95% confidence interval (CI): 58–77]. The study was on a family practice population with a prevalence of 34% (95% CI: 30–39).
If Dr Jones used Centor's study to inform on the probability of GABHS (30.1–34.1%), she could reasonably opt for further investigation such as a throat swab before initiating treatment. In contrast, the study of McIsaac et al.26 recommends empirical treatment without further investigation when the probability of GABHS is 68%.
Could the increased probability of GABHS be explained by an increase in background prevalence alone or does the test (the prediction rule) perform differently between different populations?
From Figure 1, the prevalence (pre-test probability) and the post-test probability are linked by the likelihood ratio. In the study of Centor et al.,25 the positive likelihood ratio (LR+) is 2.3 compared with McIssac26 of 4.1. Thus, the change in case mix between an emergency department and primary care population has changed the performance of the prediction rule.
A more relevant comparison may be derived from a study in a family practice setting, by Dagnelie et al.,21 which as part of evaluating a rapid antigen test for GABHS-stratified patients according to Centor's criteria.25 The prevalence of GABHS was 33% (95% CI: 29–37) (similar to McIsaac et al.26), and the positive likelihood ratio for three features or more in all patients (before applying the rapid antigen test) was only 1.8 giving a post-test probability of 47% (95% CI: 41–53)21––probably still not high enough to warrant empirical treatment before further investigation (Table 1).
|
Thus, even in similar settings with a similar disease prevalence, the prediction rule behaves differently in different subgroups. The LR+ for those in a general family practice population presenting with a sore throat is 1.8,21 compared with 4.126 in the subgroup of children, which importantly impacts on the management decision.
In effect, the studies of Dagnelie et al.21 and Centor et al.25 both derived likelihood ratios and predictive values which represented weighted averages of all the relevant subgroups, and this example demonstrates the problems of using such weighted average estimates ahead of those in relevant subgroups (Table 2).
|
|
Clinical case 2 |
|---|
|
TopAbstractIntroductionWhat is spectrum bias?The causesClinical case 1Clinical case 2DiscussionDeclarationAppendixReferences |
|---|
Dr Smith has just read an article on the performance of plasma N-terminal pro-brain natriuretic peptide (NT-proBNP) in detecting systolic heart failure in an unselected general population.27 In the study, the prevalence of systolic heart failure in patients over 70 years old was 3.7% (95% CI: 2.1–6.6) and in this group the test performed with both a sensitivity of 91% (95% CI: 62–98) and specificity of 91% (95% CI: 87–94)27 (Table 3).
|
He considers whether he may be able to apply the study to his own practice to help screen for heart failure in patients over 70. He knows he has around 2000 patients in this age group and estimates the prevalence of heart failure to be between 2% and 3%.
What he does not realize is that, even at the same cut-off for a positive result, owing to his case mix being different from the study's, the test performs with a sensitivity of 94% (95% CI: 84–98) and a specificity of 70% (95% CI: 68–72) despite the prevalence being as expected (Table 3).
If this was known, how would it affect his decision to use the test to screen for heart failure? As a screening test, NT-proBNP would be used to identify those patients in need of further investigation, which in this instance would be an echocardiogram. The success of the test could therefore be measured in terms of the absolute numbers of heart failures missed (false negatives) and numbers of patients unnecessarily referred for an echocardiogram (false positives).
Although the likelihood ratios for a positive test result differ markedly between the study and that encountered in practice, they provide less information on actual numbers of patients referred and heart failures missed. Such information is more easily derived from the sensitivities and specificities (Table 4).
|
Applying the study results to Dr Smith's population would suggest only 221 people of 2000 screened being referred for echocardiogram for four cases of heart failure missed, whereas in reality nearly three times as many (632) would be referred for the same number of patients screened with only three cases of heart failure missed (Table 4).
Notwithstanding the cost of the test itself, depending on the availability and cost of the local echocardiogram service, the difference between such figures may mean the difference between adopting NT-proBNP as a screening test and not.
The change in the test's performance between the two settings may be explained in terms of a change in the patient characteristics of the respective populations. Thus, target populations which exhibit increased numbers of elderly,28 female gender,28 hypertension,29 chronic obstructive pulmonary disease,29 hyperthyroidism29 and patients with severe heart failure30 have all been associated with increased levels of NT-proBNP.
|
Discussion |
|---|
|
TopAbstractIntroductionWhat is spectrum bias?The causesClinical case 1Clinical case 2DiscussionDeclarationAppendixReferences |
|---|
Faced with a patient of known characteristics, the clinician would like to know whether a given test result changes the probability of a particular diagnosis so that an informed decision may be taken on management. If the probability of the diagnosis prior to testing is known, then Bayes’ theorem4 allows this post-test probability to be calculated providing the likelihood ratio of the test has been estimated (Fig. 1).
Outside clinicians performing studies on their own practice populations, estimates of the likelihood ratio are obtained by critically appraising the evidence-based literature.31 Ideally, a study evaluating the test on patients with the pertinent characteristics will exist and this would provide the best estimate.
Unfortunately, such studies are often absent, as sample sizes of each subgroup are usually too small to reduce the standard error to give meaningful estimates of performance. This is further complicated by patients belonging to more than one subgroup––for example, how different is the performance of NT-proBNP in detecting heart failure in elderly patients with hypertension and hyperthyroidism compared with those who are just elderly?
Without studies on individual subgroups, the clinician must resort to using evaluations of tests on broader populations. These provide weighted averages of the test performance where the weights are proportional to the number of patients in each subgroup. However, as shown in Dr Jones’ case, using such estimates may make spectrum bias more likely and so affect patient management.
Spectrum bias may also arise when the study sample differs in case mix from that encountered in practice, as in the second clinical case. Here the resulting change in the test's performance affected the decision to adopt a new screening technology in practice.
GPs should be especially wary of this type of example when attempting to apply studies on tests conducted in secondary care settings. This situation is probably accentuated by the relative paucity of diagnostic test research in primary care where the only studies that may be available are ones conducted in more specialist settings. But such studies may not only misrepresent the overall performance of the test in a general practice population, they may affect individual patients as decisions on management are being based on incorrect estimates of their probability of disease.7
Notwithstanding other important factors which affect transferability of study results to clinical practice,12 spectrum bias alone imposes a number of limitations. To circumvent these essentially entails addressing two issues.
The first lies with research investigators designing higher powered studies so that the test performance may be separately analysed in all relevant patient subgroups. To inform clinical practice, this needs to be a routine part of reporting and when subgroup analyses are not possible, adequate detail of the case mix should be provided. Unfortunately, research so far has been mainly confined to demonstrating spectrum bias in a few limited studies8,20–24 rather than considering it as part of the norm in evaluating a test.
Secondly, responsibility lies with GPs to be mindful of the complexities of the diagnostic process. Every test's performance should be considered to be potentially affected by the presence of co-morbidities, the severity of disease and its prevalence in the patient population. This includes features extracted from the history or examination, where each targeted question or examination may be considered to be a separate diagnostic test.
Therefore, when applying studies to assist the diagnostic process, GPs should use in preference to all others, those estimates of the test's performance derived from the relevant patient subgroups. When such studies are not available, the next best estimate is likely to be a weighted average derived from a study of a broad population.
However, we should attempt where possible, to determine whether the case mix of the practice population matches that in the study sample before using such estimates.
|
Declaration |
|---|
|
TopAbstractIntroductionWhat is spectrum bias?The causesClinical case 1Clinical case 2DiscussionDeclarationAppendixReferences |
|---|
Funding: None.
Ethical approval: Not relevant.
Conflicts of interest: None
|
Appendix |
|---|
|
TopAbstractIntroductionWhat is spectrum bias?The causesClinical case 1Clinical case 2DiscussionDeclarationAppendixReferences |
|---|
Bayes' theorem4
For a binary test, where test results are either positive T+ or negative T– and subjects either have disease D+ or do not have disease D–, then Bayes' theorem may be written as
 |
The two key features of Bayes’ theorem are that it uses conditional probabilities, the probability of an event B occurring given A has occurred, written P(B|A) and provides a method of calculating P(A|B) once P(B|A) is known. This latter point proves useful in diagnosis—usually what is known from studies is the test's sensitivity P(T+|D+) and specificity P(T–|D–) and what is required in practice and unknown is the probability of disease in a patient given their test result, e.g. P(D+|T+) or P(D–|T–).
By defining the prevalence of disease P(D+) as P, the test's sensitivity as Se and the specificity as Sp, the equation may be written as
 |
Dividing by (1 – Sp)(1 – P) and defining the positive likelihood ratio as 
 |
Thus, knowing the prevalence of disease, P, and the likelihood ratio of the test LR+ from Equation (3), the post-test probability of a diagnosis (PPV) maybe calculated. This means, as shown in Figure 1, the PPV changes with both the prevalence and the likelihood ratio.
Equation (3) is usually written in the more simplified form of odds
 |
and the post-test odds, Po, is derived from the PPV, i.e. 
. |
Acknowledgments |
|---|
I would like to thank Chris Hyde, John Deeks, Martin Roland and the reviewers who have read this article for their helpful comments and suggestions.
|
Notes |
|---|
Willis BH. Spectrum bias—why clinicians need to be cautious when applying diagnostic test studies. Family Practice 2008; Pages 1–7 of 7.
|
References |
|---|
|
TopAbstractIntroductionWhat is spectrum bias?The causesClinical case 1Clinical case 2DiscussionDeclarationAppendixReferences |
|---|
1 Evidence Based Working Group. Evidence based medicine. JAMA (1992) 268:2420–2425.
2 Straus SE, McAlister FA. Evidence-based medicine: a commentary on common criticisms. CMAJ (2000) 163:837–841.
3 Furukawa TA, Guyatt GH. Sources of bias in diagnostic accuracy studies and the diagnostic process [Commentary]. CMAJ (2006) 174:481–482.
4 Stirzaker D. Elementary Probability (2003) 2nd edn. Cambridge, UK: Cambridge University Press. 54–57.
5 Sackett DL, Haynes RB, Guyatt GH, Tugwell P. Clinical Epidemiology: A Basic Science for Clinical Medicine (1991) 2nd edn. Boston, MA: Little Brown. 69–152.
6 Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology (1982) 143:29–36.
7 Goehring C, Perrier A, Morbia A. Spectrum bias: a quantitative and graphical analysis of the variability of medical diagnostic performance. Stat Med (2004) 23:125–135.[CrossRef][Web of Science][Medline]
8 Ransonhoff DF, Feinstein AR. Problems of spectrum and bias in evaluating the efficacy of diagnostic tests. N Engl J Med (1978) 299:926–930.[Abstract]
9 Koistinen MJ. Prevalence of asymptomatic myocardial ischemia in diabetic subjects. BMJ (1990) 301:92–95.
10 Mulherin SA, Miller WC. Spectrum bias or spectrum effect? Subgroup variation in diagnostic test evaluation. Ann Intern Med (2002) 137:598–602.
11 Zhou X, Obuchowski NA, McClish DK. Statistical Methods in Diagnostic Medicine (2002) New York: John Wiley and Sons. p21.
12 Irwig L, Bossuyt P, Glasziou P, Gastonis C, Lijmer J. Designing studies to ensure that estimates of test accuracy will travel. In: The Evidence Base of Clinical Diagnosis—Knottnerus JA, ed. (2002) London: BMJ Publishing Group. 95–116.
13 Rutjes AWS, Reitsma JB, Vandenbroucke JP, Glas AS, Bossuyt PMM. Case-control and two-gate designs in diagnostic accuracy studies. Clin Chem (2005) 51:1335–1341.
14 Whiting P, Rutjes AWS, Reitsma JB, Glas AS, Bossuyt PMM, Kleijnen J. Sources of variation and bias in studies of diagnostic accuracy. A systematic review. Ann Intern Med (2004) 140:189–202.
15 Rutjes AWS, Reitsma JB, Di Nisio M, Smidt N, van Rijn JC, Bossuyt PMM. Evidence of bias and variation in diagnostic accuracy studies. CMAJ (2006) 174:469–476.
16 Lijmer JG, Willem Mol B, Heisterkamp S, et al. Empirical evidence of design-related bias in studies of diagnostic tests. JAMA (1999) 282:1061–1066.
17 Pepe MS. The Statistical Evaluation of Medical Tests for Classification and Prediction (2004) Oxford: OUP. 15.
18 Brenner H, Gefeller O. Variation of sensitivity, specificity, likelihood ratios and predictive values with disease prevalence. Stat Med (1997) 16:981–991.[CrossRef][Web of Science][Medline]
19 Willis BH, Sur SD. How good are emergency department senior house officers at interpreting x-rays following radiographers' triage? Eur J Emerg Med (2007) 14(1):6–13.[CrossRef][Web of Science][Medline]
20 Lachs MS, Nachamkin I, Edelstein PH, Goldman J, Feinstein AR, Schwartz JS. Spectrum bias in the evaluation of diagnostic tests: lessons from the rapid dipstick test for urinary tract infection. Ann Intern Med (1992) 117:135–140.
21 Dagnelie CF, Bartelink ML, Van Der Graaf Y, Goessens W, De Melker RA. Towards a better diagnosis of throat infections (with group A beta-haemolytic streptococcus) in general practice. Br J Gen Pract (1998) 48:959–962.[Web of Science][Medline]
22 DiMattteo LA, Lowenstein SR, Brimhall B, Reiquam W, Gonzales R. The relationship of pharyngitis and the sensitivity of a rapid antigen test: evidence of spectrum bias. Ann Emerg Med (2001) 38:648–652.[CrossRef][Web of Science][Medline]
23 Hall MC, Kieke B, Gonzales R, Belongia EA. Spectrum bias for a rapid antigen detection test for group A β-haemolytic streptococcus pharyngitis in a paediatric population. Pediatrics (2004) 114:182–186.
24 O'Connor PW, Tansey CM, Detsky AS, Mushin AI, Kucharczyk W. The effect of spectrum bias on the utility of magnetic resonance imaging and evoked potentials in the diagnosis of suspected multiple sclerosis. Neurology (1996) 47:140–144.
25 Centor RM, Witherspoon JM, Dalton HP, Brody CE, Link K. The diagnosis of strep throat in adults in the emergency room. Med Decis Making (1981) 1:239–246.
26 McIsaac WJ, Kellner JD, Aufricht P, Vanjaka A, Low DE. Empirical validation of guidelines for the management of pharyngitis in children and adults. JAMA (2004) 291:1587–1595.
27 Groenning BA, Raymond I, Hildebrandt PR, Nilsson JC, Baumann M, Pedersen F. Diagnostic and prognostic evaluation of left ventricular systolic heart failure by plasma N-terminal pro-brain natriuretic peptide concentrations in a large sample of the general population. Heart (2004) 90:297–303.
28 Loke I, Squire IB, Davies JE, Ng LL. Reference ranges for natriuretic peptides for diagnostic use are dependent on age, gender and heart rate. Eur J Heart Fail (2003) 5:599–606.
29 Felker GM, Petersen JW, Mark DB. Natriuretic peptides in the diagnosis and management of heart failure. CMAJ (2006) 175:611–617.
30 Gustafsson F, Steensgaard-Hansen F, Badskjaer J, Poulsen AH, Corell P, Hildebrandt P. Diagnostic and prognostic performance of N-terminal ProBNP in primary care patients with suspected heart failure. J Card Fail (2005) 11(5 suppl):S15–S20.[CrossRef][Web of Science][Medline]
31 Whiting P, Rutjes AWS, Dinnes J, Reitsma JB, Bossuyt PMM, Kleijnen J. Development and validation of methods for assessing the quality of diagnostic accuracy studies. Health Technol Assess (2004) 8:59–65.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  G. M Lasseter, C. A. McNulty, F. Richard Hobbs, D. Mant, P. Little, and on behalf of the PRISM Investigators In vitro evaluation of five rapid antigen detection tests for group A beta-haemolytic streptococcal sore throat infections Fam. Pract., December 1, 2009; 26(6): 437 - 444. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||