Family Practice Vol. 21, No. 2, 155-159
Family Practice Vol. 21, No. 2 © Oxford University Press 2004, all rights reserved.
Article |
GPs' and physicians' interpretation of risks, benefits and diagnostic test results
Evidence for Population Health Unit, School of Epidemiology and Health Sciences, The Medical School, The University of Manchester, Stopford Building, Oxford Road, Manchester M13 9PT, UK
E-mail: Dick.Heller{at}man.ac.uk
Received 19 March 2003; Revised 22 October 2003; Accepted 3 November 2003.
Heller RF, Sandars JE, Patterson L and McElduff P. GPs' and physicians' interpretation of risks, benefits and diagnostic test results. Family Practice 2004; 21: 155–159.
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionAppendix: initial case scenariosReferences |
|---|
Background. Understanding pre-test probability and baseline risks helps to interpret the results of diagnostic tests and the benefits of treatment, but how good is the understanding of these concepts?
Objectives. Our aim was to assess the ability of GPs and consultant physicians to make accurate estimates and understand the application of pre-test probability and baseline risk for two common clinical conditions.
Methods. A two-stage questionnaire survey based on case scenarios of patients with angina and congestive heart failure was carried out of 202 physicians, randomly selected from the members of the Royal College of Physicians in the NW of England, 205 GPs randomly chosen from the practice list of the NW Health Authorities and 128 MRCGP examiners attending an examiners meeting. A total of 115, 106 and 81 members of these groups, respectively, responded to the first stage, and 44, 46 and 64 to the second. The main outcome measures were the stated likelihood of true ischaemic heart disease (IHD) being present and the predicted 1-year mortality; the impact of changing prevalence and baseline risk on these results; and interpretation of different methods of risk presentation.
Results. Estimates of pre-test probability of IHD being present ranged from 5 to 100% and of baseline risk of 1-year mortality from 0 to 86%. More GP examiners and consultant physicians understood the impact of increasing age on the test result than did the random sample of GPs. A majority of each group correctly said that increasing age would reduce the number needed to treat (NNT). Presentation of benefit as relative risk reduction was a greater stimulus to starting treatment than the NNT or measures of population impact.
Conclusion. Clinicians should collect data to allow a better knowledge of the likelihood of disease and of baseline risk in their patient populations. Methods to increase the understanding of the influence of pre-test probability on diagnostic test results and of how to quantify and demonstrate the impact of the benefit of interventions should be explored.
Keywords. Diagnostic test, population impact, probability, risk.
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionAppendix: initial case scenariosReferences |
|---|
Good clinical practice requires the intelligent use of diagnostic tests and decisions about appropriate treatments.1 In addition to the experiential tacit knowledge and the other factors involved in the doctor–patient relationship, some numerical aspects of clinical decision making will help the choice of treatments and diagnostic tests.
An understanding of the pre-test probability (or the likely prevalence of the condition being tested in a population of similar patients) helps to interpret the results of a diagnostic test; in low prevalence settings, a positive test is likely to be a false positive and, in high prevalence settings, a negative test is likely to be a false negative.2 For clinicians to interpret the results of diagnostic tests appropriately, they have first to be able to estimate the pre-test probability and, secondly, understand the implications of their estimate of pre-test probability on the post-test probability of the disease actually being present or not.3 Similarly, in making clinical decisions about whether or not to start a particular treatment, the clinician will want to apply the expected risk reduction (usually from trial results) to the baseline risk faced by the patient and to understand different ways of presenting the impact of this risk reduction.
We have shown recently that representative samples of both Australian and UK GPs and physicians vary considerably in their estimates of pre-test probability (Attia et al., submitted for publication), and that there were few identifiable predictors of accurate estimations. This is consistent with previous literature among physicians and medical students.3,4 We would expect that consultant physicians see a spectrum of patients, which contains a higher prevalence and severity of disease than seen by the referring GP, and that this would be incorporated in their estimates of pre-test probability and the baseline risk of subsequent mortality. This study aimed to compare GPs and consultant physicians, in terms of the accuracy of their knowledge of pre-test probability, their understanding of the effect of low and high pre-test probability on the interpretation of the test result, and their ability to predict baseline risk and to understand different presentations of risk reduction.
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionAppendix: initial case scenariosReferences |
|---|
Questionnaires were mailed to 202 physicians, randomly selected from the members of the Royal College of Physicians in the NW of England and to 205 GPs randomly chosen from the practice list of the NW Health Authorities, and handed to 128 MRCGP examiners attending an examiners meeting. There were 145 examiners (from throughout the UK) at the time, of whom 17 did not attend the meeting. The questionnaire included a case scenario of a 65-year-old man with anginal chest pain, and asked for an estimate of the likelihood of true ischaemic heart disease (IHD) being present, and a case scenario which asked for risk estimates of subsequent mortality in a 55-year-old woman with congestive heart failure (CHF) (see Appendix). Demographic factors, including age, gender, practice profile, specialty and post-graduate studies, were also included in the questionnaires. Doctors were asked to mail the replies back; one reminder was sent to non-respondents 2 weeks later. Following receipt of the response, a second questionnaire was mailed. This presented the scenarios again, but for the angina scenario probed the understanding of the influence of increasing age on the likelihood of false-positive and false-negative test results, and for the CHF scenario probed the understanding of the influence of increasing age on the number needed to treat (NNT). A separate question presented estimates of benefits in terms of NNT, relative risk reduction and the impact on the population (using treatment of CHF with ß-blockers as an example) and asked whether the form of presentation would influence the decision to start treatment.
Statistical analysis
The distribution of responses for pre-test probabilities and baseline risk are shown as box and whisker plots. Frequencies are presented for all other questions, and comparisons of responses were made using the chi-square or McNemar's test.
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionAppendix: initial case scenariosReferences |
|---|
Responses to the initial questionnaire were obtained from 115 of the 202 physicians (57%), 106 of the 205 GPs (52%) and 81 of the 128 RCGP examiners (63%). The characteristics of the three groups of respondents are shown in Table 1.
|
Second questionnaires were received from 46 of the original 115 physician respondents (40%), 44 of the 106 GP respondents (42%) and 66 of the 81 RCGP respondents (82%). The characteristics and responses to the initial questionnaires were similar in all groups between those who did and did not reply to the second questionnaire, and the level of accuracy to responses in part 1 was not found to be a predictor of likelihood to respond to the part 2 questionnaire.
Estimates of pre-test probability of IHD being present ranged from 5 to 100% and of baseline risk of 1-year mortality from 0 to 86%. Figures 1 and 2 show the mean values and the extent of the variability within each group of doctors. There were no significant differences between the groups of doctors.
|
|
Around two-thirds of the physicians and GP examiners understood that lower prevalence would be associated with more false-positive results and higher prevalence would be associated with more false-negative results. The respondents from the random sample of GPs were significantly less likely to answer these questions correctly (Table 2).
|
A majority of GPs, GP examiners and physicians (79, 83 and 76%, respectively) were correct in saying that increased baseline risk associated with increased age would reduce the NNT. The preference for methods of presentation of benefit was not statistically significantly different between doctor groups, and the analysis is presented for all groups combined (Table 3). Presentation of the benefit of using ß-blockers in relative terms was a more potent stimulus to starting treatment than presentation using the NNT (Table 3). Presenting this benefit in population terms was a very poor stimulus to treatment; however, when the population benefit was expressed in terms of numbers that related to a defined general practice population and the number of candidates for treatment, the stimulus was similar to the presentation of the NNT (Table 3).
|
There was no consistent pattern in any of the demographic variables measured among the doctors to predict the response to any of the questions.
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionAppendix: initial case scenariosReferences |
|---|
We have demonstrated that the knowledge of pre-test probability of disease varied widely, as did estimates of 1-year mortality. This is consistent with our previous report among Australian and UK clinicians which included other common and clinical conditions, and with other literature.3,4 For both pre-test probability of IHD and baseline risk of 1-year CHD mortality, estimates given by the respondents were higher than suggestions from the literature (675 and 8%,6 respectively). This comparison is less relevant than the variability, as the literature-based estimates will not take account of the different settings in which the surveyed doctors practice. There was no difference between groups of clinicians who differ in the part of the disease spectrum they see, and was not better among those who would be expected to have a particularly accurate understanding of the scientific basis of clinical practice (specialist college examiners). The understanding of the impact of disease prevalence (pre-test probability) on the chance of a positive or negative test result being true or false is also not universally understood, and significantly less well understood by GPs than by GP examiners or consultant physicians.
Both GPs and physicians had good understanding of how increasing risk (with increasing age) would reduce the NNT. This contradicts a study which showed that GPs appeared to have a poor understanding of the NNT (and other measures required for the practice of evidence-based medicine).7 As can be seen from Table 3, when the same magnitude of benefit is framed in different ways, the respondents to this survey are more influenced by the presentation of benefits in relative than absolute terms. This has been shown previously (although it is not clear if it makes a difference in practice).8,9 When our new measures, which attempt to provide a population perspective to the demonstration of risks and benefits, are presented in abstract terms of the population impact number (PIN),10 it is clear that clinicians do not see this as helping their decision-making processes. The PIN relates to a whole population rather than to candidates for treatment and may thus seem remote to clinical decision making. When the same result is expressed in terms of a defined general practice population and the number of candidates for treatment,11 the influence on decision making appears to increase and is comparable with the perceived benefit of the NNT.
The response rates to both questionnaires were not high, although the GP examiners were more likely to respond in both instances, raising the question of whether we are able to generalize from these results. The consistency of the responses to the first questionnaire between GPs, GP examiners and physicians, as well as our previous findings of similarities between the UK and Australia, and the lack of difference in answers to the first questionnaire between the responders and non-responders to the second questionnaire, suggest that similar findings might be present in other groups. Although the case scenarios initially were developed to reflect typical patients seen in clinical practice, one of the reasons for the poor response might be the feeling that such scenarios do not reflect clinical practice among the doctors surveyed. The relationship between artificial case scenarios and observations on real clinical practice may not be close, and examination of practice patterns would be preferable.
Our findings are important since they identify that many clinicians, irrespective of whether practising in primary or secondary care, may not appropriately interpret diagnostic tests or estimate baseline risk in actual practice. To do this, clinicians will need to obtain accurate estimates of prevalence and of risk in the population from which their patient comes. Few clinicians collect data from groups of patients to allow such estimates to be part of routine clinical practice, although there may be considerable variation in prevalence for certain diseases among individual general practices.12 We encourage groups of clinicians to explore how to increase the collection of data for these purposes.9 It may be, however, that doctors' numeracy skills will require updating before better availability of local data can help to guide practice.
This study adds to the previous literature since it describes similarly variable knowledge of pre-test probabilities and of risks among three groups of clinicians who work with populations that have different pre-test probabilities, different diagnostic and treatment experiences and different educational attainments.
The majority of interventions that have been introduced to alter test ordering behaviour have adopted an approach that has used guidelines and audit with feedback.13 A major limitation of guidelines is their application to a local population in which the individual clinician is working and which may have a different disease prevalence.14 In addition, there are important cognitive aspects of decision making in which a systematic bias is introduced that alters the perception of probability and that may not be reduced by the use of guidelines.15
An educational intervention to increase understanding could be offered during either medical school or higher professional training, or support mechanisms at the point of decision making. In view of the extent of the problem we have described for common conditions, we suggest that such solutions should be explored.
Conclusion
Clinicians should collect data to allow a better knowledge of the likelihood of disease and of baseline risk in their patient populations. Methods to increase the understanding of the influence of pre-test probability on diagnostic test results and of how to quantify and demonstrate the impact of the benefit of interventions should be explored.
|
Appendix: initial case scenarios |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionAppendix: initial case scenariosReferences |
|---|
Atypical angina
A 65-year-old man presents having had two episodes of retrosternal chest pain today, both precipitated by exertion, but lasting approximately 2 h despite rest. Before obtaining the rest of the history or performing a physical exam, what would you estimate his risk of true ischaemic heart disease to be?
Prognosis in congestive heart failure
A 50-year-old woman presents with symptoms consistent with congestive heart failure, developing over the last 6 weeks. She reports being breathless only with moderate exercise, but her 6 minute walk test (number of metres walked in 6 minutes) is only 300 m (the lowest quartile). An echocardiogram shows an ejection fraction of 35%. What do you estimate her risk of mortality over the subsequent year to be?
|
Acknowledgments |
|---|
This study evolved from a previous collaboration between Dr J Attia and Professor K Nair of Newcastle, Australia with RFH.
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionAppendix: initial case scenariosReferences |
|---|
1 Sox H, Blatt M, Higgins M, Marton K. Medical Decision Making. Boston: Butterworths; 1998.
2 Fletcher RH, Fletcher SW, Wagner E. Clinical Epidemiology: The Essentials. Baltimore: Williams and Wilkins; 1996.
3 Richardson WS. Five uneasy pieces about pre-test probability. J Gen Intern Med 2002; 17: 848–853.
4 Noguchi Y, Matsui K, Imura H, Kiyota M, Fukui T. Quantitative evaluation of the diagnostic thinking process in medical students. J Gen Intern Med 2002; 17: 848–853.
5 Diamond GA, Forrester JS. Analysis of probability as an aid in the clinical diagnosis of coronary artery disease. N Engl J Med 1979; 300: 1350–1358.
6 Bittner V, Weiner DH, Yusuf S et al. Prediction of mortality and morbidity with a 6 minute walk test in patients with left ventricular dysfunction. J Am Med Assoc 1993; 270: 1702–1707.
7 Young JM, Glasziou P, Ward JE. General practitioners' self rating of skills in evidence based medicine: validation study. Br Med J 2002; 324: 950–951.
8 McGettigan P, Sly K, O'Connell D, Hill S, Henry D. The effects of information framing on the practices of physicians. J Gen Intern Med 1999; 14: 633–642.
9 Heller RF, Page JH. A population perspective to evidence based medicine: ‘evidence for population health’. J Epidemiol Community Health 2002; 56: 45–47.
10 Heller RF, Dobson AJ. Disease impact number and population impact number: a population perspective to measures of risk and benefit. Br Med J 2000; 321: 950–952.
11 Heller RF, Edwards R, McElduff P. Implementing guidelines in primary care: can population impact measures help? BMC Public Health 2003; 3: 7 http://www.biomedcentral.com/ 1471–2458/3/7
12 Whitford DL, Griffin SJ, Prevost AT. Influences on the variation in prevalence of type 2 diabetes between general practices: practice, patient or socioeconomic factors? Br J Gen Pract 2003; 53: 9–14.
13 Solomon DH, Hashimoto H, Daltroy L, Liang MH. Techniques to improve physicians' use of diagnostic tests: a new conceptual framework. J Am Med Assoc 1998; 280: 2020–2027.
14 Hutchinson A, Baker R. Making Use of Guidelines in Clinical Practice. Abingdon (UK): Radcliffe Medical Press; 1999.
15 Elstein AS. Clinical problem solving and diagnostic decision making: selective review of the cognitive literature. Br Med J 2002; 324: 729–732.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  J. M. Griffith, C. L. Lewis, S. Hawley, S. L. Sheridan, and M. P. Pignone Randomized Trial of Presenting Absolute v. Relative Risk Reduction in the Elicitation of Patient Values for Heart Disease Prevention With Conjoint Analysis Med Decis Making, March 1, 2009; 29(2): 167 - 174. [Abstract] [PDF]
|
|
|
|
|
|
M. T. Bianchi, B. M. Alexander, and S. S. Cash Incorporating Uncertainty Into Medical Decision Making: An Approach to Unexpected Test Results Med Decis Making, January 1, 2009; 29(1): 116 - 124. [Abstract] [PDF]
|
|
|
|
 |
|
 M. T Bianchi and B. M Alexander Evidence based diagnosis: does the language reflect the theory? BMJ, August 26, 2006; 333(7565): 442 - 445. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||