Family Practice Vol. 21, No. 2, 160-165
Family Practice Vol. 21, No. 2 © Oxford University Press 2004, all rights reserved.
Article |
Quality indicators and variation in primary care: modelling GP referral patterns
a Department of General Practice and b Department of Public Health, Wellington School of Medicine and Health Sciences, PO Box 7343, Wellington South, New Zealand and c Tayside Centre for General Practice, University of Dundee, Kirsty Semple Way, Dundee DD2 4AD, UK
E-mail: t.love{at}tcgp.dundee.ac.uk
Received 10 March 2003; Revised 22 October 2003; Accepted 3 November 2003.
Love T, Dowell AC, Salmond C and Crampton P. Quality indicators and variation in primary care: modelling GP referral patterns. Family Practice 2004; 21: 160–165.
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionReferences |
|---|
Background. Health agencies frequently seek to develop indicators of the quality and performance of work done by clinicians. The validity of such indicators is a subject of debate among clinicians and health managers.
Objectives. Our aim was to quantify the effects of chance and small caseload on an indicator of referral behaviour for GPs.
Methods. The study used random simulation of GP referral to physiotherapy and variance components analysis of routinely collected accident insurance data. It analysed 129 079 episodes of accident-related back pain in New Zealand which were managed by 2679 GPs. The main outcome measure was the percentage of back pain cases referred for physiotherapy and for specialist assessment and by each GP.
Results. The observed number of GPs who refer to physiotherapy at high levels is satisfactorily accounted for by chance. The variability of practice among GPs within any one area is not related to the absolute level of referral.
Conclusion. The primary care setting, in which a low caseload for any one condition is the norm, presents challenges for measuring clinical performance. An emphasis upon changing the behaviour of GPs with extremely high levels on a performance indicator cannot necessarily be expected to have an impact upon the level of the indicator across a geographic area. Indicators for quality improvement should be used across whole populations of practitioners, rather than used to focus upon extremely high referring individuals.
Keywords. Back pain, computer simulation, physician's practice patterns.
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
There is an increasing interest in monitoring medical practice. In part, this has arisen from the environment of decreasing trust in health systems in general and the medical profession in particular. Such a decline in trust has been clearly documented in the USA1 and, in the wake of publicity surrounding examples of poor practice, is on the increase in the UK.2
Since the 1990s, there has been much debate about the appropriateness of indicators which can be used to monitor and improve the quality of health care.3 In 1995, Majeed and Voss noted a trend for Family Health Services Authorities to use performance indicators for general practices, and discussed both the benefits and disadvantages of league tables based upon such indicators for primary care.4
Within primary care, there has been a particular focus upon indicators for prescribing, which is a key activity for GPs and has considerable financial consequences for health funding agencies. Braybrook has described the use of indicators to monitor the effect of feedback to GPs about their prescribing,5 while Shelley has analysed some of the limitations of a specific prescribing indicator: the ratio of bronchodilators to inhaled steroids.6 Campbell has analysed a range of prescribing indicators, finding that the indicators which prescribers would accept as valid have had only a narrow focus, allowing a limited interpretation of the quality of prescribing among a group of GPs.7
Increasingly, there is a move to make indicators available to the public. In some instances, quality profiles have been released publicly, but there has been low uptake of such information. For example, while the Health Care Financing Authority released hospital mortality statistics publicly from 1986, only a quarter of consumers polled in a 1990 survey were aware of this, even though they expressed a high level of interest in such data being available.8 In the UK, recent research has found that although the service users supported the general direction of greater openness, they had concerns about the consequences of making quality information about general practice available to the public.9
The increasing interest in using indicators in a primary care setting, both confidentially to monitor the practice of clinicians within primary care organizations and publicly to inform choice about where to seek health care, raises important issues about the robustness and appropriateness of indicators. However, while it has been pointed out that indicators must be based upon evidence and should be clearly linked to a causal chain which is influenced by the primary care clinician,10,11 the role of chance in affecting primary care indicators has not been assessed.
GPs, as generalists, inherently manage a diverse caseload in which no single patient condition or procedure accounts for a large proportion of their practice.12 Starfield reports that many patient contacts are not attributable to a specific disease, and shows that the 50 most common diagnoses account for only 54.2% of general practice visits in the USA, whereas the most common diagnoses account for a much higher proportion of visits among other medical specialties.13 Where workload is so varied, the absolute numbers of cases which an individual generalist will manage in any one clinical area may be quite small. A consequence of such small numbers may be that indicator measures are unduly affected by chance: a GP with only a handful of cases in some clinical field might be observed to have an exceptionally high or low level of prescribing or referral in that field, since the low number of cases would mean that a single decision to refer (or not) would make a large difference to his or her percentage referral rate.
Moore and Roland used simulation techniques to explore the variation expected in referral rates to various specialties for GP populations, finding that for populations of the order of normal GP list size, the expected variation in referral rates could be dramatically large. These authors noted that the small numbers in general practice presented a problem for agencies which wanted to provide comparative feedback on GP referral practices.14 The current environment of increased reliance upon performance indicators to measure the appropriateness and quality of care, and even to determine funding for practitioners, means that quantifying the effect of small numbers upon primary care measurements takes on a renewed importance.
We report the findings of a quantitative investigation which uses simulation to clarify the impact of chance and small caseloads upon indicators of GP referral practice in the management of back pain. Simulation techniques have been used from time to time to investigate the chance variation of referral rates for populations,14–17 but this study uses the technique to simulate an entire national distribution of GP referral rates at the level of an individual GP. We have also used variance components analysis to measure the variance of GP referral to physiotherapy within geographic areas.
Back pain is a common condition, with a lifetime prevalence of 
58%.18 It is frequently managed in general practice, and the services to which GPs refer their patients account for considerable costs to the health system. Such services include physiotherapy, specialist assessment and radiology. In New Zealand, the Accident Compensation Corporation (ACC), a state insurer which statutorily provides national insurance coverage for all accident-related health services, uses an indicator approach to provide feedback to GPs on their management of back pain. The ACC sends information to GPs about their individual level of referral to physiotherapists, specialists and radiology providers in the management of back pain, comparing a GP's level of referral with the national average. In 1998, the year for which data were collected, GPs were statutory gatekeepers for ACC-funded physiotherapy services, although this has since changed with the introduction of new legislation. Anecdotally, GPs in New Zealand have a higher level of referral to physiotherapy for all conditions than GPs in the UK, possibly as a consequence of the open fee-for-service funding which is available for physiotherapy services from the ACC. UK GPs have been found to refer 12% of back pain events to physiotherapy,19 whereas this study found a 48% average referral rate among New Zealand GPs, a figure consistent with ACC's own analysis. Since ACC is a statutory national insurer for accident-related health care, the data which it collects are comprehensive for the whole of New Zealand.20
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
The method we used involves simulating the binomial distribution of an indicator across GPs, using the observed actual level of referral and the real levels of caseload observed in the population of GPs. The underlying null hypothesis is therefore that there is no variation in medical practice. The simulated distribution of GP referral rates is attributable entirely to chance. This shows how much variation is to be expected in the indicator if all GPs practised consistently.
Information was extracted from the databases of the ACC. The data covered all New Zealand claims for treatment of accident-related back pain which were initiated during the 1998 calendar year (n = 142 215). A claim represents payment for all the care associated with a single accident event. The information for each claim included an individual identifier for the GP who managed the claim, and indications of whether payment for physiotherapy treatment had ever been made under the claim. Since New Zealand GPs had a gatekeeping role for ACC-funded physiotherapy services in 1998, such payments may be interpreted as a GP decision to refer for physiotherapy. Claims which involved more than one GP (n = 13 129; 9.2%) were discarded, since they cannot be interpreted as the referral decision of an individual practitioner. A further seven claims with incomplete data were discarded, leaving 129 079 claims managed by 2679 GPs. The data were used to calculate percentage referral rates for back pain cases to physiotherapy for every GP in New Zealand.
Using a binomial random number generator in the Python programming language, a binomial distribution for referral was generated for every GP, using the number of back pain cases which the GP had actually managed as the number of trials, and the national proportion of claims referred to physiotherapy in question as the underlying probability of referral. From the binomial distribution for each GP, a simulated referral rate was calculated. National age/sex-specific probabilities of referral were used, with age in 10 year bands, to reflect the different likelihood of referral in those demographic groups. In order to ensure that the simulation produced stable results, it was repeated 100 times, and the standard deviation monitored. The mean of 100 simulations was compared with the distribution of observed GP referral rates.
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
Figure 1 shows the distribution of GP caseload in the data set. A total of 304 GPs (11.3%) saw five or fewer cases in a single year. Five hundred GPs (18.7%) saw less than one case per month. Many GPs see quite small caseloads of this fairly common condition, although there is a very long tail of GPs with high caseloads. Some of these extremely high caseloads may represent GP accident clinics which are inappropriately using a single clinician code for all their ACC-funded services.
|
Table 1 shows the proportion of claims referred to physiotherapy in total, and the mean and median referral rates among GPs. New Zealand GPs refer nearly half of all accident-related back pain cases to physiotherapy.
|
Figure 2 shows the number of GPs observed at different levels of physiotherapy referral rate, and the results of 100 simulations of GP referral. These data show that the simulation predicts a generally narrower distribution of practice in physiotherapy referral than is observed among New Zealand GPs. However, the simulation predicts an even greater number of extremely high referring GPs than is observed. The number of GPs referring >95% of cases to physiotherapy is the result which shows the closest agreement between real and simulated numbers of GPs, and is the only result in which the real number of GPs is within 2 SDs of the simulated number.
|
Figure 3 breaks down the simulation into two groups of GPs, those referring <1 case per month during 1998, and the remainder who refer
1 cases per month. The low caseload GPs account for all of the high and low referrers in the simulation result.
|
Figure 4 shows a comparison between an age/ sex-adjusted simulation for GP referral to specialist assessment, generated from the same ACC back pain data set as the physiotherapy simulation, and the observed national distribution of GP referral rates. While the overall shape of the distribution differs from referral to physiotherapy, as a consequence of the greater rarity of referral to specialists, the simulation once again predicts that there will be a small number of higher referrers as a consequence of small caseload.
|
A mixed model was used to estimate the variance of GP referral to physiotherapy within each of the 21 District Health Boards in New Zealand. Table 2 shows the GP variance, the number of GPs and the absolute level of referral to physiotherapy in each of the District Health Boards. The correlation coefficient between the variance of GPs and the number of GPs in each District Health Board is 0.43 (not significantly different from zero at the 5% level). The observed correlation coefficient between the variance of GPs and the absolute level of referral to physiotherapy is zero, to three decimal places.
|
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
Our results show that the number of extremely high referrers to physiotherapy for accident-related back pain can be accounted for entirely by chance, when the underlying level of referral and the caseload of individual GPs are taken into account. Indeed, GPs with low caseloads account for all of the expected high and low referrers to physiotherapy. Our results also show that the variability of GP referral to physiotherapy within a geographic area is not associated with the absolute level of referral, or with the number of GPs in each area.
This study is based upon a positive definition of medical practice variation, encapsulated both by comparison with the null hypothesis distribution of GP referral rates expected by chance if there is no variation in referral behaviour among GPs, and by the direct measurement of the variance using a mixed model with fixed and random effects. This avoids the weakness of a negative definition of medical practice variation, which attributes all residual variation to the behaviour of clinicians. However, this analysis also has limitations—the variances were measured on the logit scale, which means that they cannot be compared directly with other measurements of variance.
These results raise some new issues for debate about performance indicators for primary care. Since low caseloads for any one condition are the norm for a primary care setting, our findings suggest that indicators must be interpreted cautiously when they are applied to individual clinicians. An extreme value on an indicator does not necessarily imply that a clinician is practising differently from the norm. The distribution expected by chance is a tool which can help to decide whether indicators should be interpreted as requiring a change in clinical practice. In the case of referral to physiotherapy, practitioners who diverge from the underlying rate of referral are appearing in the centre and shoulders of the distribution, rather than at extremely high values.
Our second finding, that the variance of GPs within an area is not associated with the absolute level of referral in the whole area, also has implications for the use of indicators. If part of the motivation for using indicators is to reduce area-level variations, this result suggests that the variability per se of clinicians is not a cause of area-level variations. Rather, area-level variation is caused by the overall distribution within the population of clinicians in each area. These results suggest caution in applying a simple interpretation of indicators to the practice of individual clinicians, and in trying to modify area-level variation by concentrating upon individual clinicians with extreme values on indicators. These results also suggest that, to shift overall utilization rates, the whole population of clinicians must be considered, rather than just the tails of the distribution.
While performance indicators are an important tool for quality improvement, it is important to recognize their limitations and to make sure that their use is valid if they are to have any credibility with the public and with professionals. Our results suggest some of these limitations in the specific area of physiotherapy referral, but there is considerable scope to extend these analyses to areas such as prescribing and hospital referral which are the main focus of work on performance indicators in primary care.
|
Acknowledgments |
|---|
The data for this study were provided by the New Zealand Accident Compensation Corporation. We would like to acknowledge useful comments from anonymous referees.
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
1 Pescosolido BA, Tuch SA, Martin JK. The profession of medicine and the public: examining American' changing confidence in physician authority from the beginning of the ‘health care crisis’ to the era of health care reform. J Health Soc Behav 2001; 42: 1–16.[CrossRef][Web of Science][Medline]
2 Davies HT, Shields AV. Public trust and accountability for clinical performance: lessons from the national press reportage of the Bristol hearing. J Eval Clin Pract 1999; 5: 335–342.[CrossRef][Web of Science][Medline]
3 Campbell SM, Roland MO, Buetow SA. Defining quality of care. Soc Sci Med 2000; 51: 1611–1625.[CrossRef][Web of Science][Medline]
4 Majeed FA, Voss S. Performance indicators for general practice. Br Med J 1995; 311: 209–210.
5 Braybrook S, Walker R. Influencing prescribing in primary care: a comparison of two different prescribing feedback methods. J Clin Pharm Ther 1996; 21: 247–254.[Medline]
6 Shelley M, Croft P, Chapman S, Pantin C. Is the ratio of inhaled corticosteroid to bronchodilator a good indicator of the quality of asthma prescribing? Cross sectional study linking prescribing data to data on admissions. Br Med J 1996; 313: 1124–1126.
7 Campbell SM, Cantrill JA, Roberts D. Prescribing indicators for UK general practice: Delphi consultation study. Br Med J 2000; 321: 425–428.
8 Jensen J. Consumers see mortality data as useful tool. Mod Health 1992; 22: 82.
9 Marshall MN, Hiscock J, Sibbald B. Attitudes to the public release of comparative information on the quality of general practice care: qualitative study. Br Med J 2002; 325: 1278–82.
10 McColl A, Roderick P, Gabbay J, Smith H, Moore M. Performance indicators for primary care groups: an evidence based approach. Br Med J 1998; 317: 1354–1360.
11 Giuffrida A, Gravelle H, Roland M. Measuring quality of care with routine data: avoiding confusion between performance indicators and health outcomes. Br Med J 1999; 319: 94–98.
12 Royal College of General Practitioners. The Nature of General Medical Practice. London: Royal College of General Practitioners; 1996.
13 Starfield BH. Primary Care: Balancing Health Needs, Services, and Technology. New York: Oxford University Press; 1998.
14 Moore AT, Roland MO. How much variation in referral rates among general practitioners is due to chance? Br Med J 1989; 298: 500–502.
15 Diehr P, Grembowski D. A small area simulation approach to determining excess variation in dental procedure rates. Am J Public Health 1990; 80: 1343–1348.
16 Diehr P, Cain K, Connell F, Volinn E. What is too much variation? The null hypothesis in small-area analysis. Health Serv Res 1990; 24: 741–771.[Web of Science][Medline]
17 Bachmann MO, Bevan G. Determining the size of a total purchasing site to manage the financial risks of rare costly referrals: computer simulation model. Br Med J 1996; 313: 1054–1057.
18 Clinical Standards Advisory Group. Epidemiology Review: The Epidemiology and Cost of Back Pain. London: HMSO; 1994.
19 Mason V. The Prevalence of Back Pain in Great Britain. London: HMSO; 1994.
20 Accident Compensation Corporation. Annual Report 2002. Wellington: Accident Compensation Corporation; 2002.
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||