Family Practice Advance Access originally published online on April 11, 2005
Family Practice 2005 22(3):323-327; doi:10.1093/fampra/cmi027
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
A comparison of two methods of collecting economic data in primary care
a Health Services Research Department, Institute of Psychiatry, London and b PSSRU, LSE Health and Social Care, London School of Economics, London, UK
Correspondence to Anita Patel, Box PO24, Health Services Research Department, David Goldberg Centre, Institute of Psychiatry, De Crespigny Park, London SE5 8AF, UK; Email: a.patel{at}iop.kcl.ac.uk
Received 20 April 2004; Accepted 5 January 2005.
Patel A, Rendu A, Moran P, Leese M, Mann A and Knapp M. A comparison of two methods of collecting economic data in primary care. Family Practice 2005; 22: 323–327.
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionDeclarationReferences |
|---|
Background. There have been few attempts to assess alternative methods of collecting resource use data for economic evaluations.
Objective. This study aimed to compare two methods of collecting resource use data in primary care: GPs' case records and a self-complete postal questionnaire.
Methods. 303 primary care attenders were sent a postal survey, incorporating a questionnaire designed to collect service utilisation information for the previous six months. Data were also collected from GP case records. The reporting of GP visits between the two methods, and estimates of costs associated with those visits, were compared.
Results. There was good agreement between the number of GP visits recorded on GP case records (mean 3.03) and on the CSRI (mean 2.99) (concordance correlation coefficient = 0.756). In contrast, estimates of average costs of visits from CSRI data were higher and had greater variance compared to case record-based costs (£54.63 versus £42.37; P = 0.003). This may be explained by differences in average visit length (11.66 versus 9.36 minutes).
Conclusions. This study shows good agreement between GP case records and a self-complete questionnaire for the reporting of GP visits. However, differences in costs associated with those visits arose due to differences in the method used for calculating length of visit.
Keywords. Costs, primary care, service utilisation.
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionDeclarationReferences |
|---|
The need for a strong evidence base in primary care has previously been highlighted.1,2 Given finite resources, policy makers and clinicians require reliable information about the cost-effectiveness of health care provision. This is particularly so in the setting of primary care where 90% of contacts between the population and the National Health Service take place.1 Therefore, many evaluations of new interventions in primary care now incorporate an economic perspective. Although economic evaluations generally aim to compare the outcomes and costs of two or more interventions, different methodological approaches can be taken and these in turn can affect the robustness of study findings.
Two types of data are needed in order to calculate the costs of care. Firstly, the amount of physical resources used to provide that care, and secondly, the cost of one unit (‘unit cost’) of each resource. Methods of calculating unit costs have received much attention and the difficulties of estimating the costs of primary care in the absence of gold standard cost estimates have been highlighted.3,4 The analysis of cost data can also be problematic; common problems include insufficient sample size, missing data and skewed data. Consequently, there has been much debate over the best way to analyse such data (see Thompson and Barber5 and Drummond and McGuire6 for a detailed discussion of these issues). However, issues concerning the collection of resource use data have been relatively neglected.
Research resources are limited, encouraging researchers to seek efficient and pragmatic approaches to data collection. In a review of six approaches to collecting service use and cost data for economic evaluations, Evans and Crawford7 suggested that trial expense and effort were positively associated with validity. Data on participants' use of health and social care resources are commonly collected retrospectively, using self-report interviews and questionnaires. This is particularly the case for studies that involve people who use a broad range of services over long periods of time, as it would be too onerous to collect data prospectively, or to examine case records held by multiple agencies in a variety of formats.
Collecting data from participants or examining their case records requires their informed consent.8 Our local experience is that in order to recruit primary care patients to a research project, it is also necessary for the GP to first write to each patient to determine their willingness to be recruited into a study. These factors place necessary, but considerable, constraints on the ease with which patient information can be used for research. In the light of these issues, it is important to compare different data collection methods, in order to ensure that research resources are used cost-effectively.
Different methods for collecting economic data have their strengths and weaknesses, and to date, studies have not produced consistent findings. This is likely to reflect differences in study design (retrospective versus prospective; postal survey versus patient interview) and the methods used to collect economic data (self-report versus professional-report versus proxy-report). Indeed it has been argued that some economic evaluations would require several validation studies.9
This paper reports a comparison of two methods for primary care economic data collection: GPs' case records and a variant of the Client Service Receipt Inventory (CSRI)10 adapted to be used as a self-complete postal questionnaire.
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionDeclarationReferences |
|---|
The examination of service use data collection methods was part of a larger study that set out to compare the costs of primary care attenders with and without a diagnosis of personality disorder. A simple random sample of 303 consecutive primary care attenders was recruited from four general practices in the London area. The personalities of the cohort were rated at recruitment and they were then followed up one year later. Full details of the design and findings from the main study have been reported elsewhere.11,12
One year after recruitment, all participants received a postal survey formed of four questionnaires related to general health, life events, quality of life, use of services and other economic impacts. Two weeks after the initial mailing, non-responders were sent a reminder with a second copy of the full survey. After a further two weeks, non-responders were telephoned and data were obtained during a telephone interview. The survey included the Client Service Receipt Inventory (CSRI),10 a questionnaire for collecting information about use of health and social care services, other economic impacts (such as time off work due to illness) and socio-demographic information. It has now been used in over 150 health and social care economic evaluations, each time being tailored to suit the data requirements and broad approach to data collection for individual studies. The variant used in this study was designed to collect retrospective data on service utilisation for the previous six months. It covered the following domains: GP consultations, practice nurse visits, inpatient stays, outpatient episodes, social worker contacts, counselling contacts and therapy contacts. Participants were also asked about the duration of these contacts and any costs incurred for private treatment.
Two researchers (AR and PM) also collected further resource use data by examining GP case records for all participants, blind to participants' responses on the CSRI. A pro forma was devised specifically for this exercise in order to standardise the extraction of information. Data were collected on the number of GP consultations in the last three years. Inevitably, there were variations in the type of resource use items that could be collected using these two approaches. Therefore, the comparison in this present study is limited to the reporting of GP visits. Because the CSRI data concerned only a 6-month period, only case record data for the equivalent 6-month period were used for comparison and included in these analyses.
Statistical analysis
Analyses were performed using SPSS13 and STATA.14 Participants' characteristics were compared using one-way analysis of variance and Chi-square tests. Relative bias in CSRI and case record data in the mean number of visits and the mean cost of visits was assessed using paired sample t-tests. Overall agreement was measured using Lin's concordance correlation coefficient,15 whereby 1 indicates perfect agreement and –1 indicates perfect inverse agreement. This measure incorporates both relative bias and imprecision due to random variation. Mirandola et al.16 have previously discussed the appropriateness of this approach for their comparisons of cost data obtained from two different sources of service use data. 95% limits of agreement are also presented to indicate the range of differences to be expected for any individual person's observation. Analyses were limited to participants who had follow-up data from both case records and the CSRI.
Estimating costs
Unit costs were attached to reported GP visits using figures published in Netten et al.17 which allowed for all relevant costs associated with service provision—capital, overheads, travelling time, non-patient contact time, support services and London weighting. Case record data did not provide length of visit, so it was necessary to estimate per-visit costs based on an assumed average visit length of 9.36 minutes, as suggested by Netten et al.17 For the CSRI-based calculations, data were available on participants' own estimates of the average length of their visits, thus allowing the cost of CSRI-recorded visits to reflect differing visit durations between participants. Costs were calculated in pounds sterling at 1998/99 price levels.
Adjustments made to the data
There were some missing data concerning the number and length of visits reported in the CSRI. One participant did not report the length of their visits and was allocated the median length of contact for those who did have data. Ten participants gave vague responses regarding their number of visits (for example, ‘lots’) and were allocated the median number of consultations for those who had data.
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionDeclarationReferences |
|---|
Of the 303 participants at baseline, 229 (76%) had both CSRI and case record data at follow-up, 45 (15%) had just case record data and 29 (10%) were completely lost to follow-up having data from neither source. Those with only case record data were likely to be younger in age compared to those with both case record and CSRI data (Scheffe post-hoc test P < 0.001), and more likely to come from general practice 2 (Table 1).
|
Number and cost of GP visits
There was no significant difference in the average number of GP visits recorded on GP case records (mean = 3.03; sd = 3.49) and on the CSRI (mean = 2.99; sd = 3.67), and the agreement between the two sources was high (concordance correlation coefficient 0.756) (Table 2).
|
In contrast, average costs associated with the visits were higher when calculated on the basis of CSRI data (£54.63 versus £42.37; P = 0.003) and there was only fair overall agreement in costs between the two methods (concordance correlation coefficient 0.609). There was a higher average visit length of 11.66 minutes (standard deviation = 6.65) from the CSRI data compared with an average length of 9.36 minutes assumed for all visits reported in case records, taken from Netten et al.17 For participants who reported GP visits on the CSRI, the estimated length of visits ranged from 2 to 60 minutes (median = 10.0; interquartile range = 6.5). This variation was reflected in the standard deviation around the mean costs, which was almost twice as high in CSRI-based costs (£86) compared with case record-based costs (£49).
For the 45 subjects who did not respond to the CSRI but who had case record data, the mean number of visits recorded was 2.71 and the associated mean cost was £37.96; these figures were not significantly different to case record-based data for CSRI responders.
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionDeclarationReferences |
|---|
In this study, there was good agreement between GP case records and a self-complete questionnaire for the reporting of number of GP visits. However, costs associated with those visits differed, due to differences in the method used for calculating length of visit. Although this suggests that a self-report method allows for broader data collection than that permitted using case records, the accuracy of the participants' estimates of time is unclear. It is possible participants may have systematically over-reported the length of consultations, or that specific features of the general practices included in the study could have affected average consultation times. On the other hand, generalising national average consultation lengths to specific patient groups may be problematic, as these averages may not be generalisable to all practices. We are unable to offer a definitive judgement about whether it is better to rely on patients' self report or on national figures for the estimation of length of consultations. However, from a research perspective, there are distinct advantages of using a self-report method. Economic studies are likely to be of greater use to policy makers if they obtain the broadest possible perspective on costs; it is easier to assess a broad range of economic outcomes, such as lost productivity and out-of-pocket expenses by patients, using self-report data. In addition, case records may sometimes be misplaced and the completeness with which GP visits are recorded in records has been shown to significantly differ between practices, doctors, and patients.18 It is possible that a trade-off may need to be made between breadth of information and sample size; case record data were available for 90% of participants in this study, compared to only 76% with CSRI data. However, Kennedy et al.19 found the reverse in their study of 315 people referred for physiotherapy; 48% had data from a self-complete postal questionnaire while hospital data were available for only 30%. The participants in our study received a postal survey that included more than one questionnaire and it may be possible that response to the economic questionnaire may have been affected by the length or nature of the overall survey booklet.
This study was limited to the comparison of GP visits, although the participants in the study made use of a range of different services. Careful consideration should be given to collecting data on items of service use that are likely to form the largest components of total costs (often termed the 'cost drivers'). For example, an analysis of data from five mental health service evaluations found that the costs of the five most expensive measured services within each study contributed to between 90% and 98% of total care costs.20 At that time, GP costs contributed very little to total costs and were not among the top ten most expensive services. Kennedy et al.19 compared self-report postal questionnaires and hospital records for the reporting of physiotherapy visits. Although significantly more visits were recorded on the questionnaire, they argued that this difference may not have been of economic significance unless physiotherapy costs were dominant in the evaluation. Mirandola et al.16 compared total costs derived from two methods of collecting service use data for a range of psychiatric services, a self-report schedule and a psychiatric case register. Although there was poor agreement about individual service components, there was good agreement between the two methods in terms of total service costs. In this present study, the average cost of GP visits formed 15% of average total health care costs, compared to 55% formed by secondary care use.21 Given this small contribution to total care costs, combined with the possibility that GP case records may not contain information about use of other more expensive health care services, patient questionnaires such as the CSRI, that take a broader health care perspective, may be more appropriate for economic evaluations.
Although there have been considerable methodological developments in the calculation of unit costs and the analysis of cost data, more fundamental issues concerning the validity of the data upon which such analyses are based have been neglected. This study suggests that difficulties associated with examining individual case records in primary care-based economic evaluations could be avoided through the use of patient questionnaires, without detriment to data quality. Further research is needed to assess different data collection methods, including their acceptability to service users.
|
Declaration |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionDeclarationReferences |
|---|
Funding: the study was supported by grants from the Department of Health and the Medical Research Council.
Ethical approval: ethical committee approval was obtained from the South London and Maudsley NHS Trust LREC.
Conflicts of interest: none.
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionDeclarationReferences |
|---|
1 Mant D. R&D in primary care—an NHS priority. Br J Gen Pract 1998; 48: 871.[Web of Science][Medline]
2 Radda G. Primary care research: the MRC's proposals. Br J Gen Pract 1998; 48: 872.[Web of Science][Medline]
3 Kernick D. Costing interventions in primary care. Fam Pract 2000; 17: 66–70.
4 Kernick DP, Netten A. A methodological framework to derive the cost of the GP consultation. Fam Pract 2002; 19: 500–503.
5 Thompson SG, Barbar JA. How should cost data in pragmatic randomised trials be analysed? Br Med J 2000; 320: 1197–1200.
6 Drummond M, McGuire A (eds). Economic evaluation in health care: merging theory with practice. Oxford: Oxford University Press; 2001.
7 Evans C, Crawford B. Direct medical costing for economic evaluations: Methodologies and impact on study validity. Drug Inf J 2000; 34: 173–184.
8 Medical Research Council (2003). Personal information in medical research. Guidelines available at: www.mrc.ac.uk
9 Evans C, Crawford B. Data collection methods in prospective economic evaluations: how accurate are the results? Value Health 2000; 3: 277–286.[CrossRef][Medline]
10 Beecham J, Knapp M. Costing psychiatric interventions. In Thornicroft G (ed). Measuring Mental Health Needs. 2nd edition, London: Gaskell; 2001.
11 Moran P, Rendu A, Jenkins R, Tylee A, Mann A. The impact of personality disorder in UK primary care: a 1-year follow-up of attenders. Psychol Med 2001; 31: 1447–1454.[Web of Science][Medline]
12 Rendu A, Moran P, Patel A, Knapp M, Mann A. Economic impact of personality disorders in UK primary care attenders. Br J Psychiatry 2002; 181: 62–66.
13 Norusis M. SPSS for windows. Base system user's guide. Release 6.0. Chicago: SPSS; 1993.
14 Stata Corporation. Intercooled Stata 8.1 for Windows. College Station TX 77845 USA: Stata Corporation; 2003.
15 Lin LI-K. A concordance correlation coefficient to evaluate reproducibility. Biometrics 1989; 45: 255–268.[CrossRef][Web of Science][Medline]
16 Mirandola M, Bisoffi G, Bonizzato P, Amaddeo F. Collecting psychiatric resources utilisation data to calculate costs of care: a comparison between a service receipt interview and a case register. Soc Psychiatry Psychiatr Epidemiol 1999; 34: 541–547.[CrossRef][Web of Science][Medline]
17 Netten A, Dennett J, Knight J. Unit costs of health and social care. Canterbury: PSSRU, University of Kent; 1999.
18 Neal RD, Heywood PL, Morley S. Real world data—retrieval and validation of consultation data from four general practices. Fam Pract 1996; 13: 455–461.
19 Kennedy ADM, Leigh-Brown AP, Torgerson DJ, Campbell J, Grant A. Resource use data by patient report or hospital records: do they agree? BMC Health Serv Res 2002; 2: 2.[CrossRef][Medline]
20 Knapp M, Beecham J. Reduced-list costings: examination of an informed short cut in mental health research. Health Econ 1993; 2: 313–322.[Medline]
21 Rendu A. The cost impact of personality disorders in primary care. MPhil thesis, London: University of London; 2002.
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||