Family Practice Advance Access originally published online on June 20, 2006
Family Practice 2006 23(5):537-544; doi:10.1093/fampra/cml026
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Cluster randomized trial of a multifaceted primary care decision-support intervention for inherited breast cancer risk
a Department of Public Health, University of Aberdeen Aberdeen, UK
b Woodside Medical Practice Aberdeeen, UK
c Department of Medicine & Therapeutics, University of Aberdeen Aberdeen, UK
d Health Services Research Unit, University of Aberdeen Aberdeen, UK
e Health Economics Research Unit, University of Aberdeen Aberdeen, UK
Correspondence to Dr Brenda J Wilson, Department of Epidemiology & Community Medicine, University of Ottawa, 451 Smyth Road, Ottawa, Ontario K1H 8M5, Canada; Email: brenda.wilson{at}uottawa.ca
Received 14 June 2005; Accepted 23 May 2006.
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionDeclarationReferences |
|---|
Background. GPs are increasingly expected to meet the needs of patients concerned about their risk of inherited breast cancer, but may lack skills or confidence to use complex management guidelines. We developed an evidence-based, multifaceted intervention intended to promote confidence and skills in this area.
Objective. To evaluate the effectiveness of the intervention in improving GP confidence in managing patients concerned about genetic risk of breast cancer.
Methods.
Design. Cluster randomized controlled trial.
Setting. General practices in the Grampian region of Scotland.
Subjects. GPs and the patients they referred for genetic counselling for risk of breast cancer.
Main outcome measures. GPs’ self-reported confidence in four activities related to genetics; rates of referral of patients at elevated genetic risk; and referred patients’ understanding of cancer risk factors.
Results. No statistically significant differences were observed between intervention and control arms in the primary or secondary outcomes. A possible effect of the intervention on the proportion of referred patients who were at elevated risk could not be discounted. Only a small proportion of intervention GPs attended the educational session, were aware or the software, or made use of it in practice.
Conclusions. No convincing evidence of the effectiveness of the intervention was found, probably reflecting barriers to its use in routine practice.
Keywords. Breast neoplasms, computer-assisted decision making, family physicians, randomized controlled trials.
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionDeclarationReferences |
|---|
Emerging genetics knowledge may transform the understanding of disease and health, particularly as its relevance to the aetiology of common complex diseases becomes more apparent. It is inevitable that patients will look to their GPs for credible information about genetics and for specific advice and counselling.1–3 Whilst it is unrealistic to expect GPs to become experts in genetics, particularly in such a rapidly changing field, there are some related activities which fall well within the domain of primary care. These include taking an adequate family history as the basis for further assessment and referral decisions, providing information, reassurance and support to patients at different risk levels, and identifying appropriate general risk reduction strategies. However, there are significant barriers to achieving this role, including inadequate knowledge or confidence in genetics at even a basic level, and ethical, legal and practical issues.4 GPs find that the goal of meeting the needs of genuinely worried patients is difficult when specialist services may seek to limit the number of ‘inappropriate’ referrals. GPs currently in practice need effective interventions to help them meet patient needs in appropriate and efficient ways.
An example of such a situation is breast cancer genetics, where overall patient concern is exacerbated by the common nature of the disease. Many women have a sister, mother or cousin affected by breast cancer, but only 
5–10% are likely to be members of a family carrying a BRCA1 or BRCA2 mutation. A number of guidelines for risk assessment and management have been developed but may be complex to interpret and unsuited to occasional use in a busy practice setting.5 While this complexity suggests that IT-based approaches might be helpful, we suggest that any intervention needs to go beyond simply making complicated information more comprehensible: its more important goal should be to ‘situate’ genetics knowledge within the ‘practices of patient management [in which] GPs are the experts’6—explaining, counselling and reassuring. An intervention ultimately designed for widespread dissemination and use also needs to be cost-effective if it is to be sustainable beyond an experimental study.
Bearing these issues in mind, we designed and evaluated a multifaceted system which (i) built on a national IT initiative to assist GPs in interpreting and applying the new national guidelines for cancer genetics,6 including an explanation of the rationale underlying the risk stratification; (ii) contained elements to facilitate its use in general practice (e.g. information leaflets, contact email link); and (iii) complementary educational sessions which aimed to provide insight and skills training relating to breast cancer genetics.
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionDeclarationReferences |
|---|
Intervention and control
The decision aid component of the intervention (details in Box 1) was developed over a number of months with the involvement of GPs active in practice together with clinical geneticists. The risk assessment module gave clear instructions on the information required from a patient, and assisted users in making a rapid decision about whether or not a patient met Scottish referral guidelines. A second key element was the email query service, which had a guaranteed rapid response time.
| BOX 1 Components of the intervention software A list of the key patient information needed in order to use the guidelines A risk assessment module, presented as a set of short checklists, in which the Scottish referral guidelines for breast, ovarian and colorectal cancer (7) were embedded Background information on cancer genetics, and explanation of the evidence underlying the guidelines, prepared by local geneticists Printer-ready, locally customized patient information leaflets Selected web links for professionals and patients A contact email link with the Cancer Genetics Service, with a guaranteed response time Automated production of a draft referral letter using the regionally recommended template (8)
|
The intervention was disseminated by mailing it to all intervention practices; installing it opportunistically with routine practice IT upgrades; special visit by a research associate where requested; and distribution at the educational sessions. Every intervention GP was alerted to its existence through a personal letter, and all partners in intervention practices were invited to interactive workshops on cancer genetics designed to complement the software. This workshop carried postgraduate training points, was designed and delivered by specialist genetics staff, and was repeated on different dates in two locations. While intervention GPs were encouraged to use the software for assessment of patients, it was emphasized that all referrals would be accepted regardless of its use or not. Control practices received a baseline intervention only, i.e. the Scottish referral guidelines which were mailed to all GPs by the Scottish Office Department of Health.7 The intervention was implemented over the period 1 November 2000 to 30 June 2001 and evaluated in a cluster randomized controlled trial, in which the level of allocation was the general practice and level of analysis was GPs and their referred patients.
Setting and participants
The study was based in the Grampian Health Board area in the North East of Scotland. All practices were eligible for inclusion, except two which included GPs involved in the design and management of the study. Patient level data were included for all women referred for breast cancer genetics counselling by a participating practice in the defined pre-intervention or post-intervention periods.
Allocation
Eligible practices were stratified by number of referrals to the cancer genetics service (CGS) in the previous year (as a crude proxy for the previous level of experience with patients concerned about breast cancer genetic risk) and by urban or rural location, and randomized in a 2:1 intervention:control ratio. This uneven allocation was chosen to give greater experience of the new intervention and was taken into account in the sample size calculation. Different practices occupying the same premises were allocated to the same arm of the trial to avoid contamination. The randomization was carried out independently by the project statistician (JM) using computer-generated random numbers.
Outcomes and data collection
The primary outcome for the trial was self-reported GP confidence in activities related to managing patients concerned about genetic risk of breast cancer. Secondary outcomes were (i) awareness and use of the intervention in the intervention group; (ii) the proportion of referred patients at higher than population (‘elevated’) risk of breast cancer and (iii) referred patients' accuracy of perception of breast cancer risk factors.
We hypothesized that, compared with control GPs, intervention GPs would report higher levels of confidence in relevant activities, and that this would be reflected in more realistic appreciation of breast cancer risk factors in their patients. Since there are legitimate reasons beyond suspecting elevated genetic risk for making a referral (e.g. patient anxiety), we did not formulate a formal hypothesis regarding this outcome.
GP
A self-completion questionnaire was used to assess GP confidence (four-point scale, response choices ‘not at all/a little/moderately/very confident’) for the following activities related to patients presenting with concerns about breast cancer genetic risk: taking a family history; knowing which patients to refer; reassuring low-risk patients; and being able to answer patients' questions. The questionnaire was developed and piloted in two practices. It was mailed to all GPs pre- and 1 year post-intervention. For intervention GPs only, we collected additional post-intervention data on awareness, utility, acceptability and ease of use of the intervention software, and we identified which GPs and practices had participated in the educational workshop.
Referrals
Referral data, and classification of patients' assessed genetic risk, were obtained from the genetic clinic database, which was independent of the project, for the periods 1 January to 31 October 2000 (pre-intervention), and 1 July 2001 to 30 June 2002 (post-intervention). The database manager was blinded to intervention or control status of practices.
Patients
We used a self-completion questionnaire to assess referred patients' beliefs about breast cancer risk factors. Data were available for the periods 31 May 1998 to 31 October 2000 (pre-intervention) and 1 July 2001 to 31 May 2002 (post-intervention).
Statistical analysis
Analysis was by intention-to-treat, using Pearson's chi-square test for categorical variables. Because the units of allocation and analysis were different, intra-cluster correlation coefficients (ICC) were estimated using the analysis of variance approach8–10 for both continuous and count data, and P-values adjusted using the group-specific adjusted chi-square.10,11 Response categories to five-point scales were dichotomized. Risk ratios were calculated to compare proportions of referred patients who were subsequently assessed as elevated (moderate or high) genetic risk in control (reference) and intervention groups. Where the observed ICC was greater than zero, the P-values and 95% confidence intervals were adjusted for clustering. All P-values are from two-sided tests.
Sample size
The total possible sample size was constrained by the number of eligible practices in Grampian (N = 86). In order to maximize power, we decided to randomize all eligible practices. We estimated the ICC to be 0.05 from the pre-intervention survey. We calculated that we required 252 GPs (168 intervention, 84 control), to have 80% power of detecting an absolute difference of 20% in GPs' confidence in each of the four survey items (e.g. a shift from 40 to 60% ‘moderately/very confident’).12,13
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionDeclarationReferences |
|---|
Study participants
Figures 1 and 2 summarize the derivation of the data for the pre- and post-intervention periods. All eligible practices (n = 86) were randomized, 57 (230 GPs) to the intervention group and 29 (116 GPs) to the control group. Practice and GP characteristics are shown in Table 1. The total number of intervention and control GPs changed over the course of the trial because of natural turnover. Pre-intervention survey questionnaires were returned by 179/230 (78%) intervention GPs and 93/116 (81%) control GPs. Post-intervention survey response rates were 151/241 (63%) for intervention GPs and 92/119 (77%) for control GPs. Twenty seven (11.7%) GPs in intervention practices attended one of the three continuing medical educational sessions, representing 20 (35%) of the eligible GP practices.
|
|
|
Outcomes
GP. Table 2 summarizes the outcomes relating to measures of GP confidence. In the pre-intervention period, control and intervention GPs had generally similar patterns of responses to the four questions, reporting greatest confidence in taking an appropriate family history and least confidence in answering patients' questions about familial cancer risk. Little difference in these measures was observed post-intervention between the intervention and control groups. A total of 64/151 (42.4%) respondents in the intervention group were aware that their practice had the software and 22 (34.4% of those aware) had used it at least once (range 1–6 times per year, median 2.6). Of the different components, all 22 respondents reported using the referral guidelines, 7 the patient information leaflets, 7 the background information and 1 the email query service. When the primary outcome was examined for the latter group of respondents (those who reported use of the software), statistically significantly higher self-reported confidence was noted for the activity of ‘reassuring low-risk patients’ compared with the 127 intervention group respondents who did not use the software (‘moderately’ or ‘very’ confident, 20/22 versus 63/127, P < 0.001). No statistically significant differences were observed for the other three activities (data not shown).
|
Referrals. Data were available on 122/140 (87%) eligible patients (88 intervention, 34 control) referred by study practices in the pre-intervention period and 114/145 (79%) (85 intervention, 29 control) in the post-intervention period (Table 3). Data were missing for women who were referred but had not completed their clinical episode by the time of data collection. Post-intervention, no statistically significant between-group difference was observed in the proportions of referred patients who were subsequently assessed as high risk. In the pre-intervention period, intervention GPs were less likely than control GPs to refer patients who were eventually assessed as having elevated genetic risk, with the opposite trend observed in the post-intervention period, although these results did not reach statistical significance (Table 3). The pre-intervention difference was of borderline statistical significance. We did not adjust for clustering because the observed ICC was negative and thus truncated to zero.
|
Patients. Survey response rates were 185/209 (88%) (133/154 intervention, 52/55 control) for eligible patients referred in the pre-intervention period, and 97/117 (83%) (75/88 intervention, 22/29 control) for the post-intervention period. Table 4 summarizes beliefs about breast cancer risk factors, where no clear cut effect of the intervention was evident. As there were no statistically significant differences in any of the patient-level outcomes, we did not adjust the analysis for clustering of these data by GP and practice.
|
A cost analysis performed alongside the study indicated that the total average cost for the software development (2001 prices) was £71.69 per CD, with a marginal cost for each additional CD of £3.12.14 The cost for each GP attending the postgraduate education session was £106.07 per GP (marginal cost = £77.60).
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionDeclarationReferences |
|---|
The design, implementation and evaluation of this multifaceted intervention was deliberately pragmatic and approached from a primary care perspective. The intervention's main goal was to provide meaningful assistance to GPs who face a broad set of challenges when managing a patient concerned about her genetic risk of breast cancer6,15,16 (going beyond the need to make an ‘appropriate’ referral decision); the implementation strategy was designed to enhance its effectiveness in meeting this goal and reflected our belief that a software ‘kit’ would be inadequate in practice; and the evaluation was designed to measure the realistic achievable effects when the intervention was offered to unselected GPs for use in routine practice.
We were aware that computer-based applications could be efficacious in genetic risk assessment tasks,14 but evidence about success in routine practice was (and still is) very limited. We sought to enhance the application's potential effectiveness17 by involving end-users in its development, ensuring local identity and relevance, requiring only a small investment of learning time, and adding utilities such as the contact email, patient information leaflets, etc. However, it was not possible to integrate it with other practice systems, and it was a passive (‘on demand’) system—which may have acted as a barrier to its use, although there is no consistent evidence that active systems are any more likely to be effective.18
The dissemination and implementation plan was integral to the intervention but, while interactive workshops are reported to promote use of systems such as this,18 potential effectiveness is often undermined by low uptake,19,20 as observed in this study. We were active in promoting the workshops, and as flexible as possible in offering choices of dates and locations, more than would normally be offered in regular postgraduate education programming. Further efforts to generate uptake of the workshops were constrained by the competing goal of the study, which was that the intervention should be cost-effective and sustainable.
The evaluation was designed to maximize generalizability, which may be conceived as not only maximizing the representativeness of the study population, but also the ‘replicability’ of the intervention.21 We believe that we achieved the first goal reasonably successfully, but assessing the generalizability of the intervention depends entirely on how the latter is defined: was the intervention the supply of the software plus the offer of the workshop, or the supply of the software and the attendance at the workshop? We intended the former approach, which is more pragmatic, but less informative, than the latter; thus, there are competing interpretations of our data. A cautious exploration provides partial insight in relation to the primary outcome of confidence (but not the secondary outcomes, for which statistical power was too low). Firstly, less than half of the intervention GPs to whom it had been supplied reported awareness of its existence, and only a third of this group actually used it. Interpretation of the data for this very small group (22 respondents) is limited by low statistical power, but compared with all other intervention group GPs, they reported a level of confidence in the activity of reassuring low-risk patients which was very significantly higher than that of their colleagues. This was not observed for confidence in the other activities. While the safest interpretation of these findings is that the intervention software was ineffective, the data are also consistent with an intervention which might be at least partly effective if the target population could be persuaded to use it. However, we should also suspect that those who were able or motivated to use it might not be representative of the entire target population.
The patient outcome data are generally consistent with a null trial, but findings relating to referral rates of higher risk patients merit comment. For the intervention group GPs, we observed an increase in the proportion of referred patients who were eventually assessed as being at elevated risk by the clinic, with the opposite for the control group. This result may be entirely due to chance, or it may reflect an effect of the software in the group of GPs who used it. Because our analysis strategy was planned to formally examine between-group differences, we cannot conclude that the observed before–after differences provide positive evidence of a possible intervention effect. However, it does suggest that further evaluation of the software's effectiveness as a guideline implementation strategy might be worthwhile.
In conclusion, despite the promise of computer decision-support systems,22 and reports of GP enthusiasm for their use in genetics,14,23 this carefully conducted pragmatic trial failed to demonstrate a convincing effect of such an application on GP confidence in core activities relevant to managing patients presenting in primary care, although a possible effect on referral patterns cannot be excluded. Also, while it is also not possible to clearly separate ineffective software from an ineffective implementation strategy, the improvement in one aspect of confidence in GPs who reported actual use of the software suggests that efforts towards developing more effective implementation approaches might be worth pursuing. A qualitative study to explore these issues in more depth was not practical, but would likely have offered useful insights into the issues we have identified. We recommend that investigators planning similar studies should seriously consider developing a qualitative study in parallel with the design of the main trial.
The overall effects of decision-support systems are inextricably linked with the context in which they are used and the way they are implemented. It is therefore imperative that they are recognized as complex interventions,24 and their design and evaluation approached accordingly. They should build on data from well-designed quantitative and qualitative studies which identify the attributes most likely to render applications usable by the target population, and seek to use the most promising approaches to their wide-scale implementation.
|
Declaration |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionDeclarationReferences |
|---|
The study protocol was approved by the Joint Ethics Committee of the University of Aberdeen and Grampian Health Board, and the general practice subcommittee of Grampian Health Board. The NHS Caldicott Guardian was informed and consent for data collection was obtained from each eligible patient. The project was funded by the NHS Health Technology Assessment Programme.
Conflict of interest: none declared.
|
Notes |
|---|
Wilson BJ, Torrance N, Mollison J, Watson MS, Douglas A, Miedzybrodzka Z, Gordon R, Wordsworth S, Campbell M, Haites N and Grant A. Cluster randomized trial of a multifaceted primary care decision-support intervention for inherited breast cancer risk. Family Practice 2006; 23: 537–544
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionDeclarationReferences |
|---|
1 Shickle D, Hapgood R, Qureshi N. (2002) The genetics liaison nurse role as a means of educating and supporting primary care professionals. Fam Pract 19:193–196.
2 Women's Concerns Study Group. (2001) Raising concerns about family history of breast cancer in primary care consultations: prospective, population based study. BMJ 322:27–28.
3 De Bock GH, van Asperen CJ, de Vries JM, Hageman GCHA, Springer MP, Kievit J. (2001) How women with a family history of breast cancer and their general practitioners act on genetic advice in general practice: prospective longitudinal study. BMJ 322:26–27.
4 Emery J, Watson E, Rose P, Andermann A. (1999) A systematic review of the literature exploring the role of primary care in genetic services. Fam Pract 16:426–445.
5 Eccles DM, Evans DGR, Mackay J. (2000) on behalf of the UK Cancer Family Study Group. Guidelines for a genetic risk based approach to advising women with a family history of breast cancer. J Med Genet 37:203–209.
6 Robins R and Metcalfe S. (2004) Integrating genetics as practices of primary care. Soc Sci Med 59:223–233.[CrossRef][ISI][Medline]
7 Scottish Cancer Group Cancer Genetics Sub-Group. (2001) Cancer Genetics Services in Scotland. Guidance to Support the Implementation of Genetics Services for Breast, Ovarian and Colorectal Cancer Predisposition (Scottish Executive Health Department, Edinburgh).
8 Scottish Intercollegiate Guidelines Network. (2000) Report on a Recommended Referral Document (Scottish Intercollegiate Guidelines Network, Edinburgh) (SIGN Guideline 31).
9 Donner A. (1982) An empirical study of cluster randomization. Int J Epidemiol 11:283–286.
10 Donner A and Koval JJ. (1982) Design considerations in the estimate of intraclass correlation. Ann Hum Genet 46:271–277.[ISI][Medline]
11 Donner A and Klar N. (1994) Methods for comparing event rates in intervention studies when the unit of allocation is a cluster. Am J Epidemiol 140:279–289.
12 Machin D, Campbell M, Fayers P, Pinol A. (1997) Sample size tables for clinical studies. (Blackwell Science, Oxford).
13 Donner A, Birkett N, Buck C. (1981) Randomization by cluster: sample size requirements and analysis. Am J Epidemiol 114:906–914.
14 Emery J, Walton R, Murphy M, et al. (2000) Computer support for interpreting family histories of breast and ovarian cancer in primary care: comparative study with simulated cases. BMJ 321:28–32.
15 Kumar S. (1999) Resisting revolution: generalism and the new genetics. Lancet 354:1992–1993.[CrossRef][ISI][Medline]
16 Kumar S and Gantley M. (1999) Tensions between policy makers and general practitioners in implementing new genetics: grounded theory interview study. BMJ 319:1410–1413.
17 Wyatt J and Spiegelhalter D. (1990) Evaluating medical expert systems: what to test and how? Med Inform 15:205–217.
18 Rousseau N, McColl E, Newton J, Grimshaw J, Eccles M. (2003) Practice based, longitudinal, qualitative interview study of computerised evidence based guidelines in primary care. BMJ 326:314–326.
19 Thomas RE, Grimshaw JM, Mollison J, et al. (2003) Cluster randomized trial of a guideline-based open access urological investigation service. Fam Pract 20:646–654.
20 Davis J, Roberts R, Davidson DL, et al. (2004) Implementation strategies for a Scottish national epilepsy guideline in primary care: results of the Tayside Implementation of Guidelines in Epilepsy Randomized (TIGER) trial. Epilepsia 45:28–34.[ISI][Medline]
21 Shadish WR, Cook TD, Campbell DT. (2002) Experimental and Quasi-Experimental Designs for Generalized Causal Inference (Houghton Mifflin, Boston).
22 Grimshaw J, Shirran E, Fraser C, Winkens R, Gruintjes F, Thomas R. (2002) Systematic Review of Interventions to Improve Outpatient Referrals from Primary Care to Secondary Care (University of Aberdeen Health Services Research Unit, Aberdeen).
23 Fry A, Campbell H, Gudmunsdottir H, et al. (1999) GPs' views on their role in cancer genetics services and current practice. Fam Pract 16:468–474.
24 Campbell M, Fitzpatrick R, Haines A, et al. (2000) Framework for design and evaluation of complex interventions to improve health. BMJ 321:694–696.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  D. M. Murray, S. L. Pals, J. L. Blitstein, C. M. Alfano, and J. Lehman Design and Analysis of Group-Randomized Trials in Cancer: A Review of Current Practices J Natl Cancer Inst, April 2, 2008; 100(7): 483 - 491. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||