Family Practice Advance Access originally published online on June 5, 2008
Family Practice 2008 25(3):139-145; doi:10.1093/fampra/cmn022
Exploring the black box of change in improving test-ordering routines
a Department of General Practice/Centre for Quality of Care Research (WOK), Care and Public Health Research Institute, Maastricht University, PO Box 616, NL-6200 MD, Maastricht
b Division of Clinical Methods and Public Health, Department of General Practice, Academic Medical Center—University of Amsterdam, PO Box 22660, 1100 DD, Amsterdam
c Department of General Practice, Care and Public Health Research Institute, Maastricht University, PO Box 616, NL-6200 MD, Maastricht, The Netherlands
Correspondence to Marloes A van Bokhoven, Department of General Practice/Centre for Quality of Care Research (WOK), Care and Public Health Research Institute, Maastricht University, PO Box 616, NL-6200 MD, Maastricht, The Netherlands; Email: loes.vanbokhoven{at}hag.unimaas.nl
Received 11 July 2007; Revised 17 March 2008; Accepted 21 April 2008.
 |
Abstract |
|---|
 Top Abstract BackgroundMethodsResultsConclusionDeclarationReferences |
|---|
Background. The effects of quality improvement strategies are sometimes limited in spite of a systematic development approach. What elements play a role in the change process is not yet fully understood.
Objective. To explore this ‘black box’ of change, by analysing the barriers and facilitators GPs perceive during the change process.
Methods. Qualitative study among GPs who participated in the quality improvement strategy arm of a randomized clinical trial on blood test ordering for unexplained complaints (UCs). The strategy was based on a national guideline that advocates delayed test ordering in patients presenting with UCs. Each GP's change process was assessed by means of a semi-structured interview about barriers to and facilitators of change.
Results. Twenty-four interviews were analysed. Important themes identified in the interviews were lack of problem awareness, the time and effort it takes to change, influence of patients and the pros and cons of the changed behaviour.
Conclusion. The themes can be summarized into one comprehensive issue: the GPs lack a sense of urgency to change. An important explanation seems to be that two questions from the problem analysis prior to the development of the strategy had not been adequately answered: "Is the GPs' current behaviour a problem and does the problem have consequences for patients?" and if so, "What is the extent of the problem?." As a result, insufficient attention was given to applicability issues, such as time investment, costs and patient and practitioner satisfaction and anxiety.
Keywords. Family practice, health care, health services research, laboratory techniques and procedures, needs assessment, quality assurance.
|
Background |
|---|
|
TopAbstractBackgroundMethodsResultsConclusionDeclarationReferences |
|---|
Most strategies to improve professional behaviour are only moderately effective.1,2 Possible reasons are engrained routines and the use of inadequate methods to design and evaluate change strategies.3–5 A relevant solution is to use a systematic approach in designing a change strategy.6 Nevertheless, the effects of such strategies sometimes remain disappointing because a prior determinant analysis is frequently lacking and the innovation strategies that are used are mostly not theory driven.7 Sometimes, barriers and facilitators have been analysed in advance but not used in the design.8,9
Several models about changing individuals' behaviour indicate that change is usually a stepwise process, during which the target group can experience various problems in achieving the proposed change.10,11 The precise elements that play a role in this process are not yet fully understood. Although Flottorp et al.12,13 carefully designed a quality improvement strategy based on barriers identified previously, even this strategy had little effect in changing practice. Though they tried to explain this in a process evaluation, no detailed insight was obtained into the change process itself, which largely remains a black box.14,15
The objective of this paper is to explore this black box of change at a time when the practitioners involved are actually in the middle of the change process, by analysing the barriers and facilitators that GPs perceive ‘during’ such a change process. To our knowledge, such analyses have scarcely been reported in the literature. We took account of the interactions between a quality improvement strategy, its participants and their environment by focussing on GPs who participated in a strategy to improve their blood test–ordering behaviour for patients presenting with unexplained complaints (UCs), as defined in a national guideline.16 This strategy was applied in a randomized clinical trial in which three groups of GPs were compared: Group 1 was instructed to order blood tests directly at the first consultation. GPs in Group 2 were instructed to propose a watchful waiting approach to their patients and GPs in Group 3 were instructed to try and postpone test ordering as well, but they also participated in a quality improvement strategy that supported them in postponing test ordering. The theory-driven strategy had been systematically designed and was based on a problem analysis that has been published previously (Box 1).6,17
| BOX 1 Quality improvement strategy Topic UCs in general practice. Problem When patients present to a GP with UCs, diagnostic blood tests are frequently ordered. As the risk of serious pathology in these patients is low, the probability of false-positive test results is relatively high. This may lead to a cascade of further diagnostic testing, anxiety and somatization. Recommended behaviour Applying a 4-week watchful waiting period (based on a guideline from the Dutch College of General Practitioners). If complaints persist, a limited set of tests (erythrocyte sedimentation rate, haemoglobin, thyroid stimulating hormone, glucose) is indicated. Most important barriers (prior to the quality improvement strategy) Personal routines, tolerance of diagnostic uncertainty, pressure by patients to perform laboratory testing (even if patients do not explicitly ask for this), time pressure and tactical motives for test ordering. Educational methods selected on the basis of barriers and facilitators General Repeated message provision, education based on problems experienced by GPs in daily practice, offering opportunities to practice new behaviour. Two small group interactive meetings and one personalized practice visit. Supplying GPs with written course materials. Routines Formulating routines and alternative routine, setting goals for change, practicing in GPs’ own practice. Tolerance of diagnostic uncertainty Case illustrating diagnostic accuracy to overcome knowledge deficiencies about (limited) diagnostic accuracy of tests and to visualize risks of pathology. Skills training to discuss uncertainty with patients and colleagues providing examples of successful explanations. Perceived pressure by patients Communication skills training based on GPs' personal barriers, growing complexity of training situation; opportunities to practice in GPs' own practice and receive feedback. Educational materials for patients. Time pressure Analysing the sources of time pressure. Colleagues giving each other suggestions. Tactical motives Provision of alternative tactical strategies. Format Small group meeting 1 Part 1: Interactive explanation of diagnostic value of tests for patients with UCs and effect of watchful waiting approach on diagnostic value. Part 2: Discussion of difficulties experienced in practice when dealing with patients presenting with UCs. Individual goal setting to change behaviour in GPs’ own practice. Small group meeting 2 Part 1: Discussion about experiences with behaviour change. Searching for solutions to barriers that have arisen. Part 2: Practicing difficult situations by means of video vignettes. Setting new individual goals to change one's own behaviour. Practice visit Discussing individuals’ barriers to change and providing suggestions to overcome these, based on stage of change. In between meetings, GPs get the opportunity to work on their goals to change their behaviour. Course materials Course book; leaflets for patients with information about UCs; diaries about complaints and food intake to hand out to patients to fill in and later discuss together; video message for the waiting room, explaining to patients the use of watchful waiting.
|
The question we wanted to answer was what are crucial barriers and facilitators during the change process among GPs who have been stimulated to postpone blood test ordering in patients presenting with UCs?
|
Methods |
|---|
|
TopAbstractBackgroundMethodsResultsConclusionDeclarationReferences |
|---|
Comprehensive, semi-structured interviews were conducted with GPs as a part of the practice visit that belonged to the quality improvement strategy. We chose for this method because it gave us the opportunity to discuss thoroughly the determinants which we expected to be different in each individual GP. The interviews took place during the practice visit, which was part of the quality improvement strategy.
Selection of participants
The study population consisted of all GPs who were participating in a trial on blood test ordering for patients with UCs and had been randomized to the quality improvement strategy arm of this trial. Randomization in the trial took place at the practice level. The full protocol of the study has been published elsewhere.18 The GPs were recruited in the southern and western parts of the Netherlands by mail and telephone using address lists of all GPs in a certain area. Participating practices were situated in rural as well as urban areas. All participants (N = 26) were visited at their practices by one of the researchers (MAB or HK), as part of the quality improvement strategy.
Data collection
During the practice visit, each GP's change process was assessed by means of a semi-structured interview. All interviews were done by two female interviewers, MAB and HK. Both are medical doctors. The first graduated as a GP, the second ended vocational training in general practice prematurely. Practice visits were scheduled at least 3 months after the second small group meeting. The small group meetings were moderated by GP–epidemiologists for the epidemiological part of the first meeting and by behavioural scientists who teached at the institutes of vocational training in general practice for the communication parts of both meetings. Approximately four GPs participated in a meeting at the same time. GPs were informed that the interview was not intended to judge their functioning as physicians but to see how change was progressing and to make suggestions to support the process if possible. The topics included in the scenario of the interviews are summarized in Table 1 and are based on models of changing behaviour. The first two questions were rather open ended: "Do you remember the previous meeting, and if so, what did you plan to change?" and "How did things go after this meeting?" Depending on the answers of the GPs, topics were further elaborated. When a GP did not address the topics of the scenario spontaneously, the interviewer introduced them. When barriers to change came up during the interview, which was always the case, these were subsequently used to provide suggestions to overcome them.
|
The interviews were recorded on audiotape and field notes with key issues were taken as a backup. The tapes were transcribed verbatim. Age and number of years of experience as a GP were recorded.
Analysis
The interviews were uploaded and coded using Atlas/ti software (version 4.2). The approach resembles a framework approach. Open coding was used as much as possible, though the authors naturally were aware of the literature on stages of change and the interview had been structured according to the existing models of changing behaviour. Thus, some deductive elements were added to inductive elements. One author (MAB) coded all interviews. Each interview was also coded by one of two other authors (HK or TW), independently from both the first coder and the other second coder. A cyclic approach was used to add and adapt codes. Single passages of text could get different codes. Differences in coding were discussed and consensus was sought. Similar codes were combined. Subsequently, we used an eclectic model of change, developed by Grol and Wensing,10 to structure the findings after the coding process was finished, for two reasons (Table 1).19,20 Firstly, stages are widely used to describe the change processes of individuals and the model we used summarizes these. Secondly, it reflects the basic structure of the interviews. If codes did not fit into the model, we somewhat adjusted the categories of the model.
As proposed by the same authors, the barriers and facilitators within each stage were labelled with one of the following six levels: the innovation itself (applying watchful waiting), the individual professional, the patient, the social context, the organizational context and the economic and political context.10
|
Results |
|---|
|
TopAbstractBackgroundMethodsResultsConclusionDeclarationReferences |
|---|
Population
All 26 GPs participating in quality improvement strategy of the trial were interviewed. None of the GPs had any objections against the interview. The length of the interviews was
45 minutes. The majority (80%) were male. Regions and settings were equally divided between the south (54%) and west of the country (46%) and between urban (50%) and rural settings (50%). The mean age of the respondents was 47.5 years and they had an average of 13.5 years of experience as a GP. The interviews were recorded after an average of 7.3 months (range 3–10) into the intervention period. Of the participating practices, 10 were single handed. Two GPs who were running a joint practice decided not to complete the study, though some of the topics were discussed in one joint interview with these two GPs. From the other practices, only one GP per practice participated in the study. Data from all but one interview (24) were available for analysis due to technical problems. All participants set objectives to change their behaviour. The objectives ranged from ‘to suggest patients with UCs to come back in the afternoon, because then I have more time’ to ‘better clarify the patient's fears’.
Change process
Barriers and facilitators have been summarized in an appendix which is available online.
Each participant mentioned several barriers and facilitators that fitted into more than one stage of the change process. Most of the codes and comments related to the ‘positive attitude’ stage. Barriers generally outnumbered the facilitators. All stages are illustrated below with quotations from the interviews.
Awareness and interest.. The first stage in the change process is particularly related to becoming aware of the problem and the innovation and then becoming interested in them. The fact that all participants voluntarily applied to participate in the trial implies that they were most likely aware of and interested in the innovation. This is reflected by the limited number of comments relating to this stage and the facilitators referred to. However, several GPs mentioned, contrary to their prior expectations, that they did not see many patients with clear UCs. In addition, a few GPs were not greatly interested in this topic because they did not experience many problems with testing in daily practice (Box 2).
| BOX 2 Comments on awareness and interest (GP 11, female) "Sure, but I'd expected to see far more of such people than I did. May be I'm being more selective." (GP 5, male) "We don't see it as a problem. If people ask for blood tests, that's no problem to us, nor to the patients. You get problems if you start to change the pattern that's developed over the years. Because that's time-consuming."
|
Understanding of innovation and insight into one's own routines.. In the next stage of the change process, GPs acquire a clear understanding of what the improvement entails and develop insight into their own performance, including the deviations from the target behaviour. Several participants perceived the information that was given as part of the quality improvement strategy, especially the numerical illustrations of the limited diagnostic value of immediate test ordering, as an important facilitator of change. Recent vocational training was also perceived as a facilitator of adopting the watchful waiting approach. However, even after the instructions provided as part of the quality improvement strategy, several GPs overestimated the accuracy of blood tests for UCs.
A determinant of insight into their own routines that was mentioned several times was whether or not the participants saw room for improvement in their behaviour. Other determinants reflected skills and personal characteristics (Box 3).
| BOX 3 Comments on understanding the innovation and insight into one's own routines (GP 24, male) "I never used to order tests for people who had felt tired for a week anyway." (GP 15, male) "I think I'm a bit of a perfectionist. And such a vague diagnosis, that's hard to sell." (GP 19, female) "I think that, above all, I want to do the right thing, in the medical or technical sense of the word. And I often feel that's not what I'm doing when I'm doing consultations and I find myself ordering blood tests just to reassure the patient for the moment. You think to yourself: right, these tests are not going to produce any abnormal results. And that's the end of it." (GP 23, female) "Well, I do things like PSA [prostate-specific antigen], as I don't trust my own rectal examination."
|
Positive attitude and intentions or decisions to change.. In the following stage, GPs balance the advantages and disadvantages of the watchful waiting approach against those of immediate blood test ordering. Many GPs claimed that the advantages of a watchful waiting approach were outweighed by the advantages of immediate test ordering, and the disadvantages of watchful waiting. They found that testing was efficient, satisfied patients, had diagnostic or strategic value, reassured both patients and GPs and was not costly. Actually, several GPs did not experience any disadvantages of their current approach of immediate test ordering. According to the GPs, patients had a strong influence on their test-ordering behaviour.
Few participants mentioned clear intentions to change their routines (Box 4).
| BOX 4 Comments on positive attitude and intentions or decisions to change (GP 7, male) "These patients often come with certain ideas in their heads which make me think I'll need some additional arguments to talk them out of that." (GP 9, male) "The point is, if I have the idea that I don't feel like waiting for another two weeks or a month while nothing is being done. Then I just want to get going. It might be a thyroid problem or a kidney problem, and then I find it hard to say why don't you wait for another month and endure the itch." (GP 22, female) "What I normally do is give them the form for the lab tests, and nine out of ten times, that's the end of it." (GP8, male) "People are becoming more and more worried, what with the Internet and all that stuff. They read all kinds of things and become more and more uncertain about their own health. And they want clarity as soon as possible. And you notice that, so I have a greater tendency to order tests." (GP 10, male) "Of course it also depends on other factors, internal and external. If you're very busy, you're more likely to say right let's have some tests done. That means the problem is deferred for the moment, I'll get some more information to go on and some time to think about it." (GP 17, male) "So you see the results of tests ordered by your colleague, who's taken over, and you think, goodness, that's a lot of tests, a whole lot of tests."
|
Actual adoption in practice and confirmation of benefits.. After GPs decide to change, they need to try out the watchful waiting approach on a small scale and evaluate whether they are satisfied with the new approach or not. GPs who had a successful tryout found that their skills improved and failed to find the anticipated negative effects of watchful waiting. GPs who relapsed mentioned several barriers, especially as regards the feasibility of a watchful waiting approach (Box 5).
| BOX 5 Comments on actual adoption in practice and confirmation of benefits (GP 22, female) "Well, there was no one saying they didn't want that, no. I think they all liked it." (GP 7, male) "And if you don't order them the first time round and explain to them why, that tests would probably show nothing and their complaint is probably something that will resolve spontaneously or has a different kind of origin, you find that people start to think about it and don't return." (GP 20, female) "I think the main disadvantage is that it takes a lot of time. That I think is a major disadvantage of my way of working." (GP 14, male) "And I've found several times, after I had actually postponed it and then did it anyway after a while, the tests did show up something."
|
Integration in routines and embedding within the organization.. Finally, the new approach needs to become routine behaviour and must be embedded in the organization. Few GPs mentioned barriers or facilitators relating to the integration stage, and none of them mentioned any factors influencing embedding. Facilitators included the availability of tactics to prevent relapse into old routines and the structure of the quality improvement strategy with its repeated educational meetings (Box 6).
| BOX 6 Comments on integration in routines and embedding within the organization (GP 22, female) "I should say that the way you people organize things does help you keep it up. Because of course research into the way people learn has shown that repetition and returning to it time and again does make you actually do it, right?" (GP 6, male) "I can assure you that the routine of blood tests is really gone."
|
|
Conclusion |
|---|
|
TopAbstractBackgroundMethodsResultsConclusionDeclarationReferences |
|---|
Summary of main findings
The aim of this study was to look inside the black box of behavioural change by identifying crucial barriers to and facilitators of change during the actual change process that GPs went through. Important aspects mentioned by the interviewees appear to be the lack of problem awareness, the time and effort it takes to change, the influence of the patients and considerations about the pros and cons of the behaviour change.
Strengths and limitations
The main strength of this study was that, as far as we know, it was among the first to explore experienced barriers and facilitators during the course of a change process instead of perceived barriers beforehand. No purposive sampling was done, but the sample consisted of GPs with a range of characteristics, e.g. both sexes, different levels of experience as a GP and different practice types. The group might, however, represent a selection of unusually motivated GPs, as they participated voluntarily. This might negatively influence transferability of results. However, even this well-motivated group saw mostly barriers in the early stages of the change process. The credibility of our findings is high. The types of barriers and facilitators that were mentioned by the participants were also mentioned in the change process of other behaviours. In addition, it is known that guideline adherence is also negatively influenced by similar factors, such as when practice routines need to be changed.21 To triangulate our findings in order to check for consistency of our data, we planned to compare the change process as reported by the GP to the actual change in test ordering as measured by the numbers of tests ordered. Unfortunately, the test-ordering information was insufficient to draw meaningful conclusions. However, there were no quotes in our data set that contradict our general findings.
Interviewing GPs implies the limitation of only retrieving determinants of behaviour that participants are aware of. Unconscious barriers, e.g. lack of skills, do not always emerge. The model we used to categorize the barriers and facilitators is restricted to changes by individual practitioners and does not, for instance, describe organizational change. This made it suitable for our findings as individual change is the target of this strategy. While categorizing the barriers and facilitators within the stages of the model, we felt that the ‘innovation’ aspect in the ‘orientation’ stage covers not only the new routine itself but also the comparison with previous behaviour, the appreciation of the quality improvement strategy and, in a research setting, the influence of the trial on the intended behaviour. We also decided to add ‘awareness of the problem’ to the ‘awareness of the innovation’ substage.
Comparison with existing literature
Our study revealed barriers to and facilitators of change across the consecutive stages of the change process. However, determinants relating to the stages at both extremities of the change process were mentioned less frequently than those relating to the intermediate stages. The absence of factors relating to the early stages can be explained by the fact that the interviewees were participating in a trial and were therefore almost by definition aware of the topic. The finding that few of the determinants that were mentioned related to the final stage might be a sign that the interviews were held somewhat too soon as reaching the maintenance stage takes time. As a result, the results on the advanced stages may be incomplete. In addition, the chosen methodology, interviewing GPs, tends to emphasize the insight and acceptance stages because these are more ‘cognitive’ than the others. However, a more likely explanation is that the strategy has been less successful than intended. Most GPs were not yet at the stage of actually changing their behaviour. This is supported by the large numbers of barriers mentioned by the interviewees that related the insight and acceptance stages. A look at these barriers shows that they can be summarized into one comprehensive issue across the stages, namely the fact that the GPs lacked a sense of urgency to change. Apparently, the participants did not become permanently convinced that their test-ordering behaviour needed to change. The limited difference in determinants between our interviews and the diagnostic analysis on which the quality improvement strategy was based, corroborates this.17 One explanation for this lack of a sense of urgency is that the participants were as a rule not willing to invest in any quality improvement of their functioning and generally preferred to stick to old habits. On the other hand, the participants did voluntarily and actively participate in the quality improvement strategy meetings. An alternative explanation for the GPs' lack of a sense of urgency is that two questions from the problem analysis prior to the development of the strategy had not been adequately answered: "Is the GPs' current behaviour a problem and does the problem have consequences for patients?" and if so, "What is the extent of the problem?" The sense of urgency, in the sense of an individual's motivation to change specific behaviour, has been recognized as a facilitator before.22 The guideline on which the quality improvement strategy is based discusses the low diagnostic accuracy of tests and the value of a watchful waiting approach, from the viewpoint of Bayesian decision theory. However, support for this theory with scientific evidence from practice-based research is still limited. In addition, when we compare the guideline with the Appraisal of Guidelines for Research and Evaluation (AGREE) criteria for the methodological quality of clinical guidelines, it appears that relatively little attention has been devoted to its applicability and to patients' experiences and expectations.23 Greenhalgh et al.24 also emphasized that complex behavioural change needs to be seen as a process rather than an event, a process in which GPs interact with the innovation and their environment. To add to the complexity, not only evidence but also the GPs' intuition may be of value in this process. Greenhalgh25 suggested that systematic critical reflection about intuitive judgements in dialogue with professional colleagues may improve these ‘intuitive powers’.
Implications
When guideline developers take these aspects into account, they may produce different, more easily applicable, guidelines, providing better answers to GPs' practical needs. Quality improvement strategies based on these improved guidelines may then be more effective in creating a sense of urgency to change, which in turn appears to be a condition priming true behavioural change. This implies that evidence about applicability issues, such as time investment, costs and patient and practitioner satisfaction and anxiety, needs to be made available to guideline developers. In addition, differences in patient preferences should be evaluated, so that the GPs' ideal of delivering tailored patient care can be met. In conclusion, to develop an effective quality improvement strategy, a thorough problem analysis is needed in advance. This should address the size of the problem and its consequences especially for patients, the applicability of the target behaviour and differences in patient preferences.
|
Declaration |
|---|
|
TopAbstractBackgroundMethodsResultsConclusionDeclarationReferences |
|---|
Funding: Dutch Health Care Insurance Board (CVZ); the Centraal Ziekenfonds health insurance company; the Netherlands Organization for Scientific Research (NWO) (grant 920-03-157). None of the sources of funding influenced the study design or the writing of the manuscript or the decision to submit the manuscript for publication.
Ethical approval: Not applicable.
Conflicts of interest: The authors declare that they have no conflict of interest.
|
Acknowledgments |
|---|
The authors would like to thank all participating GPs for their willingness to take part in this study.
|
Notes |
|---|
van Bokhoven MA, Koch H, Dinant G-J, Bindels PJE, Grol RPTM and van der Weijden T. Exploring the black box of change in improving test-ordering routines. Family Practice 2008; 25: 139–145.
|
References |
|---|
|
TopAbstractBackgroundMethodsResultsConclusionDeclarationReferences |
|---|
1 Grimshaw JM, Shirran L, Thomas R, et al. Changing provider behaviour: an overview of systematic reviews of interventions. Med Care (2001) 39:II2–45.[Web of Science][Medline]
2 Bero LA, Grilli R, Grimshaw JM, Harvey E, Oxman AD, Thomson MA. Getting research findings into practice: closing the gap between research and practice: an overview of systematic reviews of interventions to promote the implementation of research findings. BMJ (1998) 317:465–468.
3 Wensing M, van der Weijden T, Grol R. Implementing guidelines and innovations in general practice: which interventions are effective? Br J Gen Pract (1998) 48:991–997.[Web of Science][Medline]
4 Davis DA, Thomson MA, Oxman AD, Haynes RB. Changing physician performance. A systematic review of the effect of continuing medical education strategies. JAMA (1995) 274:700–705.
5 Campbell M, Fitzpatrick R, Haines A, et al. Framework for design and evaluation of complex interventions to improve health. BMJ (2000) 321:694–696.
6 van Bokhoven MA, Kok G, van der Weijden T. Designing a quality improvement intervention: a systematic approach. Qual Saf Health Care (2003) 12:215–220.
7 Fleuren MAH, Wiefferink K, Paulussen TGWM. Determinants of innovation within health care organizations: literature review and Delphi study. Int J Qual Health Care (2004) 16:107–123.
8 Bosch M, van der Weijden T, Wensing M, Grol R. Tailoring quality improvement interventions to identified barriers: a multiple case analysis. J Eval Clin Pract (2007) 13:181–188.
9 Shaw B, Cheater F, Baker R, et al. Tailored interventions to overcome identified barriers to change: effects on professional practice and health care outcomes. Cochrane Database Syst Rev (2005) 3. CD005470.
10 Grol R, Wensing M. What drives change? Barriers to and incentives for achieving evidence-based practice. Med J Aust (2004) 180:S57–S60.[Web of Science][Medline]
11 Grol R, Wensing M. Effective implementation: a model. In: Improving Patient Care: The Implementation of Change in Clinical Practice—Grol R, Wensing M, Eccles M, eds. (2005) Edinburgh, UK: Elsevier.
12 Flottorp S, Oxman AD. Identifying barriers and tailoring interventions to improve the management of urinary tract infections and sore throat: a pragmatic study using qualitative methods. BMC Health Serv Res (2003) 3:3.[CrossRef][Medline]
13 Flottorp S, Oxman AD, Håvelstrud K, Treweek S, Herrin J. Cluster randomised controlled trial of tailored interventions to improve the management of urinary tract infections in women and sore throat. BMJ (2002) 325:367.
14 Flottorp S, Håvelstrud K, Oxman AD. Process evaluation of a cluster randomized trial of tailored interventions to implement guidelines in primary care—why is it so hard to change practice? Fam Pract (2003) 20:333–339.
15 Grol R, Baker R, Moss F. Quality improvement research: understanding the science of change in health care. Qual Saf Health Care (2002) 11:110–111.
16 Dinant GJ, van Wijk MAM, Janssens HJEM, et al. NHG-standaard bloedonderzoek. Algemene principes en uitvoering in eigen beheer. [Dutch College of General Practitioners guideline ‘Blood testing. General principles and use by GP’]. Huisarts Wet (1994) 37:202–211.
17 van der Weijden T, van Bokhoven MA, Dinant GJ, van Hasselt CM, Grol RPTM. Understanding laboratory testing in diagnostic uncertainty: a qualitative study in general practice. Br J Gen Pract (2002) 52:974–980.[Web of Science][Medline]
18 van Bokhoven MA, Koch H, van der Weijden T, Bindels PJE, Grol RPTM, Dinant G-J. Blood test ordering for unexplained complaints in general practice, the VAMPIRE randomised clinical trial protocol. [ISRCTN55755886]. BMC Fam Pract (2006) 7:20.[CrossRef][Medline]
19 Grol RPTM. Implementing guidelines in general practice care. Qual Health Care (1992) 1:184–191.
20 Grol R, Wensing M, Eccles M. Improving patient care. In: The implementation of change in clinical practice (2005) London: Elsevier.
21 Grol R, Dalhuijsen J, Thomas S, Veld C, Rutten G, Mokkink H. Attributes of clinical guidelines that influence use of guidelines in general practice: observational study. BMJ (1998) 317:858–861.
22 Kennedy T, Regehr G, Rosenfield J, Roberts W, Lingard L. Exploring the gap between knowledge and behavior: a qualitative study of clinician action following an educational intervention. Acad Med (2004) 79:386–393.[CrossRef][Web of Science][Medline]
23 AGREE-Collaboration. Appraisal of Guidelines for Research and Evaluation (AGREE) Instrument 2004. http://www.agreecollaboration.org (accessed on July 14, 2006).
24 Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q (2004) 82:581–629.[CrossRef][Web of Science][Medline]
25 Greenhalgh T. Intuition and evidence—uneasy bedfellows? Br J Gen Pract (2002) 52:395–400.[Web of Science][Medline]
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||