Family Practice Vol. 21, No. 2, 183-188
Family Practice Vol. 21, No. 2 © Oxford University Press 2004, all rights reserved.
Article |
Learning with computerized guidelines in general practice?
A randomized controlled trial
Medical Knowledge Network evidence.de and a Grönemeyer Institute of Microtherapy, Faculty of Medicine, University Witten/Herdecke, Germany
E-mail: butzlaff{at}uni-wh.de
Received 7 January 2003; Revised 8 August 2003; Accepted 3 November 2003.
Butzlaff M, Vollmar HC, Floer B, Koneczny N, Isfort J and Lange S. Learning with computerized guidelines in general practice? A randomized controlled trial. Family Practice 2003; 21: 183–188.
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionReferences |
|---|
Background. Evidence-based guidelines are seen as an important instrument to transfer scientifically generated knowledge into daily clinical practice and to ensure high standards of clinical care. Despite wide promulgation, clinical guidelines so far have a limited impact on individual professional learning and on changing daily medical practice.
Objectives. Our aims were (i) to study a potential knowledge increase among German GPs after implementation of web- and evidence-based guidelines and (ii) to identify and analyse potential barriers to individual professional learning with computerized guidelines.
Methods. A prospective, randomized controlled trial was conducted including 72 GPs (21% female, 79% male). The intervention group (n = 38) had access to clinical guidelines via the Internet or CD-ROM, the control group had not (n = 34). Both groups received a standardized two-part questionnaire. An increase of knowledge was measured with 25 multiple choice questions related to four different medical topics. In addition, reasons for using or not using computerized guidelines were analysed after access to guidelines was open to all participating physicians.
Results. There was no significant knowledge increase in the intervention group (P = 0.69). Twenty-two (58%) GPs of the intervention group had used the guidelines. Unspecified curiosity (76%) and a specific medical question (38%) were predominant motives for usage among physicians who had used the guidelines. Among ‘non-users’, 78% stated ‘lack of time’ as the main reason for not using guidelines.
Conclusion. An efficient knowledge transfer through computerized guidelines was not achieved. Usage, individual learning and potential implementation depend on adequate incentives and pragmatic aspects of clinical practice: easy and quick access.
Keywords. Clinical practice guidelines, evidence-based medicine, general practice, Internet, randomized controlled trial.
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
Like all other physicians, GPs have to cope with a rapidly growing amount of new medical knowledge. Apart from relevant and high-quality publications, they are confronted with an increasing amount of irrelevant and useless information. Sifting the chaff from the wheat is becoming a key competence for the individual learner.1 Since clinical guidelines are, ideally, ‘condensing’ a wide body of literature into a concise and evidence-based document, new, relevant and useful findings may reach general practice more efficiently through computerized guidelines than printed articles, manuals or books.2,3 After several years of exponential growth, a multitude of clinical practice guidelines (CPGs) is available for a similarly wide range of indications, diagnoses and therapies.2,4 Thus, availability, especially via the Internet, is no longer a major barrier to successful implementation. Instead, internal barriers, such as lack of awareness or missing capabilities to use existing resources, or external barriers, such as lack of time, today are predominant roadblocks for the implementation process. Overlapping and synergistic implementation strategies as well as attractive incentives for individual learning are to be worked out.5–7 To date, however, the current literature of successful implementation strategies can be summarized as ‘not one size fits all’.6
The medical knowledge network www.evidence.de of the University of Witten-Herdecke (North Rhine-Westphalia) in Germany, is a department within the Medical Faculty that uses web- and evidence-based guidelines as a major tool for knowledge transfer to a network of some 400 faculty members plus 120 GPs. Dissemination of guidelines mainly via the Internet was chosen because: (i) content can be adequately presented, interactively used and easily updated; (ii) individual communication may follow with the same medium via e-mail or a discussion forum; and (iii) the significance of the Internet as a platform of information and knowledge transfer and as a learning tool within the medical profession is growing.8
This randomized controlled trial aimed to asses whether a web-based academic information platform can accelerate medical knowledge transfer and guideline implementation, if it is (i) accessible either on- or off-line (Internet/CD-ROM); (ii) introduced and explained by a peer physician; (iii) easy to use (plain algorithm, linked with corresponding text, references and patient information); and (iv) connected with the University.
The primary outcome for the first successful step towards implementation was the increase of individual medical knowledge regarding four clinical topics covered by web- and evidence-based guidelines. Usage of guidelines, quality rating and motives to use or not to use available guidelines were secondary outcomes.
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
From April until June 2001, 72 GPs (20.8% female, 79.2% male) of the academic teaching network of the University Witten-Herdecke answered a two-part questionnaire within this prospective, randomized trial. Power calculation of sample size was based on the assumption that at least five additionally correct answers were considered necessary for a significant and relevant knowledge gain.
The first part of the questionnaire consisted of 25 pre-tested multiple choice questions referring to four clinical topics covered by CPGs (dementia, congestive heart failure, urinary tract infection and prevention of colorectal carcinoma). These questions were designed as multiple choice questions and captured the major and relevant aspects of each guideline.
In February of 2001, all 120 GPs of the network had received a questionnaire regarding availability and use of a personal computer. Eighty-eight (73.3%) answered, out of which 78 (88.6%) had access to a PC with a CD-ROM drive or a PC with Internet access. No distinction was made between modem or ISDN access. All 78 practitioners with PC access were asked to participate in the study; 72 finally participated (see Fig. 1).
|
All practitioners used a Microsoft Windows® or an Apple Macintosh® System.
Allocation numbers were associated with intervention and control groups by use of a computer-generated allocation schedule. To protect against unintended individual knowledge transfer between intervention and controls, participants were block-randomized with regard to single or group practice.
Each practitioner was visited by a peer physician from the University in her/his office. During this visit, both the intervention (n = 38) and the control group (n = 34) answered the first part of the questionnaire (analysis Ia). Having answered the questionnaire and following a brief personal introduction to access and usage of the guidelines by the visiting physician (20 ± 10 minutes), the intervention group was given access to the four web- and evidence-based guidelines. There were two ways of using the guidelines: (i) a password-protected web site (access time 45 ± 10 s depending on access via modem or ISDN); or (ii) a CD-ROM with the HTML files of the guidelines. The CD-ROM contained a small self-starting Web Browser (Opera 3.61®) opening the start site of the guidelines automatically in an off-line modus (access time 20 ± 10 s depending on equipment).
The guidelines consisted of four parts or levels, respectively: (i) an algorithm, serving as a condensed summary of the document; (ii) a full text version; (iii) a reference list; and (iv) a patient guideline.
These four levels were linked at relevant points, e.g. from a drug of interest, the user could ‘click’ directly onto the full text version including dosage, side effects and cost, and from there to the corresponding reference list including strength of evidence rating. In addition, guidelines were presented as PDF files on the web and on CD-ROM for simple print-outs of the guidelines.
The same multiple choice questions were repeated during individual visits after an average time interval of 70 days (analysis Ib) for both groups. Throughout the intervention phase, the intervention and control group were unaware of the fact that the same questions would be asked again.
Part two of the questionnaire analysed the reasons for using or not using the guidelines in the intervention group and in the control group after a second time interval of 
70 days, during which access to the guidelines was also permitted.
The second part (analysis II) was completed by 62 participants (35 of the intervention, 27 of the control group) and included a 5-point Likert scale regarding utilization, individual motives and perception of the guidelines.
Statistical analyses
A commercial software package (SPSS, SPSS Inc., Chicago, IL) was used for statistical analyses. Dependency of two or more categorical variables was analysed applying Fisher's exact test. Differences of the distributions of continuous variables in independent groups was analysed by applying the Mann–Whitney U-test.
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
Baseline characteristics
Seventy-two GPs participated in this study. The baseline characteristics are shown in Table 1. The median age of participants was 47 years (range 30–61); 20.8% were women. Intervention and control groups were well balanced with respect to baseline characteristics (age, sex, practice size and academic title based on scientific dissertation ‘Dr med.’).
|
All participating physicians had either Internet access (in practice or at home) or a computer with a CD-ROM. Twenty-nine (40.3%) of all physicians had Internet both in their practice and at home; 53 (73.6%) had a CD-ROM both in their practice and at home. Three (4.2%) physicians had no Internet access; one (1.4%) had no CD-ROM (Table 2).
|
Part I: knowledge gain
This was assessed by the number of participants with knowledge increase between the intervention and control group (intention to treat) (Table 3). A slight difference between all intervention and control group participants remained non-significant when only those physicians of the intervention group (n = 22) were analysed who had actually used the guidelines (P = 0.50).
|
Equally, the percentage of participating physicians whose knowledge increased by >2 points (maximum 25 points) did not show any significant difference.
Part II: utilization and motives
Users. Analysis of data from both the intervention and control group (n = 62) after 5 months (analysis II) showed that 23 (37.1%) had not used the guidelines, 14 (22.6%) had used them once, 24 (38.7%) 2–5 times and one (1.6%) >5 times. Differentiating those users with regard to age, sex and practice size did not show any differences.
Usage during daytime and practice hours (8 a.m. to 6 p.m.) was higher than during evening hours or at night (6 p.m. to 8 a.m.) (Table 4). Those physicians who had used the guidelines were asked about their motives for utilization with three pre-specified categories: unspecified ‘curiosity’, ‘specific medical question’ and ‘sense of duty’ (with respect to the group of researchers and the University). For 76% of all users, curiosity was a major motive to use the guidelines, 38% had a specific medical question which led to the use of the guidelines, and 26% indicated that they felt obliged to use the guidelines (Fig. 2).
|
|
When asked to rate the overall quality, 80% of the participating and guidelines-using physicians gave a ‘high grade’ mark. Sixty-two per cent found them clearly arranged and understandable, 59% stated that the guidelines were helpful in daily practice, 51% thought they were helpful when counselling patients, 31% stated that their therapeutic decisions were influenced by the guidelines and 28% stated that they were quickly accessible.
Non-users. Participating physicians who had not used the guidelines were also asked about their motives: 18 (78.3%) of the non-users (n = 23) stated that they had not found the time to do so. Five (21.7%) indicated that they had technical difficulties respecting access to the guidelines, one (4.3%) stated that he/she found it too complicated to use the guidelines, and one (4.3%) stated that he/she felt restricted in her/his therapeutic decisions.
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
The efficient transfer of new and relevant research findings into daily clinical practice remains a major challenge. What could be helpful to physicians in their individual learning process regarding new and relevant scientific findings remains a key question.
Clinical guidelines are systematically developed tools to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances.9 Expectations remain high that in combination with modern information technology, CPGs will play an important role as instruments for medical knowledge transfer. However, little is known about the successful process and factors that lead physicians to change and adapt their daily routine when they become aware of a guideline.
While some general aspects are beginning to emerge, i.e. the relative efficiency of ‘reminders’, successful individual learning and guideline implementation will also depend on specific regional or local needs and preferences of physicians.6
In this study, four web- and evidence-based guidelines were introduced by an academic department through peer presentation.
First of all, the lack of a significant knowledge increase is striking. Neither the intervention group as a whole nor the subgroup that had actually used the guidelines showed a significant knowledge increase. Although individual learning and knowledge transfer are only a first step towards clinical improvement, the intervention did not show any lasting impact.
These results are in line with recent rather unsuccessful interventions especially in combination with electronic resources.10,11 On the other hand, Shiffmann et al. in a systematic review found that in 14 of 18 studies about computer-based guideline implementation, guideline adherence improved with computer-based strategies. However, most of these studies focused on guidelines that were integrated in clinical information systems and only three studies had their focus on the primary care sector.3
Two other studies found that the method of presentation (electronic or paper) did not have an impact on guideline utilization or physician decision making.12,13 Although the study by Bell and colleagues found that physicians working with computerized guidelines (versus content-equivalent printed versions) needed less time and were more satisfied with their learning, immediate and long-term post-test scores did not show a favourable comparative knowledge increase.12,13
Secondly, usage of guidelines in our study appeared relatively low during the intervention phase. More than a third of all participants had not used the guidelines, and only a small minority had used them regularly (>5 times). This result might be influenced by the limited number of guidelines (only four medical topics) and the rather short period of intervention (only 2.5 months). A broader spectrum of high quality and evidence-based guidelines and a longer time frame may expedite ‘regular’ use and implementation. The same problems/restrictions were described by Eccles and colleagues.10,11
Also, there was no systematic reminder of the option to use clinical guidelines during the intervention phase, i.e. a peer visit, telephone call or letter reminding physicians. Possibly, physicians need more than one contact or conversation to have their attention and curiosity raised to a level where they test a new option for their individual learning.
A potentially significant barrier to individual learning with web-based resources and guidelines appeared during discussions after the study period: several physicians indicated personal hesitation to use the web from their office, because personalized patient data unintentionally and illegally might become available to hackers/third parties during their ‘on-line time’. However, we are not able to quantify these considerations and their impact on utilization of the web-based guidelines.
Thirdly, the combination of unspecific curiosity and a specific medical question as major motives to use the guidelines is encouraging. At least in this set of practitioners there was no major scepticism or negative attitude towards guidelines. This may indicate that the general acceptance of CPGs as information and quality improvement tools is growing.
Fourthly, the factor ‘lack of time’ in this study was an obvious ‘roadblock’ to more frequent utilization of web-based guidelines, especially when compared with other barriers such as technical difficulties or general reservations. Extrapolating from this finding, the successful implementation of a new information tool depends on its ability to generate quick and relevant answers in daily practice.5,14 Most of the participating physicians used the guidelines between 8 a.m. and 6 p.m., thus during regular office hours. In other words, an electronic CPG that is not user friendly and, most of all, quickly accessible in practice will not work.
In general, however, the factor ‘lack of time’ as well as the increasing daily workload of GPs are issues beyond the scope of scientific guideline generation and presentation.15 As long as continuous professional development remains a private issue of individual physicians without regulatory consequences nor any mechanisms to compensate for time invested and income lost, any educational intervention will have a limited effect.4,6,7,14 New regulations implemented by 1 January 2004 require mandatory continuous medical education (CME) from GPs in Germany, with lower levels of reimbursement if physicians fail to demonstrate their individual learning efforts.
Limitations
In addition to utilization barriers outlined by participating practitioners themselves (see Fig. 3), several aspects may have contributed to the negative outcome on individual learning and knowledge transfer:
- The intervention may have been too broad and non-specific by introducing extensive and evidence-based documents (CPGs) containing several levels rather than few and focused new findings.
- Reinforcement or reminders of web access and off-line guideline utilization during the normal daily routine might haven strengthened the intervention.
- For several physicians, the combination of a relatively young technology (web) and a new set of guidelines meant two steps at the same time, possibly one too many.
- The introduction of and physician instruction about usability of computerized guidelines may have been inadequate and/or too short.
- The time interval might have been too short to become familiar with a new and web-based information tool.7
|
Conclusions
This study re-enforces the fact that passive diffusion of information alone, even if provided by modern and easily accessible technology, has limited or no influence on individual learning and guideline implementation. Our conclusions for future studies result from the above-described limitations.
Since acceptance of evidence-based guidelines appears to be high in general and use of the Internet continues to grow rapidly among GPs, future studies on learning efficiency and guideline implementation should focus on aspects of practicality and adequate combinations of incentives, individual instruction to use web-based resources, quick access, security in routine daily use, financial incentives and practical and focused reminder systems.
Which combination, if any, works in real life is an open question. Many approaches and hypotheses need to be tested.
|
Acknowledgments |
|---|
We thank Stephan Quentin, MD, for a careful review of the manuscript, and the network of primary care physicians at the University of Witten-Herdecke for their participation.
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
1 Smith R. Where is the wisdom...? Br Med J 1991; 303: 798–799.[ISI][Medline]
2 Bero LA, Grilli R, Grimshaw JM, Harvey E, Oxman AD, Thomson MA. Closing the gap between research and practice: an overview of systematic reviews of interventions to promote the implementation of research findings. The Cochrane Effective Practice and Organization of Care Review Group. Br Med J 1998; 317: 465–468.
3 Shiffman RN, Liaw Y, Brandt CA, Corb GJ. Computer-based guideline implementation systems: a systematic review of functionality and effectiveness. J Am Med Inform Assoc 1999; 6: 104–114.
4 Grimshaw JM, Shirran L, Thomas R et al. Changing provider behavior: an overview of systematic reviews of interventions. Med Care 2001; 39: II2–II45.[ISI][Medline]
5 Grol R. Successes and failures in the implementation of evidence-based guidelines for clinical practice. Med Care 2001; 39 [8 Suppl 2]: II46–II54.[ISI][Medline]
6 Grimshaw GM, Thomas R, Maclennan G et al. Efficient and effective implementation strategies. In: Annual Meeting of the International Society of Technological Assessment of Health Care. Satellite Symposium: Clinical Practice Guidelines 2002. Berlin: Urban & Fischer; 2002: 241.
7 Mamary EM, Charles P. On-site to on-line: barriers to the use of computers for continuing education. J Contin Educ Health Prof 2000; 20: 171–175.[CrossRef][Medline]
8 Harris JM, Jr, Salasche SJ, Harris RB. Infopoints: the internet and the globalisation of medical education. Br Med J 2001; 323: 1106.
9 Field MJ, Lohr KM. Guidelines for Clinical Practice: From Development to Use. Washington: National Academy Press; 1992.
10 Eccles M, McColl E, Steen N et al. Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. Br Med J 2002; 325: 941–947.
11 Rousseau N, McColl E, Newton J, Grimshaw J, Eccles M. Practice based, longitudinal, qualitative interview study of computerised evidence based guidelines in primary care. Br Med J 2003; 326: 314–321.
12 Bell DS, Fonarow GC, Hays RD, Mangione CM. Self-study from web-based and printed guideline materials. A randomized, controlled trial among resident physicians. Ann Intern Med 2000; 132: 938–946.
13 Jousimaa J, Makela M, Kunnamo I, MacLennan G, Grimshaw JM. Primary care guidelines on consultation practices: the effectiveness of computerized versus paper-based versions. A cluster randomized controlled trial among newly qualified primary care physicians. Int J Technol Assess Health Care 2002; 18: 586–596.[ISI][Medline]
14 Dickmann C, Habermeyer E, Spitzer K. WWW-based continuing medical education: how do general practitioners use it? In: Hasman A (ed.) Medical Infobahn for Europe. IOS Press; 2000.
15 Butzlaff M, Floer B, Koneczny N et al. [www.evidence.de—assessment and utilization of clinical guidelines by primary care physicians and internists]. Z Arztl Fortbild Qualitatssich 2002; 96: 127–133.[Medline]
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  H. Varonen, J. Jousimaa, A. Helin-Salmivaara, and I. Kunnamo Electronic primary care guidelines with links to Cochrane reviews--EBM Guidelines Fam. Pract., August 1, 2005; 22(4): 465 - 469. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||