Family Practice Vol. 19, No. 5, 547-556
© Oxford University Press 2002
Information Technology |
An evaluation of general practice websites in the UK
Warders Medical Centre, East Street, Tonbridge, Kent TN9 1LA,
a Department of Psychiatry and
b Department of General Practice, St George’s Hospital Medical School, Cranmer Terrace, London SW17 0RE,
c 180 Sandgate Road, Folkestone, Kent CT20 2HN,
d Woodbridge Hill Surgery, Guildford, Surrey GU2 6AT and
Dr AJ Howitt, Warders Medical Centre, East Street, Tonbridge, Kent TN9 1LA, UK; E-mail: ajhowitt{at}warders.co.uk
Howitt A, Clement S, de Lusignan S, Thiru K, Goodwin D and Wells S. An evaluation of general practice websites in the UK. Family Practice 2002; 19: 547–556.
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionImplications for practiceAppendixReferences |
|---|
Background. General practice websites are an emerging phenomenon, but there have been few critical evaluations of their content. Previously developed rating instruments to assess medical websites have been criticized for failing to report their reliability and validity.
Objectives. The purpose of this study was to develop a rating instrument for assessing UK general practice websites, and then to evaluate them critically.
Methods. The STaRNet Website Assessment Tool (SWAT) was developed listing criteria that general practice websites may meet, which was then used to evaluate a random sample of websites drawn from an electronic database. A second assessor rated a subsample of the sites to assess the tool’s inter-rater reliability. The setting was an information technology group of a general practice research network using a random sample of 108 websites identified from the database. The main outcome measures were identification of rating criteria and frequency counts from the website rating instrument.
Results. Ninety (93.3%) sites were accessible, of which 84 were UK general practice websites. Criteria most frequently met were those describing the scope of the website and their functionality. Apart from e-mail to practices, criteria related to electronic communication were rarely met. Criteria relating to the quality of information were least often met. Inter-rater reliability kappa values for the items in the tool ranged from –0.06 to 1.0 (mean 0.59). Values were >0.6 for 15 out of 25 criteria assessed in 40 sites which were rated by two assessors.
Conclusions. General practice websites offer a wide range of information. They are technically satisfactory, but do not exploit fully the potential for electronic doctor–patient communication. The quality of information they provide is poor. The instrument may be developed as a template for general practices producing or revising their own websites.
Keywords. Communication, family practice, information services, Internet, medical informatics.
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionImplications for practiceAppendixReferences |
|---|
Currently >400 general practices in the UK have a website.1 With the growth in use of the Internet generally and the Department of Health’s aims for every general practice to have an Internet connection,2 the use of this medium for communication in primary care and the number of practice websites is likely to increase substantially in the near future.
There has, however, been little attempt made to evaluate general practice websites, both in the range of services and information they offer and of the quality of information provided.
Concern has been expressed about the quality of medical information displayed generally on the Internet. Some sites provide inaccurate or misleading information3,4 and others purporting to offer impartial advice have failed to disclose that they have a financial interest in persuading users to follow their advice.5 These concerns have led to the development of ethical codes for providers of medical information on the Internet.6–8 Although these vary depending on the specific purpose and context of the proposed codes, there are common themes between many of them.
Rating instruments have been developed to help evaluate medical websites.9 These instruments themselves have been criticized for failing to provide details on inter-rater reliability and how their content validity was established.10,11
We aimed to develop a rating instrument for assessing general practice websites and then to evaluate a randomly selected sample of UK websites using the instrument.
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionImplications for practiceAppendixReferences |
|---|
Development of the rating instrument
The study was conducted by an information technology group of the South Thames Area Research Network (STaRNet). We selected practices from the largest database of general practice websites.1 At the time of the study, in February 2000, it listed 324 general practice websites in the UK.
We identified themes within the sites in a preliminary survey of a random sample of 20 sites, choosing every fifteenth site from a list on the database, which arranged sites alphabetically by the name of the practice. Additionally, we sought out existing health website rating instruments for items which were relevant to general practice, from both printed and Internet sources.9,10 From these sources, we produced the STaRNet Website Assessment Tool. This is a rating instrument which specifically aims to evaluate websites of National Health Service (NHS) general practices in the UK, produced primarily for their patients. It contained criteria regarding many aspects of general practice websites. Figure 1
presents an overview of the stages in the development of the instrument and its evaluation.
|
We modified the instrument after a pilot survey on a random sample of another 20 websites. Then, 11 members of the STaRNet Clinical Computerisation and Data Quality (CCDQ) group evaluated the same 20 sites using the rating instrument. The membership of the group includes GPs, a health services researcher, an informatics researcher, practice managers, nurses and administrative staff. Additionally, a single website was evaluated by 14 members of the group. After these evaluations, members of the group examined the criteria and assessed the face and content validity of the instrument. We then produced a final version of the rating instrument (see Appendix). It included information on the site’s content, design and functionality, quality of information provided and the extent to which it enabled electronic communication between the practice and its patients.
Evaluation of the sites using the instrument
AH then surveyed a random sample of 108 of the remaining websites. Sites evaluated in the development of the tool were excluded. Members of the CCDQ also evaluated a sample of 52 of these sites, to test the inter-rater reliability of the instrument.
Data were entered into Microsoft Access and transferred to SPSS (Version 10.0) for analysis.
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionImplications for practiceAppendixReferences |
|---|
Website evaluation
Of the 108 sites identified, 83.3% (90) were accessible. Of these, 93.3% (84/90) were from practices fulfilling the inclusion criteria of being based in the UK and offering services under the NHS.
Table 1 shows frequencies of response for each item in the rating instrument. Criteria for practice information, which were drawn mainly from the statutory requirements for practices to produce printed practice leaflets, were met most frequently. Two related additional aspects of website content were noted in the ‘other features’ section of SWAT. These were practice charters and complaint procedures, which were observed in 
10% of sites in the main study.
|
Of the 48.8% (41/84) of sites which gave the date of the last update, the mean time elapsed until the evaluation date was 249 days (SD 260.7, range 0–874).
Treatment of minor illness and health promotion were the most common topics on the sites which displayed medical information. Another theme identified was preventative advice, although only one site gave specific information on coronary heart disease prevention. In the sites which offered medical information, a total of 44 additional areas of health information were identified. These included meningitis (15/44), self-help advice (11/44) and other areas of health promotion including sexual health (12/44).
Of the 52 sites shown in Table 1
as including ‘advertisements’, which we defined as details of services or products that have to be paid for by the patient, almost all were for services offered in the practice; 13.5% (7/52) had banner advertisements for external organizations. Advertisements were either for services offered by the practice team outside the scope of the NHS [travel vaccines 36.5% (19/52), private medical services, including employment and sports examinations, 53.8% (28/52)] or for practitioners working in the practice building, e.g. physiotherapy 5.8% (3/52) and alternative medicine 15.4% (8/52). Only seven out of the total number of 76 (9.2%) advertisements noted in all the sites could be distinguished from the rest of the site, by either style or context. Three sites stated that they conformed to ‘Health on The Net’ (HoN) ethical guidelines, although none of these fulfilled all the HoN criteria.7 Of the practices which offered links to other Internet medical sites, these were most often direct to distant rather than local ones.
Reliability data
Forty of the 52 sites chosen for assessment of inter-rater reliability were accessible by both evaluators. Cross-tabulations were made of items shown in Table 1 that were evaluated by both assessors for these 40 sites. Of 1080 judgements made, the two assessors agreed in 87.4% (944) of cases. It was possible to calculate Cohen’s kappa for 25 items. These are marked in Table 1. Kappa values for the items in the tool ranged from –0.06 to 1.0 (mean 0.59). Table 2 shows the distribution of kappa values in quintile bands.
|
Quality of graphics and ease of navigating around the website were recorded using a five-point scale. A bivariate analysis using Spearman’s correlation showed almost no agreement (rho = 0.005 and 0.021, P = 0.976 and 0.898, respectively) between assessors in their evaluation of these items.
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionImplications for practiceAppendixReferences |
|---|
General practice websites appear to fulfil a wide range of functions: an on-line practice leaflet; a source of medical advice and information; a means of electronic communication between patients and their practice; as a link to other medical websites; and as an advertising medium. However, little attention is given to demonstrating the quality of the information provided and, as a means of facilitating electronic communication, general practice websites are still in their infancy.
Strengths and limitations of this study
Medical website evaluation instruments have been criticized for not reporting how their validity and reliability were assessed. We report how both of these issues were addressed. Although both assessors made identical judgements for the majority of criteria, kappa values for some individual criteria were low. Although this may have been due in part to ambiguity or lack of clarity in the instrument, much of the explanation may reflect the heterogeneous nature of the websites. There was little standardization in the structure and content of the sites, requiring the assessor to make a judgement rather than simply recording an observation.
In this study, primary care professionals have developed criteria to judge website content and quality. A weakness of the study is the omission of the perspective of the patient. A future development of this study could include their views on the type and range of services and information they would find most useful. However, lay views on websites place strong reliance on design, appearance and technical qualities of the site, none of which have any direct link with the quality of the information.12
The websites chosen for study were obtained from a single database. We were concerned that practices entered into this database may not be representative of all general practice websites in the UK. However, we were unable to find another source which was sufficiently specific. For example, using a single search engine, www.google.com, to identify sites including the words ‘general practice’, ‘medical’ and ‘NHS’ yielded 41 900 results. Furthermore, the variable nature of titles of general practices would raise concerns about the sensitivity of searches using the site name alone.
Comparison with existing literature
Quality of information..
Other rating instruments identify three key concepts in defining the quality of information on medical Internet sites: authorship, attribution and disclosure.10,13,14
- Authorship requires that the site states who is responsible for the information.
- Attribution requires that the source of the information is stated.
- Disclosure requires that funding or sponsorship of the site is stated and that advertisements and other potential conflicts of interest are explicit
Our data suggest that this concept of quality of information is almost entirely lacking from general practice websites. Although professionals routinely expect these requirements to be fulfilled for information they use, this study illustrates the challenge to develop methods of enabling patients to evaluate critically medical information on the Internet.3,15
Other measures of quality of information, such as when the site was last updated and a disclaimer to warn users about the limitations of the material, were also usually absent from the sites we surveyed, or, if present, often suggested that the content of the site was out of date. Adherence to ethical guidelines is more contentious. The difficulty of policing and enforcing such codes has been raised elsewhere.10 In this study, very few sites claimed to adhere to an ethical code, whereas none fulfilled all the requirements of the code.
Purpose and reliability of the instrument.. The fundamental value of producing and promoting quality or ‘kite’ marks for medical websites has been questioned.11 Although it was not the purpose of this study to produce such an instrument, we feel that practices should have the opportunity to evaluate their websites against objective standards, as emulating current practice will simply perpetuate the status quo. SWAT provides a reliable and valid tool for this purpose.
As we know of no similar measures of reliability from other website assessment instruments, it is difficult to compare the reliability of our instrument with others. Our findings suggest that criteria based on subjective assessments of issues such as design and functionality are inherently unreliable and should be omitted from future versions. These are, however, important issues for users of websites, and we suggest that practices needing help with them should consult industry standard sources of information such as www.useit.com or www.uie.com.
|
Implications for practice |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionImplications for practiceAppendixReferences |
|---|
The purpose of general practice websites.
Practices producing their own websites need to decide what the fundamental purpose of their site is. Are they a public service or advertisements to attract patients? The data show that many sites are a mixture of information and advertisements. The information includes details, in many cases incomplete, about the practice and medical advice. Many sites enabled e-mail communication with users. There was little evidence of sites offering explicit information about security of e-mails and protection of patient confidentiality, which are issues which need addressing.
Linking with other websites.
Producing high quality information for a wide range of subjects may be beyond the resources of many practices, for whom it may be more appropriate simply to provide links to other sites, such as NHS Direct.16 Also, our data show that information provided does not match national priorities, such as coronary heart disease prevention.17 Perhaps a role for practice websites would be to link to larger medical websites and add information specific to their practice population or detail services the practice provides in tackling a particular area. Likewise, practices could concentrate on producing links to local medical websites, which would enable users to access information in parallel to that found on national and international sites.
Advertisements.
Many sites displayed advertisements for services which are available in the practice but which needed to be paid for. These advertisements were rarely distinguishable from purely informational material. This is of particular importance where there may be financial conflicts of interest within the practices, e.g. where the practice charges a therapist for use of a room. Practices should ensure that advertisements can be readily discerned for what they are, e.g. by the use of a different style from that of the rest of the site.7
Quality of information.
The Internet is set fundamentally to change the doctor–patient relationship, through its ability to disseminate information.18 It has been suggested that a patient-centred approach to evidence-based medicine requires both patients and professionals to have access to information of equal quality in making medical decisions.19 General practice websites may therefore have a role in redefining the relationship between a practice and its patients by enabling patients to have access to appropriate information.
In summary, we wish to emphasize to practices setting up their own websites the concept of providing quality evidence-based information for patients and that their ethical duties with regard to disclosure of competing interests apply as much to new technologies as to old.
|
Appendix |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionImplications for practiceAppendixReferences |
|---|
Members of the CCDQ who evaluated websites: Julia Ashford, Ann Buckwell, Clodagh Cashman, Sarah Clement, Maggie Dobson, Martin Edwards, Daryl Goodwin, Brian Higginson, Alistair Howitt, Simon de Lusignan, Siyana Shaffie, Chau Shum, Frank Sullivan, Krish Thiru and Sally Wells
|
Acknowledgments |
|---|
We wish to thank Dr Alun Price for allowing us to use his database for this study. STaRNet (supported by London and South East Regions of the NHS Executive) funded the study.
|
Notes |
|---|
e Present address: Fisher Medical Research Unit, Skipton, North Yorkshire BD23 2TH, UK.
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionImplications for practiceAppendixReferences |
|---|
1 Internet-GP. www.internet-gp.com
2 NHS Executive. Information for Health. An Information Strategy for the Modern NHS 1998–2005. A National Strategy for Local Implementation. London: NHSE, 1998. www.doh.gov.uk/nhsexipu/strategy/index.htm
3 Eysenbach G, Diepgen TL. Towards quality management of medical information on the internet: evaluation, labelling, and filtering of information. Br Med J 1998; 317: 1496–1500.
4 Shepperd S, Charnock D, Gann B. Helping patients access high quality health information. Br Med J 1999; 319: 764–766.
5 Eysenbach G. Towards ethical guidelines for e-health: JMIR Theme Issue on eHealth Ethics. J Med Internet Res 2000; 2: e7. www.jmir.org/2000/1/e7/
6 Health Internet Ethics: Ethical Principles For Offering Internet Health Services to Consumers. www.hiethics.com/Principles/index.asp
7 Health on the net foundation code of conduct for medical and health web sites. www.hon.ch/HONcode/Conduct.html
8 e-Health Ethics Initiative. e-Health Ethics Draft Code. J Med Internet Res 2000; 2: e2. www.jmir.org/2000/1/e2/
9 Kim P, Eng TR, Deering MJ, Maxfield A. Published criteria for evaluating health related web sites: review. Br Med J 1999; 318: 647–649.
10 Jadad AR, Gagliardi A. Rating health information on the internet: navigating to knowledge or to Babel? J Am Med Assoc 1998; 279: 611–614.
11 Delamothe T. Quality of websites: kitemarking the west wind (Editorial). Br Med J 2000; 321: 843–844.
12 Jadad AR. Promoting partnerships: challenges for the internet age. Br Med J 1999; 319: 761–763.
13 Eysenbach G, Sa ER, Diepgen DL. Shopping around the internet today and tomorrow: towards the millennium of cybermedicine. Br Med J 1999; 319: 1294.
14 Alexander J, Tate MA. Checklist for an Informational Web Page 1996. www2.widener.edu/Wolfgram-Memorial-Library/webevaluation/inform.htm
15 Smith AG. Testing the Surf: Criteria for Evaluating Internet Information Resources. The Public-Access Computer Systems Review 8,3 1997. http://lib-04.lib.uh.edu/pacsrev/1997/smit8n3.htm
16 Charnock D, Shepperd S. Quality Criteria for Online Consumer Health Information on Treatment Choices. DISCERN ON THE INTERNET (February 1999). Health Information on the Internet. www.discern.org.uk/HOTI.htm
17 NHS Direct. www.nhsdirect.nhs.uk
18 Our Healthier Nation. National Service Framework for Coronary Heart Disease. NHS Executive 2000. www.ohn.gov.uk/ohn/ohn.htm
19 Hawker A. Trust me—I’m a website. Primary Health Care Specialist Group of the British Computer Society. Annual Conference Proceedings 1999. www.phcsg.org.uk/conferences/Cambridge1999/camb99-07.htm
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  K. Schmidt and E. Ernst Assessing websites on complementary and alternative medicine for cancer Ann. Onc., May 1, 2004; 15(5): 733 - 742. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||