Family Practice Vol. 20, No. 5, 545-551
© Oxford University Press 2003
Information in practice |
Using the Internet to conduct surveys of health professionals: a valid alternative?
General Practice and Primary Care Research Unit, Institute of Public Health, University of Cambridge, Robinson Way, Cambridge CB2 2SR and
a Department of General Practice and Primary Care, St Georges Hospital Medical School, Cranmer Terrace, London SW17 0RE, UK.
Correspondence to Ms Dejana Braithwaite; E-mail: dkb22{at}medschl.cam.ac.uk
Braithwaite D, Emery J, de Lusignan S and Sutton S. Using the Internet to conduct surveys of health professionals: a valid alternative? Family Practice 2003; 20: 545–551.
Received 31 October 2002; Accepted 19 May 2003.
 |
Abstract |
|---|
 Top Abstract IntroductionSystematic review of Internet...Case study: electronic survey...DiscussionReferences |
|---|
Objective. The purpose of this study was to examine whether Internet-based surveys of health professionals can provide a valid alternative to traditional survey methods.
Methods. (i) Systematic review of published Internet-based surveys of health professionals focusing on criteria of external validity, specifically sample representativeness and response bias. (ii) Internet-based survey of GPs, exploring attitudes about using an Internet-based decision support system for the management of familial cancer.
Results. The systematic review identified 17 Internet-based surveys of health professionals. Whilst most studies sampled from professional e-directories, some studies drew on unknown denominator populations by placing survey questionnaires on open web sites or electronic discussion groups. Twelve studies reported response rates, which ranged from nine to 94%. Sending follow-up reminders resulted in a substantial increase in response rates. In our own survey of GPs, a total of 268 GPs participated (adjusted response rate = 52.4%) after five e-mail reminders. A further 72 GPs responded to a brief telephone survey of non-respondents. Respondents to the Internet survey were more likely to be male and had significantly greater intentions to use Internet-based decision support than non-respondents.
Conclusions. Internet-based surveys provide an attractive alternative to postal and telephone surveys of health professionals, but they raise important technical and methodological issues which should be carefully considered before widespread implementation. The major obstacle is external validity, and specifically how to obtain a representative sample and adequate response rate. Controlled access to a national list of NHSnet e-mail addresses of health professionals could provide a solution.
Keywords. Decision support, GPs, health survey, Internet.
|
Introduction |
|---|
|
TopAbstractIntroductionSystematic review of Internet...Case study: electronic survey...DiscussionReferences |
|---|
Postal surveys of health professionals have traditionally been used to assess their knowledge, views and attitudes to inform service planning and provision. In the UK, the shift in emphasis on a primary care-led NHS has meant that GPs receive increasing numbers of questionnaires from academics, industry and other health service providers, leading to a reduction in response rates.1
The potential to use the Internet as a research tool and not merely an information resource is growing2 as more doctors go on-line. It is estimated that more than one in eight UK GPs are connected to and familiar with the Internet.3 The advent of the NHSnet, the NHS computer network, and the establishment of the new NHS e-mail and directory service for all NHS staff in the UK4 could enable implementation of large-scale Internet-based surveys of GPs and other health professionals. Commercial interest is growing in this area, with doctors receiving invitations to join reward schemes in return for completing web-based surveys, mainly for the pharmaceutical industry (www.Medix-uk.com). But what are the methodological and technical implications of conducting survey research via the Internet? In this paper we review published Internet-based surveys of health professionals, focusing on issues of external validity, specifically sample representativeness and response rate. In addition, we present a more detailed account of the technical issues in conducting an Internet-based survey of GPs.
|
Systematic review of Internet-based surveys of health professionals |
|---|
|
TopAbstractIntroductionSystematic review of Internet...Case study: electronic survey...DiscussionReferences |
|---|
Methods
We searched Medline and Embase from inception to October 2002 using the following search terms: ‘Internet’, ‘(World Wide) Web’, ‘Survey’ and ‘Questionnaire’. Additional papers were identified through examining reference lists from identified papers. Papers were included if they reported survey data of health professionals that were collected via e-mail or a specific web site.
Results
Of the 115 Internet-based surveys identified, there were 17 surveys of health professionals, all of which were published between 1999 and 2002.5–19 All other Internet-based surveys were of populations other than health professionals. We present background details of each survey and specific information pertaining to the external validity of these studies (Table 1
). Survey topics ranged from interphysician communication and issues in the referral process,5 to practices in intraspinal therapy for pain management,9 to doctors’ experience of patients using the Internet.16 While most studies utilized professional e-directories of specified health professionals as their sampling frames,6,7,9–11,13–15,17,18 some surveys drew on commercial organization e-mail directories16 or targeted volunteers recruited via web sites8,12,19 or electronic discussion groups.5 Twelve of the studies reported response rates, which ranged from nine11 to 94%.16 Five studies could not estimate response rates as participants were solicited via web sites with unknown denominator populations.5,6,8,15,19 Relatively low response rates in Internet surveys have been reported in studies that have directly compared them with traditional paper methods.10,11,17 A substantial increase in response rate following reminders has been found in other electronic surveys of health professionals.6,13,18
|
|
Case study: electronic survey of GPs |
|---|
|
TopAbstractIntroductionSystematic review of Internet...Case study: electronic survey...DiscussionReferences |
|---|
Methods
We conducted an electronic survey of GPs to investigate their attitudes and expectations about using an Internet-based decision support system for the management of familial cancer. The results of this survey are reported separately.20 We describe here its design, development and administration. An Internet-based method was chosen as we wanted to sample an Internet- and computer-literate population who are most likely to be early adopters of the technology.21
Sample
The survey was hosted by UKPractice (www.ukpractice.net), a password-protected Internet portal aimed at primary care with 22 000 registered users, all of whom are registered with the UK General Medical Council. Doctors who had accessed the UKPractice site at least once during the 3 months prior to the study were invited by e-mail to participate. The e-mail invitation was sent by the UKPractice Education Adviser, thus maintaining confidentiality of the users’ e-mail addresses.
Survey design
The e-mail invitation contained a link to the Internet-based questionnaire. The questionnaire contained 47 questions written in Active Server Page and used JavaScript to verify the authenticity of responses. Eight hours of programming time were required to implement the questionnaire on the Internet and establish the underlying database. Data from completed forms were inserted into an SQL database automatically, and were then imported into a Microsoft Excel spreadsheet for analysis.
To complete the survey, respondents were required to answer questions by clicking in specific buttons that represented points on a Likert scale, or entering free text in specific boxes (Fig. 1). Users scrolled down the entire questionnaire and finally clicked on the ‘submit’ button when the survey was completed. The questionnaire was piloted initially with 30 UKPractice users, all of whom completed the questionnaire successfully. Electronic reminders to non-respondents were sent 1 month after the initial invitation, and a further four sent 2, 4, 6 and 8 weeks later. The survey was concluded with a brief telephone follow-up of non-respondents.
|
Response rate
Of the 1003 e-mail invitations sent, 268 were rejected by the server due to e-mail addresses being either incorrectly spelt or no longer valid. Ninety-one replied stating that they were either not practising GPs or not GPs at all, and a further 31 GPs stated that they did not wish to participate. A technical error over a weekend with the server occurred during the data collection, losing 95 responses, and a further 38 questionnaires were received blank. Some of these were possibly caused by respondents erroneously clicking the ‘reset’ button before the ‘submit’ button. After the first e-mail invitation, the adjusted response rate, accounting for invalid e-mail addresses and lost data, was just under 30% (n = 149), but rose to 52.4% (n = 268) following five reminders. A telephone-based survey of non-respondents (n = 72) yielded a response rate of 63%.
Sample representativeness
In this study, we deliberately chose to sample GPs from a technically literate sampling frame. Respondents were based in practices throughout the UK; 70% were male and 
50% had practised for <10 years. The sample is therefore unrepresentative of UK GPs as a whole. In the telephone survey of non-respondents, participants were significantly less likely to be male or use Internet-based decision support than respondents in the electronic survey, suggesting a response bias within our target sample of GPs.
|
Discussion |
|---|
|
TopAbstractIntroductionSystematic review of Internet...Case study: electronic survey...DiscussionReferences |
|---|
For health service researchers, there are clearly several benefits of conducting surveys of health professionals via the Internet which make it appear an attractive alternative to postal or telephone methods. Key advantages of using electronic surveys are their relative ease of implementation, and the potential to conduct large-scale surveys whilst eliminating the costs of stationery, postage and administration.22 Simple questionnaires do not require extensive programming skills or time, and the cost of sending multiple e-mail invitations and reminders is negligible. Electronic surveys allow tighter control of the order in which respondents see specific questions, thus preventing respondents from returning to change their answers. More sophisticated programming allows validation checks as data are collected or randomization of respondents to different versions of the questionnaire. Although automated systems of electronic entry of data into a database exist for paper-based surveys, these are relatively expensive (e.g. www.formic.co.uk). Internet-based surveys allow simple automatic transfer of data into a database, thus eliminating the need for manual inputting and avoiding potential errors of data entry.
As demonstrated in our review of Internet-based surveys and our own experience, some key technical and methodological issues raised by Internet-based survey research should be considered carefully before researchers jump onto the bandwagon.18,22 Creating a web-based survey instrument is not simply a case of reproducing an e-version of the paper survey. Formatting may need to be changed to simplify data entry, clarify possible responses and avoid the possibility of submitting data before completing the survey. Because of variation in individual respondents’ computer hardware and software, decisions must be made about the complexity of visual design, and therefore speed of download, and the ability to view the whole questionnaire on a range of screen settings. The option to print out and reply to a hard copy of the survey should also be considered. While postal surveys may lose the occasional response, our study demonstrates the potential to lose a much larger amount of data through a brief server failure.
The principal methodological issue arising from our review concerns sample representativeness and therefore external validity of the data obtained, in relation to both the original sample and response bias. Sending reminders represents a potential solution. In our study, the response rate almost doubled following five reminders and achieved rates comparable with those obtained in postal surveys of GPs.1 As acknowledged in most papers in the review, respondents to electronic surveys are usually unrepresentative of the general population of health professionals even within a certain health care specialty. In a large UK-based commercial scheme that conducts web-based survey research, female doctors are significantly underrepresented.16 One cannot assume that registered users with a specific web site will necessarily reflect the expected group of health professionals. Although UKPractice is designed for use by GPs, many of its users work in other medical specialities, as we discovered in our survey.
When conducting postal surveys, it is often possible to obtain a complete list of eligible health professionals in a given specialty or geographical location from which to sample. For example, a list of GPs’ postal addresses and telephone numbers can be obtained from Health Authorities. Additional information about gender and date of registration can be obtained from the Medical Register, thus enabling sample stratification or selection of specific groups. In the UK, the establishment of the NHS e-mail and directory service for NHS staff raises the possibility of creating an ‘e-phonebook’ of electronic addresses of NHS health professionals from which researchers could recruit more representative samples for Internet-based survey research. Ideally, this would also contain basic demographic information such as that recorded in the Medical Register. However, if such a resource were created, careful management of who could access this information would be required. General access to this information raises the spectre of electronic overload, which might well be of far greater magnitude than in postal surveys, with a resultant effect on response rates. Alternatively, health professionals could be invited to have their details included in such a database, but this would clearly jeopardize the rationale for its creation in the first place.
Using Internet technology to conduct surveys of health professionals offers many advantages. However, the technical and methodological implications of using this approach should not be underestimated. Further research is required into methods to improve the external validity of electronic surveys, including approaches to increase the representativeness of invited samples and limit response bias. Until it is possible to obtain access to more representative samples of health professionals, the validity of Internet-based survey research remains in doubt.
|
Acknowledgments |
|---|
The authors would like to thank Cancer Research UK for financial support and Dr Henry Smithson for assistance with the recruitment of GPs into the case study.
|
References |
|---|
|
TopAbstractIntroductionSystematic review of Internet...Case study: electronic survey...DiscussionReferences |
|---|
1 McAvoy BR, Kaner EF. General practice postal surveys: a questionnaire too far? Br Med J 1996; 313:732–733.
2 Eysenbach G, Diepgen TL. Epidemiological data can be gathered with world wide web. Br Med J 1998; 316:72.
3 Roscoe T. Two surveys of internet use in primary care. Health Info Internet 1998; 2–3.
4 The NHS Information Authority website (www.nhsia.nhs.uk). Accessed on 28 July 2003.
5 Angelo SJ, Citkowitz E. An electronic survey of physicians using online clinical discussion groups: a brief report. Conn Med 2001; 65:135–138.[Medline]
6 Fischbacher C, Chappel D, Edwards R, Summerton N. Health surveys via the Internet: quick and dirty or rapid and robust? J R Soc Med 2000; 93:356–359.
7 Gandhi TK, Sittig DF, Franklin M, Sussman AJ, Fairchild DG, Bates DW. Communication breakdown in the outpatient referral process. J Gen Intern Med 2000; 15:626–631.[CrossRef][ISI][Medline]
8 Gandsas A, Draper K, Chekan E et al. Laparoscopy and the internet. A surgeon survey. Surg Endosc 2001; 15:1044–1048.[Medline]
9 Hassenbusch SJ, Portenoy RK. Current practices in intraspinal therapy—a survey of clinical trends and decision making. J Pain Symptom Manage 2000; 20:S4–S11.[ISI][Medline]
10 Hollowell CM, Patel RV, Bales GT, Gerber GS. Internet and postal survey of endourologic practice patterns among American urologists. J Urol 2000; 163:1779–1782.[CrossRef][ISI][Medline]
11 Kim HL, Hollowell CM, Patel RV, Bales GT, Clayman RV, Gerber GS. Use of new technology in endourology and laparoscopy by American urologists: internet and postal survey. Urology 2000; 56:760–765.[CrossRef][ISI][Medline]
12 Kim HL, Benson DA, Stern SD, Gerber GS. Practice trends in the management of prostate disease by family practice physicians and general internists: an internet-based survey. Urology 2002; 59:266–271.[Medline]
13 McLean SA, Feldman JA. The impact of changes in HCFA documentation requirements on academic emergency medicine: results of a physician survey. Acad Emerg Med 2001; 8:880–885.[Medline]
14 Muhumuza R, Moles DR, Bedi R. A survey of dental practitioners on their use of electronic mail. Br Dent J 1999; 186:131–134.[CrossRef][ISI][Medline]
15 Pereira J, Bruera E, Quan H. Palliative care on the net: an online survey of health care professionals. J Palliat Care 2001; 17:41–45.[ISI][Medline]
16 Potts HWW, Wyatt JC. Survey of doctors’ experience of patients using the Internet. J Med Internet Res 2002; 4:e5.[Medline]
17 Raziano DB, Jayadevappa R, Valenzula D, Weiner M, Lavizzo-Mourey R. E-mail versus conventional postal mail survey of geriatric chiefs. Gerontologist 2001; 41:799–804.
18 Schleyer TKL, Forrest JL. Methods for the design and administration of web-based surveys. J Am Med Inform Assoc 2000; 7:416–425.
19 Wright S, Neill K. Using the World Wide Web for research data collection. Clin Excell Nurse Pract 1999; 3:362–365.[Medline]
20 Braithwaite D, Sutton S, Smithson WH, Emery J. Internet-based risk assessment and decision support for the management of familial cancer in primary care: survey of GPs’ attitudes and intentions. Fam Pract 2002; 19:587–590.
21 Rogers EM. Diffusion of Innovations, 3rd edn. New York: Free Press; 1983.
22 Wyatt JC. When to use web-based surveys. J Am Med Inform Assoc 2000; 7:426–429.
CiteULike 
Connotea 
Del.icio.us What's this?
This article has been cited by other articles:
|
|
 |
|
  J. B. VanGeest, T. P. Johnson, and V. L. Welch Methodologies for Improving Response Rates in Surveys of Physicians: A Systematic Review Eval Health Prof, December 1, 2007; 30(4): 303 - 321. [Abstract] [PDF]
|
|
|
|
 |
|
 J. S. Shapiro, J. Kannry, A. W. Kushniruk, G. Kuperman, and The New York Clinical Information Exchange (NYCLIX Emergency Physicians' Perceptions of Health Information Exchange J. Am. Med. Inform. Assoc., November 1, 2007; 14(6): 700 - 705. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 L. Barnard, D. Ferriday, N. Guenther, B. Strauss, A.H. Balen, and L. Dye Quality of life and psychological well being in polycystic ovary syndrome Hum. Reprod., August 1, 2007; 22(8): 2279 - 2286. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
C. Lusk, G. L. Delclos, K. Burau, D. D. Drawhorn, and L. A. Aday Mail Versus Internet Surveys: Determinants of Method of Response Preferences Among Health Professionals Eval Health Prof, June 1, 2007; 30(2): 186 - 201. [Abstract] [PDF]
|
|
|
|
|
|
K. Hartford, R. Carey, and J. Mendonca Sampling Bias in an International Internet Survey of Diversion Programs in the Criminal Justice System Eval Health Prof, March 1, 2007; 30(1): 35 - 46. [Abstract] [PDF]
|
|
|
|
 |
|
 P. L. Bailey, L. G. Glance, M. P. Eaton, B. Parshall, and S. McIntosh A Survey of the Use of Ultrasound During Central Venous Catheterization Anesth. Analg., March 1, 2007; 104(3): 491 - 497. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
G. R. d. O. Filho and L. Schonhorst Attitudes of residents and anesthesiologists toward basic sciences. Anesth. Analg., July 1, 2006; 103(1): 137 - 143. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 G. Umefjord, K. Hamberg, H. Malker, and G. Petersson The use of an Internet-based Ask the Doctor Service involving family physicians: evaluation by a web survey Fam. Pract., April 1, 2006; 23(2): 159 - 166. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 J. Elms, R. O'Hara, S. Pickvance, D. Fishwick, M. Hazell, T. Frank, M. Henson, P. Marlow, G. Evans, L. Bradshaw, et al. The perceptions of occupational health in primary care Occup. Med., October 1, 2005; 55(7): 523 - 527. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 D. W. Wheeler, D. D. Remoundos, K. D. Whittlestone, M. I. Palmer, S. J. Wheeler, T. R. Ringrose, and D. K. Menon Doctors' confusion over ratios and percentages in drug solutions: the case for standard labelling J R Soc Med, August 1, 2004; 97(8): 380 - 383. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||